Add a new environment varible
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS_DYNAMICE_LIST that specifies a text
file containing a list of RO fossilize caches to load. The list file
is modifiable at runtime to allow for loading RO caches after
initialization unlike MESA_DISK_CACHE_READ_ONLY_FOZ_DBS.
The implementation spawns an updater thread that uses inotify to monitor
the list file for modifications, attempting to load new foz dbs added to
the list. Removing files from the list will not evict a loaded cache.
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS_DYNAMIC_LIST takes an absolute path.
The file must exist at initialization for updating to occur.
File names of foz dbs in the list file are new-line separated and take
relative paths to the default cache directory like
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS.
The maximum number of RO foz dbs is kept to 8 and is shared between
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS_DYNAMIC_LIST and
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS.
The intended use case for this feature is to allow prebuilt caches
to be downloaded and loaded asynchronously during app runtime.
Prebuilt caches be large (several GB) and depending on network
conditions would otherwise present extended wait time for caches
to be availible before app launch.
This will be used in Chrome OS.
Signed-off-by: Juston Li <justonli@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19328>
The flock is per-fd, not per thread, and we do it outside of the main mutex. This was
done to avoid having to wait in the mutex, but we can get a case where one ends up running
the body with the flock unlocked.
Fix this by adding a mutex that doesn't need to be locked for reads.
Fixes: 4f0f8133a3 "util/fossilize_db: Do not lock the fossilize db permanently."
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12266>
We now handle compression in the shared cache item creation code.
Compressing the cache item header with the already compressed blob
doesn't help much so lets just remove it.
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9593>
My benchmarking shows no significant change in cache load times with a
single shader cache file vs the existing cache implementation of many
small files (tested with my spinning rust HDD).
However this new single file cache implementation does reduce the total
size on disk used by the shader cache. We have a problem with the existing
cache where writing tiny files to disk causes more disk space to be used than
is actually needed for the files due to the minimum size required for a file.
In pratice this tends to inflate the size of the cache on disk to over 3x
larger.
There are other advantages of using a single file for shader cache entries
such as allowing better removal of cache entries once we hit the max cache
size limit (although we don't implement any max cache size handling in this
initial implementation).
The primary reason for implementing a single file cache for now is to allow
better performance and handling by third party applications such as steam
that collect and distribute precompiled cache entries.
For this reason we also implement a new environment variable
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS which allows a user to pass in a path
to a number of external read only shader cache dbs. There is an initial
limit of 8 dbs that can be passed to mesa like so:
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS=/full_path/filename1, ... ,/full_path/filename8
Where the filename represents the cache db and its index file e.g.
filename1.foz and filename1_idx.foz
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7725>
