Just because the user / system-integrater doesn't want shader disk
cache, doesn't mean they don't want EGL_ANDROID_blob_cache to work.
We've kind of already solved this for the android case, so just
generalize that solution.
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/9520
Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24636>
ftruncate() allocates disk space lazily. If the disk is full and it is
unable to allocate disk space when accesed via mmap(), it will crash
with a SIGBUS.
Switch to posix_fallocate(), which ensures disk space is allocated
otherwise it fails if there isn't enough disk space. The disk cache
won't be enabled in this case.
For normal cases, a small increase in disk usage as the 1.3MB index
file will be fully allocated when initialized now.
fallback to ftruncate() if posix_fallocate() isn't found.
Signed-off-by: Juston Li <justonli@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22097>
Needed to build on Solaris
Fixes: f58e6fee74 ("util/disk_cache: delete more cache items in one go when full")
Signed-off-by: Alan Coopersmith <alan.coopersmith@oracle.com>
Reviewed-by: Eric Engestrom <eric@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21110>
Replace single file mesa-db cache with multipart mesa-db cache.
Each part of the multipart cache essentially is a single file
mesa-db cache, aka database shard. Multipart cache brings much
more optimized cache eviction times in comparison to a single file
cache.
Acked-by: Timothy Arceri <tarceri@itsqueeze.com>
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20256>
Store values of all environment variables related to disk caching within
struct disk_cache. This makes code cleaner and also will allow us to
combine read-only single-file cache with read-write caches.
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18551>
stack-trace by asan
Direct leak of 238 byte(s) in 6 object(s) allocated from:
#0 0x7f101e1dbe8f in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
#1 0x55ce3a959e2c in create_cache_item_header_and_blob ../src/util/disk_cache_os.c:647
#2 0x55ce3a95c217 in disk_cache_write_item_to_disk_foz ../src/util/disk_cache_os.c:967
#3 0x55ce3a97bce1 in util_queue_thread_func ../src/util/u_queue.c:306
#4 0x55ce3a989c3b in impl_thrd_routine ../src/c11/impl/threads_posix.c:67
#5 0x7f101ddc7ea6 in start_thread nptl/pthread_create.c:477
Fixes: 2a9b4ad177 ("util/disk_cache: Add option to disable compression")
Cc: mesa-stable
Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Eric Engestrom <eric@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18024>
Introduce new cache type, the Mesa-DB. This is a single-file read/write
cache that is based on the read-only Fossilize DB cache. Mesa-DB supports
cache size capping. It's a much more efficient cache than the multi-file
cache because Mesa-DB doesn't have the inode overhead. The plan is to make
Mesa-DB the default cache implementation once it will be deemed as stable
and well tested. For now users have to set the new MESA_DISK_CACHE_DATABASE
environment variable in order to active the Mesa-DB cache.
Mesa-DB cache is resilient to corrupted cache files and doesn't require
maintenance from users and developers. The size capping is implemented
by evicting least recently used cache items and compacting the cache
database files with the evicted entries. In order to prevent frequent
compaction of the cache, at minimum a half of cache is evicted when cache
is full.
Acked-by: Timothy Arceri <tarceri@itsqueeze.com>
Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16888>
Relocate struct cache_entry_file_data to the header file, making it
accessible to the cache testing code. This is a preparatory step towards
addition of the new Mesa-DB cache type.
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16888>
The disk_cache_load_cache_index() is solely about the foz-db cache and
we're going to add the new cache type. Append the function name with
_foz postfix to improve the naming, making clear what that function is
about.
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16888>
The shader cache files produced by multi-file cache have a minimum size
limitation imposed by filesystem, usually it's 4KB. In order to make
cache tests suitable for a single file caches, we need to bypass the data
compression, making size of written out data determined for the eviction
testing.
The disk_cache_create() now checks whether driver_id name is set to
"make_check_uncompressed" in order to disable cache data compression.
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16888>
Rename MESA_GLSL_CACHE to MESA_SHADER_CACHE, as the on-disk cache can
store not only GLSL but also SPIR-V shaders.
v2:
- Keep old envvar as deprecated (Mike)
Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15390>
Currently the cache just deletes enough items when the cache is
full to make room for the new item being stored. This hasn't
been too much of a problem in practice but for things like running
piglit where we have thousands of unique shaders and all threads
being utilised we end up with a pretty big bottle neck.
With this change rather than just brute forcing our way to having
enough room for the new item, we instead grab 10% of the least
recently used items in the random directory we chose and delete
them all. This should only be around 0.04% of total cache items
but should hopefully releave some of the pressure on system calls
like fstatat().
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11523>
This refactors the cache code so that we create the full cache
item in memory before writing it to disk. The result is slightly
cleaner code and the ability to share this code between the single
and multi file cache implementations.
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9443>
This will allow us to share validation between the single file cache
and the multifile cache.
It also reduces the number of malloc() calls.
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9443>
This makes compression use more consistent between the zstd and
zlib libraries. It also reduces the amount of code required for
zlib use.
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9593>
When the MESA_DISK_CACHE_SINGLE_FILE environment variable is set
we make use of the new single file shader cache implementation.
The new cache uses the following directory structure based on the
first defined name as follows:
$MESA_GLSL_CACHE_DIR/driver_id/gpu_name/foz_cache.foz
$MESA_GLSL_CACHE_DIR/driver_id/gpu_name/foz_cache_idx.foz
$XDG_CACHE_HOME/mesa_shader_cache_sf/driver_id/gpu_name/foz_cache.foz
$XDG_CACHE_HOME/mesa_shader_cache_sf/driver_id/gpu_name/foz_cache_idx.foz
<pwd.pw_dir>/.cache/mesa_shader_cache_sf/driver_id/gpu_name/foz_cache.foz
<pwd.pw_dir>/.cache/mesa_shader_cache_sf/driver_id/gpu_name/foz_cache_idx.foz
Where foz_cache_idx.foz is a database of offsets pointing to the location of
the shader cache entries in foz_cache.foz
This initial implementation doesn't have any max cache size handling and is
initially intended to be use by applications such as steam that will handle
cache management for us.
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7725>
This allows us to guarantee the different cache implementations will not
interfere with each other and should make it more clear that the max cache
size limits are applied separately for each cache implementation.
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7725>
This code is already OS agnostic and moving it here enables code
sharing in the following patches.
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7725>
This will be faster and avoids checking for errors with the
compression implementation which we shouldn't need to do. Instead
we trust the compression library does the correct thing and simply
error check the data loaded from disk.
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7725>
On some systems it is problematic to have the shader cache enabled
by default. This adds a build option to support the disk cache but
keep it disabled unless the environment variable
MESA_GLSL_CACHE_DISABLE=false.
For example, on Chrome OS, Chrome already has it's own shader
disk cache implementation so it disables the mesa feature. Tests
do not want the shader disk cache enabled because it can cause
inconsistent performance results and the default 1GB for the
disk cache could lead to problems that require more effort to
work around. The Mesa shader disk cache is useful for VMs though,
where it is easy to configure the feature with environment
variables. With the current version of Mesa, Chrome OS would need
to have a system-wide environment variable to disable the disk
cache everywhere except where needed. More elegant to just build
Mesa with the cache feature disabled by default.
Reviewed-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6967>
Remove filename ralloc comment. filename is allocated by asprintf.
Clean up disk_cache_get dead code left over from 367ac07efc
("disk_cache: move cache item loading code into
disk_cache_load_item() helper").
Fix defect reported by Coverity Scan.
Logically dead code (DEADCODE)
dead_error_line: Execution cannot reach this statement:
free(filename);
Signed-off-by: Vinson Lee <vlee@freedesktop.org>
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6738>
This should be helpful if someone chooses to implement cache support on
windows. Also providing this greater level of abstraction makes it easier
to implement alterative cache layouts in future.
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6197>
This pulls out the cache item writing code from cache_put() into
a new helper. In this patch we also move various functions called
by this code into the new disk_cache_os.c file.
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6197>
This will make windows support easier to add in future. To avoid code
churn this temporarily duplicates the mkdir_if_needed() function, we
will delete the duplicate in a following patch.
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6197>
