This refactors the cache code so that we create the full cache
item in memory before writing it to disk. The result is slightly
cleaner code and the ability to share this code between the single
and multi file cache implementations.
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9443>
This will allow us to share validation between the single file cache
and the multifile cache.
It also reduces the number of malloc() calls.
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9443>
We now handle compression in the shared cache item creation code.
Compressing the cache item header with the already compressed blob
doesn't help much so lets just remove it.
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9593>
This makes compression use more consistent between the zstd and
zlib libraries. It also reduces the amount of code required for
zlib use.
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9593>
This will be used by the following patch. It allows us to detangle
compression from the disk cache code, and abstract the underlying
compression libraries we use.
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9593>
Calling code to pack/unpack with overlap would be already be undefined.
Cuts 50k of text on x86_64 release builds from the compiler having more
freedom in the src/dst loads knowing that they don't interfere with each
other.
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Adam Jackson <ajax@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9500>
Now the texture virtual memory usage is less of a problem,
we can use this workaround permanently.
In the spirit of the API it's certainly not the proper way
of implementing DYNAMIC textures (it seems they are ok
to have hidden copies in driver managed memory, but not have
virtual addressing space reduced), but it makes sense for us,
both performance wise, and to avoid bugs.
Signed-off-by: Axel Davy <davyaxel0@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9377>
The cache has been detangled from glsl and used outside it (with Vulkan drivers)
for years now.
This also cleans up the dependancies in the build file. The test doesn't
depend on the glsl lib but rather the util lib.
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9327>
Fixes the following building error:
FAILED: out/target/product/x86_64/obj_x86/STATIC_LIBRARIES/libmesa_util_intermediates/xmlconfig.o
...
external/mesa/src/util/xmlconfig.c:1030:12: fatal error: 'driconf_static.h' file not found
^~~~~~~~~~~~~~~~~~
1 error generated.
Fixes: a6b0ceb ("driconf: Generate a static table when no xmlconfig")
Acked-by: Rob Clark <robclark@freedesktop.org>
Tested-by: Marijn Suijten <marijn.suijten@somainline.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9294>
MR to move to python3 in Android build gen rules is still pending
The change is to avoid following building error:
FAILED: out/target/product/x86_64/gen/STATIC_LIBRARIES/libmesa_util_intermediates/driconf_static.h
/bin/bash -c "/usr/bin/python external/mesa/src/util/driconf_static.py external/mesa/src/util/00-mesa-defaults.conf > out/target/product/x86_64/gen/STATIC_LIBRARIES/libmesa_util_intermediates/driconf_static.h"
File "external/mesa/src/util/driconf_static.py", line 2
SyntaxError: Non-ASCII character '\xc2' in file external/mesa/src/util/driconf_static.py on line 2, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details
Fixes: a6b0ceb ("driconf: Generate a static table when no xmlconfig")
Acked-by: Rob Clark <robclark@freedesktop.org>
Tested-by: Marijn Suijten <marijn.suijten@somainline.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9294>
In release builds, there should be no change, but in debug builds the
assert will help us catch undefined behavior resulting from using
util_cpu_caps before it is initialized.
With fix for u_half_test for MSVC from Jesse Natalie squashed in.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9266>
I noticed that we were hitting this before st_create_context() called
util_cpu_detect() and so num_cpu_mask_bits was zero. But there is no
harm in calling util_cpu_detect(), so lets just call it here to be safe.
Fixes: d877451b48 ("util/u_queue: add UTIL_QUEUE_INIT_SET_FULL_THREAD_AFFINITY")
Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9266>
We don't need to close the file here check_files_opened_successfully()
already does that for us if needed.
This fixes a crash when invalid filename/paths are provided to the
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS environment variable.
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9251>
After commit a46b73e in order to build mesa for Android
we need to generate $(intermediates)/util/format/u_format_pack.h
instead of $(intermediates)/format/u_format_pack.h
Fixes the following building error in Android build:
external/mesa/src/util/format/u_format_fxt1.c:28:10: fatal error: 'util/format/u_format_pack.h' file not found
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
Fixes: a46b73e ("mesa: Move the FXT1 compressor/decompressor to util/")
Reviewed-by: Adam Jackson <ajax@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9240>
this is a standardized (and very slightly improved for usability) version
of the macro that has been copied into every vulkan driver
includes fixup from Rob Clark <robclark@freedesktop.org>
Reviewed-by: Rob Clark <robclark@freedesktop.org>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9191>
For platforms which do not have support for parsing driconf from xml
files on the filesystem, build in driconf tables generated from
00-mesa-defaults.conf at compile time and use that for option matching.
This allows us to have game/engine specific overrides built in to mesa.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9179>
For the static-table alternative to WITH_XMLCONFIG, we are going to want
to re-use the element attribute processing, to avoid duplicating things
like engine name regexp matching and version range matching. This just
shuffles things around a bit so we can re-use useful parts in the next
patch.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9179>
For builds without runtime xmlconfig parsing, generate a static table
from 00-mesa-defaults.conf.
Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9179>
When the MESA_DISK_CACHE_SINGLE_FILE environment variable is set
we make use of the new single file shader cache implementation.
The new cache uses the following directory structure based on the
first defined name as follows:
$MESA_GLSL_CACHE_DIR/driver_id/gpu_name/foz_cache.foz
$MESA_GLSL_CACHE_DIR/driver_id/gpu_name/foz_cache_idx.foz
$XDG_CACHE_HOME/mesa_shader_cache_sf/driver_id/gpu_name/foz_cache.foz
$XDG_CACHE_HOME/mesa_shader_cache_sf/driver_id/gpu_name/foz_cache_idx.foz
<pwd.pw_dir>/.cache/mesa_shader_cache_sf/driver_id/gpu_name/foz_cache.foz
<pwd.pw_dir>/.cache/mesa_shader_cache_sf/driver_id/gpu_name/foz_cache_idx.foz
Where foz_cache_idx.foz is a database of offsets pointing to the location of
the shader cache entries in foz_cache.foz
This initial implementation doesn't have any max cache size handling and is
initially intended to be use by applications such as steam that will handle
cache management for us.
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7725>
My benchmarking shows no significant change in cache load times with a
single shader cache file vs the existing cache implementation of many
small files (tested with my spinning rust HDD).
However this new single file cache implementation does reduce the total
size on disk used by the shader cache. We have a problem with the existing
cache where writing tiny files to disk causes more disk space to be used than
is actually needed for the files due to the minimum size required for a file.
In pratice this tends to inflate the size of the cache on disk to over 3x
larger.
There are other advantages of using a single file for shader cache entries
such as allowing better removal of cache entries once we hit the max cache
size limit (although we don't implement any max cache size handling in this
initial implementation).
The primary reason for implementing a single file cache for now is to allow
better performance and handling by third party applications such as steam
that collect and distribute precompiled cache entries.
For this reason we also implement a new environment variable
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS which allows a user to pass in a path
to a number of external read only shader cache dbs. There is an initial
limit of 8 dbs that can be passed to mesa like so:
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS=/full_path/filename1, ... ,/full_path/filename8
Where the filename represents the cache db and its index file e.g.
filename1.foz and filename1_idx.foz
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7725>
This allows us to guarantee the different cache implementations will not
interfere with each other and should make it more clear that the max cache
size limits are applied separately for each cache implementation.
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7725>
This code is already OS agnostic and moving it here enables code
sharing in the following patches.
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7725>
This will be faster and avoids checking for errors with the
compression implementation which we shouldn't need to do. Instead
we trust the compression library does the correct thing and simply
error check the data loaded from disk.
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7725>
Anything higher than INT_MAX results in overflow although the parameter is declared as size_t.
Worse, with (size_t)-1 it is silently ignored and Woverflow is not emitted.
Closes#4226
Reviewed-by: Jose Fonseca <jfonseca@vmware.com>
Tested-by: Prodea Alexandru-Liviu <liviuprodea@yahoo.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9134>
../src/util/softfloat.c: In function ‘_mesa_shift_right_jam_m’:
../src/util/softfloat.c:432:16: warning: ‘tmp’ may be used uninitialized in this function [-Wmaybe-uninitialized]
432 | *tmp++ = 0;
You could actually hit this if you called _mesa_shift_right_jam_m with
size_words = 0 and dist < 32. Not that you'd _do_ that, but. In this
case do nothing instead of writing through an uninitialized pointer.
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8724>
