The standard way to query options in mesa is `os_get_option()` which
abstracts platform-specific mechanisms to get config variables.
However in quite a few places `getenv()` is still used and this may
preclude controlling some options on some systems.
For instance it is not generally possible to use `MESA_DEBUG` on
Android.
So replace most `getenv()` occurrences with `os_get_option()` to
support configuration options more consistently across different
platforms.
Do the same with `secure_getenv()` replacing it with
`os_get_option_secure()`.
The bulk of the proposed changes are mechanically performed by the
following script:
-----------------------------------------------------------------------
#!/bin/sh
set -e
replace() {
# Don't replace in some files, for example where `os_get_option` is defined,
# or in external files
EXCLUDE_FILES_PATTERN='(src/util/os_misc.c|src/util/u_debug.h|src/gtest/include/gtest/internal/gtest-port.h)'
# Don't replace some "system" variables
EXCLUDE_VARS_PATTERN='("XDG|"DISPLAY|"HOME|"TMPDIR|"POSIXLY_CORRECT)'
git grep "[=!( ]$1(" -- src/ | cut -d ':' -f 1 | sort | uniq | \
grep -v -E "$EXCLUDE_FILES_PATTERN" | \
while read -r file;
do
# Don't replace usages of XDG_* variables or HOME
sed -E -e "/$EXCLUDE_VARS_PATTERN/!s/([=!\( ])$1\(/\1$2\(/g" -i "$file";
done
}
# Add const to os_get_option results, to avoid warning about discarded qualifier:
# warning: initialization discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
# but also errors in some cases:
# error: invalid conversion from ‘const char*’ to ‘char*’ [-fpermissive]
add_const_results() {
git grep -l -P '(?<!const )char.*os_get_option' | \
while read -r file;
do
sed -e '/^\s*const/! s/\(char.*os_get_option\)/const \1/g' -i "$file"
done
}
replace 'secure_getenv' 'os_get_option_secure'
replace 'getenv' 'os_get_option'
add_const_results
-----------------------------------------------------------------------
After this, the `#include "util/os_misc.h"` is also added in files where
`os_get_option()` was not used before.
And since the replacements from the script above generated some new
`-Wdiscarded-qualifiers` warnings, those have been addressed as well,
generally by declaring `os_get_option()` results as `const char *` and
adjusting some function declarations.
Finally some replacements caused new errors like:
-----------------------------------------------------------------------
../src/gallium/auxiliary/gallivm/lp_bld_misc.cpp:127:31: error: no matching function for call to 'strtok'
127 | for (n = 0, option = strtok(env_llc_options, " "); option; n++, option = strtok(NULL, " ")) {
| ^~~~~~
/android-ndk-r27c/toolchains/llvm/prebuilt/linux-x86_64/bin/../sysroot/usr/include/string.h:124:17: note: candidate function not viable: 1st argument ('const char *') would lose const qualifier
124 | char* _Nullable strtok(char* _Nullable __s, const char* _Nonnull __delimiter);
| ^ ~~~~~~~~~~~~~~~~~~~
-----------------------------------------------------------------------
Those have been addressed too, copying the const string returned by
`os_get_option()` so that it could be modified.
In particular, the error above has been fixed by copying the `const
char *env_llc_options` variable in
`src/gallium/auxiliary/gallivm/lp_bld_misc.cpp` to a `char *` which can
be tokenized using `strtok()`.
Reviewed-by: Eric Engestrom <eric@igalia.com>
Reviewed-by: Yonggang Luo <luoyonggang@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38128>
The check function will free the db or the file if the other is NULL,
however, in this case db and the file are passed to foz_destroy, which
will also free them. This results in a double free. Instead, check that
either is NULL, and then goto the error case to ensure proper cleanup.
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29881>
Add a new environment varible
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS_DYNAMICE_LIST that specifies a text
file containing a list of RO fossilize caches to load. The list file
is modifiable at runtime to allow for loading RO caches after
initialization unlike MESA_DISK_CACHE_READ_ONLY_FOZ_DBS.
The implementation spawns an updater thread that uses inotify to monitor
the list file for modifications, attempting to load new foz dbs added to
the list. Removing files from the list will not evict a loaded cache.
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS_DYNAMIC_LIST takes an absolute path.
The file must exist at initialization for updating to occur.
File names of foz dbs in the list file are new-line separated and take
relative paths to the default cache directory like
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS.
The maximum number of RO foz dbs is kept to 8 and is shared between
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS_DYNAMIC_LIST and
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS.
The intended use case for this feature is to allow prebuilt caches
to be downloaded and loaded asynchronously during app runtime.
Prebuilt caches be large (several GB) and depending on network
conditions would otherwise present extended wait time for caches
to be availible before app launch.
This will be used in Chrome OS.
Signed-off-by: Juston Li <justonli@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19328>
When loading multiple RO foz dbs, if a db fails to load, continue trying
to load other RO foz dbs instead of destroying the foz cache.
Preserve destroying the foz cache and not preceding to load RO caches
if the RW cache fails to load.
Signed-off-by: Juston Li <justonli@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19328>
Mesa utilizes only one type of cache at a time. This patch enables support
for combined reading from read-only Fossilize cache + non-foz read-write
caches.
From now on, a non-foz read-write caches will first try to retrieve data
from a read-only foz cache if new MESA_DISK_CACHE_COMBINE_RW_WITH_RO_FOZ
environment variable is set to true, otherwise the caching behaviour is
unchanged. The new flag has no effect when MESA_DISK_CACHE_SINGLE_FILE=1,
i.e. when the single-file foz cache is used.
This change allows us to ship a prebuilt RO caches for a certain
applications, while the rest of applications will benefit from the
regular RW caching that supports cache-size limitation. This feature
will be used by ChromeOS.
Usage example #1:
MESA_DISK_CACHE_DATABASE=0
MESA_DISK_CACHE_SINGLE_FILE=0
MESA_DISK_CACHE_COMBINE_RW_WITH_RO_FOZ=1
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS=rocache1,rocache2
Usage example #2:
MESA_DISK_CACHE_DATABASE=1
MESA_DISK_CACHE_SINGLE_FILE=0
MESA_DISK_CACHE_COMBINE_RW_WITH_RO_FOZ=1
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS=rocache1,rocache2
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18551>
The foz_prepare() doesn't perform cleanup on failure and then foz_destroy()
is never invoked for the foz_db, causing minor memory and FD leaks. Add the
cleanup to foz_prepare() error code paths. Make foz_destroy() to clear the
foz_db struct for consistency, right now the destroying is invoked only once,
but Mesa cache error code paths aren't trivial and may change in the future.
Suggested-by: Timothy Arceri <tarceri@itsqueeze.com>
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18551>
If the file ptr is not NULL then foz_destroy will also try to destroy it.
Fixes: eca6bb9540 ("util/fossilize_db: add basic fossilize db util to read/write shader caches")
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14671>
The flock is per-fd, not per thread, and we do it outside of the main mutex. This was
done to avoid having to wait in the mutex, but we can get a case where one ends up running
the body with the flock unlocked.
Fix this by adding a mutex that doesn't need to be locked for reads.
Fixes: 4f0f8133a3 "util/fossilize_db: Do not lock the fossilize db permanently."
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12266>
Don't anticipate seeing any partial written headers but just in case we
should probably wait on the lock to make sure whatever header was being
written is finished being written.
Fixes: 4f0f8133a3 "util/fossilize_db: Do not lock the fossilize db permanently."
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12204>
If things went perfectly parsed_offset was never updated for the
final entry and we'd seek_set to the start of the last entry. Is
fun when appending to the file next.
Fixes: 2ec1bff0f3 "util/fossilize_db: Split out reading the index."
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12204>
This thing is entirely opt-in wrt caring about it when writing to
a file anyway. Since we also lock the two at the same time and they
have an 1-1 relation we can just lock one of the two files. Saves
some syscalls.
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11485>
This avoids all locks for reads and using lock only while actually
writing.
This is enabled by doing two things:
1) Reading the index incrementally. This way we get new entries
written by other processes and do not write duplicate entries.
2) Taking the lock only during writes, and applying the incremental
read while holding the lock so we always append to the actual end of the file.
Fixes: eca6bb9540 ("util/fossilize_db: add basic fossilize db util to read/write shader caches")
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11485>
For non 64bit devices the key stored in hash_table_u64 is wrapped in
hash_key_u64 structure, which is never free.
This commit fixes this issue by just removing the user-defined
`delete_function` parameter in hash_table_u64_{destroy,clear} (which
nobody is using) and using instead a delete function to free this
structure.
Fixes: 608257cf82 ("i965: Fix INTEL_DEBUG=bat")
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10480>
We now handle compression in the shared cache item creation code.
Compressing the cache item header with the already compressed blob
doesn't help much so lets just remove it.
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9593>
We don't need to close the file here check_files_opened_successfully()
already does that for us if needed.
This fixes a crash when invalid filename/paths are provided to the
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS environment variable.
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9251>
My benchmarking shows no significant change in cache load times with a
single shader cache file vs the existing cache implementation of many
small files (tested with my spinning rust HDD).
However this new single file cache implementation does reduce the total
size on disk used by the shader cache. We have a problem with the existing
cache where writing tiny files to disk causes more disk space to be used than
is actually needed for the files due to the minimum size required for a file.
In pratice this tends to inflate the size of the cache on disk to over 3x
larger.
There are other advantages of using a single file for shader cache entries
such as allowing better removal of cache entries once we hit the max cache
size limit (although we don't implement any max cache size handling in this
initial implementation).
The primary reason for implementing a single file cache for now is to allow
better performance and handling by third party applications such as steam
that collect and distribute precompiled cache entries.
For this reason we also implement a new environment variable
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS which allows a user to pass in a path
to a number of external read only shader cache dbs. There is an initial
limit of 8 dbs that can be passed to mesa like so:
MESA_DISK_CACHE_READ_ONLY_FOZ_DBS=/full_path/filename1, ... ,/full_path/filename8
Where the filename represents the cache db and its index file e.g.
filename1.foz and filename1_idx.foz
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7725>
