Useful for potentially huge bitsets that are expected to be mostly
filled with zeroes, reducing memory consumption by assuming bits being
zero by default (without wasting memory to store zeroes).
Co-authored-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/37908>
The standard way to query options in mesa is `os_get_option()` which
abstracts platform-specific mechanisms to get config variables.
However in quite a few places `getenv()` is still used and this may
preclude controlling some options on some systems.
For instance it is not generally possible to use `MESA_DEBUG` on
Android.
So replace most `getenv()` occurrences with `os_get_option()` to
support configuration options more consistently across different
platforms.
Do the same with `secure_getenv()` replacing it with
`os_get_option_secure()`.
The bulk of the proposed changes are mechanically performed by the
following script:
-----------------------------------------------------------------------
#!/bin/sh
set -e
replace() {
# Don't replace in some files, for example where `os_get_option` is defined,
# or in external files
EXCLUDE_FILES_PATTERN='(src/util/os_misc.c|src/util/u_debug.h|src/gtest/include/gtest/internal/gtest-port.h)'
# Don't replace some "system" variables
EXCLUDE_VARS_PATTERN='("XDG|"DISPLAY|"HOME|"TMPDIR|"POSIXLY_CORRECT)'
git grep "[=!( ]$1(" -- src/ | cut -d ':' -f 1 | sort | uniq | \
grep -v -E "$EXCLUDE_FILES_PATTERN" | \
while read -r file;
do
# Don't replace usages of XDG_* variables or HOME
sed -E -e "/$EXCLUDE_VARS_PATTERN/!s/([=!\( ])$1\(/\1$2\(/g" -i "$file";
done
}
# Add const to os_get_option results, to avoid warning about discarded qualifier:
# warning: initialization discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
# but also errors in some cases:
# error: invalid conversion from ‘const char*’ to ‘char*’ [-fpermissive]
add_const_results() {
git grep -l -P '(?<!const )char.*os_get_option' | \
while read -r file;
do
sed -e '/^\s*const/! s/\(char.*os_get_option\)/const \1/g' -i "$file"
done
}
replace 'secure_getenv' 'os_get_option_secure'
replace 'getenv' 'os_get_option'
add_const_results
-----------------------------------------------------------------------
After this, the `#include "util/os_misc.h"` is also added in files where
`os_get_option()` was not used before.
And since the replacements from the script above generated some new
`-Wdiscarded-qualifiers` warnings, those have been addressed as well,
generally by declaring `os_get_option()` results as `const char *` and
adjusting some function declarations.
Finally some replacements caused new errors like:
-----------------------------------------------------------------------
../src/gallium/auxiliary/gallivm/lp_bld_misc.cpp:127:31: error: no matching function for call to 'strtok'
127 | for (n = 0, option = strtok(env_llc_options, " "); option; n++, option = strtok(NULL, " ")) {
| ^~~~~~
/android-ndk-r27c/toolchains/llvm/prebuilt/linux-x86_64/bin/../sysroot/usr/include/string.h:124:17: note: candidate function not viable: 1st argument ('const char *') would lose const qualifier
124 | char* _Nullable strtok(char* _Nullable __s, const char* _Nonnull __delimiter);
| ^ ~~~~~~~~~~~~~~~~~~~
-----------------------------------------------------------------------
Those have been addressed too, copying the const string returned by
`os_get_option()` so that it could be modified.
In particular, the error above has been fixed by copying the `const
char *env_llc_options` variable in
`src/gallium/auxiliary/gallivm/lp_bld_misc.cpp` to a `char *` which can
be tokenized using `strtok()`.
Reviewed-by: Eric Engestrom <eric@igalia.com>
Reviewed-by: Yonggang Luo <luoyonggang@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38128>
Use os_get_option_dup in
src/gallium/auxiliary/gallivm/lp_bld_misc.cpp
src/util/tests/process_test.c
because the string is going to be modified.
Use os_get_option_dup in device_select_layer.c are because the string is being assigned to a
struct member (protection for the future), and also because the consecutive usages (protection for the present).
Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Yiwei Zhang <zzyiwei@chromium.org>
Acked-by: Antonio Ospite <antonio.ospite@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38128>
Most of the time, we can infer the type to append in
util_dynarray_append using __typeof__, which is standardized in C23 and
support in Jesse's MSMSVCV. This patch drops the type argument most of
the time, making util_dynarray a little more ergonomic to use.
This is done in four steps.
First, rename util_dynarray_append -> util_dynarray_append_typed
bash -c "find . -type f -exec sed -i -e 's/util_dynarray_append(/util_dynarray_append_typed(/g' \{} \;"
Then, add a new append that infers the type. This is much more ergonomic
for what you want most of the time.
Next, use type-inferred append as much as possible, via Coccinelle
patch (plus manual fixup):
@@
expression dynarray, element;
type type;
@@
-util_dynarray_append_typed(dynarray, type, element);
+util_dynarray_append(dynarray, element);
Finally, hand fixup cases that Coccinelle missed or incorrectly
translated, of which there were several because we can't used the
untyped append with a literal (since the sizeof won't do what you want).
All four steps are squashed to produce a single patch changing every
util_dynarray_append call site in tree to either drop a type parameter
(if possible) or insert a _typed suffix (if we can't infer). As such,
the final patch is best reviewed by hand even though it was
tool-assisted.
No Long Linguine Meals were involved in the making of this patch.
Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Acked-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/38038>
Wasn't 100% sure about some of these derivations so let's add tests.
Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@intel.com>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/37200>
For getting clc_helpers.cpp can be compiled with gcc/mingw
Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Eric Engestrom <eric@igalia.com>
Reviewed-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36597>
Over past months a performance issue was found with the Mesa-DB cache
implementation that results in a too slow cache startup time when cache is
full. A better indexing strategy will need to be invented to mitigate the
issue. Until then, let's default back to the multi-file cache.
Suggested-by: Michel Dänzer <mdaenzer@redhat.com>
Acked-by: Timothy Arceri <tarceri@itsqueeze.com>
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34199>
Instead of using unreliable polling to wait for foz db updater to parse
and load from the dynamic list, also use inotify to wait for foz db
updater close the list file after its done updating.
Fixes: 4dfd306454 ("disk_cache: Disable the "List" test for RO disk cache.")
Signed-off-by: Juston Li <justonli@google.com>
Reviewed-by: David Heidelberg <david@ixit.cz>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32237>
Add Mesa-DB regression test for a segfault bug that happened when a cache
entry bigger than size-limit of the cache is added to empty cache.
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Acked-by: Timothy Arceri <tarceri@itsqueeze.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30988>
Now we have a few extra methods and things are diverging a bit between
Linux and Windows. Add a few unit tests to make sure this works.
Signed-off-by: Daniel Almeida <daniel.almeida@collabora.com>
Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30594>
Only create 8 threads rather than 16, since otherwise that can hammer
fork a bit too much. Since it creates a bunch of threads, don't try to
run it in parallel with other tests either.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31059>
In !25925 I attempted to fix an error spam caused by the disk cache not
creating parent directories of the cache directory when needed,
resulting in failure to create the disk cache. Presumably that is bad
for performance.
Unfortunately I did a really bad job and only fixed the edge case where
the cache dir is given by $MESA_SHADER_CACHE_DIR rather than the general
case. Here I attempt to be more comprehensive.
Fixes#8294
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30160>
The `capture_not_overwritten` unit test captures and compares two
backtraces -- one from inside a call to `func_c` and one outside -- and
confirms that they are not identical. That is, that `func_c` is in the
backtrace.
On 32-bit x86, without `-fno-omit-frame-pointer`, the function will not
emit a stack frame. As a result, the unit test fails.
The fix is to compile `func_c` with the flag `-fno-omit-frame-pointer`
to prevent the compiler from optimizing out the stack frame which is
otherwise unneeded.
Bug: https://bugs.gentoo.org/823774
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/4091
Fixes: d0d14f3f64 ("util: Add unit test for stack backtrace caputure")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30622>
Allows a driver to declare indirect arguments for its tracepoints and
pass an address. u_trace will request a copy of the data which should
be implemented on the command processor.
Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Co-Authored-by: Danylo Piliaiev <dpiliaiev@igalia.com>
Reviewed-by: Danylo Piliaiev <dpiliaiev@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29944>
Looking at each usage of DETECT_OS_UNIX, it's more about the POSIX API usage, not the
Unix-like OS, so let's rename it
And for POSIX it's a standard to claim which API present, but for UNIX there is no such thing
Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Acked-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Gurchetan Singh <gurchetansingh@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29555>
except leaving u_endian.h behind to use __ANDROID__ directly to be
consistent with the rest in that file, which deserves a different
refactor
Signed-off-by: Yiwei Zhang <zzyiwei@chromium.org>
Reviewed-by: Yonggang Luo <luoyonggang@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27374>
Currently the shader cache is broken when running under flatpak-spawn
because the shader cache's parent directory will not exist. For example,
the shader cache directory might be:
/home/mcatanzaro/.var/app/org.gnome.Epiphany.Devel/cache
If /home/mcatanzaro/.var/app/org.gnome.Epiphany.Devel/ does not already
exist, we fail. Let's create the directories recursively, as if by
'mkdir -p', rather than just fail.
Fixes#8294
Reviewed-by: Adam Jackson <ajax@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25925>
The default value remains 2048, which is also used for rounding up
any user provided size.
The option is useful in cases where there's a better idea of the
amount of data that's going to be used.
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25279>
An "augmented tree" is a tree with extra data attached which flows from
the leaves to the root. An "interval tree" is a datastructure of
(potentially-overlapping) intervals where, in addition to inserting and
removing intervals, we can quickly lookup all the intervals which
overlap a given interval.
After describing red-black trees, CLRS explains how it's possible to
implement an interval tree using an augmented red-black tree where the
nodes are ordered by interval start and each node also stores the
maximum interval end for its entire subtree.
Implement the interval tree extension described by CLRS. Iterating over
all overlapping intervals is actually an exercise, so we have to solve
the exercise. The recursive solution has been re-written to use the
parent pointers to avoid needing a stack, similarly to rb_tree_first()
and rb_node_next().
For now, we only implement unsigned intervals, but the core algorithms
are all abstracted to allow other types. There's still some boilerplate,
but it's the best that can be done in C.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22071>
In the linear allocator, when a size larger than the minimum
buffer size is allocated, we currently create the new buffer
to fit exactly the requested size.
In that case, don't bother updating the `latest` pointer, since
this newly created buffer is already full. In the worst case,
the current `latest` is full and it would be the same; in the
best case, there's still room for small allocations, so we avoid
wasting space if the next allocations would fit.
Reviewed-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25517>
Not comprehensive, but those were the ones used to work on the
previous linear_alloc changes. Also having a test already
set up lower the barrier to add more tests for future in case of
bugs.
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25280>
The disk_cache_create() now always returns valid cache even when disk
cache is disabled. In a case of disabled cache, the disk cache is NO-OP.
Test whether get/put() work as expected for the disabled cache.
Reviewed-by: Rob Clark <robdclark@chromium.org>
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24985>
Previous commit decoupled EGL_ANDROID_blob_cache from the disk cache
and haven't updated the SHADER_CACHE_DISABLE_BY_DEFAULT test-case that
is failing because now cache is always created even if disk cache is
disabled, such cache is NO-OP in this case. Fix the failing test.
Fixes: 39f26642 ("util: Decouple disk cache from EGL_ANDROID_blob_cache")
Reviewed-by: Rob Clark <robdclark@chromium.org>
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24985>
Test for disabled cache was removed when we decoupled
EGL_ANDROID_blob_cache from the disk cache because test was failing
since it became outdated. Add the updated test.
Fixes: 39f26642 ("util: Decouple disk cache from EGL_ANDROID_blob_cache")
Reviewed-by: Rob Clark <robdclark@chromium.org>
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24985>
Previously rb_tree_search would return the first node encountered, but
that may not be the first node that would be encoutnered by, say,
rb_tree_foreach.
Two test cases are added.
v2: Add more curly braces. Suggested by Faith.
Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24856>
Just because the user / system-integrater doesn't want shader disk
cache, doesn't mean they don't want EGL_ANDROID_blob_cache to work.
We've kind of already solved this for the android case, so just
generalize that solution.
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/9520
Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24636>
It uses a poll function that waits for a second hoping for another thread
to catch up, which is not a reliable way to do synchronization. The test
has been spuriously failing merges on a regular basis recently.
This is issue #9222, which I'm leaving open until the author can fix the test.
Fixes: 3b69b67545 ("util/fossilize_db: add runtime RO foz db loading via FOZ_DBS_DYNAMIC_LIST")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24755>
Guard util to be standalone by move subdir('util') just behind subdir('android_stub')
texcompress_rgtc_tmp.h is not a standard c header, do not treat it as source
Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19522>
Make the `error` variable static to prevent a clash with
the glibc error() function when LTO is used.
Fixes the LTO build.
Otherwise, it'll fail in the linking phase with a conflict:
```
[411/2321] Linking target src/util/process_test
FAILED: src/util/process_test
c++ -o src/util/process_test src/util/process_test.p/tests_process_test.c.o -flto -Wl,--as-needed -Wl,--no-undefined -Wl,--fatal-warnings -Wl,--start-group src/util/libmesa_util.a src/util/format/libmesa_format.a src/util/libmesa_util_sse41.a src/c11/impl/libmesa_util_c11.a subprojects/perfetto/libperfetto.a /usr/lib/x86_64-linux-gnu/libz.so -pthread -lm -ldl /usr/lib/x86_64-linux-gnu/libunwind.so -Wl,--end-group
mold: error: symbol type mismatch: error
>>> defined in /tmp/process_test.SLc9I6.ltrans0.ltrans.o as STT_OBJECT
>>> defined in /lib/x86_64-linux-gnu/libc.so.6 as STT_FUNC
```
Reviewed-by: Yonggang Luo <luoyonggang@gmail.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21511>
Whenever a single file mesa-db cache hits max size limit, a half of cache
is evicted and the cache file is defragmented. The downside of this eviction
strategy is that it causes high disk IO usage during eviction if mesa-db
cache file size is large.
In order to mitigate this downside, we will split mesa-db into multiple
part such that only one part will be evicted at a time. Each part will be
an individual single file mesa-db cache, like a DB shard. The new multipart
mesa-db cache will merge the parts into a single virtual cache.
This patch introduces two new environment variables:
1. MESA_DISK_CACHE_DATABASE_NUM_PARTS:
Controls number of mesa-db cache file parts. By default 50 parts will be
created. The old pre-multipart mesa-db cache files will be auto-removed
if they exist, i.e. Mesa will switch to the new DB version automatically.
2. MESA_DISK_CACHE_DATABASE_EVICTION_SCORE_2X_PERIOD:
Controls the eviction score doubling time period. The evicted DB part
selection is based on cache entries size weighted by 'last_access_time' of
the entries. By default the cache eviction score is doubled for each month
of cache entry age, i.e. for two equally sized entries where one entry is
older by one month than the other, the older entry will have x2 eviction
score than the other entry. Database part with a highest total eviction
score is selected for eviction.
This patch brings x40 performance improvement of cache eviction time using
multipart cache vs a single file cache due to a smaller eviction portions
and more optimized eviction algorithm.
Acked-by: Timothy Arceri <tarceri@itsqueeze.com>
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20256>
The cache_exists() function adds a dummy cache entry that may affect cache
eviction testing results. Remove this entry.
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20256>
The environment variables are persistent and must be explicitly unset by
each cache test, otherwise next test will fail. Add the missing unsets to
the Cache.List test.
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20256>
Allow the loading process to affect driconf option matching without
changing the behavior throughout mesa common code or leaking the name of
the loading process to logs, artifact storage, or in sub-thread naming,
as can be the case with the broader MESA_PROCESS_NAME override.
This new MESA_DRICONF_EXECUTABLE_OVERRIDE takes higher precedence over
MESA_PROCESS_NAME in the case where both are set.
Signed-off-by: Ryan Neph <ryanneph@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20779>
Also deprecate GALLIUM_PROCESSS_NAME in favor of MESA_PROCESS_NAME,
while maintaining existing functionality for use cases relying on
GALLIUM_PROCESSS_NAME.
GALLIUM_PROCESSS_NAME takes higher precedence over MESA_PROCESS_NAME in
the case where both are set.
Signed-off-by: Ryan Neph <ryanneph@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20779>
