On some systems it is problematic to have the shader cache enabled
by default. This adds a build option to support the disk cache but
keep it disabled unless the environment variable
MESA_GLSL_CACHE_DISABLE=false.
For example, on Chrome OS, Chrome already has it's own shader
disk cache implementation so it disables the mesa feature. Tests
do not want the shader disk cache enabled because it can cause
inconsistent performance results and the default 1GB for the
disk cache could lead to problems that require more effort to
work around. The Mesa shader disk cache is useful for VMs though,
where it is easy to configure the feature with environment
variables. With the current version of Mesa, Chrome OS would need
to have a system-wide environment variable to disable the disk
cache everywhere except where needed. More elegant to just build
Mesa with the cache feature disabled by default.
Reviewed-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6967>
We need the disk cache enabled in Android to get EGL_ANDROID_blob_cache's
callbacks called, but we don't actually want to store anything on disk.
Fixes "Failed to create //.cache for shader cache (Read-only file
system)---disabling." spam on init.
Reviewed-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6762>
Doesn't really change anything to the test though ¯\_(ツ)_/¯
CID: 1429511
Fixes: e8495646af "glsl/tests: changes to test_disk_cache_create test"
Signed-off-by: Eric Engestrom <eric.engestrom@imgtec.com>
Reviewed-by: Brian Paul <brianp@vmware.com>
Reviewed-by: Tapani Pälli <tapani.palli@intel.com>
Next patch will allow disk_cache instance to be created without
path set for it, modify some test cases that assume disk_cache
creation to fail with invalid path. Creation should succeed but
simple put/get test fail.
v2: leave tests as is but check that both cache struct exists
and try simple put/get that should fail with invalid path set
(Emil)
Signed-off-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com> (v1)
Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
Patch moves functions higher so that we can utilize them from
test_disk_cache_create which is modified by next patch.
Signed-off-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
Instead of setting based on set/unset, allow users to use boolean values.
In the docs and tests, use `DISABLE=true` instead of `DISABLE=1` as it's
clearer IMO.
Signed-off-by: Eric Engestrom <eric.engestrom@imgtec.com>
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
Steam is already analysing cache items, unfortunatly we did not
introduce a versioning mechanism for identifying structural changes
to cache entries earlier so the only way to do so is to rename the
cache directory.
Since we are renaming it we take the opportunity to give the directory
a more meaningful name.
Adding a version field to the header of cache entries will help us to
avoid having to rename the directory in future. Please note this is
versioning for the internal structure of the entries as defined in
disk_cache.{c,h} as opposed to the structure of the data provided to
the disk cache by the GLSL compiler and the various driver backends.
Reviewed-by: Nicolai Hähnle <nicolai.haehnle@amd.com>
This will be used for things such as adding driver specific environment
variables to the key. Allowing us to set environment vars that change
the shader and not have the driver ignore them if it finds existing
shaders in the cache.
Reviewed-by: Eduardo Lima Mitev <elima@igalia.com>
When using an overlayfs system (like a Docker container), rmrf_local()
fails because part of the files to be removed are in different mount
points (layouts). And thus cache-test fails.
Letting crossing mount points is not a big problem, specially because
this is just for a test, not to be used in real code.
Reviewed-by: Nicolai Hähnle <nicolai.haehnle@amd.com>
While at it, also fix up a failure message to not reference timestamp
and gpu dirs as those are no longer being made.
Signed-off-by: Grazvydas Ignotas <notasas@gmail.com>
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
This allows to get rid of the arch and gpu name directories.
v2: (Timothy Arceri) don't use an opaque data type to store
pointer size and gpu name.
v3: (Timothy Arceri) use blob to store driver keys just make sure
to store null terminator for strings, and make sure blob is
defined by disk_cache and not it's users.
v4: (Timothy Arceri) fix typo, and make ptr_size a uint8_t.
Signed-off-by: Grazvydas Ignotas <notasas@gmail.com>
Reviewed-by: Nicolai Hähnle <nicolai.haehnle@amd.com>
Instead of using a directory, hash the timestamps into the cache keys
themselves. Since there is no more timestamp directory, there is no more
need for deleting the cache of other mesa versions and we rely on
eviction to clean up the old cache entries. This solves the problem of
using several incarnations of disk_cache at the same time, where one
deletes a directory belonging to the other, like when both OpenGL and
gallium nine are used simultaneously (or several different mesa
installations).
v2: using additional blob instead of trying to clone sha1 state
v3: (Timothy Arceri) don't use an opaque data type to store
timestamp.
V4: (Timothy Arceri) use blob to store driver keys just make sure
to store null terminator for strings, and make sure blob is
defined by disk_cache and not it's users.
Signed-off-by: Grazvydas Ignotas <notasas@gmail.com>
Reviewed-by: Nicolai Hähnle <nicolai.haehnle@amd.com>
Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=100091
There is no need to hardcode it, we can just use blob_key[0].
This is needed because the next patches are going to change how cache
keys are computed.
Signed-off-by: Grazvydas Ignotas <notasas@gmail.com>
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
This will allow to hash additional data into the cache keys or even
change the hashing algorithm easily, should we decide to do so.
v2: don't try to compute key (and crash) if cache is disabled
Signed-off-by: Grazvydas Ignotas <notasas@gmail.com>
Reviewed-by: Timothy Arceri <tarceri@itsqueeze.com>
Currently only a one in one out eviction so if at max_size and
cache files were to constantly increase in size then so would the
cache. Restrict to limit of 8 evictions per new cache entry.
V2: (Timothy Arceri) fix make check tests
Reviewed-by: Grazvydas Ignotas <notasas@gmail.com>
This should help reduce any overhead added by the shader cache
when programs are not found in the cache.
To avoid creating any special function just for the sake of the
tests we add a one second delay whenever we call dick_cache_put()
to give it time to finish.
V2: poll for file when waiting for thread in test
V3: fix poll delay to really be 100ms, and simplify the wait function
Reviewed-by: Grazvydas Ignotas <notasas@gmail.com>
This reverts commit 0f60c6616e.
Piglit and all games tested so far seem to be working without
issue. This change will allow wide user testing and we can decided
before the next release if we need to turn it off again.
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
In order to avoid costly fallback recompiles when cache items are
created with an old version of Mesa or for a different gpu on the
same system we want to create directories that look like this:
./{TIMESTAMP}_{LLVM_TIMESTAMP}/{GPU_ID}
Note: The disk cache util will take a single timestamp string, it is
up to the backend to concatenate the llvm string with the mesa string
if applicable.
Reviewed-by: Nicolai Hähnle <nicolai.haehnle@amd.com>
The shader cache is expected to be developed incrementally over a
fairly long series of commits. For that period of instability, we
require users to opt into the shader cache by setting:
MESA_GLSL_CACHE_ENABLE=1
In the future, when the shader cache is complete, we can revert this
commit so that the cache will be on by default.
The user can always disable the cache with
MESA_GLSL_CACHE_DISABLE=1. That functionality is not affected by this
commit, (nor will it be affected by the future revert).
Reviewed-by: Eric Anholt <eric@anholt.net>
Earlier commit imported a SHA1 implementation and relaxed the SHA1 and
disk cache handling, broking the Windows builds.
Restrict things for now until we get to a proper fix.
Fixes: d1efa09d34 "util: import sha1 implementation from OpenBSD"
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
At the moment we support 5+ different implementations each with varying
amount of bugs - from thread safely problems [1], to outright broken
implementation(s) [2]
In order to accommodate these we have 150+ lines of configure script and
extra two configure toggles. Whist an actual implementation being
~200loc and our current compat wrapping ~250.
Let's not forget that different people use different code paths, thus
effectively makes it harder to test and debug since the default
implementation is automatically detected.
To minimise all these lovely experiences, import the "100% Public
Domain" OpenBSD sha1 implementation. Clearly document any changes needed
to get building correctly, since many/most of those can be upstreamed
making future syncs easier.
As an added bonus this will avoid all the 'fun' experiences trying to
integrate it with the Android and SCons builds.
v2: Manually expand __BEGIN_DECLS/__END_DECLS and document (Tapani).
Furthermore it seems that some games (or surrounding runtime) static
link against OpenSSL resulting in conflicts. For more information see
the discussion thread [3]
Bugzilla [1]: https://bugs.freedesktop.org/show_bug.cgi?id=94904
Bugzilla [2]: https://bugs.freedesktop.org/show_bug.cgi?id=97967
[3] https://lists.freedesktop.org/archives/mesa-dev/2017-January/140748.html
Cc: Mark Janes <mark.a.janes@intel.com>
Cc: Vinson Lee <vlee@freedesktop.org>
Cc: Tapani Pälli <tapani.palli@intel.com>
Cc: Jonathan Gray <jsg@jsg.id.au>
Tested-by: Jonathan Gray <jsg@jsg.id.au>
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
Acked-by: Tapani Pälli <tapani.palli@intel.com> (v1)
Acked-by: Jason Ekstrand <jason@jlekstrand.net> (v1)
It's not dependent on GLSL and it can be useful for shader caches that don't
deal with GLSL.
v2: address review comments
v3: keep the other 3 lines in configure.ac
Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
This code provides for an on-disk cache of objects. Objects are stored
and retrieved via names that are arbitrary 20-byte sequences,
(intended to be SHA-1 hashes of something identifying for the
content). The directory used for the cache can be specified by means
of environment variables in the following priority order:
$MESA_GLSL_CACHE_DIR
$XDG_CACHE_HOME/mesa
<user-home-directory>/.cache/mesa
By default the cache will be limited to a maximum size of 1GB. The
environment variable:
$MESA_GLSL_CACHE_MAX_SIZE
can be set (at the time of GL context creation) to choose some other
size. This variable is a number that can optionally be followed by
'K', 'M', or 'G' to select a size in kilobytes, megabytes, or
gigabytes. By default, an unadorned value will be interpreted as
gigabytes.
The cache will be entirely disabled at runtime if the variable
MESA_GLSL_CACHE_DISABLE is set at the time of GL context creation.
Many thanks to Kristian Høgsberg <krh@bitplanet.net> for the initial
implementation of code that led to this patch. In particular, the idea
of using an mmapped file, (indexed by a portion of the SHA-1), for the
efficent implementation of cache_has_key was entirely his
idea. Kristian also provided some very helpful advice in discussions
regarding various race conditions to be avoided in this code.
Signed-off-by: Timothy Arceri <timothy.arceri@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
