fdo-mirrors/mesa
1
0
Fork 0
mirror of https://gitlab.freedesktop.org/mesa/mesa.git synced 2026-05-17 13:58:05 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 2
mesa/src/util/mesa_cache_db_multipart.h

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

49 lines
1.4 KiB
C
Raw Normal View History

util/mesa-db: Introduce multipart mesa-db cache Whenever a single file mesa-db cache hits max size limit, a half of cache is evicted and the cache file is defragmented. The downside of this eviction strategy is that it causes high disk IO usage during eviction if mesa-db cache file size is large. In order to mitigate this downside, we will split mesa-db into multiple part such that only one part will be evicted at a time. Each part will be an individual single file mesa-db cache, like a DB shard. The new multipart mesa-db cache will merge the parts into a single virtual cache. This patch introduces two new environment variables: 1. MESA_DISK_CACHE_DATABASE_NUM_PARTS: Controls number of mesa-db cache file parts. By default 50 parts will be created. The old pre-multipart mesa-db cache files will be auto-removed if they exist, i.e. Mesa will switch to the new DB version automatically. 2. MESA_DISK_CACHE_DATABASE_EVICTION_SCORE_2X_PERIOD: Controls the eviction score doubling time period. The evicted DB part selection is based on cache entries size weighted by 'last_access_time' of the entries. By default the cache eviction score is doubled for each month of cache entry age, i.e. for two equally sized entries where one entry is older by one month than the other, the older entry will have x2 eviction score than the other entry. Database part with a highest total eviction score is selected for eviction. This patch brings x40 performance improvement of cache eviction time using multipart cache vs a single file cache due to a smaller eviction portions and more optimized eviction algorithm. Acked-by: Timothy Arceri <tarceri@itsqueeze.com> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20256>
2022-10-24 13:24:16 +03:00
/*
* Copyright © 2022 Collabora, Ltd.
*
* SPDX-License-Identifier: MIT
*/
#ifndef MESA_CACHE_DB_MULTIPART_H
#define MESA_CACHE_DB_MULTIPART_H
#include "mesa_cache_db.h"
util/mesa-db-multipart: Open one cache part at a time Open one cache DB part at a time for a multi-part cache to reduce number of FDs used by the cache. Previously multi-part DB cache instance was consuming 100 FDs, now it's 2 and cache files are opened when cache is read or written instead of opening them at the init time. Fixes: fd9f7b748e2e ("util/mesa-db: Introduce multipart mesa-db cache") Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/11776 Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Acked-by: Timothy Arceri <tarceri@itsqueeze.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30988>
2024-09-03 13:51:18 +03:00
#include "simple_mtx.h"
util/mesa-db: Introduce multipart mesa-db cache Whenever a single file mesa-db cache hits max size limit, a half of cache is evicted and the cache file is defragmented. The downside of this eviction strategy is that it causes high disk IO usage during eviction if mesa-db cache file size is large. In order to mitigate this downside, we will split mesa-db into multiple part such that only one part will be evicted at a time. Each part will be an individual single file mesa-db cache, like a DB shard. The new multipart mesa-db cache will merge the parts into a single virtual cache. This patch introduces two new environment variables: 1. MESA_DISK_CACHE_DATABASE_NUM_PARTS: Controls number of mesa-db cache file parts. By default 50 parts will be created. The old pre-multipart mesa-db cache files will be auto-removed if they exist, i.e. Mesa will switch to the new DB version automatically. 2. MESA_DISK_CACHE_DATABASE_EVICTION_SCORE_2X_PERIOD: Controls the eviction score doubling time period. The evicted DB part selection is based on cache entries size weighted by 'last_access_time' of the entries. By default the cache eviction score is doubled for each month of cache entry age, i.e. for two equally sized entries where one entry is older by one month than the other, the older entry will have x2 eviction score than the other entry. Database part with a highest total eviction score is selected for eviction. This patch brings x40 performance improvement of cache eviction time using multipart cache vs a single file cache due to a smaller eviction portions and more optimized eviction algorithm. Acked-by: Timothy Arceri <tarceri@itsqueeze.com> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20256>
2022-10-24 13:24:16 +03:00
struct mesa_cache_db_multipart {
util/mesa-db-multipart: Open one cache part at a time Open one cache DB part at a time for a multi-part cache to reduce number of FDs used by the cache. Previously multi-part DB cache instance was consuming 100 FDs, now it's 2 and cache files are opened when cache is read or written instead of opening them at the init time. Fixes: fd9f7b748e2e ("util/mesa-db: Introduce multipart mesa-db cache") Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/11776 Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Acked-by: Timothy Arceri <tarceri@itsqueeze.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30988>
2024-09-03 13:51:18 +03:00
struct mesa_cache_db **parts;
util/mesa-db: Introduce multipart mesa-db cache Whenever a single file mesa-db cache hits max size limit, a half of cache is evicted and the cache file is defragmented. The downside of this eviction strategy is that it causes high disk IO usage during eviction if mesa-db cache file size is large. In order to mitigate this downside, we will split mesa-db into multiple part such that only one part will be evicted at a time. Each part will be an individual single file mesa-db cache, like a DB shard. The new multipart mesa-db cache will merge the parts into a single virtual cache. This patch introduces two new environment variables: 1. MESA_DISK_CACHE_DATABASE_NUM_PARTS: Controls number of mesa-db cache file parts. By default 50 parts will be created. The old pre-multipart mesa-db cache files will be auto-removed if they exist, i.e. Mesa will switch to the new DB version automatically. 2. MESA_DISK_CACHE_DATABASE_EVICTION_SCORE_2X_PERIOD: Controls the eviction score doubling time period. The evicted DB part selection is based on cache entries size weighted by 'last_access_time' of the entries. By default the cache eviction score is doubled for each month of cache entry age, i.e. for two equally sized entries where one entry is older by one month than the other, the older entry will have x2 eviction score than the other entry. Database part with a highest total eviction score is selected for eviction. This patch brings x40 performance improvement of cache eviction time using multipart cache vs a single file cache due to a smaller eviction portions and more optimized eviction algorithm. Acked-by: Timothy Arceri <tarceri@itsqueeze.com> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20256>
2022-10-24 13:24:16 +03:00
unsigned int num_parts;
volatile unsigned int last_read_part;
volatile unsigned int last_written_part;
util/mesa-db-multipart: Open one cache part at a time Open one cache DB part at a time for a multi-part cache to reduce number of FDs used by the cache. Previously multi-part DB cache instance was consuming 100 FDs, now it's 2 and cache files are opened when cache is read or written instead of opening them at the init time. Fixes: fd9f7b748e2e ("util/mesa-db: Introduce multipart mesa-db cache") Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/11776 Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Acked-by: Timothy Arceri <tarceri@itsqueeze.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30988>
2024-09-03 13:51:18 +03:00
const char *cache_path;
uint64_t max_cache_size;
simple_mtx_t lock;
util/mesa-db: Introduce multipart mesa-db cache Whenever a single file mesa-db cache hits max size limit, a half of cache is evicted and the cache file is defragmented. The downside of this eviction strategy is that it causes high disk IO usage during eviction if mesa-db cache file size is large. In order to mitigate this downside, we will split mesa-db into multiple part such that only one part will be evicted at a time. Each part will be an individual single file mesa-db cache, like a DB shard. The new multipart mesa-db cache will merge the parts into a single virtual cache. This patch introduces two new environment variables: 1. MESA_DISK_CACHE_DATABASE_NUM_PARTS: Controls number of mesa-db cache file parts. By default 50 parts will be created. The old pre-multipart mesa-db cache files will be auto-removed if they exist, i.e. Mesa will switch to the new DB version automatically. 2. MESA_DISK_CACHE_DATABASE_EVICTION_SCORE_2X_PERIOD: Controls the eviction score doubling time period. The evicted DB part selection is based on cache entries size weighted by 'last_access_time' of the entries. By default the cache eviction score is doubled for each month of cache entry age, i.e. for two equally sized entries where one entry is older by one month than the other, the older entry will have x2 eviction score than the other entry. Database part with a highest total eviction score is selected for eviction. This patch brings x40 performance improvement of cache eviction time using multipart cache vs a single file cache due to a smaller eviction portions and more optimized eviction algorithm. Acked-by: Timothy Arceri <tarceri@itsqueeze.com> Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20256>
2022-10-24 13:24:16 +03:00
};
bool
mesa_cache_db_multipart_open(struct mesa_cache_db_multipart *db,
const char *cache_path);
void
mesa_cache_db_multipart_close(struct mesa_cache_db_multipart *db);
void
mesa_cache_db_multipart_set_size_limit(struct mesa_cache_db_multipart *db,
uint64_t max_cache_size);
void *
mesa_cache_db_multipart_read_entry(struct mesa_cache_db_multipart *db,
const uint8_t *cache_key_160bit,
size_t *size);
bool
mesa_cache_db_multipart_entry_write(struct mesa_cache_db_multipart *db,
const uint8_t *cache_key_160bit,
const void *blob, size_t blob_size);
void
mesa_cache_db_multipart_entry_remove(struct mesa_cache_db_multipart *db,
const uint8_t *cache_key_160bit);
#endif /* MESA_CACHE_DB_MULTIPART_H */
Reference in a new issue Copy permalink