mesa/.gitlab-ci/lava/utils
Deborah Brouwer 816c835c84 ci/lava: Detect a6xx gpu recovery failures
Sporadically a6xx gpu will fail to recover causing the lava job
a660_vk_full to loop on error messages for three hours before timing
out.

A few sporadic error messages may still be recoverable, but when multiple
errors occur over a short period, successful recovery is unlikely. Parse
the logs to look for repeated error messages within a short time period.
If found, cancel the lava job and rerun it.

Also add unit tests for this behaviour.

cc: mesa-stable

Reported-by: Valentine Burley <valentine.burley@gmail.com>
Acked-by: Daniel Stone <daniel.stone@collabora.com>
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Signed-off-by: Deborah Brouwer <deborah.brouwer@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30032>
(cherry picked from commit 72c182f873)
2024-07-23 22:28:07 +02:00
..
__init__.py ci/lava: Create LAVAJobDefinition 2023-11-02 03:31:50 +00:00
console_format.py ci/lava: Add canceled job status 2022-07-08 12:26:05 +00:00
constants.py ci/lava: Detect a6xx gpu recovery failures 2024-07-23 22:28:07 +02:00
gitlab_section.py ci/lava: Update LogFollower for better section handling and history 2023-04-19 14:36:37 +00:00
lava_farm.py ci/lava: Add LavaFarm class to find LAVA farm from runner tag 2023-02-16 13:08:41 +00:00
lava_job.py ci/lava: Introduce unretriable exception handling 2024-04-22 21:20:07 +00:00
lava_job_definition.py ci: Use id_tokens for JWT auth 2024-05-15 15:37:05 +02:00
lava_log_hints.py ci/lava: Detect a6xx gpu recovery failures 2024-07-23 22:28:07 +02:00
lava_proxy.py ci/lava: Extract LAVA proxy and LAVAJob abstractions 2023-04-19 14:36:37 +00:00
log_follower.py ci/lava: Ignore DUT feedback messages 2024-01-24 18:39:17 +00:00
log_section.py ci/lava: Distinguish test suites in DUT vs Docker 2023-05-19 14:45:17 +00:00
ssh_job_definition.py ci/lava: Add unit tests covering job definition 2023-11-02 03:31:50 +00:00
uart_job_definition.py svga/ci: land vmware mesa-ci lava farm 2024-03-27 10:30:23 +00:00