These stages are for the jobs that are skipped in merge pipelines,
automatically run in nightly pipelines, and are available to run
manually in other pipelines.
None of these ever run in post-merge pipelines.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34590>
The execution_times structure was filled the same way in all the cases of an
if. As a side effect, the retry evaluation doesn't include the last result and
can lead, as a race condition, into triggering one extra time than the
indication in the stress argument.
Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33750>
When one launches a stress test on a single job, the script behaves like the
stress number is not set. After this wrong end, relaunch the command works
only if stress is bigger than 2. In case 2, it can confuse the number of
executions.
When in stress mode, don't exit the monitor_pipeline method as if there were
only one job run. One job run, prints in std the job trace, but in stress
mode, there are more than one job execution.
The stress_status_counter structure lost the information about job IDs, and
the bug happens when it counts twice the same job.
Reported-by: Martin Roukala (né Peres) <martin.roukala@mupuf.org>
Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33750>
The former version was problematic because:
- time.perf_counter() returns seconds relative to an arbitrary point in
time (monotonic clock)
- time.mktime() converts to epoch time (seconds since 1970)
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Reviewed-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33750>
Modify the pretty_wait function to use a two-digit width for seconds
display, ensuring consistent and aligned output when showing the
countdown timer.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Reviewed-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33750>
Add error handling for job retry/play actions with proper exception
handling and improve status tracking. Introduce a maximum retry
limit with MAX_ENABLE_JOB_ATTEMPTS to prevent infinite loops when
jobs cannot be enabled.
Change enable_job to return a boolean status rather than job objects
to avoid race conditions with stale job state. Update callers to
properly handle the new return value and wait for fresh job data
in the next monitoring loop.
Address race conditions in the CI monitor script:
Reported-by: Timur Kristóf <timur.kristof@gmail.com>
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Reviewed-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33750>
Extract the target job processing logic from the monitor_pipeline
function into a dedicated run_target_job helper. This improves
readability and maintainability by reducing the complexity of the
monitor_pipeline function.
Also add type casting to ensure proper typing for the job objects and
import the necessary Callable type.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Reviewed-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33750>
Include the difference between start and end times, so it is easy to
check how much time it took.
This can be used for things like ensuring the test phase is under 10
minutes, as suggested in the documentation.
Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Reviewed-by: Vignesh Raman <vignesh.raman@collabora.com>
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34478>
Replace global path variables with ProjectPaths dataclass
- Add explicit file existence check before loading YAML
- Enhance tag retrieval by checking environment variables first
- Add logging for better debugging of tag selection process
- Remove redundant file existence check in main function
- Improve error handling for missing conditional tags file
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33863>
This reverts commit 1cc2c738bb
We originally changed some aliases into functions so scripts could use
them without needing to be sourced, keeping the environment cleaner.
However, this broke `x_off`, which is supposed to stop debug logs
(xtrace output) from showing in the console. The function version still
triggered xtrace before disabling it, while the alias correctly
redirected the logs to `/dev/null`.
It also fixes the `bin/ci/update_tag.py` script to be able to reuse the
aliases via double sourcing the setup-test-env.sh and the respective
build script.
Reported-by: Eric Engestrom <eric@igalia.com>
Reviewed-by: Eric Engestrom <eric@igalia.com>
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33863>
Make structural tagging functions available for both test and build
scripts.
Introduces the update_tag.sh helper for listing, checking, and updating
deterministic tags.
Also adds the ci_tag_build_time_check and ci_tag_test_time_check
functions to validate tags during build and test phases, ensuring
consistent component versioning.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33421>
The lavacli version 1.5.2 has been released in December 2022.
Use the most recent version 2.2.0, released in October 2023, instead.
Notable changes since 1.5.2:
- Authentication tokens are now stripped from exceptions when HTTP
requests fail. (1.6)
Signed-off-by: Leonard Göhrs <l.goehrs@pengutronix.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33266>
Update the trace update script to use get_token_from_default_dir for
reading the GitLab token from the default location (~/.config/gitlab-token),
fixing the script and aligning it with our other tools.
Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33061>
Currently marge will ignore an error message if it is just the word
"error" without any further information. However, if she never finds a
more informative message, then she will just print that meaningless
error message.
Instead of an empty error message, just leave the message blank.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32773>
If marge doesn't find deqp-runner results or trace errors, she will parse
a failed job log for error messages. The parsing ignores certain phrases
in an attempt to find more relevant errors.
Add a test to check marge's error log parsing to make sure that we don't
create regressions when adding new phrases to ignore.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32773>
Currently the update_traces_checksum script prints a label verification
request with a line that is 167 characters long.
Split the long line to make it more readable. Update the flake8
configuration to enforce a maximum line length of 159 characters to ensure
consistency.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32602>
Currently the ci scripts and tests don't have any linting checks. Add
.flake8 linting to start adding some consistency to the scripts. Ignore
most of the existing errors until they can be addressed on an individual
basis.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32602>
When the gantt chart service was first implemented, the read_token()
function would attempt to read from the default token file if the token
argument was missing.
Now it's necessary to call a separate function to look in the default
token file, so if the token argument is not provided, expressly call
get_token_from_default_dir().
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32637>
The gantt charts have a vertical line indicating the time when marge-bot
stops waiting for CI to finish. Currently the ci-timeout is hard-coded at
60 minutes. But marge-bot's timeout is configurable, so allow this value
to be configured as well.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32637>
If a job is still running when the gantt chart is generated, the job's
phases won't be represented as a bar on the chart.
To show these jobs on the gantt chart, introduce a new phase "In-Progress"
to capture the current status of these jobs. Rename the "Running" phase
as "Time spent running" to emphasize that the running phase is complete.
This provides a more complete picture of the pipeline jobs when CI times
out.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32637>
The gantt chart script currently expect the pipeline url to be the last
line of marge-bot's comment on a failed pipeline. The pipeline summary
hook now adds additional information that was preventing the gantt chart
script from finding the pipeline url.
Change the regex for finding the pipeline url to account for pipeline
summary comments.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32637>
Add a pipeline-id to the gantt filename so that the filename is unique.
Reduce the additional info provided in the gantt comments since the tool
is now more familiar to developers.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32637>
Currently the gantt chart script will post a comment to any project that
uses the default marge bot user id. Although the gantt chart is flexible
enough to work for any project, restrict it to mesa/mesa by default. Add
the --project-id option to add a custom project or list of projects to
analyze.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32637>
The gantt chart currently hard-codes the marge user id, but since the
script can be used for any marge-bot user, add an option to specify the
marge-bot user id. If no option is specified, the default marge-bot user
id is still used.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32637>
Move the tool to summarize a failed pipeline to a generic .marge/hooks
directory. This will allow the fdo-bots repo to handle all marge hooks in
a consistent way across repositories that use this service.
Add a symlink to the bin/ci directory so that the pipeline summary tool
can still be run locally as well.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32413>
The former version was presenting some bugs running fixtures in parallel
testing.
With the new version, we need to change the side effects on mocks a
little bit and fix a start time to make the tests reproducible.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32163>
Start by finding our known prefix, stripping only that off, then
validating that the rest of the data makes sense. This fixes certain
tokens which were erroneously getting rejected.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31270>
Now that we can exclude by stages instead, we don't need --force-manual.
Having mandatory arguments which also do the wrong thing is not great.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30784>
Add an argument to ci_run_n_monitor specifying certain stages to be
excluded from consideration, defaulting to the one with post-merge and
performance jobs. This allows, e.g., to run all Panfrost pre-merge jobs:
./ci_run_n_monitor.py --target 'panfrost.*'
or to run all Freedreno pre-merge jobs:
./ci_run_n_monitor.py --target '.*' --include-stage freedreno
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30784>
