Check the arguments to have a consistent definition of metavars to control the
string shown, and have them all lowercase. Also define the types expected on
those arguments.
Signed-off-by: Sergi Blanch-Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35828>
The --rev argument mentions that it accepts a git revision. I think it would
be practical to describe that it accepts a commit id, as well as a tag or
branch name.
Signed-off-by: Sergi Blanch-Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35828>
ci-run_n_monitor tool can work with more projects than Mesa. It would be split
to its own repo with transparent access from the scripts in Mesa. But
meanwhile, this simple change allows using more GitLab instances than the
default gl.fd.o.
Signed-off-by: Sergi Blanch-Torne <sergi.blanch.torne@collabora.com>
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Acked-by: Eric Engestrom <eric@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35826>
8f557b84f6 was done because `text` is sometimes just an int,
but the fix was only applies to the padding calculation.
Unfortunately, the padding direction is also different between strings
and integers, which means the behaviour is now incoherent.
Let's convert `text` to a string before we start doing anything so that
everything afterwards is coherent.
Fixes: 8f557b84f6 ("ci: crnm: fix hyperlink format")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35587>
Some text paddings are not enough during the execution of the script. Having
an adaptative way for this variable can help the structure and visibility of
the output.
Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34856>
It's another example of a long list of job names, when the jobs waiting for a
status update is printed. The print formatted list can help to present it.
Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34856>
Rewrite how the jobs are cancelled within a pool of threads to simplify how
it can then use the print formatted list to for a better presentation of the
results.
Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34856>
When we are printing a long list, and it needs more than one line, it gets
hard to review the content. This is an idea to group the elements to be
printed in columns to make it easier to identify individual elements in
the output.
Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34856>
These stages are for the jobs that are skipped in merge pipelines,
automatically run in nightly pipelines, and are available to run
manually in other pipelines.
None of these ever run in post-merge pipelines.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34590>
The execution_times structure was filled the same way in all the cases of an
if. As a side effect, the retry evaluation doesn't include the last result and
can lead, as a race condition, into triggering one extra time than the
indication in the stress argument.
Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33750>
When one launches a stress test on a single job, the script behaves like the
stress number is not set. After this wrong end, relaunch the command works
only if stress is bigger than 2. In case 2, it can confuse the number of
executions.
When in stress mode, don't exit the monitor_pipeline method as if there were
only one job run. One job run, prints in std the job trace, but in stress
mode, there are more than one job execution.
The stress_status_counter structure lost the information about job IDs, and
the bug happens when it counts twice the same job.
Reported-by: Martin Roukala (né Peres) <martin.roukala@mupuf.org>
Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33750>
The former version was problematic because:
- time.perf_counter() returns seconds relative to an arbitrary point in
time (monotonic clock)
- time.mktime() converts to epoch time (seconds since 1970)
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Reviewed-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33750>
Modify the pretty_wait function to use a two-digit width for seconds
display, ensuring consistent and aligned output when showing the
countdown timer.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Reviewed-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33750>
Add error handling for job retry/play actions with proper exception
handling and improve status tracking. Introduce a maximum retry
limit with MAX_ENABLE_JOB_ATTEMPTS to prevent infinite loops when
jobs cannot be enabled.
Change enable_job to return a boolean status rather than job objects
to avoid race conditions with stale job state. Update callers to
properly handle the new return value and wait for fresh job data
in the next monitoring loop.
Address race conditions in the CI monitor script:
Reported-by: Timur Kristóf <timur.kristof@gmail.com>
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Reviewed-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33750>
Extract the target job processing logic from the monitor_pipeline
function into a dedicated run_target_job helper. This improves
readability and maintainability by reducing the complexity of the
monitor_pipeline function.
Also add type casting to ensure proper typing for the job objects and
import the necessary Callable type.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Reviewed-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33750>
Now that we can exclude by stages instead, we don't need --force-manual.
Having mandatory arguments which also do the wrong thing is not great.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30784>
Add an argument to ci_run_n_monitor specifying certain stages to be
excluded from consideration, defaulting to the one with post-merge and
performance jobs. This allows, e.g., to run all Panfrost pre-merge jobs:
./ci_run_n_monitor.py --target 'panfrost.*'
or to run all Freedreno pre-merge jobs:
./ci_run_n_monitor.py --target '.*' --include-stage freedreno
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30784>
Let's be defensive and use `frozenset` from Python to avoid changing
global variables during the runtime (or any static part of code).
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30526>
The skip_follow_statuses variable, used to check if we need to stay
monitoring the pipeline instead of jumping to the target job traces, is
based on COMPLETED_STATUSES set. But, in Python, we do shallow copies by
default, and changes on skip_follow_statuses reflected on
COMPLETED_STATUSES, which was making manual dependencies stop playing
when --force-manual was not given.
Fixes: 84d401aebf0832741716f947dd7e2e9aac1221ac
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30526>
Some recurrent messages are only cluttering the ci_run_n_monitor's logs,
so let's use the easily provided @cache decorator to print stuff that
the user will only needs to see once.
Also removes some junk that can happen during the pipeline monitoring
loop, like the "----" line break, and newlines.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30361>
We stop monitoring pipeline changes when we find out that the target job
is already running, or is in a complete state.
But when we `--force-manual` we should consider that the job is not
complete and we should iterate a little more in the monitor pipeline
stage until it gets ready for trace following.
Closes: #11552
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30361>
Jobs that we want to run can get the `created` status when its
dependencies are still running.
So let's gather this information and ensure that we will wait these jobs
to reach the `manual` or `running` status before jumping to monitor
target jobs trace.
Closes: #11517
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30361>
This reverts commit 032d4a20f9.
The `if not to_cancel: return` was a red herring as what actually matters
is the job status, which is checked in each cancel_job() call, so we
can't know in advance whether anything will be cancelled, so let's just
drop this text explanation.
In the meantime we've also improved the emoji next to cancelled jobs, so
let's hope there is no longer any need to explain what this long list of
job names means.
Fixes: 032d4a20f9 ("bin/ci_run_n_monitor: explain that the 'Universal Recycling symbol' ♲ emoji means these jobs were cancelled")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30265>
IMO emojis are nice to add next to the information to recognize things
easily in cases where they are visible, but they should not *replace*
the information.
This adds a readable text next to all emojis (some already had one,
like the "job duration" ones).
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30223>
The job loops, w/o and explicit sort, are sorted by the job id. As they produce
logs, humans could feel an improvement by sorting by name.
Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29444>
Formatting the output lines with a padding in the job name fields may help
the readability of the information when one tries to follow the progress of
a specific job.
Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29444>
Coding style homogenous for all the methods in the file. Document
unicode symbols used.
Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29444>
A couple of these were trivially missing the `created` status which
usually doesn't stay for long enough for this very slow script to notice.
The `cancel_job()` function will no longer cancel manual jobs waiting to
be started.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29917>
The job duration is printed during the test progress. It can be collected and
summarized at the end of the process. It is interesting when doing a stress
test as one will have this information together at the end.
Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29419>
When the tool is used to stress test a pipeline, if there are jobs already ran,
use their information like it does when the stress flag is not set.
This provides consistency between the behavior when stress argument is not set,
to when it is set. When it is not set, it uses the information about jobs that
are already done. When it is set, it has to use the information about the
already ran jobs. Also, it saves resources by triggering the minimum required.
Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29432>
The stress counter after enable_job(retry), often stores the new status
instead of the job complete. So, the summary printed later doesn't show the
real evolution of the test.
Signed-off-by: Sergi Blanch Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29416>
When running on a detached head (eg. checkout of a tag or a specific
commit), there is no active branch, so we can't perform this check; just
skip it and assume the user knows what they're doing.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27578>
This avoids the surprising behaviour where `--target jobname` works for
some jobs but not others, because gitlab adds `X/N` at the end of these
job names.
If the user does specify something like `jobname 1/.*` to only run the
first, the extra `\d+/\d+` is ignored, just like if the job isn't
`parallel:` and therefore doesn't end with `X/N`.
If the user really wants to fail to match parallel jobs (previous
behaviour), they can simply add a `$` at the end of the job name/regex
(but also, I don't see why someone would want that behaviour).
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27530>
