Moving `ci-tron:priority:` out of the variable because an empty value
will not be authorized, and this makes it obvious if that bug ever
happens (job will not be picked up and gitlab will complain that
`ci-tron:priority:` is not a tag registered by any runner), instead of
getting picked up by any runner that will then reject (fail) the job.
(This is caused by GitLab's API not allowing tags to be enforced when
picking up jobs, resulting in jobs with missing tags being picked up by
any runner, like the bug we had with the generic fd.o runners a few
months ago.)
v2 (Martin Roukala):
* use the priority tags in all amdgpu jobs
* add missing tags in etnaviv jobs
* add missing tags in broadcom jobs
Cc: mesa-stable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/37897>
This also includes support for the new job cookie system, verifying that jobs
running originate from the claimed source. This is useful internally but won't
affect users, unlike the feature in the title :)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/37609>
This isn’t related to the Debian update, but this is a good opportunity
to uprev ci-templates as well since we’re rebuilding all the containers
anyway.
Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35853>
Now the `subject` from S3 id_token JWT refers to the target project when
we are in an MR pipeline, due to this patch [1].
So, let's use workflow variables to help us define where the S3 paths
should be pointing at.
[1] 7e9f329e8f
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Suggested-by: Benjamin Tissoires <benjamin.tissoires@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36832>
Rebuilding all the test containers for a change that only affects the
build containers is costly and unnecessary. Split the `DEBIAN_BASE_TAG`
into `DEBIAN_BUILD_BASE_TAG` and `DEBIAN_TEST_BASE_TAG` to make it
possible to rebuild only the relevant half of the containers.
Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36736>
GitLab added *.log to the allow list back in 11.4 (October 2018 [1]),
but also both GitLab's code and this workaround have been broken anyway
for a very long time now (they just serve everything as `content-type:
application/octet-stream`), so let's just drop it.
[1]: 129a60e79d
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36556>
The variables are evaluated by GitLab before the job starts, while the
intent here seemed to have been to append the path at run time; that's
not a thing if done here, but luckily this was also not necessary, so
let's drop the invalid literal `$PYTHONPATH` from the `PYTHONPATH` list.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/36078>
The workflow rules added in 7bfb51a7e6 catch the `if` ladder too early and
bypass the `is-scheduled-pipeline`, `is-push-to-upstream-default-branch`,
and `is-push-to-upstream-staging-branch` rules setting the correct job
priority.
Fixes: 7bfb51a7e6 ("ci: Fix missing pipelines on user pipelines in MRs")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35501>
Following the idea to distribute in a tree of files to include and split
between the files with or without hidden job definitions, some jobs in the
root file can be moved to files made specific to describe build or test jobs.
Signed-off-by: Sergi Blanch-Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35427>
This is a quick job using the .fdo.ci-fairy image.
Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Reviewed-by: Sergi Blanch-Torne <sergi.blanch.torne@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35427>
The job definitions for lava-related jobs are encapsulated in a directory,
while the other two farm managers were in the generic test directory. Having a
directory for ci-tron places it side by side with other farm managers. For
bare-metal, it has another advantage, as this encapsulates elements related to
this farm manager in a single place.
To maintain simplicity and consistency in file naming, the gitlab-ci file in
the lava directory is renamed as it has a prefix that corresponds with the
directory hosting it. The other farm equivalent files don't include this
duplication as a prefix, and there isn't such a prefix in any other case of
the CI.
Signed-off-by: Sergi Blanch-Torne <sergi.blanch.torne@collabora.com>
Acked-by: Martin Roukala (né Peres) <martin.roukala@mupuf.org>
Reviewed-by: Eric Engestrom <eric@igalia.com>
Reviewed-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35427>
Oops. We were being a bit too extensive with our rules and also skipping
the container stage for user (non-marge) pipelines in MRs unless
image-tags.yml had changed. We can't actually do this, because we might
still need to run the jobs to copy the containers into the user's
namespace, even if we don't need to rebuild them.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35491>
This runs in 11 seconds and doesn't even clone the git repo, so we can
use the placeholder tag here to get it through instantly.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35203>
For pre-merge pipelines, we know that we don't need to run the
container-check jobs unless image-tags.yml has been modified. In this
case, we can just skip the container jobs, because we 'know' that the
containers are present in the upstream repo.
For downstream or off-main pipelines, we still need to run the check
job, because the changed-file tracking is not reliable on other
branches, and because we may need to copy the container image from the
upstream container storage into the user's namespace.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35203>
Introduce find_s3_project_artifact (and curl-with-retry) helpers to
simplify locating S3 artifacts across upstream mesa/mesa and forks.
Use these helpers in the Fluster build process and for fetching LAVA
rootfs artifacts.
Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35167>
Compared to the existing Debian-based x86_64_pyutils container, this
Alpine-based variant reduces the image size by approximately 83%.
Include all the necessary python artifacts, including lava_job_submitter
in the container to avoid having to download them at the start of each
test job.
Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34980>
Instead of the current allowlist in export-gitlab-job-env-for-dut.sh,
filter out unwanted environment variables and forward the rest to
bare-metal and LAVA DUTs, as well as crosvm.
Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35051>
Firmware packages continue to grow in size, so stop installing them in
the test-base image.
The necessary firmware is now collected and uploaded per vendor in an
external repository.
LAVA devices can opt into optional firmware by specifying the name of the
archive via LAVA_FIRMWARE.
For bare-metal, Qualcomm firmware required for DUTs in the Google lab is
included in the baremetal image.
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/13051
Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34861>
This whole thing of dumping the env var in a file and unsetting it so
that it wouldn't be visible in an env dump anymore? Yeah, we kinda
failed here 😅
Note: setting it to an empty string instead of unsetting it allows for
redundant `eval "$S3_JWT_FILE_SCRIPT"` calls without failing over an
unset variable.
Reported-by: @alatiera
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35050>
The EXTERNAL_KERNEL_TAG variable is no longer needed.
For LAVA and bare-metal, we can override the KERNEL_TAG variable to fetch
both the kernel image and modules from a different tag than the default
mainline gfx-ci/linux kernel.
For LAVA, this also avoids the issue where jobs using EXTERNAL_KERNEL_TAG
would still have mainline kernel modules downloaded by the LAVA overlay.
Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34873>
Up until now, every project using CI-Tron had to write their own job
submission flow because CI-Tron itself was not providing any official
way of interacting with it via GitLab.
This however changed, and the solution is vastly superior to what we
have been using in Mesa:
* Ability to pass all the environment variables of the job to the DUT,
so no need to remember to add variables in
`export-gitlab-job-env-for-dut.sh` anymore
* No dependency on Mesa code, which means no need to wait on
python-artifacts and the ability to replicate a run by just copying
the job description outputted by the job \o/
* Ability to have as many initrd, HTTP, and TFTP artifacts as wanted
* Ability to expose a variable through a TFTP/HTTP endpoint or as an
initrd
* Ability to overwrite the platform environment (machine-specific FW)
* Ability to have as many kernel cmdline variables, all merged when
generating the final cmdline. This makes it easy to share some
snippets of cmdline between jobs
Transitioning from the custom to the generic template is however pretty
involved. This commit does the minimum changes needed to switch to the
new model, often simply replacing the B2C_ prefix with CI_TRON_.
Further renaming of "b2c" prefixes into "ci-tron" is left for future
commits.
Co-authored-by: Eric Engestrom <eric@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34795>
These stages are for the jobs that are skipped in merge pipelines,
automatically run in nightly pipelines, and are available to run
manually in other pipelines.
None of these ever run in post-merge pipelines.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34590>
The /templates/ci-fairy.yml include was pinned to an old revision of
ci-templates that wasn't able to upload to S3 with s3cp following the
infrastructure transition.
Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34689>
Note that this code is executed on the generic FDo gitlab runners, not
in our docker images. This change is merely to avoid the confusion that
lead to the code in the previous commit.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34684>
Right now, `deploy-docs` (updating the docs.mesa3d.org website) is the
only job that can exist in that pipeline (along with `alpine/x86_64_build`,
but that job is always a no-op).
Right now this job can get stuck for a long time waiting for a runner,
but it's very short, and we kinda want the website to be updated quickly.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34554>
Enforce a default job timeout of 1 second, to make jobs which don't
explicitly specify a timeout insta-fail, rather than potentially hanging
around for an hour.
Container builds get the full hour as they can run long and are not run
in pre-merge context, and LAVA jobs also get the full hour as they have
multiple internal timeout mechanisms which aim to fast-fail jobs once
they actually start. However, as they just queue jobs to an external
host (shared with other projects like KernelCI), these timeouts aren't
reflected into the GitLab CI definitions.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34280>
This affects:
- generic jobs (sanity, rustfmt, shader-db, docs, etc.)
- linux image builds
- linux mesa builds
- software renderer tests
- android tests
- virgl & venus tests
Marge pipelines get high priority, nightly pipelines get low priority,
and everything else is in between.
(Hardware test farms have their own mechanisms.)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34264>
That tag was supposed to allow these jobs to run faster, but these
runners are currently having disk issues, and the normal runners look
like they're plenty fast enough (at least right now since almost nobody
runs ci jobs ^^).
We might revert this later, but for now let's merge this to unblock CI.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34120>
