spread-gitlab-ci
spread, reports, reruns for CI operators
spread-gitlab-ci is a GitLab CI component whose job runs spread in the image-garden container with spread, spread-plus and spread-gitlab-report added. Your spread.yaml's garden backend calls image-garden allocate in the job; the job reports tasks as JUnit and annotations, stops spread before the timeout, and retries only the tasks that did not pass.
On the map: built on it
▶ Watch this deck — 6 scenes, ~4 min
Worth seeing first
- The adhoc backend: allocate and discard — the garden backend your spread.yaml needs
- Running the container in CI — runners with /dev/kvm and how jobs pick them
Part of journeys
- From your laptop to a git forge, step 4 of 6 · previous: Running the container in CI · next: image-garden-action for GitHub
01 spread, the GitLab way
Include the component and your pipeline gets a spread job: failed tasks in the merge request and test report, a page per failure, a folded log, and a stop before the timeout.
Your spread tests run fine on your machine. In GitLab, you'd like a job that runs them on every push and shows the results where you look anyway: in the merge request, in the pipeline's test report, in a log you can read. That job already exists. spread-gitlab-ci is a CI component: you include it with a couple of inputs, and your pipeline gets a spread job. A failing task shows up in the merge request with the line that says why, every failure gets a page of its own, and spread is stopped before the job runs out of time.
Details and sources
Details
- spread-gitlab-ci is a GitLab CI/CD component; including it gives the pipeline a job that runs spread under spread-gitlab-report, with a JUnit report (pipeline test report and merge request widget, the line that says why), an HTML page per failing task, a folded job log, and spread stopped before the job times out (README.md:9-21).
- The usage example includes
gitlab.com/zygoon/spread-gitlab-ci/spread@0.4.0withsystemsandtags(README.md:25-31). - Known discrepancy: the component's README example names fedora-42. Its usage example and the
systemsinput's description namefedora-42(README.md:29, spread.yml:33-36); image-garden's system isfedora-cloud-42. The slides use image-garden's name. - This map read spread-gitlab-ci at commit
3d522d4.
Read more
02 image-garden's image, with spread added
registry.gitlab.com/zygoon/spread-gitlab-ci is image-garden's image plus spread, spread-plus and spread-gitlab-report, pinned by checksum, for amd64 and arm64. Pick KVM runners with tags.
The job runs in the component's own image, and that image starts from image-garden's container. On top it adds three programs: spread, spread-plus, Canonical's fork of spread with more options, and spread-gitlab-report, which watches spread and writes the reports. Each one is downloaded by a pinned checksum, so one version of the component is one version of each tool. Like image-garden's own image, it's built for x86_64 and arm sixty-four hosts. And like any image-garden job, it needs a runner that gives it the KVM device, picked with tags.
Details and sources
Details
- The image is
FROM registry.gitlab.com/zygoon/image-garden:v0.7.0and addsspread(canonical/spread32cad07, as spread-dist builds it),spread-plus(2ff7bdb3) andspread-gitlab-reportv0.3.0, each withADD --checksum, for amd64 and arm64 (Dockerfile:6-52). - The component's default image is
registry.gitlab.com/zygoon/spread-gitlab-ci:0.4.0, the component's own version (spread.yml:18-22). - It needs runners that give jobs
/dev/kvm, picked withtags; GitLab.com's hosted runners do not (README.md:38-50). - Known discrepancy: spread-gitlab-ci builds on an older image-garden. At this commit its base is image-garden v0.7.0, not v0.8.0, and older than the v0.7.2 the craft components use; it lacks 0.8.0's +tpm systems and loongarch64, and 0.7.2's allocate without nc.
03 Your spread.yaml does the rest
The job runs spread with the garden backend; your spread.yaml defines it, with allocate and discard calling image-garden inside the job.
Notice what the component doesn't contain: any knowledge of image-garden. It simply runs spread on your project, with the garden backend by default. The garden backend is the one in your own spread.yaml, the adhoc one whose allocate hook execs image-garden allocate. That runs inside the job's container, the VM boots there, and spread logs in on the container's loopback. So the file you test with at home is the file CI uses, unchanged. If it works locally, it works here, given a runner with KVM.
Details and sources
Details
- The
backendinput defaults togarden; the job builds selectors from it and runsspread, orspread-VARIANT(spread.yml:27-31, spread.yml:126-151). - The component defines no backend; the
gardenbackend is yourspread.yaml's,type: adhoc, withallocate: exec image-garden allocate --spread "$SPREAD_SYSTEM"."$(uname -m)"anddiscard: exec image-garden discard "$SPREAD_SYSTEM_ADDRESS", as image-garden's README shows (README.md:740-751). - allocate forwards SSH on
127.0.0.1inside the job container, where spread runs too.
Read more
04 The inputs you'll set
tags for KVM runners; systems and tests become selectors garden:SYSTEM:TEST; spread-variant: plus; spread-args; before-script.
A few inputs shape the job. tags pick the runner, so put your KVM tag there. systems and tests narrow the run: for each system and each test you give, the job asks spread for backend, colon, system, colon, test, and with neither it runs everything. The system names are the spread names, with -plus- and -at-. spread-variant set to plus runs spread-plus instead. spread-args passes options such as -v, and before-script runs commands first, such as making a file your tests need.
Details and sources
Details
- Inputs and defaults:
tags[],backendgarden,systemsandtestsempty for all,spread-args,spread-variant""orplus,spread,before-script(spread.yml:10-92). - For each system and each test the job adds the selector
backend:system:test, leaving out the parts not given; with none it passes no selector (spread.yml:126-138, README.md:33-36). - The command is
spread, orspread-pluswhenspread-variantisplus, unless thespreadinput names another (spread.yml:146-147). - System names here are spread's names: see spread names.
Read more
05 Stopping in time, retrying what failed
timeout 2h, stop-before 10m, kill-after 5m. Up to two retries; rerun.txt, kept in the job's cache, narrows each retry to the tasks that did not pass.
Two things make the job kind to your pipeline. First, time. The job's timeout is two hours by default, and spread is asked to stop ten minutes before it, killed five minutes later if it won't. That leaves time to discard the machines and upload the reports, so a slow run still tells you what happened. Second, retries. When spread fails, GitLab retries the job, up to twice. The report lists the tasks that didn't pass, the job keeps that list in its cache, and a retry runs only those tasks.
Details and sources
Details
timeout2h,stop-before10m,kill-after5m: spread is stoppedstop-beforethe timeout so that it restores its machines and the reports are uploaded (spread.yml:76-86).retry: max: retries(default 2, GitLab's limit)when: script_failure(spread.yml:100-103).- spread-gitlab-report writes
junit.xml,annotations.jsonandrerun.txt; the job copiesrerun.txtto.spread-rerun/selectors, kept in the cachespread-rerun-$CI_PIPELINE_ID-$CI_JOB_NAME_SLUGeven when the job fails, and a retry runs only those tasks (spread.yml:115-169). - A retry finds the list only with a shared cache or on the same runner; otherwise it runs every task (README.md:74-90). The numbers on the slide are an example.
Read more
06 Where to go from here
Next: the adhoc backend, spread names, the container in CI, the GitHub action, the contract, and spread in the snap.
So spread-gitlab-ci is a spread job in image-garden's image, driven by your own spread.yaml, reporting the GitLab way and retrying only what failed. Next, the adhoc backend deck shows the hooks this job relies on, and the names deck how systems are written. The CI deck covers runners and pkgproxy. On GitHub, the action runs the same spread with the snap. And the contract deck lists what all these jobs expect from image-garden.
Details and sources
Read more
Basis of the statements on this page
Read in spread-gitlab-ci at commit 3d522d4 ("code"): its README, templates/spread.yml and Dockerfile. The example was not run for this map.
An example
.gitlab-ci.yml:
include: - component: gitlab.com/zygoon/spread-gitlab-ci/spread@0.4.0 inputs: systems: ubuntu-cloud-24.04 fedora-cloud-42 tags: [kvm, x86_64]
spread.yaml, the backend the job runs:
backends: garden: type: adhoc allocate: exec image-garden allocate --spread "$SPREAD_SYSTEM"."$(uname -m)" discard: exec image-garden discard "$SPREAD_SYSTEM_ADDRESS" systems: - ubuntu-cloud-24.04: username: ubuntu password: ubuntu - fedora-cloud-42: username: fedora password: fedora
Inputs
| Input | Default | Meaning |
|---|---|---|
systems | all | systems, separated by spaces, as spread names them |
tests | all | suites or tasks, separated by spaces |
backend | garden | the spread backend; empty for every backend |
tags | [] | runner tags: pick runners with /dev/kvm |
spread-variant | empty | plus runs spread-plus |
spread-args | empty | more spread options, such as -v |
before-script | [] | commands to run before spread |
timeout | 2h | the job's timeout |
stop-before | 10m | stop spread this long before the timeout |
kill-after | 5m | kill spread this long after asking it to stop |
retries | 2 | GitLab retries of a failed job: 0, 1 or 2 |
rerun | true | a retry runs only the tasks that did not pass |
report-dir | spread-report | where the reports go |
image | registry.gitlab.com/zygoon/spread-gitlab-ci:0.4.0 | the job's image |
The full list, with job-name, stage, spread and expire-in, is in the component's README (README.md:103-124).
Reports
| File | Shown as |
|---|---|
junit.xml | the pipeline's test report and the merge request widget |
annotations.json | links on the job page |
| one HTML page per failed task, and an index | the job's artifacts |
rerun.txt | the tasks a retry runs, through the job's cache |
Roads from here
SPREAD · tests
- The adhoc backend: allocate and discard — the allocate and discard hooks the job relies on
- spread names and garden names — system names in spread.yaml and in inputs
BUILT ON IT · CI integrations
- image-garden-action for GitHub — the same spread run on GitHub, with the snap
- What the integrations rely on — what this job relies on in image-garden
SNAP · snap-only features
- Choosing a spread — spread and spread-plus in the snap
CONTAINER · OCI image
- Running the container in CI — runners, tags and pkgproxy for the job