spread-gitlab-ci

spread, reports, reruns for CI operators

spread-gitlab-ci is a GitLab CI component whose job runs spread in the image-garden container with spread, spread-plus and spread-gitlab-report added. Your spread.yaml's garden backend calls image-garden allocate in the job; the job reports tasks as JUnit and annotations, stops spread before the timeout, and retries only the tasks that did not pass.

On the map: built on it

▶ Watch this deck — 6 scenes, ~4 min

Worth seeing first

Part of journeys

01 spread, the GitLab way

SPREAD IN GITLAB spread: passes at home include: - component: gitlab.com/zygoon/spread-gitlab-ci/spread@0.4.0 inputs: systems: ubuntu-cloud-24.04 fedora-cloud-42 tags: [kvm] a spread job in the merge request: the failed task, and why a page per failed task, linked from the job spread stopped before the timeout, reports still uploaded
spread tests that pass on your machine; in the project's pipeline configuration an include of the spread-gitlab-ci component with systems and tags gives the pipeline a spread job; failed tasks appear in the merge request with the reason, each failure gets a page, and spread is stopped before the job times out

Include the component and your pipeline gets a spread job: failed tasks in the merge request and test report, a page per failure, a folded log, and a stop before the timeout.

Your spread tests run fine on your machine. In GitLab, you'd like a job that runs them on every push and shows the results where you look anyway: in the merge request, in the pipeline's test report, in a log you can read. That job already exists. spread-gitlab-ci is a CI component: you include it with a couple of inputs, and your pipeline gets a spread job. A failing task shows up in the merge request with the line that says why, every failure gets a page of its own, and spread is stopped before the job runs out of time.

Details and sources

Details

  • spread-gitlab-ci is a GitLab CI/CD component; including it gives the pipeline a job that runs spread under spread-gitlab-report, with a JUnit report (pipeline test report and merge request widget, the line that says why), an HTML page per failing task, a folded job log, and spread stopped before the job times out (README.md:9-21).
  • The usage example includes gitlab.com/zygoon/spread-gitlab-ci/spread@0.4.0 with systems and tags (README.md:25-31).
  • Known discrepancy: the component's README example names fedora-42. Its usage example and the systems input's description name fedora-42 (README.md:29, spread.yml:33-36); image-garden's system is fedora-cloud-42. The slides use image-garden's name.
  • This map read spread-gitlab-ci at commit 3d522d4.

02 image-garden's image, with spread added

THE JOB'S IMAGE registry.gitlab.com/zygoon/spread-gitlab-ci FROM image-garden's container spread spread-plus spread-gitlab-report watches spread, writes the reports each pinned by its checksum built for amd64 (x86_64) arm64 (aarch64) runs on a runner with /dev/kvm, picked with tags
The component's image starts FROM image-garden's container and adds spread, spread-plus and spread-gitlab-report, each pinned by checksum; it is built for amd64 and arm64; a runner with /dev/kvm, picked with tags, runs it

registry.gitlab.com/zygoon/spread-gitlab-ci is image-garden's image plus spread, spread-plus and spread-gitlab-report, pinned by checksum, for amd64 and arm64. Pick KVM runners with tags.

The job runs in the component's own image, and that image starts from image-garden's container. On top it adds three programs: spread, spread-plus, Canonical's fork of spread with more options, and spread-gitlab-report, which watches spread and writes the reports. Each one is downloaded by a pinned checksum, so one version of the component is one version of each tool. Like image-garden's own image, it's built for x86_64 and arm sixty-four hosts. And like any image-garden job, it needs a runner that gives it the KVM device, picked with tags.

Details and sources

Details

  • The image is FROM registry.gitlab.com/zygoon/image-garden:v0.7.0 and adds spread (canonical/spread 32cad07, as spread-dist builds it), spread-plus (2ff7bdb3) and spread-gitlab-report v0.3.0, each with ADD --checksum, for amd64 and arm64 (Dockerfile:6-52).
  • The component's default image is registry.gitlab.com/zygoon/spread-gitlab-ci:0.4.0, the component's own version (spread.yml:18-22).
  • It needs runners that give jobs /dev/kvm, picked with tags; GitLab.com's hosted runners do not (README.md:38-50).
  • Known discrepancy: spread-gitlab-ci builds on an older image-garden. At this commit its base is image-garden v0.7.0, not v0.8.0, and older than the v0.7.2 the craft components use; it lacks 0.8.0's +tpm systems and loongarch64, and 0.7.2's allocate without nc.

03 Your spread.yaml does the rest

YOUR BACKEND the job: spread garden:… the component knows nothing of image-garden your spread.yaml backends: garden: type: adhoc allocate: exec image-garden allocate … discard: … image-garden discard … the job container QEMU: the VM spread, on 127.0.0.1:PORT the file you test with at home is the file CI uses given a runner with KVM
The job runs spread with the garden backend; the garden backend is the adhoc one in your spread.yaml, whose allocate hook execs image-garden allocate; the VM boots inside the job container and spread logs in on its loopback; the same file works at home and in CI

The job runs spread with the garden backend; your spread.yaml defines it, with allocate and discard calling image-garden inside the job.

Notice what the component doesn't contain: any knowledge of image-garden. It simply runs spread on your project, with the garden backend by default. The garden backend is the one in your own spread.yaml, the adhoc one whose allocate hook execs image-garden allocate. That runs inside the job's container, the VM boots there, and spread logs in on the container's loopback. So the file you test with at home is the file CI uses, unchanged. If it works locally, it works here, given a runner with KVM.

Details and sources

Details

  • The backend input defaults to garden; the job builds selectors from it and runs spread, or spread-VARIANT (spread.yml:27-31, spread.yml:126-151).
  • The component defines no backend; the garden backend is your spread.yaml's, type: adhoc, with allocate: exec image-garden allocate --spread "$SPREAD_SYSTEM"."$(uname -m)" and discard: exec image-garden discard "$SPREAD_SYSTEM_ADDRESS", as image-garden's README shows (README.md:740-751).
  • allocate forwards SSH on 127.0.0.1 inside the job container, where spread runs too.

04 The inputs you'll set

INPUTS inputs: tags: [kvm, x86_64] systems: ubuntu-cloud-24.04 fedora-cloud-42 tests: tests/main/ becomes garden:ubuntu-cloud-24.04:tests/main/ garden:fedora-cloud-42:tests/main/ spread names: -plus-, -at- spread-variant: plus spread-args: -v before-script: - make testdata
The inputs: tags pick a KVM runner; systems and tests become spread selectors garden:SYSTEM:TEST, written with -plus- and -at-; spread-variant plus runs spread-plus; spread-args passes options such as -v; before-script runs commands first

tags for KVM runners; systems and tests become selectors garden:SYSTEM:TEST; spread-variant: plus; spread-args; before-script.

A few inputs shape the job. tags pick the runner, so put your KVM tag there. systems and tests narrow the run: for each system and each test you give, the job asks spread for backend, colon, system, colon, test, and with neither it runs everything. The system names are the spread names, with -plus- and -at-. spread-variant set to plus runs spread-plus instead. spread-args passes options such as -v, and before-script runs commands first, such as making a file your tests need.

Details and sources

Details

  • Inputs and defaults: tags [], backend garden, systems and tests empty for all, spread-args, spread-variant "" or plus, spread, before-script (spread.yml:10-92).
  • For each system and each test the job adds the selector backend:system:test, leaving out the parts not given; with none it passes no selector (spread.yml:126-138, README.md:33-36).
  • The command is spread, or spread-plus when spread-variant is plus, unless the spread input names another (spread.yml:146-147).
  • System names here are spread's names: see spread names.

05 Stopping in time, retrying what failed

TIME AND RETRIES job starts timeout: 2h spread runs stop: -10m kill: +5m machines discarded, reports uploaded attempt 1: 40 tasks 3 fail rerun.txt, in the cache attempt 2: 3 tasks up to two retries
A timeline of the job: spread runs; ten minutes before the two-hour timeout it is asked to stop, and killed five minutes later if needed, leaving time to discard machines and upload reports; on failure GitLab retries the job up to twice, and each retry runs only the tasks in rerun.txt, kept in the job's cache

timeout 2h, stop-before 10m, kill-after 5m. Up to two retries; rerun.txt, kept in the job's cache, narrows each retry to the tasks that did not pass.

Two things make the job kind to your pipeline. First, time. The job's timeout is two hours by default, and spread is asked to stop ten minutes before it, killed five minutes later if it won't. That leaves time to discard the machines and upload the reports, so a slow run still tells you what happened. Second, retries. When spread fails, GitLab retries the job, up to twice. The report lists the tasks that didn't pass, the job keeps that list in its cache, and a retry runs only those tasks.

Details and sources

Details

  • timeout 2h, stop-before 10m, kill-after 5m: spread is stopped stop-before the timeout so that it restores its machines and the reports are uploaded (spread.yml:76-86).
  • retry: max: retries (default 2, GitLab's limit) when: script_failure (spread.yml:100-103).
  • spread-gitlab-report writes junit.xml, annotations.json and rerun.txt; the job copies rerun.txt to .spread-rerun/selectors, kept in the cache spread-rerun-$CI_PIPELINE_ID-$CI_JOB_NAME_SLUG even when the job fails, and a retry runs only those tasks (spread.yml:115-169).
  • A retry finds the list only with a shared cache or on the same runner; otherwise it runs every task (README.md:74-90). The numbers on the slide are an example.

06 Where to go from here

ROADS spread-gitlab-ci The adhoc backend allocate and discard Names in spread -plus-, -at-, no arch KVM in CI runners, tags, pkgproxy GitHub action spread with the snap The contract what it relies on spread-variant spread in the snap
Six roads from spread-gitlab-ci: the adhoc backend, names in spread, KVM in CI, the GitHub action, the contract and spread in the snap

Next: the adhoc backend, spread names, the container in CI, the GitHub action, the contract, and spread in the snap.

So spread-gitlab-ci is a spread job in image-garden's image, driven by your own spread.yaml, reporting the GitLab way and retrying only what failed. Next, the adhoc backend deck shows the hooks this job relies on, and the names deck how systems are written. The CI deck covers runners and pkgproxy. On GitHub, the action runs the same spread with the snap. And the contract deck lists what all these jobs expect from image-garden.

Details and sources

Read more

Basis of the statements on this page

Read in spread-gitlab-ci at commit 3d522d4 ("code"): its README, templates/spread.yml and Dockerfile. The example was not run for this map.

An example

.gitlab-ci.yml:

include: - component: gitlab.com/zygoon/spread-gitlab-ci/spread@0.4.0 inputs: systems: ubuntu-cloud-24.04 fedora-cloud-42 tags: [kvm, x86_64]

spread.yaml, the backend the job runs:

backends: garden: type: adhoc allocate: exec image-garden allocate --spread "$SPREAD_SYSTEM"."$(uname -m)" discard: exec image-garden discard "$SPREAD_SYSTEM_ADDRESS" systems: - ubuntu-cloud-24.04: username: ubuntu password: ubuntu - fedora-cloud-42: username: fedora password: fedora

Inputs

InputDefaultMeaning
systemsallsystems, separated by spaces, as spread names them
testsallsuites or tasks, separated by spaces
backendgardenthe spread backend; empty for every backend
tags[]runner tags: pick runners with /dev/kvm
spread-variantemptyplus runs spread-plus
spread-argsemptymore spread options, such as -v
before-script[]commands to run before spread
timeout2hthe job's timeout
stop-before10mstop spread this long before the timeout
kill-after5mkill spread this long after asking it to stop
retries2GitLab retries of a failed job: 0, 1 or 2
reruntruea retry runs only the tasks that did not pass
report-dirspread-reportwhere the reports go
imageregistry.gitlab.com/zygoon/spread-gitlab-ci:0.4.0the job's image

The full list, with job-name, stage, spread and expire-in, is in the component's README (README.md:103-124).

Reports

FileShown as
junit.xmlthe pipeline's test report and the merge request widget
annotations.jsonlinks on the job page
one HTML page per failed task, and an indexthe job's artifacts
rerun.txtthe tasks a retry runs, through the job's cache

Roads from here

SPREAD · tests

BUILT ON IT · CI integrations

SNAP · snap-only features

CONTAINER · OCI image