Running spread on image-garden
machines for every task for CI operators
spread runs test tasks on machines that a backend gives it. With image-garden each machine is a VM booted in snapshot mode from a prepared image, reached over SSH on a local port, and stopped by discard, so every task starts from a clean system.
On the map: spread
▶ Watch this deck — 5 scenes, ~4 min
Part of journeys
- Testing with spread, step 2 of 4 · previous: image-garden allocate · next: The adhoc backend: allocate and discard
01 What spread asks of a backend
spread asks a backend for a machine per system: an SSH address, a user and a password. image-garden gives all three.
Say you have a test suite, and you want each task to run on a fresh Ubuntu, Debian and Fedora machine. spread runs the tasks, but it doesn't make machines itself. It asks a backend for one machine per system, and wants three things back: an address it can reach over SSH, a user name and a password. When the tasks are done, it hands the machine back. image-garden fits that shape well. It turns a system name into a running machine with SSH on a local port, and it already knows the user and password every image was given.
Details and sources
Details
- The README's backend gives each system a
usernameandpasswordand gets the address fromimage-garden allocate --spread(README.md:736-756). - image-garden's own spread.yaml declares the
gardenbackend this way (spread.yaml:24-72). - What spread does with the address and credentials (connect over SSH, run the task) is spread's behaviour, described here from its configuration, not from image-garden's code.
Read more
02 One prepared image, many clean machines
The image is prepared once. Each allocation boots it with -snapshot in the background, on its own port, and nothing it writes is kept.
Here's what happens when spread asks for Ubuntu 24.04. The first time, image-garden makes the image: the download, the check and the one provisioning boot. That's the slow part, and it happens once. Every allocation after that boots the prepared image in snapshot mode, so whatever a test writes stays in memory and is gone when the machine stops. The next task gets the same clean image, not one left over from the last run. Allocations run in the background, each on its own local port, so spread can keep several machines going at once.
Details and sources
Details
- Unless
--premadeis given, allocate makesSYSTEM.runandSYSTEM.qcow2first, under a per-systemflock, then checks memory against the launcher (image-garden:1057-1068). - It picks a free port in 5000-9999, forwards
127.0.0.1:PORTto the guest's port 22 and starts the launcher with${QEMU_SNAPSHOT_OPTION=-snapshot},-daemonizeand a pid file (image-garden:1095-1163). QEMU_SNAPSHOT_OPTION=""keeps a machine's changes instead.- The port numbers on the slide are examples.
03 A task and a run
spread garden:ubuntu-cloud-24.04:tests/smoke/uname allocates one machine, runs the task on it and collects uname.txt.
A task is a small YAML file. This one, from image-garden's own demo, runs uname and keeps its output as an artifact. To run it, you name the backend, the system and the task, separated by colons: garden, then ubuntu-cloud-24.04, then the task's directory. spread calls the backend's allocate hook, gets an address back, logs in with the user and password listed for that system, copies the project over and runs the task. Leave the system out, and spread runs the task on every system the backend lists.
Details and sources
Details
- The task is image-garden's demo task
tests/smoke/uname:summary,artifacts: [uname.txt],execute: uname -a | tee uname.txt(spread-demo/tests/smoke/uname/task.yaml:3-7). - The demo's Makefile runs
spread garden:<system>:for one system at a time (spread-demo/Makefile:4-8). - Logging in, copying the project and running every listed system when none is named are spread's behaviour (derived, not from image-garden's code).
Read more
04 Handing the machine back
discard stops the machine and frees its port. allocate cleans up after itself when QEMU or its caller goes away; image-garden ps lists what is left.
When the task is finished, spread runs the backend's discard hook with the machine's address. image-garden finds the QEMU behind that port, stops it, and removes its pid file, so the port is free again. Because the machine ran in snapshot mode, nothing it did reaches the prepared image. If a machine dies while it boots, or the spread that asked for it goes away, allocate stops it itself and says why, so nothing is left running in the background. And to see what is still running, image-garden ps lists every machine with its address.
Details and sources
Details
discard localhost:PORTreadsport.PORT.pidunder$XDG_RUNTIME_DIR/image-garden/, kills that process and removes the.pidand.systemfiles and a TPM run directory (image-garden:1254-1272).- While waiting for SSH, allocate removes the pid files and prints
<FATAL … exited before SSH became available, see … serial.log>when QEMU is gone, and discards the machine itself when its caller went away (image-garden:1181-1196). image-garden ps [--all|--stale]lists allocated machines (image-garden:1276-1358).
Read more
05 Where to go from here
Next: the adhoc backend, system names, spread in the snap, spread-gitlab-ci and cloud-init profiles.
That's the whole loop: allocate a clean machine from a prepared image, run a task over SSH, discard it. Next, the adhoc backend deck goes through the allocate and discard hooks line by line, and explains why each line is there. The names deck shows how spread's system names map to image-garden's, with modifiers and instances. For CI, there's spread in the snap, and spread-gitlab-ci on the container. And if your tasks spend their time installing software, profiles build it into the image once.
Details and sources
Roads from here
RUNTIME · make rules and the CLI
- image-garden allocate — what allocate does on the host
SPREAD · tests
- The adhoc backend: allocate and discard — the allocate and discard hooks, line by line
- spread names and garden names — how spread names map to image-garden names
BUILT ON IT · CI integrations
- spread-gitlab-ci — spread in GitLab CI on the image-garden container
FIRST BOOT · cloud-init changes
- Several systems from one base image — machines with heavy software installed once
SNAP · snap-only features
- Choosing a spread — spread and spread-plus from the snap