Running spread on image-garden

machines for every task for CI operators

spread runs test tasks on machines that a backend gives it. With image-garden each machine is a VM booted in snapshot mode from a prepared image, reached over SSH on a local port, and stopped by discard, so every task starts from a clean system.

On the map: spread

▶ Watch this deck — 5 scenes, ~4 min

Part of journeys

01 What spread asks of a backend

WHAT SPREAD NEEDS a fresh machine for every task Ubuntu Debian Fedora spread runs the tasks one machine per system a backend an address it reaches over SSH a user name a password when the tasks are done, the machine is handed back image-garden localhost:PORT · ubuntu / ubuntu
spread wants a fresh Ubuntu, Debian and Fedora machine for its tasks; it asks a backend for one machine per system and gets back an SSH address, a user name and a password; image-garden is that backend, answering with localhost and a port and the user every image was given

spread asks a backend for a machine per system: an SSH address, a user and a password. image-garden gives all three.

Say you have a test suite, and you want each task to run on a fresh Ubuntu, Debian and Fedora machine. spread runs the tasks, but it doesn't make machines itself. It asks a backend for one machine per system, and wants three things back: an address it can reach over SSH, a user name and a password. When the tasks are done, it hands the machine back. image-garden fits that shape well. It turns a system name into a running machine with SSH on a local port, and it already knows the user and password every image was given.

Details and sources

Details

  • The README's backend gives each system a username and password and gets the address from image-garden allocate --spread (README.md:736-756).
  • image-garden's own spread.yaml declares the garden backend this way (spread.yaml:24-72).
  • What spread does with the address and credentials (connect over SSH, run the task) is spread's behaviour, described here from its configuration, not from image-garden's code.

02 One prepared image, many clean machines

PREPARED ONCE, ALLOCATED OFTEN download and check provisioning boot the prepared image the slow part, and it happens once localhost:5123 -snapshot: writes stay in memory, gone when the machine stops localhost:6411 localhost:8070 the same clean image in the background, each on its own local port
The first time, image-garden downloads and checks the image and runs the provisioning boot, giving a prepared image once; every allocation then boots that image in snapshot mode as its own machine on its own local port, in the background, and nothing it writes is kept

The image is prepared once. Each allocation boots it with -snapshot in the background, on its own port, and nothing it writes is kept.

Here's what happens when spread asks for Ubuntu 24.04. The first time, image-garden makes the image: the download, the check and the one provisioning boot. That's the slow part, and it happens once. Every allocation after that boots the prepared image in snapshot mode, so whatever a test writes stays in memory and is gone when the machine stops. The next task gets the same clean image, not one left over from the last run. Allocations run in the background, each on its own local port, so spread can keep several machines going at once.

Details and sources

Details

  • Unless --premade is given, allocate makes SYSTEM.run and SYSTEM.qcow2 first, under a per-system flock, then checks memory against the launcher (image-garden:1057-1068).
  • It picks a free port in 5000-9999, forwards 127.0.0.1:PORT to the guest's port 22 and starts the launcher with ${QEMU_SNAPSHOT_OPTION=-snapshot}, -daemonize and a pid file (image-garden:1095-1163).
  • QEMU_SNAPSHOT_OPTION="" keeps a machine's changes instead.
  • The port numbers on the slide are examples.

03 A task and a run

A TASK AND A RUN summary: Just run uname artifacts: - uname.txt execute: | uname -a | tee uname.txt tests/smoke/uname/task.yaml $ spread garden:ubuntu-cloud-24.04:tests/smoke/uname backend : system : task allocate log in copy the project run the task no system named: the task runs on every system the backend lists
A task.yaml that runs uname and keeps uname.txt as an artifact; the command spread garden:ubuntu-cloud-24.04:tests/smoke/uname names the backend, the system and the task; spread allocates, logs in, copies the project and runs the task; without a system it runs on every system the backend lists

spread garden:ubuntu-cloud-24.04:tests/smoke/uname allocates one machine, runs the task on it and collects uname.txt.

A task is a small YAML file. This one, from image-garden's own demo, runs uname and keeps its output as an artifact. To run it, you name the backend, the system and the task, separated by colons: garden, then ubuntu-cloud-24.04, then the task's directory. spread calls the backend's allocate hook, gets an address back, logs in with the user and password listed for that system, copies the project over and runs the task. Leave the system out, and spread runs the task on every system the backend lists.

Details and sources

Details

  • The task is image-garden's demo task tests/smoke/uname: summary, artifacts: [uname.txt], execute: uname -a | tee uname.txt (spread-demo/tests/smoke/uname/task.yaml:3-7).
  • The demo's Makefile runs spread garden:<system>: for one system at a time (spread-demo/Makefile:4-8).
  • Logging in, copying the project and running every listed system when none is named are spread's behaviour (derived, not from image-garden's code).

04 Handing the machine back

HANDING IT BACK spread $ image-garden discard localhost:5123 port.5123.pid stop QEMU the port is free again snapshot mode: the prepared image is untouched QEMU died while booting, or the caller went away: allocate stops the machine itself and says why $ image-garden ps every machine and its address
spread runs image-garden discard with the address; image-garden reads the port's pid file, stops QEMU and frees the port, and the prepared image is untouched; when QEMU dies while booting or the caller goes away, allocate stops the machine itself; image-garden ps lists what is still running

discard stops the machine and frees its port. allocate cleans up after itself when QEMU or its caller goes away; image-garden ps lists what is left.

When the task is finished, spread runs the backend's discard hook with the machine's address. image-garden finds the QEMU behind that port, stops it, and removes its pid file, so the port is free again. Because the machine ran in snapshot mode, nothing it did reaches the prepared image. If a machine dies while it boots, or the spread that asked for it goes away, allocate stops it itself and says why, so nothing is left running in the background. And to see what is still running, image-garden ps lists every machine with its address.

Details and sources

Details

  • discard localhost:PORT reads port.PORT.pid under $XDG_RUNTIME_DIR/image-garden/, kills that process and removes the .pid and .system files and a TPM run directory (image-garden:1254-1272).
  • While waiting for SSH, allocate removes the pid files and prints <FATAL … exited before SSH became available, see … serial.log> when QEMU is gone, and discards the machine itself when its caller went away (image-garden:1181-1196).
  • image-garden ps [--all|--stale] lists allocated machines (image-garden:1276-1358).

05 Where to go from here

ROADS spread on a garden The adhoc backend allocate and discard Names in spread -plus-, -at-, no arch spread in the snap spread or spread-plus spread-gitlab-ci on the container Profiles software built in once
Five roads from this deck: the adhoc backend, system names, spread in the snap, spread-gitlab-ci and cloud-init profiles

Next: the adhoc backend, system names, spread in the snap, spread-gitlab-ci and cloud-init profiles.

That's the whole loop: allocate a clean machine from a prepared image, run a task over SSH, discard it. Next, the adhoc backend deck goes through the allocate and discard hooks line by line, and explains why each line is there. The names deck shows how spread's system names map to image-garden's, with modifiers and instances. For CI, there's spread in the snap, and spread-gitlab-ci on the container. And if your tasks spend their time installing software, profiles build it into the image once.

Details and sources

Roads from here

RUNTIME · make rules and the CLI

SPREAD · tests

BUILT ON IT · CI integrations

FIRST BOOT · cloud-init changes

SNAP · snap-only features