The adhoc backend: allocate and discard
allocate and discard for CI operators
image-garden plugs into spread as a type adhoc backend. allocate execs image-garden allocate --spread SYSTEM.ARCH, which waits for SSH and prints an ADDRESS or FATAL line; discard passes SPREAD_SYSTEM_ADDRESS back; every system lists the user and password that image-garden's cloud-init created; a non-root user means scripts run as root through sudo, inside that user's session.
On the map: spread
▶ Watch this deck — 8 scenes, ~5 min
Worth seeing first
- Running spread on image-garden — what spread asks of a backend
Part of journeys
- Testing with spread, step 3 of 4 · previous: Running spread on image-garden · next: spread names and garden names
- From your laptop to a git forge, step 1 of 6 · next: Running image-garden in a container
01 A backend you write yourself
The garden backend is type: adhoc: two shell hooks, allocate and discard, and a list of systems.
spread has backends for clouds and containers, but none that knows image-garden. You don't need one. The adhoc backend lets you write the two hooks yourself, as shell scripts in spread.yaml. allocate must produce a machine and report its address. discard gets that address back and gets rid of the machine. So the backend for image-garden is type adhoc, with a few lines in each hook and a list of systems. This deck goes through those lines one at a time, and why each one is there.
Details and sources
Details
- image-garden's own backend:
garden:withtype: adhoc, anallocate: |and adiscard: |script andsystems:(spread.yaml:24-72). - The README's shortest form puts each hook on one line (README.md:736-753).
- spread's other backend types are spread's own; image-garden documents only the adhoc one.
Read more
02 The allocate line
exec image-garden allocate --spread "$SYSTEM"."$ARCH": --spread prints <ADDRESS localhost:PORT> or <FATAL …>, and exec hands the output and exit status to spread.
The heart of allocate is one line: exec image-garden allocate, the spread option, then the system name, a dot and the architecture. Each part matters. The spread option switches image-garden to spread's way of talking. It prints the address on an ADDRESS line, and a problem on a FATAL line, which is what spread reads from an adhoc hook. exec replaces the shell with image-garden, so that output and its exit status go straight to spread. And the architecture comes from uname, because spread system names don't carry one.
Details and sources
Details
- The line is
exec image-garden allocate --spread "$SYSTEM"."$ARCH", afterARCH="${ARCH:-$(uname -m)}"(spread.yaml:43-47). --spreadsets the spread output mode (image-garden:1023-1024, image-garden:1114-1116); in that mode allocate prints<ADDRESS localhost:PORT>(image-garden:1203) or<FATAL …>and exits 213 (image-garden:1181-1190).- The README: "The
execkeyword allows image-garden to handle the equivalent of spread shell functions likeADDRESS,FATAL, orERROR. Use--spreadto opt into that output mode explicitly." (README.md:755-756). - Without
--spread, allocate guesses from its parent processes whether spread called it, and prints a plainlocalhost:PORTotherwise (image-garden:1117-1147).
Read more
03 Waiting for SSH
In spread mode allocate waits for the SSH- banner, up to GARDEN_SSH_WAIT_TIMEOUT (600 seconds), before it prints the address.
Why does allocate wait at all? spread allows only one minute to connect after a machine is allocated. QEMU accepts connections on the forwarded port long before the guest's SSH server is running, and an emulated guest can take more than a minute to boot. So, in spread mode, allocate waits until the port answers with an SSH banner, up to ten minutes by default, and only then prints the address. If QEMU exits first, allocate prints a FATAL line that points at the serial log, and spread gives up on that machine instead of waiting.
Details and sources
Details
- "Spread allows only one minute to connect after allocation, but QEMU accepts connections on the forwarded port long before sshd in the guest is running" (image-garden:1167-1171).
- The wait probes the port for an
SSH-banner untilGARDEN_SSH_WAIT_TIMEOUT(default 600 seconds) runs out; at the deadline it prints the address anyway (image-garden:1172-1201). - If QEMU's pid is gone, it prints
<FATAL SYSTEM exited before SSH became available, see SYSTEM.PORT.serial.log>and exits 213 (image-garden:1185-1190).
Read more
04 The lines before exec
QEMU_MEM_OPTION and QEMU_SMP_OPTION size the machine. From the snap, restore PATH from SPREAD_HOST_PATH in both hooks.
A few lines before exec set the machine up. QEMU_MEM_OPTION picks the memory: image-garden's own spread.yaml gives four gigabytes, or whatever SPREAD_QEMU_MEM says. QEMU_SMP_OPTION sets the CPU count the same way. If you run spread from the image-garden snap, there's one more step. spread puts /snap/bin first on PATH, and a snap's commands can't run the entry points there. The snap's spread app saves your original PATH in SPREAD_HOST_PATH, and the hook puts it back before calling image-garden.
Details and sources
Details
export QEMU_MEM_OPTION="-m ${SPREAD_QEMU_MEM:-4096}"and, when set,QEMU_SMP_OPTION="-smp $SPREAD_QEMU_SMP"(spread.yaml:37-42). The demo uses-smp 2and 1024 MB, 2048 MB for Ubuntu Core (spread-demo/spread.yaml:18-23).- Both hooks restore
PATHfromSPREAD_HOST_PATH(spread.yaml:27-35, spread.yaml:49-57). - The snap's
spreadapp setsSPREAD_HOST_PATH: $PATH(snap/snapcraft.yaml:211-226).
Read more
05 The discard line
exec image-garden discard "$SPREAD_SYSTEM_ADDRESS" stops QEMU by its pid file, frees the port and removes a TPM system's runtime state. discard -a stops them all.
Discard is one line too: exec image-garden discard, with SPREAD_SYSTEM_ADDRESS, the address allocate printed. exec does the same job as in allocate: image-garden's output and exit status go straight to spread. image-garden takes the port from the address, reads the pid file it wrote when it started QEMU, stops that process, and removes the pid file and the note of which system ran there. For a TPM system it also stops swtpm and removes its runtime state. Without discard, machines would keep running after the tests, each holding a port and gigabytes of memory, until you stopped them all with discard -a.
Details and sources
Details
- The hook:
exec image-garden discard "$SPREAD_SYSTEM_ADDRESS"(spread.yaml:59), as in the README's backend (README.md:745) and the demo (spread-demo/spread.yaml:40). Withexec, the shell is replaced by image-garden, so its output and exit status are the hook's. - discard accepts
localhost:PORT, kills the pid inport.PORT.pidand removesport.PORT.pid,port.PORT.systemand the TPM run directory (image-garden:1254-1272); any other address exits 1. discard_tpm_run_dirwaits for swtpm, stops it and removes its run directory (image-garden:836-859).discard -adoes the same for every pid file (image-garden:1221-1250).
Read more
06 Systems and their users
Each system lists username and password: the distribution's name for both, or root/root; Ubuntu Core takes ubuntu/ubuntu only. SYSTEM.run --query prints them.
Last comes the list of systems. Each entry names a system the way spread wants it, and gives the user and password spread logs in with. They have to match what image-garden's cloud-init created: ubuntu and ubuntu on Ubuntu, debian and debian on Debian, and the distribution's own name for both on the others. root with password root works too, except on Ubuntu Core, which doesn't allow root password logins, so list ubuntu there. When in doubt, the .run script prints the user and password of its system.
Details and sources
Details
- cloud-init creates
root/rootand a user named after the group, with that name as password (mk/010-qemu.mk:200-211). - Ubuntu Core does not allow root password authentication (README.md:708); the demo lists
ubuntu/ubuntufor every Ubuntu Core system (spread-demo/spread.yaml:135-172). SYSTEM.run --queryprintssystem.users.N.usernameand.password(mk/010-qemu.mk:63-92); a named instance's script prints its base system's users (mk/020-instances.mk:89).
Read more
07 Root through sudo, in the user's session
With a non-root username, spread runs scripts with sudo: root, but in the user's login session. For tests that look at sessions, list root/root; Ubuntu Core has no root login.
There's a subtlety with that user. When the username isn't root, spread logs in as that user and runs every prepare, execute and restore script through sudo. So your scripts still run as root, but inside a login session that belongs to ubuntu. The session, the login user ID, the runtime directory and the session bus all still say ubuntu, while the process says root. Most tests never notice. Tests of anything that looks at sessions, such as logind, polkit, user services or the user side of snapd, can behave differently. Where that matters, log in as root, with password root. On Ubuntu Core you can't, so keep it in mind there.
Details and sources
Details
- image-garden's cloud-init gives the distribution user passwordless sudo,
sudo: ["ALL=(ALL) NOPASSWD:ALL"], and createsrootwith passwordroot(mk/010-qemu.mk:200-211). Either can be listed in spread.yaml. - That spread runs each script through
sudowhen the system's username is not root is spread's behaviour, not image-garden's; it is stated here as the maintainer's experience, not read in spread's source for this map. - Derived, from how sudo and Linux sessions work: sudo changes the user ID but not the logind session, the audit login user ID (
/proc/self/loginuid), or variables such asXDG_RUNTIME_DIRandDBUS_SESSION_BUS_ADDRESSthat the SSH login set up for the user (sudo resets most of the environment, so which survive depends on its configuration)./run/user/1000on the slide assumes the user has UID 1000. - Ubuntu Core allows no root password login (README.md:708), so its systems are always reached as
ubuntuthrough sudo.
Read more
08 Where to go from here
Next: system names, allocate and discard on the host, the users, spread in the snap and spread-gitlab-ci.
So the backend is type adhoc, an allocate hook that execs image-garden allocate in spread mode, a discard hook that hands the address back, and a list of systems with their users, root when sessions matter. Next, the names deck explains how the system name in allocate is put together, with modifiers and instances. The allocate and discard decks show what happens on the host, and the users deck where those passwords come from. For CI, there's spread in the snap, and the same backend in spread-gitlab-ci.
Details and sources
Read more
Read more
Basis of the statements on this page
The backend below is image-garden's own garden backend from spread.yaml (lines 24-72) and the README's "Spread Integration Note" and "Spread Test Framework" sections, at the commit this map describes ("code"). The comments are this map's. The systems list is shortened; nothing here was run.
An annotated garden backend
backends:
garden:
# spread has no image-garden backend of its own: write the hooks.
type: adhoc
allocate: |
# From the image-garden snap, spread puts /snap/bin first on
# PATH; the snap's spread app saved the original PATH.
if [ -n "${SPREAD_HOST_PATH-}" ]; then
PATH="${SPREAD_HOST_PATH}"
fi
# Size the machine. These are read by the .run script.
export QEMU_MEM_OPTION="-m ${SPREAD_QEMU_MEM:-4096}"
if [ -n "${SPREAD_QEMU_SMP:-}" ]; then
export QEMU_SMP_OPTION="-smp $SPREAD_QEMU_SMP"
fi
# spread names carry no architecture: add the host's.
ARCH="${ARCH:-$(uname -m)}"
# spread names cannot hold + or @: turn -plus- and -at- back.
SYSTEM="${SPREAD_SYSTEM//-plus-/+}"
SYSTEM="${SYSTEM/-at-/@}"
# exec: image-garden's output and exit status go to spread.
# --spread: wait for SSH, print <ADDRESS …> or <FATAL …>.
exec image-garden allocate --spread "$SYSTEM"."$ARCH"
discard: |
if [ -n "${SPREAD_HOST_PATH-}" ]; then
PATH="${SPREAD_HOST_PATH}"
fi
# The address allocate printed, localhost:PORT; exec as above.
exec image-garden discard "$SPREAD_SYSTEM_ADDRESS"
systems:
# The user and password image-garden's cloud-init created.
- ubuntu-cloud-24.04:
username: ubuntu
password: ubuntu
- debian-cloud-13:
username: debian
password: debian
- fedora-cloud-43:
username: fedora
password: fedora
# Ubuntu Core allows no root password login.
- ubuntu-core-24-plus-tpm:
username: ubuntu
password: ubuntu
# An instance defined in .image-garden.mk.
- ubuntu-cloud-24.04-at-mc:
username: ubuntu
password: ubuntuWhat each hook gets and gives
| Hook | spread gives it | It must give spread |
|---|---|---|
allocate | SPREAD_SYSTEM, the system's name in spread.yaml | <ADDRESS host:port> on success, <FATAL message> on failure |
discard | SPREAD_SYSTEM_ADDRESS, the address allocate printed | nothing; the machine is gone |
What allocate prints and when
| Situation | Output | Exit status |
|---|---|---|
| SSH banner seen | <ADDRESS localhost:PORT> | 0 |
GARDEN_SSH_WAIT_TIMEOUT (600 s) ran out | <ADDRESS localhost:PORT>, after a warning | 0 |
| QEMU gone before SSH answered | <FATAL SYSTEM exited before SSH became available, see SYSTEM.PORT.serial.log> | 213 |
| the port cannot be probed | <FATAL cannot wait for SSH on port PORT of SYSTEM> | 213 |
| QEMU failed to start | <FATAL last line of SYSTEM.PORT.stderr.log> | 213 |
| the caller of allocate went away | the machine is discarded | 1 |
Environment the hooks can use
| Variable | Read by | Meaning |
|---|---|---|
SPREAD_QEMU_MEM | the allocate hook above | memory in MB, 4096 by default |
SPREAD_QEMU_SMP | the allocate hook above | CPU count, QEMU's default if unset |
ARCH | the allocate hook above | the architecture, uname -m by default |
QEMU_MEM_OPTION, QEMU_SMP_OPTION | the .run script | QEMU options for memory and CPUs |
QEMU_SNAPSHOT_OPTION | image-garden allocate | -snapshot by default; empty keeps changes |
GARDEN_SSH_WAIT_TIMEOUT | image-garden allocate | seconds to wait for SSH, 600 by default |
SPREAD_HOST_PATH | both hooks | the PATH spread was started with, set by the snap's spread app |
Root or a user
| username | How scripts run | When to use it |
|---|---|---|
root / root | as root, in root's own login session | tests of sessions, logind, polkit, user services, or anything that reads the login user ID |
the distribution user, such as ubuntu / ubuntu | through passwordless sudo: root, but in the user's login session | most tests; the only choice on Ubuntu Core, which has no root password login |
Roads from here
RUNTIME · make rules and the CLI
- image-garden allocate — the rest of what allocate does on the host
- Stopping and listing VMs — discard and ps in full
SPREAD · tests
- spread names and garden names — how the system name in allocate is put together
BUILT ON IT · CI integrations
- spread-gitlab-ci — the same backend in GitLab CI
FIRST BOOT · cloud-init changes
- The users every system gets — the users every image gets
SNAP · snap-only features
- Choosing a spread — spread and spread-plus from the snap