The adhoc backend: allocate and discard

allocate and discard for CI operators

image-garden plugs into spread as a type adhoc backend. allocate execs image-garden allocate --spread SYSTEM.ARCH, which waits for SSH and prints an ADDRESS or FATAL line; discard passes SPREAD_SYSTEM_ADDRESS back; every system lists the user and password that image-garden's cloud-init created; a non-root user means scripts run as root through sudo, inside that user's session.

On the map: spread

▶ Watch this deck — 8 scenes, ~5 min

Worth seeing first

Part of journeys

01 A backend you write yourself

TYPE: ADHOC no built-in backend knows image-garden backends: garden: type: adhoc you write the hooks yourself allocate: | make a machine, print its address discard: | take the address back, stop it systems: who spread logs in as a few lines in each hook: the rest of this deck
A spread.yaml backend named garden of type adhoc, with an allocate hook that makes a machine and prints its address, a discard hook that takes the address back and stops the machine, and a list of systems with their users

The garden backend is type: adhoc: two shell hooks, allocate and discard, and a list of systems.

spread has backends for clouds and containers, but none that knows image-garden. You don't need one. The adhoc backend lets you write the two hooks yourself, as shell scripts in spread.yaml. allocate must produce a machine and report its address. discard gets that address back and gets rid of the machine. So the backend for image-garden is type adhoc, with a few lines in each hook and a list of systems. This deck goes through those lines one at a time, and why each one is there.

Details and sources

Details

  • image-garden's own backend: garden: with type: adhoc, an allocate: | and a discard: | script and systems: (spread.yaml:24-72).
  • The README's shortest form puts each hook on one line (README.md:736-753).
  • spread's other backend types are spread's own; image-garden documents only the adhoc one.

02 The allocate line

THE ALLOCATE LINE exec image-garden allocate --spread "$SYSTEM"."$ARCH" --spread speak spread's way: an ADDRESS line, or a FATAL line <ADDRESS localhost:5123> <FATAL …> what spread reads from an adhoc hook exec output and exit status go straight to spread $ARCH from uname: spread names carry none
The line exec image-garden allocate --spread SYSTEM.ARCH: --spread makes image-garden print an ADDRESS line or a FATAL line for spread; exec hands that output and the exit status straight to spread; the architecture comes from uname because spread names carry none

exec image-garden allocate --spread "$SYSTEM"."$ARCH": --spread prints <ADDRESS localhost:PORT> or <FATAL …>, and exec hands the output and exit status to spread.

The heart of allocate is one line: exec image-garden allocate, the spread option, then the system name, a dot and the architecture. Each part matters. The spread option switches image-garden to spread's way of talking. It prints the address on an ADDRESS line, and a problem on a FATAL line, which is what spread reads from an adhoc hook. exec replaces the shell with image-garden, so that output and its exit status go straight to spread. And the architecture comes from uname, because spread system names don't carry one.

Details and sources

Details

  • The line is exec image-garden allocate --spread "$SYSTEM"."$ARCH", after ARCH="${ARCH:-$(uname -m)}" (spread.yaml:43-47).
  • --spread sets the spread output mode (image-garden:1023-1024, image-garden:1114-1116); in that mode allocate prints <ADDRESS localhost:PORT> (image-garden:1203) or <FATAL …> and exits 213 (image-garden:1181-1190).
  • The README: "The exec keyword allows image-garden to handle the equivalent of spread shell functions like ADDRESS, FATAL, or ERROR. Use --spread to opt into that output mode explicitly." (README.md:755-756).
  • Without --spread, allocate guesses from its parent processes whether spread called it, and prints a plain localhost:PORT otherwise (image-garden:1117-1147).

03 Waiting for SSH

WAITING FOR SSH spread allows one minute to connect QEMU accepts on the port the guest's SSH answers emulated: often after a minute allocate --spread waits for the SSH- banner, up to 600 s <ADDRESS localhost:5123> QEMU exits first: <FATAL … exited before SSH, see the serial log>
A timeline: spread gives one minute to connect; QEMU accepts on the forwarded port at once, but the guest's SSH server answers later, after more than a minute on an emulated guest; allocate in spread mode waits for the SSH banner, up to 600 seconds, then prints the ADDRESS line; if QEMU exits first it prints a FATAL line pointing at the serial log

In spread mode allocate waits for the SSH- banner, up to GARDEN_SSH_WAIT_TIMEOUT (600 seconds), before it prints the address.

Why does allocate wait at all? spread allows only one minute to connect after a machine is allocated. QEMU accepts connections on the forwarded port long before the guest's SSH server is running, and an emulated guest can take more than a minute to boot. So, in spread mode, allocate waits until the port answers with an SSH banner, up to ten minutes by default, and only then prints the address. If QEMU exits first, allocate prints a FATAL line that points at the serial log, and spread gives up on that machine instead of waiting.

Details and sources

Details

  • "Spread allows only one minute to connect after allocation, but QEMU accepts connections on the forwarded port long before sshd in the guest is running" (image-garden:1167-1171).
  • The wait probes the port for an SSH- banner until GARDEN_SSH_WAIT_TIMEOUT (default 600 seconds) runs out; at the deadline it prints the address anyway (image-garden:1172-1201).
  • If QEMU's pid is gone, it prints <FATAL SYSTEM exited before SSH became available, see SYSTEM.PORT.serial.log> and exits 213 (image-garden:1185-1190).

04 The lines before exec

BEFORE EXEC export QEMU_MEM_OPTION="-m ${SPREAD_QEMU_MEM:-4096}" memory: 4 GB by default export QEMU_SMP_OPTION="-smp $SPREAD_QEMU_SMP" CPUs, the same way if [ -n "${SPREAD_HOST_PATH-}" ]; then PATH="${SPREAD_HOST_PATH}" fi spread puts /snap/bin first on PATH the snap's spread app SPREAD_HOST_PATH=$PATH the hook puts PATH back before calling image-garden
Lines before exec: QEMU_MEM_OPTION set from SPREAD_QEMU_MEM, default 4096, and QEMU_SMP_OPTION from SPREAD_QEMU_SMP; from the snap, spread puts /snap/bin first on PATH, the snap's spread app saved the original PATH in SPREAD_HOST_PATH, and the hook restores it before calling image-garden

QEMU_MEM_OPTION and QEMU_SMP_OPTION size the machine. From the snap, restore PATH from SPREAD_HOST_PATH in both hooks.

A few lines before exec set the machine up. QEMU_MEM_OPTION picks the memory: image-garden's own spread.yaml gives four gigabytes, or whatever SPREAD_QEMU_MEM says. QEMU_SMP_OPTION sets the CPU count the same way. If you run spread from the image-garden snap, there's one more step. spread puts /snap/bin first on PATH, and a snap's commands can't run the entry points there. The snap's spread app saves your original PATH in SPREAD_HOST_PATH, and the hook puts it back before calling image-garden.

Details and sources

Details

05 The discard line

THE DISCARD LINE exec image-garden discard "$SPREAD_SYSTEM_ADDRESS" exec, as in allocate: output and exit status go to spread the port: 5123 port.5123.pid stop QEMU files removed TPM system: stop swtpm, remove its state without discard: a port and gigabytes of memory per machine $ image-garden discard -a
exec image-garden discard with SPREAD_SYSTEM_ADDRESS hands image-garden's output and exit status to spread; image-garden takes the port, reads its pid file, stops QEMU and removes the pid and system files; for a TPM system it also stops swtpm and removes its state; without discard each machine keeps a port and gigabytes of memory, until discard -a stops them all

exec image-garden discard "$SPREAD_SYSTEM_ADDRESS" stops QEMU by its pid file, frees the port and removes a TPM system's runtime state. discard -a stops them all.

Discard is one line too: exec image-garden discard, with SPREAD_SYSTEM_ADDRESS, the address allocate printed. exec does the same job as in allocate: image-garden's output and exit status go straight to spread. image-garden takes the port from the address, reads the pid file it wrote when it started QEMU, stops that process, and removes the pid file and the note of which system ran there. For a TPM system it also stops swtpm and removes its runtime state. Without discard, machines would keep running after the tests, each holding a port and gigabytes of memory, until you stopped them all with discard -a.

Details and sources

Details

  • The hook: exec image-garden discard "$SPREAD_SYSTEM_ADDRESS" (spread.yaml:59), as in the README's backend (README.md:745) and the demo (spread-demo/spread.yaml:40). With exec, the shell is replaced by image-garden, so its output and exit status are the hook's.
  • discard accepts localhost:PORT, kills the pid in port.PORT.pid and removes port.PORT.pid, port.PORT.system and the TPM run directory (image-garden:1254-1272); any other address exits 1.
  • discard_tpm_run_dir waits for swtpm, stops it and removes its run directory (image-garden:836-859).
  • discard -a does the same for every pid file (image-garden:1221-1250).

06 Systems and their users

SYSTEMS AND USERS systems: - ubuntu-cloud-24.04: username: ubuntu password: ubuntu what image-garden's cloud-init created in the image - debian-cloud-13: username: debian password: debian root / root works too - ubuntu-core-24: username: ubuntu password: ubuntu Ubuntu Core: no root password login $ ./ubuntu-cloud-24.04.x86_64.run --query
The systems list of the garden backend: ubuntu-cloud-24.04 with ubuntu and ubuntu, debian-cloud-13 with debian and debian, ubuntu-core-24 with ubuntu and ubuntu because Ubuntu Core allows no root password login; root and root works elsewhere; the .run script prints the user and password with --query

Each system lists username and password: the distribution's name for both, or root/root; Ubuntu Core takes ubuntu/ubuntu only. SYSTEM.run --query prints them.

Last comes the list of systems. Each entry names a system the way spread wants it, and gives the user and password spread logs in with. They have to match what image-garden's cloud-init created: ubuntu and ubuntu on Ubuntu, debian and debian on Debian, and the distribution's own name for both on the others. root with password root works too, except on Ubuntu Core, which doesn't allow root password logins, so list ubuntu there. When in doubt, the .run script prints the user and password of its system.

Details and sources

Details

07 Root through sudo, in the user's session

ROOT THROUGH SUDO SSH login as ubuntu: ubuntu's session sudo … prepare, execute, restore your script, running as root still ubuntu's: login session (logind) login user ID (loginuid) XDG_RUNTIME_DIR=/run/user/1000 the session bus the process: root may differ: logind, polkit, user services, snapd's user side username: root password: root where sessions matter Ubuntu Core: no root login
spread logs in over SSH as ubuntu and runs each script with sudo; the script runs as root inside ubuntu's login session; the session, login user ID, runtime directory and session bus still belong to ubuntu; tests of logind, polkit, user services or snapd's user side may behave differently; listing root and root avoids it except on Ubuntu Core

With a non-root username, spread runs scripts with sudo: root, but in the user's login session. For tests that look at sessions, list root/root; Ubuntu Core has no root login.

There's a subtlety with that user. When the username isn't root, spread logs in as that user and runs every prepare, execute and restore script through sudo. So your scripts still run as root, but inside a login session that belongs to ubuntu. The session, the login user ID, the runtime directory and the session bus all still say ubuntu, while the process says root. Most tests never notice. Tests of anything that looks at sessions, such as logind, polkit, user services or the user side of snapd, can behave differently. Where that matters, log in as root, with password root. On Ubuntu Core you can't, so keep it in mind there.

Details and sources

Details

  • image-garden's cloud-init gives the distribution user passwordless sudo, sudo: ["ALL=(ALL) NOPASSWD:ALL"], and creates root with password root (mk/010-qemu.mk:200-211). Either can be listed in spread.yaml.
  • That spread runs each script through sudo when the system's username is not root is spread's behaviour, not image-garden's; it is stated here as the maintainer's experience, not read in spread's source for this map.
  • Derived, from how sudo and Linux sessions work: sudo changes the user ID but not the logind session, the audit login user ID (/proc/self/loginuid), or variables such as XDG_RUNTIME_DIR and DBUS_SESSION_BUS_ADDRESS that the SSH login set up for the user (sudo resets most of the environment, so which survive depends on its configuration). /run/user/1000 on the slide assumes the user has UID 1000.
  • Ubuntu Core allows no root password login (README.md:708), so its systems are always reached as ubuntu through sudo.

08 Where to go from here

ROADS The adhoc backend Names in spread -plus-, -at-, no arch allocate what it does on the host discard and ps stop and list machines Users and passwords set by cloud-init spread in the snap spread or spread-plus spread-gitlab-ci the same backend in CI
Six roads from this deck: system names, allocate, discard, the users, spread in the snap and spread-gitlab-ci

Next: system names, allocate and discard on the host, the users, spread in the snap and spread-gitlab-ci.

So the backend is type adhoc, an allocate hook that execs image-garden allocate in spread mode, a discard hook that hands the address back, and a list of systems with their users, root when sessions matter. Next, the names deck explains how the system name in allocate is put together, with modifiers and instances. The allocate and discard decks show what happens on the host, and the users deck where those passwords come from. For CI, there's spread in the snap, and the same backend in spread-gitlab-ci.

Details and sources

Read more

Basis of the statements on this page

The backend below is image-garden's own garden backend from spread.yaml (lines 24-72) and the README's "Spread Integration Note" and "Spread Test Framework" sections, at the commit this map describes ("code"). The comments are this map's. The systems list is shortened; nothing here was run.

An annotated garden backend

backends:
    garden:
        # spread has no image-garden backend of its own: write the hooks.
        type: adhoc
        allocate: |
            # From the image-garden snap, spread puts /snap/bin first on
            # PATH; the snap's spread app saved the original PATH.
            if [ -n "${SPREAD_HOST_PATH-}" ]; then
                PATH="${SPREAD_HOST_PATH}"
            fi

            # Size the machine. These are read by the .run script.
            export QEMU_MEM_OPTION="-m ${SPREAD_QEMU_MEM:-4096}"
            if [ -n "${SPREAD_QEMU_SMP:-}" ]; then
                export QEMU_SMP_OPTION="-smp $SPREAD_QEMU_SMP"
            fi

            # spread names carry no architecture: add the host's.
            ARCH="${ARCH:-$(uname -m)}"

            # spread names cannot hold + or @: turn -plus- and -at- back.
            SYSTEM="${SPREAD_SYSTEM//-plus-/+}"
            SYSTEM="${SYSTEM/-at-/@}"

            # exec: image-garden's output and exit status go to spread.
            # --spread: wait for SSH, print <ADDRESS …> or <FATAL …>.
            exec image-garden allocate --spread "$SYSTEM"."$ARCH"
        discard: |
            if [ -n "${SPREAD_HOST_PATH-}" ]; then
                PATH="${SPREAD_HOST_PATH}"
            fi
            # The address allocate printed, localhost:PORT; exec as above.
            exec image-garden discard "$SPREAD_SYSTEM_ADDRESS"
        systems:
            # The user and password image-garden's cloud-init created.
            - ubuntu-cloud-24.04:
                username: ubuntu
                password: ubuntu
            - debian-cloud-13:
                username: debian
                password: debian
            - fedora-cloud-43:
                username: fedora
                password: fedora
            # Ubuntu Core allows no root password login.
            - ubuntu-core-24-plus-tpm:
                username: ubuntu
                password: ubuntu
            # An instance defined in .image-garden.mk.
            - ubuntu-cloud-24.04-at-mc:
                username: ubuntu
                password: ubuntu

What each hook gets and gives

Hookspread gives itIt must give spread
allocateSPREAD_SYSTEM, the system's name in spread.yaml<ADDRESS host:port> on success, <FATAL message> on failure
discardSPREAD_SYSTEM_ADDRESS, the address allocate printednothing; the machine is gone

What allocate prints and when

SituationOutputExit status
SSH banner seen<ADDRESS localhost:PORT>0
GARDEN_SSH_WAIT_TIMEOUT (600 s) ran out<ADDRESS localhost:PORT>, after a warning0
QEMU gone before SSH answered<FATAL SYSTEM exited before SSH became available, see SYSTEM.PORT.serial.log>213
the port cannot be probed<FATAL cannot wait for SSH on port PORT of SYSTEM>213
QEMU failed to start<FATAL last line of SYSTEM.PORT.stderr.log>213
the caller of allocate went awaythe machine is discarded1

Environment the hooks can use

VariableRead byMeaning
SPREAD_QEMU_MEMthe allocate hook abovememory in MB, 4096 by default
SPREAD_QEMU_SMPthe allocate hook aboveCPU count, QEMU's default if unset
ARCHthe allocate hook abovethe architecture, uname -m by default
QEMU_MEM_OPTION, QEMU_SMP_OPTIONthe .run scriptQEMU options for memory and CPUs
QEMU_SNAPSHOT_OPTIONimage-garden allocate-snapshot by default; empty keeps changes
GARDEN_SSH_WAIT_TIMEOUTimage-garden allocateseconds to wait for SSH, 600 by default
SPREAD_HOST_PATHboth hooksthe PATH spread was started with, set by the snap's spread app

Root or a user

usernameHow scripts runWhen to use it
root / rootas root, in root's own login sessiontests of sessions, logind, polkit, user services, or anything that reads the login user ID
the distribution user, such as ubuntu / ubuntuthrough passwordless sudo: root, but in the user's login sessionmost tests; the only choice on Ubuntu Core, which has no root password login

Roads from here

RUNTIME · make rules and the CLI

SPREAD · tests

BUILT ON IT · CI integrations

FIRST BOOT · cloud-init changes

SNAP · snap-only features