What the integrations rely on

what they rely on for CI operators for contributors

The integrations call image-garden and read what it prints: allocate --spread and its one ADDRESS line, discard by address, the working directory's .image-garden/ and .image-garden.mk, the --query user keys, four environment variables, and, for the action, make targets, rebase and the snap's settings.

On the map: built on it

▶ Watch this deck — 7 scenes, ~5 min

Worth seeing first

01 Five projects, one small interface

THE CONTRACT five integrations run it, read it image-garden testdata/bin/image-garden, the stand-in allocate --spread SYSTEM → <ADDRESS localhost:5555> writes .image-garden/SYSTEM.run: system.users.0.username=ubuntu … discard → (nothing) enough for the whole build script to run inside image-garden: free to change these few things: a CI job breaks
The five integrations contain no image-garden code; they run it and read what it prints; the craft components' tests replace it with a thirty-line stand-in that answers allocate --spread with an ADDRESS line, writes a .run script reporting a user and password, and accepts discard; changes inside image-garden are free, changes to these few things break CI jobs

The integrations only run image-garden and read its output. The craft components' tests replace it with a 30-line stand-in that answers allocate --spread, writes a .run that answers --query, and accepts discard.

None of the five projects built on image-garden contains a line of its code. They run it and read what it prints. That makes the boundary small, and worth knowing exactly. The craft components' own tests show it best: they replace image-garden with a thirty-line stand-in. It answers allocate --spread with an address line, writes a .run script that reports a user and a password, and accepts discard. That's enough for the whole build script to run. Anything inside image-garden can change freely. Change one of these few things, and a CI job somewhere stops working. This deck lists them.

Details and sources

Details

  • The stand-in "records how it is called … and allocates a VM that is only the fake ssh next to it": allocate must be called with --spread, prints <ADDRESS localhost:5555> (or <FATAL …> and exit 213 with FAKE_ALLOCATE=fail), and writes .image-garden/SYSTEM.run, which prints system.users.0.username=ubuntu and system.users.0.password=ubuntu; discard does nothing (snapcraft-gitlab-ci testdata/bin/image-garden:6-30). It also logs PKGPROXY_URL, QEMU_MEM_OPTION and QEMU_SMP_OPTION, which the tests check.
  • The five projects: snapcraft-, rockcraft-, charmcraft- and spread-gitlab-ci, and image-garden-action. spread-gitlab-ci calls nothing itself: your spread.yaml's garden backend runs allocate and discard (sp-adhoc).
  • The full list, with who relies on what, is in this deck's reference below.

02 allocate's one line, and discard

ALLOCATE AND DISCARD $ image-garden allocate --spread ubuntu-cloud-24.04.x86_64 standard error Making ubuntu-cloud-24.04.x86_64... Waiting for SSH on port 5123... standard output: one line, after SSH answers <ADDRESS localhost:5123> sed -n -E 's/^<ADDRESS (.*)>$/\1/p' port=${address##*:} ssh -p 5123 …@localhost forwarded on the loopback address: hostfwd=tcp:127.0.0.1:5123-:22 <FATAL ubuntu-cloud-24.04.x86_64 exited before SSH became available, …> exit 213 $ image-garden discard localhost:5123 anything else: exit 1
image-garden allocate --spread writes its progress to standard error and one line to standard output, ADDRESS localhost:5123, once SSH answers; the craft scripts cut the port out of that line and ssh to localhost because the port is forwarded on 127.0.0.1; on failure the line is FATAL with exit 213; discard takes the same localhost:PORT address and refuses anything else

allocate --spread prints progress on stderr and one line on stdout, <ADDRESS localhost:PORT> or <FATAL …> with exit 213. SSH is forwarded on 127.0.0.1. discard takes only localhost:PORT.

The heart of it is allocate, with --spread. Everything it says while it makes and starts the machine goes to standard error. Standard output gets exactly one line: ADDRESS, in angle brackets, with localhost, a colon and a port, and only once the guest's SSH server has answered. The craft scripts cut that line out with a pattern, take the port after the colon, and connect to localhost, because the port is forwarded on the loopback address. On failure, the line says FATAL instead, and the exit code is two hundred and thirteen. Later, discard takes that same address back. It refuses anything but localhost, colon, port.

Details and sources

Details

  • --spread selects spread's protocol (image-garden:1023-1026). The make that prepares the system writes to stderr (>&2, image-garden:1063), as do the progress messages.
  • The port is forwarded with hostfwd=tcp:127.0.0.1:PORT-:22 (image-garden:1104-1107), so whatever connects must run on the same host or in the same container.
  • allocate waits for the SSH- banner for up to GARDEN_SSH_WAIT_TIMEOUT seconds (600), then prints <ADDRESS localhost:PORT>; after the timeout it prints the address anyway. A machine that dies first, or a port probe that fails, gives <FATAL …> and exit 213, as does a QEMU that does not start (image-garden:1167-1216).
  • The craft scripts parse sed -n -E 's/^<ADDRESS (.*)>$/\1/p' and take ${address##*:} as the port; without an address line the job fails with "image-garden printed no address" (snapcraft-vm:370-373).
  • discard localhost:PORT kills the QEMU of that pid file and removes it; any other form exits 1; discard -a stops every machine of the user (image-garden:1221-1268). spread's garden backend calls it with exec and $SPREAD_SYSTEM_ADDRESS (spread.yaml:59).
  • The slide's port 5123 is an example; allocate picks one between 5000 and 9999.

03 The directory you run it in

THE WORKING DIRECTORY $PWD/ .image-garden/ ubuntu-cloud-24.04.x86_64.qcow2 ubuntu-cloud-24.04.x86_64.run ubuntu-cloud-24.04.x86_64.5123.serial.log … .stdout.log, .stderr.log .image-garden.mk included in the rules the craft scripts make that shape in a scratch directory, copy the logs out at the end the GitHub action caches .image-garden/ without the logs, uploads the logs when a job fails
image-garden works in the directory it is run from; its .image-garden/ subdirectory holds SYSTEM.qcow2, SYSTEM.run and logs named by system and port, and .image-garden.mk beside it is included in the rules; the craft scripts build a scratch directory with that shape and collect the logs; the GitHub action caches .image-garden/ and uploads its logs on failure

image-garden works in the current directory: .image-garden/ there holds the image, the .run script and the logs, and .image-garden.mk there is included. The craft scripts and the action rely on those paths.

The second part is where things land. image-garden works in the directory you run it from. If that directory has a .image-garden directory, the image, the .run script and the logs go in there, named after the system, and the logs after the port too. And a .image-garden.mk next to it is read as part of the rules. The craft scripts lean on all three: they make a scratch directory with .image-garden inside, copy your .image-garden.mk beside it, and later collect the logs from there. The GitHub action caches that same directory, and uploads its logs when a job fails.

Details and sources

Details

  • GARDEN_PROJECT_DIR="$(pwd)" (image-garden:7); mk/999-local.mk has -include $(GARDEN_PROJECT_DIR)/.image-garden.mk (999-local.mk:5).
  • allocate makes SYSTEM.run and SYSTEM.qcow2 in .image-garden/ when it exists, otherwise in the current directory, and locks SYSTEM.lock there (image-garden:1057-1068); it starts ./SYSTEM.run from there, with SYSTEM.PORT.serial.log, .stdout.log and .stderr.log beside it (image-garden:1149-1160).
  • The craft scripts create $work/.image-garden, copy the image-garden-mk input to $work/.image-garden.mk, run allocate and discard from $work, read $work/.image-garden/SYSTEM.run --query and copy $work/.image-garden/*.log (snapcraft-vm:337-349, snapcraft-vm:282-286, snapcraft-vm:374).
  • The action caches SUBDIR/.image-garden without *.log, keyed on the hash of SUBDIR/.image-garden.mk, and uploads .image-garden/*.log when the job fails (action.yaml:116-123, action.yaml:480-489).

04 Two keys and four variables

KEYS AND VARIABLES out: --query vm.memory_mb=2048 system.users.0.username=root system.users.0.password=root system.users.1.username=ubuntu … rename either: no login in: the environment QEMU_MEM_OPTION="-m 2048" QEMU_SMP_OPTION="-smp 2" the memory and cpus inputs PKGPROXY_URL=http://HOST:PORT the proxy the guests use GARDEN_DL_PKGPROXY=http://… image downloads; the action sets it too
The .run script's --query output is key=value lines, of which the craft scripts read system.users.0.username and system.users.0.password; four environment variables go in: QEMU_MEM_OPTION and QEMU_SMP_OPTION from the memory and cpus inputs, PKGPROXY_URL for the guests' proxy, and GARDEN_DL_PKGPROXY for image downloads, which the action also sets

The craft scripts read system.users.0.username and .password from --query, and set QEMU_MEM_OPTION, QEMU_SMP_OPTION and PKGPROXY_URL; the runner or the action sets GARDEN_DL_PKGPROXY.

Then the names. The .run script answers --query with plain key equals value lines, and the craft scripts read two keys: system.users.0.username, and system.users.0.password. Rename either, and they can't log in. On the way in, four environment variables carry settings. QEMU_MEM_OPTION and QEMU_SMP_OPTION size the machine; the craft components set them from their memory and cpus inputs. PKGPROXY_URL names the proxy the guests use, and GARDEN_DL_PKGPROXY the one that image-garden downloads images through. The action sets that last one too.

Details and sources

Details

  • SYSTEM.run --query prints vm.memory_mb=N, from QEMU_MEM_OPTION or the default, and the users as system.users.N.username= and system.users.N.password= lines, then exits (010-qemu.mk:57-92). allocate itself reads vm.memory_mb for its memory check.
  • The craft scripts read system.users.0.username and system.users.0.password (snapcraft-vm:374-377).
  • The .run script uses QEMU_MEM_OPTION and QEMU_SMP_OPTION when set, word-split into QEMU's arguments (010-qemu-x86_64.mk:73-98); the craft scripts export -m MEMORY and -smp CPUS (snapcraft-vm:362-368).
  • PKGPROXY_URL (http://host[:port]) is read when the image is made (010-qemu.mk:20-29). GARDEN_DL_PKGPROXY is read by the download helpers; at v0.8.0 it defaults to PKGPROXY_URL, or the snap's pkgproxy, and set but empty turns it off (image-garden:31-58). The action sets GARDEN_DL_PKGPROXY=http://127.0.0.1:3142 (action.yaml:391-393).
  • The craft components' README asks for pkgproxy 0.12 or later for the image downloads (snapcraft-gitlab-ci README.md:213-214).

for contributors

In the code

  • The --query preamble is EMIT_QEMU_RUN_SCRIPT_QUERY_PREAMBLE; rulegen.py passes root distro for cloud images, mk/050-ubuntu-core.mk passes only:ubuntu (rulegen.py:127, 050-ubuntu-core.mk:25). A new user model must keep user 0 able to log in with a password.

05 What the GitHub action adds

THE ACTION'S SIDE image-garden-action uses the snap: a wider contract image-garden make SYSTEM.ARCH.qcow2 .run .user-data .meta-data .seed.iso image-garden rebase SYSTEM.ARCH.qcow2 the file name; SYSTEM.ARCH works too snap set image-garden pkgproxy=snap snap set image-garden spread-variant=plus snap connect image-garden:pkgproxy-sockets pkgproxy:sockets snap alias image-garden.spread spread ~/snap/image-garden/common/cache/dl created before the first make
The action relies on image-garden make of five files named by system and architecture, rebase of the .qcow2 file by file name, a form rebase accepts as well as the system name, the snap options pkgproxy and spread-variant, the pkgproxy-sockets plug, the image-garden.spread app under an alias, and the snap's download directory, which it creates first

The action relies on image-garden make of SYSTEM.ARCH .qcow2 .run .user-data .meta-data .seed.iso, rebase SYSTEM.ARCH.qcow2, the pkgproxy and spread-variant options, the pkgproxy-sockets plug and the image-garden.spread app.

The GitHub action uses the snap, so its part of the contract is wider. It calls image-garden make with five files named by system and architecture: the qcow2 disk, the .run script, the user-data, the meta-data and the seed image. It runs rebase on the qcow2 file, by its file name, which rebase takes as well as the system name. It sets two snap options, pkgproxy and spread-variant, connects the snap's pkgproxy-sockets plug, and runs the snap's spread under an alias. And it expects the downloads in the snap's cache directory, which it creates before the first make.

Details and sources

Details

  • image-garden make --debug --dry-run SYSTEM.$(uname -m).qcow2, then image-garden make of .qcow2, .run, .user-data, .meta-data and .seed.iso; with cache-prepared-images, image-garden rebase SYSTEM.$(uname -m).qcow2; it first creates ~/snap/image-garden/common/cache/dl (action.yaml:411-442).
  • snap connect image-garden:pkgproxy-sockets pkgproxy:sockets, snap set image-garden pkgproxy=snap or pkgproxy=http://HOST-IP:3142, snap alias image-garden.spread spread, snap set image-garden spread-variant=… (action.yaml:376-401).
  • In the snap: the spread app runs spread-chooser (snapcraft.yaml:211-224); the pkgproxy-sockets content plug (snapcraft.yaml:254-266); the configure hook accepts pkgproxy as component, snap, none or a URL (configure:11-36) and installs the spread component for spread-variant (configure:83-101).
  • rebase finds the image in .image-garden/, takes the system name or the .qcow2 file name, and rewrites its backing path with qemu-img rebase -u (image-garden:943-980). The usage text, man page and README write rebase SYSTEM.ARCH; the action's .qcow2 form works the same.

06 What would break them

WHAT BREAKS THEM a name, a line or a path changes, or how allocate behaves allocate: nc -w 3 … aarch64, privileged container AppArmor breaks nc 10 minutes per VM a release without nc the craft images move to it GitLab components FROM …/zygoon/image-garden:TAG a change arrives when they update the GitHub action snap install --channel=latest/stable a release arrives in the next job
Changing a name, line or path, or how allocate behaves, breaks integrations; allocate once waited for SSH with nc, which the host's AppArmor broke in privileged containers on aarch64 runners, making each VM start take ten minutes; a release without nc fixed it and the craft components moved to it; GitLab components pin the container by tag and get changes when they update, the action follows a snap channel and gets a release in its next job

Renaming or reshaping any of these breaks an integration, and so can a change in behaviour: allocate's use of nc once cost ten minutes per VM on aarch64 runners. The GitLab components pin a container tag; the action follows a snap channel.

So what breaks them? Changing any of those names, lines or paths, and also changing how allocate behaves underneath. That has happened. allocate once waited for SSH with nc, and on arm sixty-four runners, inside a privileged container, the host's AppArmor profile broke nc: starting a VM took ten minutes, for an SSH server that was up in seconds. A release without nc fixed it, and the craft components moved their images to it. That's the other half of the contract. The GitLab components pin an image-garden container by its tag, so a change reaches them when they update. The action installs the snap from a channel, so a release reaches its next job.

Details and sources

Details

  • "The component's image is built on image-garden v0.7.2, which fixes timeouts on arm64. On aarch64 runners, such as Raspberry Pi 5 machines with Ubuntu 26.04, starting the VM no longer takes ten minutes waiting for SSH that was up in seconds: image-garden's allocate no longer uses nc, which the host's AppArmor profile breaks in a privileged container." (snapcraft-gitlab-ci NEWS:10-16). rockcraft- and charmcraft-gitlab-ci carry the same entry.
  • At the commit in garden.commit, allocate probes ports and the SSH banner with bash's /dev/tcp (tcp_probe, image-garden:785-801); v0.7.1 used nc -w for both.
  • The GitLab components' images are FROM registry.gitlab.com/zygoon/image-garden:v0.7.2 (snapcraft-gitlab-ci Dockerfile:9), and each component's default image is its own image of the same version, so a new image-garden reaches a pipeline when it moves to a new component release.
  • Known discrepancy: spread-gitlab-ci builds on an older image-garden. It is FROM image-garden:v0.7.0 (spread-gitlab-ci Dockerfile:20), before the nc fix; none of the four uses v0.8.0 yet.
  • The action installs image-garden from image-garden-channel, latest/stable by default, in every job (action.yaml:12-14, action.yaml:366-375).

for contributors

Before changing image-garden

07 Where to go from here

ROADS the contract allocate how the line is made The adhoc backend spread calls it *craft-gitlab-ci packing in a VM GitHub action the snap's side .run --query users and memory rebase a restored image, this cache
Six roads from the contract: allocate, the adhoc backend, the craft components, the GitHub action, the --query output and rebase

Next: allocate, the adhoc backend, the craft components, the GitHub action, --query and rebase.

So the contract is short: allocate with --spread and its one line, discard by address, the working directory's .image-garden, the query keys, four variables, and for the action, a few make targets, rebase and the snap's settings. To see how allocate arrives at its line, take the allocate deck, and the adhoc backend for how spread calls it. The craft deck and the action deck show the two biggest users. And the query and rebase decks cover the last two pieces in full.

Details and sources

Read more

Basis of the statements on this page

Read in the integrations at the commits in integrations.commits and in image-garden at the commit in garden.commit ("code"). Which project relies on which item was read in their scripts and templates; that a change would break them is derived. Nothing was run for this page.

What each integration relies on

ItemWhere image-garden does itcraft componentsspread-gitlab-ciimage-garden-action
image-garden allocate --spread SYSTEM.ARCHimage-garden:996-1219called directlythrough your spread.yamlthrough your spread.yaml
one stdout line <ADDRESS localhost:PORT>, progress on stderrimage-garden:1063, :1203parsed with sedspread reads itspread reads it
<FATAL …> and exit 213image-garden:1181-1213the job fails with the linespread reports itspread reports it
SSH forwarded on 127.0.0.1:PORTimage-garden:1107ssh -p PORT USER@localhostspread connects therespread connects there
image-garden discard localhost:PORTimage-garden:1249-1268in the exit trapthe discard hook, with execthe discard hook, with exec
state in .image-garden/ of the current directoryimage-garden:1058-1068, :1149-1162the scratch directoryyour projectcached and uploaded
.image-garden.mk in the current directoryimage-garden:7, mk/999-local.mk:5the image-garden-mk inputyour projectits hash keys the image cache
SYSTEM.run --query: system.users.0.username, .passwordmk/010-qemu.mk:57-92the loginno (credentials in spread.yaml)no
QEMU_MEM_OPTION, QEMU_SMP_OPTIONmk/010-qemu-x86_64.mk:73-74 and the other archesthe memory and cpus inputsyour environmentno
PKGPROXY_URLmk/010-qemu.mk:20-29exported from pkgproxy-urlthe runner's environmentno (snap set image-garden pkgproxy=)
GARDEN_DL_PKGPROXYimage-garden:31-58the runner's environmentthe runner's environmenthttp://127.0.0.1:3142
image-garden make SYSTEM.ARCH.{qcow2,run,user-data,meta-data,seed.iso}the generated rulesnonoyes
image-garden rebase SYSTEM.ARCH.qcow2image-garden:943-980nonowith cache-prepared-images
snap: pkgproxy, spread-variant, pkgproxy-sockets, image-garden.spreadsnap/hooks/configure, snap/snapcraft.yamlnonoyes
~/snap/image-garden/common/cache/dlimage-garden:20-29nonocreated before make

How each integration gets image-garden

Integrationimage-garden it usesHow a new release arrives
snapcraft-gitlab-ci 91b5e29container v0.7.2a new component release
rockcraft-gitlab-ci e8fd725container v0.7.2a new component release
charmcraft-gitlab-ci 6fc8543container v0.7.2a new component release
spread-gitlab-ci 3d522d4container v0.7.0a new component release
image-garden-action 2891c7asnap, image-garden-channel (latest/stable)the next job

Known discrepancies that touch the contract

  • spread-gitlab-ci builds on an older image-garden: v0.7.0, before allocate stopped using nc.

The README, image-garden's own spread.yaml and spread-demo/spread.yaml run exec image-garden in both hooks, so the <ADDRESS> and <FATAL> lines are the hook's own output and image-garden's exit status is the hook's (spread.yaml:45-59). rebase takes SYSTEM.ARCH and SYSTEM.ARCH.qcow2 alike; the action passes the second.

Roads from here

RUNTIME · make rules and the CLI

SPREAD · tests

BUILT ON IT · CI integrations