image-garden allocate

a VM in the background for CI operators for users

allocate makes the system if needed, checks memory, picks a free port, starts the .run script in the background with -snapshot and SSH forwarded to localhost, and prints the address, waiting for the SSH banner and answering in spread's ADDRESS and FATAL lines with --spread. discard stops the machine.

On the map: runtime

▶ Watch this deck — 7 scenes, ~5 min

Worth seeing first

Part of journeys

01 A machine in the background

IMAGE-GARDEN ALLOCATE in the background reachable over SSH thrown away after $ image-garden allocate ubuntu-cloud-24.04.x86_64 localhost:PORT 1 make 2 memory 3 port 4 start 5 answer
Tests need a machine in the background, reachable over SSH and thrown away afterwards; image-garden allocate ubuntu-cloud-24.04.x86_64 answers with localhost and a port, after five steps: make, memory, port, start and answer

image-garden allocate ubuntu-cloud-24.04.x86_64 starts the machine in the background and prints localhost:PORT.

The .run script is fine when you sit in front of the machine. Tests need something else. They want a machine running in the background, that they can reach over SSH, and that they can throw away when they're done, with nothing left on the image. That's image-garden allocate. You give it a system name, such as Ubuntu 24.04 on x86, and it answers with an address, localhost and a port. Behind that one line are a few careful steps, and this deck goes through them, in order.

Details and sources

Details

  • A machine started by its .run script has no SSH port; the README points to allocate for one that "runs in the background and is available over ssh" (README.md:86-89).
  • The command and its options: image-garden allocate [--spread] [-p PORT] SYSTEM.ARCH [QEMU-OPTION...], plus --premade and --nice (image-garden:995-1039). Arguments after the system name are passed on to the .run script (image-garden:1160).
  • x86-64 in the name is turned into x86_64 (image-garden:1041).

02 Make it, then check the memory

1 MAKE · 2 MEMORY flock X.lock image-garden make X.run X.qcow2 --premade skips it one lock per system: two allocates never prepare it twice first time: download and provisioning boot after that: already made, instant X.run --query → vm.memory_mb (MemAvailable − reserve) ÷ 1.5 counts -m only, not what the guest writes too tight: a warning, 30 seconds to stop it, then allocate goes on anyway
allocate runs image-garden make for X.run and X.qcow2 under a per-system lock, the first time a download and provisioning boot and later instant; then it compares the memory the script reports with free host memory minus a reserve, counting only the machine's memory and not what the guest writes, and if it is tight it warns, waits 30 seconds and goes on

flock X.lock image-garden make X.run X.qcow2, unless --premade. A memory warning waits 30 seconds, then allocate goes on.

First, allocate makes sure the image exists. It runs image-garden make for the .run script and the qcow2 disk, holding a lock named after the system, so two allocations of the same system don't prepare it twice. The first time, that's the download and the provisioning boot; after that it's instant. Then it asks the script how much memory the machine wants and compares that with what your host has free, keeping some in reserve. It counts only that memory, not what the guest will write. If it looks tight, allocate says so and waits thirty seconds, so you can stop it, before going on anyway.

Details and sources

Details

  • Unless --premade, allocate runs flock X.lock image-garden make X.run X.qcow2, in .image-garden/ when that directory exists; the comment explains that a file shared by several systems is locked separately, in the download helpers (image-garden:1057-1064).
  • The memory check runs on the launcher once it exists (image-garden:1066-1068). It reads vm.memory_mb from --query, subtracts a reserve (GARDEN_MEMORY_RESERVE, else 2048 MB in a graphical session and 1024 MB otherwise) from MemAvailable, divides by 1.5 and, if the machine needs more, prints a warning and sleeps 30 seconds before continuing (image-garden:711-782).
  • The check counts only the machine's memory, -m; the -snapshot overlay that holds the guest's writes is not counted, wherever it is. The README says so for the snap (README.md:155-161); see living with strict confinement.
  • allocate first warns when /dev/kvm exists but is not writable (kvm_check, image-garden:670-710).

03 A port of its own

3 A PORT $XDG_RUNTIME_DIR/image-garden/allocate.lock one for all your machines shuf -i 5000-9999 is anything there? yes: pick again -p PORT or choose it yourself 127.0.0.1:PORT hostfwd port 22 in the guest QEMU_DISPLAY_OPTION=-display none nobody is watching
Under one allocate.lock in the runtime directory, allocate picks a random port between 5000 and 9999, probes it and tries again until it is free, or uses the port given with -p; QEMU forwards 127.0.0.1:PORT to the guest's port 22 and the display is turned off

A random free port from 5000 to 9999, or -p PORT. hostfwd=tcp:127.0.0.1:PORT-:22, and -display none.

Next comes a port. Every allocate of yours, from any directory, takes the same lock in your runtime directory before it picks one, so two machines never get the same port. It picks a random port between five thousand and ten thousand, checks that nothing listens there, and tries again if something does. With -p, you choose the port yourself. Then it tells QEMU to forward that port on localhost to port twenty-two in the guest, and turns the display off, since nobody is watching.

Details and sources

Details

  • The lock is $XDG_RUNTIME_DIR/image-garden/allocate.lock, taken with flock --exclusive -n in a loop (image-garden:1070-1080). XDG_RUNTIME_DIR falls back to /run, and in the snap to /run/snap.NAME when the directory snapd names does not exist (image-garden:64-72).
  • If the caller has gone away while waiting, allocate stops without starting anything (image-garden:1081-1084).
  • Ports: -p PORT is tried first; otherwise shuf -i 5000-9999 -n 1 until the probe finds it free (image-garden:1088-1102, probe image-garden:798-823).
  • QEMU_DISPLAY_OPTION defaults to -display none and QEMU_NETDEV_USER_EXTRA=",hostfwd=tcp:127.0.0.1:$PORT-:22" (image-garden:1104-1107). Only localhost can reach the port.
  • For a +tpm system, the swtpm runtime directory is tied to the port (image-garden:1108-1112).

04 Started, detached, logged

4 START ./ubuntu-cloud-24.04.x86_64.run -snapshot image unchanged, writes in a scratch file -parallel none -monitor none -serial file:X.PORT.serial.log the console -pidfile $XDG_RUNTIME_DIR/image-garden/port.PORT.pid -daemonize >X.PORT.stdout.log 2>X.PORT.stderr.log X stands for ubuntu-cloud-24.04.x86_64 QEMU is up: the port lock is released for the next machine
allocate runs ./X.run with -snapshot so the image is not changed and the guest's writes go to a scratch file, -monitor none, -serial to X.PORT.serial.log, a pid file in the runtime directory and -daemonize, with stdout and stderr in X.PORT logs; then it releases the port lock

./X.run -snapshot -monitor none -serial file:X.PORT.serial.log -pidfile …/port.PORT.pid -daemonize, with stdout and stderr in X.PORT.*.log.

Now allocate runs the same .run script, with a few arguments of its own. Dash snapshot keeps every write in a scratch file in QEMU's temporary directory, so the image on disk is never changed. The monitor is off, and the serial console goes to a log file named after the system and the port. QEMU writes its process ID to a pid file in your runtime directory, and daemonizes. Its own output goes to two more log files. As soon as QEMU is up, allocate lets go of the port lock, so the next machine can start.

Details and sources

Details

  • The invocation: ./X.run ${QEMU_SNAPSHOT_OPTION=-snapshot} -parallel none -monitor none -serial file:X.PORT.serial.log -pidfile $XDG_RUNTIME_DIR/image-garden/port.PORT.pid -daemonize "$@" >X.PORT.stdout.log 2>X.PORT.stderr.log, run in .image-garden/ when it exists (image-garden:1151-1163).
  • The lock file descriptor is closed for QEMU (4<&-) and released when the script returns (exec 4>&-, image-garden:1163-1164); the system name is saved in port.PORT.system for ps (image-garden:1165-1166).
  • Derived, from QEMU: with -snapshot, the guest's writes go to a temporary overlay in QEMU's temporary directory, $TMPDIR or else /var/tmp, which is deleted when QEMU exits. In the snap, a wrapper sets TMPDIR to ~/snap/image-garden/common/tmp, on disk, or to the snap's private /tmp when that cannot be made (qemu-var-tmp:6-20).

05 The answer, plain or for spread

5 THE ANSWER plain localhost:PORT as soon as QEMU runs with --spread wait for the SSH banner up to 600 seconds (GARDEN_SSH_WAIT_TIMEOUT) spread allows only a minute to connect <ADDRESS localhost:PORT> <FATAL …> exit 213 the machine died or never started
Plain allocate prints localhost:PORT as soon as QEMU runs; with --spread it waits up to ten minutes for the SSH banner, because spread allows a minute to connect, then prints ADDRESS localhost:PORT in angle brackets, or FATAL with the reason and exit code 213

Plain: localhost:PORT. With --spread: wait for the SSH banner, then <ADDRESS localhost:PORT>, or <FATAL …> and exit 213.

What allocate prints depends on who asked. Plain, it prints localhost, colon, and the port, as soon as QEMU runs. With --spread, it speaks spread's language. It first waits until the guest's SSH server greets it, for up to ten minutes, because spread gives up if it can't connect within a minute. Then it prints ADDRESS in angle brackets with the address inside. If the machine dies first, or never starts, it prints FATAL with the reason instead, and exits with code two hundred and thirteen.

Details and sources

Details

  • With --spread, allocate probes the port for an SSH- banner until GARDEN_SSH_WAIT_TIMEOUT (default 600 seconds) runs out, then continues anyway; the comment says spread allows only one minute to connect (image-garden:1167-1202).
  • If the probe cannot run, or QEMU exits, it prints <FATAL …> and exits 213; if the caller goes away, it discards the machine and exits 1 (image-garden:1175-1196).
  • Success prints <ADDRESS localhost:PORT> with --spread, else localhost:PORT (image-garden:1203-1205). A QEMU that fails to start gives <FATAL last stderr line> and exit 213, or "Allocation failed" and exit 1 (image-garden:1208-1218).
  • Without --spread, allocate still guesses that spread called it from the process tree and prints a warning that this will be removed (image-garden:1113-1148).

06 Log in, then discard

LOG IN, THEN DISCARD $ ssh -p PORT ubuntu@localhost the user from --query run your tests, look around $ image-garden discard localhost:PORT QEMU stopped, pid file removed the snapshot's changes go with it $ QEMU_SNAPSHOT_OPTION="" image-garden allocate ubuntu-cloud-24.04.x86_64 keeps the changes, for example to debug a failing boot
ssh -p PORT ubuntu@localhost logs in as the user that X.run --query reports; after the tests image-garden discard localhost:PORT stops QEMU and removes the pid file and the snapshot's changes go with it; QEMU_SNAPSHOT_OPTION set to empty keeps them

ssh -p PORT ubuntu@localhost, then image-garden discard localhost:PORT. QEMU_SNAPSHOT_OPTION="" keeps the changes.

With the port, you log in with SSH as one of the users the script lists when you ask it with --query: root, or ubuntu, each with its own name as the password. Do what you need, run your tests, look around. When you're done, run image-garden discard with the same address. It stops QEMU and removes the pid file, and because of the snapshot, every change goes with it. If you want the changes to stay, for example to debug a failing boot, set QEMU_SNAPSHOT_OPTION to nothing before you allocate.

Details and sources

Details

  • --query prints system.users.N.username and .password; for most systems user 0 is root and user 1 the distribution's user, for Ubuntu Core only ubuntu (010-qemu.mk:63-92).
  • discard localhost:PORT kills the PID in port.PORT.pid, removes the pid and system files and a TPM runtime directory; discard -a does it for every machine (image-garden:1221-1274).
  • Persistent mode: QEMU_SNAPSHOT_OPTION="" with the script or with allocate (README.md:789-803). The changes then go into X.qcow2, the prepared image itself.

07 Where to go from here

ROADS allocate The adhoc backend how spread calls it discard and ps stop and list machines --query the user and password The serial console now in a log file
Four roads from image-garden allocate: spread's adhoc backend, discard and ps, --query, and the serial console

Next: spread's adhoc backend, discard and ps, --query, and the serial console.

That's allocate: make the image if needed, check the memory, pick a port, start the script in the background with a snapshot, and answer with an address, waiting for SSH when spread asks. From here, see how spread calls allocate and discard through its adhoc backend. The discard deck covers stopping and listing machines, and the query deck shows how tools learn the user and password. And the serial log is still the console, as the console deck explains.

Details and sources

Details

  • spread calls allocate with --spread and discards by the address it was given.

Read more

Basis of the statements on this page

Read in the image-garden source at the commit this map describes ("code"). Nothing was run for this page.

Options

OptionEffect
--spreadwait for SSH, answer with <ADDRESS …> or <FATAL …> and exit 213
-p PORTuse this port if it is free
--premadedo not run make; the image and launcher must exist
--nicerun make and QEMU under nice (automatic in an x11 or wayland session)
SYSTEM.ARCHthe system, such as ubuntu-cloud-24.04.x86_64
QEMU-OPTION…passed on to the .run script, so to QEMU

Source: image-garden:995-1039.

Environment

VariableEffect
QEMU_SNAPSHOT_OPTION-snapshot unless set; empty keeps the changes
QEMU_MEM_OPTION, QEMU_SMP_OPTION, …passed to the .run script
QEMU_DISPLAY_OPTION-display none unless set
GARDEN_SSH_WAIT_TIMEOUTseconds to wait for SSH with --spread, default 600
GARDEN_MEMORY_RESERVEMB kept for the host in the memory check
XDG_RUNTIME_DIRwhere pid files and the lock go; /run if unset

Files

FileWhereWhat
allocate.lock$XDG_RUNTIME_DIR/image-garden/the port lock, shared by all your machines
port.PORT.pid$XDG_RUNTIME_DIR/image-garden/QEMU's process ID; discard kills it
port.PORT.system$XDG_RUNTIME_DIR/image-garden/the system name, for ps
port.PORT.tpm/$XDG_RUNTIME_DIR/image-garden/+tpm systems: swtpm's runtime directory
X.lockproject dir or .image-garden/the per-system make lock
X.PORT.serial.logproject dir or .image-garden/the guest console
X.PORT.stdout.log, X.PORT.stderr.logproject dir or .image-garden/QEMU's output
the -snapshot overlay$TMPDIR, else /var/tmp; in the snap ~/snap/image-garden/common/tmpthe guest's writes, deleted when QEMU exits; not counted by the memory check

X is the system name. Sources: image-garden:1057-1068, image-garden:1070-1080, image-garden:1108-1112, image-garden:1151-1166.

Roads from here

RUNTIME · make rules and the CLI

SPREAD · tests