image-garden allocate
a VM in the background for CI operators for users
allocate makes the system if needed, checks memory, picks a free port, starts the .run script in the background with -snapshot and SSH forwarded to localhost, and prints the address, waiting for the SSH banner and answering in spread's ADDRESS and FATAL lines with --spread. discard stops the machine.
On the map: runtime
▶ Watch this deck — 7 scenes, ~5 min
Worth seeing first
- The .run script — the script allocate starts
Part of journeys
- Testing with spread, step 1 of 4 · next: Running spread on image-garden
01 A machine in the background
image-garden allocate ubuntu-cloud-24.04.x86_64 starts the machine in the background and prints localhost:PORT.
The .run script is fine when you sit in front of the machine. Tests need something else. They want a machine running in the background, that they can reach over SSH, and that they can throw away when they're done, with nothing left on the image. That's image-garden allocate. You give it a system name, such as Ubuntu 24.04 on x86, and it answers with an address, localhost and a port. Behind that one line are a few careful steps, and this deck goes through them, in order.
Details and sources
Details
- A machine started by its
.runscript has no SSH port; the README points to allocate for one that "runs in the background and is available over ssh" (README.md:86-89). - The command and its options:
image-garden allocate [--spread] [-p PORT] SYSTEM.ARCH [QEMU-OPTION...], plus--premadeand--nice(image-garden:995-1039). Arguments after the system name are passed on to the.runscript (image-garden:1160). x86-64in the name is turned intox86_64(image-garden:1041).
Read more
02 Make it, then check the memory
flock X.lock image-garden make X.run X.qcow2, unless --premade. A memory warning waits 30 seconds, then allocate goes on.
First, allocate makes sure the image exists. It runs image-garden make for the .run script and the qcow2 disk, holding a lock named after the system, so two allocations of the same system don't prepare it twice. The first time, that's the download and the provisioning boot; after that it's instant. Then it asks the script how much memory the machine wants and compares that with what your host has free, keeping some in reserve. It counts only that memory, not what the guest will write. If it looks tight, allocate says so and waits thirty seconds, so you can stop it, before going on anyway.
Details and sources
Details
- Unless
--premade, allocate runsflock X.lock image-garden make X.run X.qcow2, in.image-garden/when that directory exists; the comment explains that a file shared by several systems is locked separately, in the download helpers (image-garden:1057-1064). - The memory check runs on the launcher once it exists (image-garden:1066-1068). It reads
vm.memory_mbfrom--query, subtracts a reserve (GARDEN_MEMORY_RESERVE, else 2048 MB in a graphical session and 1024 MB otherwise) fromMemAvailable, divides by 1.5 and, if the machine needs more, prints a warning and sleeps 30 seconds before continuing (image-garden:711-782). - The check counts only the machine's memory,
-m; the-snapshotoverlay that holds the guest's writes is not counted, wherever it is. The README says so for the snap (README.md:155-161); see living with strict confinement. - allocate first warns when
/dev/kvmexists but is not writable (kvm_check, image-garden:670-710).
Read more
03 A port of its own
A random free port from 5000 to 9999, or -p PORT. hostfwd=tcp:127.0.0.1:PORT-:22, and -display none.
Next comes a port. Every allocate of yours, from any directory, takes the same lock in your runtime directory before it picks one, so two machines never get the same port. It picks a random port between five thousand and ten thousand, checks that nothing listens there, and tries again if something does. With -p, you choose the port yourself. Then it tells QEMU to forward that port on localhost to port twenty-two in the guest, and turns the display off, since nobody is watching.
Details and sources
Details
- The lock is
$XDG_RUNTIME_DIR/image-garden/allocate.lock, taken withflock --exclusive -nin a loop (image-garden:1070-1080).XDG_RUNTIME_DIRfalls back to/run, and in the snap to/run/snap.NAMEwhen the directory snapd names does not exist (image-garden:64-72). - If the caller has gone away while waiting, allocate stops without starting anything (image-garden:1081-1084).
- Ports:
-p PORTis tried first; otherwiseshuf -i 5000-9999 -n 1until the probe finds it free (image-garden:1088-1102, probe image-garden:798-823). QEMU_DISPLAY_OPTIONdefaults to-display noneandQEMU_NETDEV_USER_EXTRA=",hostfwd=tcp:127.0.0.1:$PORT-:22"(image-garden:1104-1107). Only localhost can reach the port.- For a +tpm system, the swtpm runtime directory is tied to the port (image-garden:1108-1112).
Read more
04 Started, detached, logged
./X.run -snapshot -monitor none -serial file:X.PORT.serial.log -pidfile …/port.PORT.pid -daemonize, with stdout and stderr in X.PORT.*.log.
Now allocate runs the same .run script, with a few arguments of its own. Dash snapshot keeps every write in a scratch file in QEMU's temporary directory, so the image on disk is never changed. The monitor is off, and the serial console goes to a log file named after the system and the port. QEMU writes its process ID to a pid file in your runtime directory, and daemonizes. Its own output goes to two more log files. As soon as QEMU is up, allocate lets go of the port lock, so the next machine can start.
Details and sources
Details
- The invocation:
./X.run ${QEMU_SNAPSHOT_OPTION=-snapshot} -parallel none -monitor none -serial file:X.PORT.serial.log -pidfile $XDG_RUNTIME_DIR/image-garden/port.PORT.pid -daemonize "$@" >X.PORT.stdout.log 2>X.PORT.stderr.log, run in.image-garden/when it exists (image-garden:1151-1163). - The lock file descriptor is closed for QEMU (
4<&-) and released when the script returns (exec 4>&-, image-garden:1163-1164); the system name is saved inport.PORT.systemforps(image-garden:1165-1166).
- Derived, from QEMU: with
-snapshot, the guest's writes go to a temporary overlay in QEMU's temporary directory,$TMPDIRor else/var/tmp, which is deleted when QEMU exits. In the snap, a wrapper setsTMPDIRto~/snap/image-garden/common/tmp, on disk, or to the snap's private/tmpwhen that cannot be made (qemu-var-tmp:6-20).
05 The answer, plain or for spread
Plain: localhost:PORT. With --spread: wait for the SSH banner, then <ADDRESS localhost:PORT>, or <FATAL …> and exit 213.
What allocate prints depends on who asked. Plain, it prints localhost, colon, and the port, as soon as QEMU runs. With --spread, it speaks spread's language. It first waits until the guest's SSH server greets it, for up to ten minutes, because spread gives up if it can't connect within a minute. Then it prints ADDRESS in angle brackets with the address inside. If the machine dies first, or never starts, it prints FATAL with the reason instead, and exits with code two hundred and thirteen.
Details and sources
Details
- With
--spread, allocate probes the port for anSSH-banner untilGARDEN_SSH_WAIT_TIMEOUT(default 600 seconds) runs out, then continues anyway; the comment says spread allows only one minute to connect (image-garden:1167-1202). - If the probe cannot run, or QEMU exits, it prints
<FATAL …>and exits 213; if the caller goes away, it discards the machine and exits 1 (image-garden:1175-1196). - Success prints
<ADDRESS localhost:PORT>with--spread, elselocalhost:PORT(image-garden:1203-1205). A QEMU that fails to start gives<FATAL last stderr line>and exit 213, or "Allocation failed" and exit 1 (image-garden:1208-1218). - Without
--spread, allocate still guesses that spread called it from the process tree and prints a warning that this will be removed (image-garden:1113-1148).
Read more
06 Log in, then discard
ssh -p PORT ubuntu@localhost, then image-garden discard localhost:PORT. QEMU_SNAPSHOT_OPTION="" keeps the changes.
With the port, you log in with SSH as one of the users the script lists when you ask it with --query: root, or ubuntu, each with its own name as the password. Do what you need, run your tests, look around. When you're done, run image-garden discard with the same address. It stops QEMU and removes the pid file, and because of the snapshot, every change goes with it. If you want the changes to stay, for example to debug a failing boot, set QEMU_SNAPSHOT_OPTION to nothing before you allocate.
Details and sources
Details
--queryprintssystem.users.N.usernameand.password; for most systems user 0 is root and user 1 the distribution's user, for Ubuntu Core onlyubuntu(010-qemu.mk:63-92).discard localhost:PORTkills the PID inport.PORT.pid, removes the pid and system files and a TPM runtime directory;discard -adoes it for every machine (image-garden:1221-1274).- Persistent mode:
QEMU_SNAPSHOT_OPTION=""with the script or with allocate (README.md:789-803). The changes then go intoX.qcow2, the prepared image itself.
Read more
07 Where to go from here
Next: spread's adhoc backend, discard and ps, --query, and the serial console.
That's allocate: make the image if needed, check the memory, pick a port, start the script in the background with a snapshot, and answer with an address, waiting for SSH when spread asks. From here, see how spread calls allocate and discard through its adhoc backend. The discard deck covers stopping and listing machines, and the query deck shows how tools learn the user and password. And the serial log is still the console, as the console deck explains.
Details and sources
Details
- spread calls allocate with
--spreadand discards by the address it was given.
Read more
Read more
Basis of the statements on this page
Read in the image-garden source at the commit this map describes ("code"). Nothing was run for this page.
Options
| Option | Effect |
|---|---|
--spread | wait for SSH, answer with <ADDRESS …> or <FATAL …> and exit 213 |
-p PORT | use this port if it is free |
--premade | do not run make; the image and launcher must exist |
--nice | run make and QEMU under nice (automatic in an x11 or wayland session) |
SYSTEM.ARCH | the system, such as ubuntu-cloud-24.04.x86_64 |
QEMU-OPTION… | passed on to the .run script, so to QEMU |
Source: image-garden:995-1039.
Environment
| Variable | Effect |
|---|---|
QEMU_SNAPSHOT_OPTION | -snapshot unless set; empty keeps the changes |
QEMU_MEM_OPTION, QEMU_SMP_OPTION, … | passed to the .run script |
QEMU_DISPLAY_OPTION | -display none unless set |
GARDEN_SSH_WAIT_TIMEOUT | seconds to wait for SSH with --spread, default 600 |
GARDEN_MEMORY_RESERVE | MB kept for the host in the memory check |
XDG_RUNTIME_DIR | where pid files and the lock go; /run if unset |
Files
| File | Where | What |
|---|---|---|
allocate.lock | $XDG_RUNTIME_DIR/image-garden/ | the port lock, shared by all your machines |
port.PORT.pid | $XDG_RUNTIME_DIR/image-garden/ | QEMU's process ID; discard kills it |
port.PORT.system | $XDG_RUNTIME_DIR/image-garden/ | the system name, for ps |
port.PORT.tpm/ | $XDG_RUNTIME_DIR/image-garden/ | +tpm systems: swtpm's runtime directory |
X.lock | project dir or .image-garden/ | the per-system make lock |
X.PORT.serial.log | project dir or .image-garden/ | the guest console |
X.PORT.stdout.log, X.PORT.stderr.log | project dir or .image-garden/ | QEMU's output |
the -snapshot overlay | $TMPDIR, else /var/tmp; in the snap ~/snap/image-garden/common/tmp | the guest's writes, deleted when QEMU exits; not counted by the memory check |
X is the system name. Sources: image-garden:1057-1068, image-garden:1070-1080, image-garden:1108-1112, image-garden:1151-1166.
Roads from here
RUNTIME · make rules and the CLI
- Stopping and listing VMs — stopping and listing allocated machines
- Asking a .run script about its system — the users a tool logs in with
- Working on the serial console — the console of an allocated machine, in a log file
- Where image-garden keeps its files — where the pid files and logs live
SPREAD · tests
- The adhoc backend: allocate and discard — how spread calls allocate and discard