Stopping and listing VMs
discard and ps for CI operators for users
discard stops the machine at localhost:PORT, or every machine with -a, by killing the PID in its pid file in $XDG_RUNTIME_DIR/image-garden and removing that file, the system file and a TPM runtime directory; ps lists the machines from the same files, with --stale and --all for the ones whose QEMU is gone.
On the map: runtime
▶ Watch this deck — 7 scenes, ~5 min
Worth seeing first
- image-garden allocate — the machines and the files that discard and ps work on
01 What allocate leaves behind
allocate leaves port.5555.pid and port.5555.system in $XDG_RUNTIME_DIR/image-garden/. discard and ps read nothing else.
Every machine you allocate runs on in the background, detached from your terminal, until something stops it. To stop it, and to see which ones are running, image-garden has to remember them. It does that with two small files in your runtime directory, under image-garden, named after the port. The pid file holds QEMU's process ID, written by QEMU itself. The system file holds the system's name, Ubuntu 24.04 on x86 here. Two commands read them: discard stops machines, and ps lists them. This deck follows the machine allocate started on port 5555.
Details and sources
Details
- allocate passes
-pidfile $XDG_RUNTIME_DIR/image-garden/port.PORT.pidto the.runscript, so QEMU writes its own PID there, then saves the system name inport.PORT.system"for ps command" (image-garden:1153-1166). XDG_RUNTIME_DIRfalls back to/run, and in the snap to/run/snap.NAMEwhen snapd's directory does not exist (image-garden:61-73). discard and ps must see the same value as the allocate that started the machine.- The PID 41872 and the port 5555 on these slides are examples: allocate picks a free port from 5000 to 9999.
02 Discard one machine
image-garden discard localhost:5555 kills the PID, removes port.5555.pid, port.5555.system and port.5555.tpm/. The logs stay. Any other address form exits 1 silently.
To stop it, run image-garden discard with the address allocate printed, localhost:5555. discard takes the port from the address, reads the pid file and sends that process the usual termination signal. Then it removes the pid file and the system file. A machine with a TPM also has a runtime directory, and that goes too. The log files in your project stay, so you can still read the console afterwards. That's what spread runs in its discard hook. An address in any other form, such as 127.0.0.1, makes discard exit with status one, without a word.
Details and sources
Details
- With no argument discard prints "usage: image-garden discard ADDRESS | -a" and exits 1 (image-garden:1250-1253).
- For
localhost:PORTit readsport.PORT.pid, runskill PID || true(SIGTERM, the default signal), removesport.PORT.pidandport.PORT.systemand callsdiscard_tpm_run_dir; when the pid file is missing it does nothing and exits 0. Any other address form exits 1 without a message (image-garden:1254-1272). Run:discard 127.0.0.1:5555exited 1 and printed nothing. - The serial, stdout and stderr logs that allocate writes in the project directory, or in
.image-garden/, are not touched (image-garden:1157-1162). - The machine runs with
-snapshotunlessQEMU_SNAPSHOT_OPTIONwas set, so stopping it drops every change made in the guest (image-garden:1154). - spread's discard hook in the repository runs
exec image-garden discard "$SPREAD_SYSTEM_ADDRESS", as the README does, so discard's exit status is the hook's (spread.yaml:48-59); the craft components discard from an exit trap (snapcraft-vm:268-281). - The man page says discard removes the files "then kills its process"; the code kills first (image-garden.1:589-591). The outcome is the same.
Read more
03 Discard them all
image-garden discard -a discards the machine of every port.*.pid in the runtime directory, from every project, and exits 0.
Sometimes you don't have the address any more. A job was cancelled, a script crashed between allocate and discard, or you simply lost track. discard -a walks over every pid file in the runtime directory, in order, and does for each one what discard does for a single address: kill, remove both files, remove a TPM directory. It exits with success even when there was nothing to stop. Be careful where you run it: it stops every machine you allocated, from any project, not only this one. image-garden's own integration suite does the same by hand after each test, discarding each address it finds.
Details and sources
Details
discard -aexits 0 when$XDG_RUNTIME_DIR/image-gardendoes not exist; otherwise, for eachport.*.pidin glob order, it kills the PID, removes the pid and system files and the TPM directory, and exits 0 (image-garden:1223-1247).- A failed kill prints the shell's error, such as "kill: No such process", and the loop goes on (run).
- The runtime directory is per user, not per project, so
-areaches every machine your user allocated (derived from image-garden:61-73). WhereXDG_RUNTIME_DIRis unset and falls back to/run, as in many CI containers, every user of that/runshares it. - image-garden's integration suite discards each
localhost:PORTit finds inrestore-each(spread.yaml:81-90).
Read more
04 What is running
image-garden ps prints ADDRESS, PID, RSS(MB) and SYSTEM for each pid file whose process runs. RSS comes from /proc/PID/status.
To see what's running, use image-garden ps. It prints a header and one row per machine: the address, QEMU's process ID, how much memory that process holds right now, in megabytes, and the system name. The memory column is the resident size, read from the kernel's view of the process, so it shows what the guest has really touched, not the two gigabytes it was given. By default, ps lists only machines whose process still exists. It doesn't ask the guest anything and it doesn't touch the network: it only reads the files allocate left behind.
Details and sources
Details
- ps always prints the header
ADDRESS PID RSS(MB) SYSTEMwithprintf '%-16s %-8s %-8s %s', and exits 0 with only the header when the runtime directory does not exist (image-garden:1300-1307). - A row is printed for each
port.*.pidwhose PID runs:/proc/PIDexists, orkill -0succeeds (image-garden:1309-1356, image-garden:867-878). - RSS is
VmRSSfrom/proc/PID/status, divided by 1024, one decimal (image-garden:880-885). The system name comes fromport.PORT.system, or-without one (image-garden:1347-1350). - Run: with one live process the output was a header and one row. The PID and RSS on the slide are examples.
- That QEMU's resident size grows with the memory the guest touches, not with
-m, is derived from how Linux counts resident memory. - The usage line:
image-garden ps [--all] [--stale]; any other option gives "image-garden: unknown option" and exit 1 (image-garden:1276-1299).
Read more
05 Stale entries
ps --stale lists entries whose process is gone, or whose pid file is malformed; ps --all lists everything. discard removes a stale entry.
Not every machine ends with discard. Shut the guest down from inside, or let QEMU crash, and the process is gone, but its pid file stays. ps hides such an entry. --stale shows only those, marked stale, and --all shows them next to the running ones. A pid file that doesn't hold a number is marked malformed. To clean up, discard the address as usual: the kill fails, it says so, and the files go anyway. One caution: discard trusts the number in the file, and process numbers get reused. An old stale entry may name another of your processes.
Details and sources
Details
- An entry is
(stale)when its PID is a number that does not run,(malformed)when the pid file does not hold a number. Default: running entries only;--stale: stale and malformed ones only;--all: every entry, with the suffix (image-garden:1309-1355). - Run: with one live and one dead PID,
psshowed one row,ps --allboth with(stale)on the dead one and an empty RSS column,ps --staleonly the dead one. - A guest that powers off ends QEMU, and with it the process; the pid file stays because nothing removes it but discard and allocate's failure paths (image-garden:1186-1190, image-garden:1208-1210).
- discard does not check that the PID belongs to QEMU before
kill(image-garden:1260-1265). PID reuse is derived from how Linux assigns process IDs.
Read more
06 Machines with a TPM
For port.5555.tpm/, discard waits up to 5 s for swtpm to exit, kills it, and removes the directory, retrying for 2 s while the launcher's watcher does the same.
A machine with a TPM has one more thing to clean up. Its launcher started swtpm, the TPM emulator, in a directory of its own, named after the port. When discard has stopped QEMU, swtpm notices the disconnect and exits by itself. discard gives it up to five seconds, then kills it if it's still there. Then it removes the directory, with the copy of the TPM state inside. The launcher's watcher may be removing the same directory at the very same moment, so discard keeps trying for two seconds before it reports a failure.
Details and sources
Details
- allocate names the directory
$XDG_RUNTIME_DIR/image-garden/port.PORT.tpm, exports it asGARDEN_TPM_RUN_DIRand removes a leftover one first (image-garden:825-831, image-garden:1108-1112). discard_tpm_run_dirreadsswtpm.pid, waits up to 50 × 0.1 s while swtpm runs and is not a zombie, then kills it; then it triesrm -rfup to 20 times, 0.1 s apart, and a last time that may fail (image-garden:833-865).- The launcher's watcher,
tail --pid=… -f /dev/null; rm -rf …, removes the directory when swtpm exits (010-qemu-x86_64.mk:203). - Only the four
+tpmUbuntu Core systems have such a directory.
Read more
07 Where to go from here
Next: allocate, spread's adhoc backend, swtpm per launch, and where image-garden keeps its files.
That's discard and ps: two small files per machine, one command that stops machines by address or all at once, and one that lists them, with stale entries on request. From here, the allocate deck shows where those files come from, and spread's adhoc backend shows the discard hook that calls this for you. The swtpm deck explains the directory a machine with a TPM keeps. And the cache deck shows every other file image-garden leaves around, and which ones you can delete.
Details and sources
Read more
Basis of the statements on this page
Read in the image-garden source at the commit in garden.commit ("code"). The outputs of ps, ps --all, ps --stale and discard were run with the v0.8.0 script, whose ps and discard are unchanged at that commit, against a scratch XDG_RUNTIME_DIR holding one pid file of a live process and one of a dead one; no VM was started. PIDs, ports and RSS values on the slides are examples.
Commands
| Command | Effect | Exit | |
|---|---|---|---|
image-garden discard localhost:PORT | kill the PID in port.PORT.pid, remove it, port.PORT.system and port.PORT.tpm/ | 0, also when there is no pid file | |
image-garden discard -a | the same for every port.*.pid | 0 | |
image-garden discard | prints "usage: image-garden discard ADDRESS \ | -a" | 1 |
image-garden discard OTHER | nothing | 1 | |
image-garden ps | header and running entries | 0 | |
image-garden ps --stale | header and stale or malformed entries | 0 | |
image-garden ps --all | header and every entry, with (stale) or (malformed) | 0 |
Sources: image-garden:1221-1358.
Files
File in $XDG_RUNTIME_DIR/image-garden/ | Written by | Removed by |
|---|---|---|
port.PORT.pid | QEMU (-pidfile) | discard, allocate when the machine fails to come up |
port.PORT.system | allocate | discard, allocate when the machine dies before SSH |
port.PORT.tpm/ | the +tpm launcher | the launcher's watcher, discard, the next allocate on that port |
allocate.lock | allocate | nothing; it is a lock |
Sources: image-garden:1070-1080, image-garden:1108-1112, image-garden:1151-1215, 010-qemu-x86_64.mk:173-203.
Example output
$ image-garden ps --all ADDRESS PID RSS(MB) SYSTEM localhost:5555 41872 812.6 ubuntu-cloud-24.04.x86_64 localhost:7021 38110 fedora-cloud-42.x86_64 (stale)
Roads from here
RUNTIME · make rules and the CLI
- image-garden allocate — where the pid and system files come from
- How a TPM system is launched — the TPM directory a machine with a TPM keeps
- Where image-garden keeps its files — every other file image-garden leaves, and what you can delete
- Working on the serial console — the serial log that stays after discard
SPREAD · tests
- The adhoc backend: allocate and discard — the discard hook spread runs for you
BUILT ON IT · CI integrations
- Building snaps, rocks and charms in a VM — a CI job that always discards its VM, even when it fails