Stopping and listing VMs

discard and ps for CI operators for users

discard stops the machine at localhost:PORT, or every machine with -a, by killing the PID in its pid file in $XDG_RUNTIME_DIR/image-garden and removing that file, the system file and a TPM runtime directory; ps lists the machines from the same files, with --stale and --all for the ones whose QEMU is gone.

On the map: runtime

▶ Watch this deck — 7 scenes, ~5 min

Worth seeing first

01 What allocate leaves behind

TWO FILES PER MACHINE QEMU: ubuntu-cloud-24.04.x86_64 in the background, localhost:5555 $XDG_RUNTIME_DIR/image-garden/ port.5555.pid 41872 QEMU's PID port.5555.system ubuntu-cloud-24.04.x86_64 image-garden discard stops machines image-garden ps lists them
QEMU running ubuntu-cloud-24.04.x86_64 in the background on localhost:5555 writes its PID to port.5555.pid in $XDG_RUNTIME_DIR/image-garden; allocate writes the system name to port.5555.system. image-garden discard and image-garden ps read these two files.

allocate leaves port.5555.pid and port.5555.system in $XDG_RUNTIME_DIR/image-garden/. discard and ps read nothing else.

Every machine you allocate runs on in the background, detached from your terminal, until something stops it. To stop it, and to see which ones are running, image-garden has to remember them. It does that with two small files in your runtime directory, under image-garden, named after the port. The pid file holds QEMU's process ID, written by QEMU itself. The system file holds the system's name, Ubuntu 24.04 on x86 here. Two commands read them: discard stops machines, and ps lists them. This deck follows the machine allocate started on port 5555.

Details and sources

Details

  • allocate passes -pidfile $XDG_RUNTIME_DIR/image-garden/port.PORT.pid to the .run script, so QEMU writes its own PID there, then saves the system name in port.PORT.system "for ps command" (image-garden:1153-1166).
  • XDG_RUNTIME_DIR falls back to /run, and in the snap to /run/snap.NAME when snapd's directory does not exist (image-garden:61-73). discard and ps must see the same value as the allocate that started the machine.
  • The PID 41872 and the port 5555 on these slides are examples: allocate picks a free port from 5000 to 9999.

02 Discard one machine

DISCARD ADDRESS $ image-garden discard localhost:5555 port.5555.pid: 41872 kill 41872: QEMU stops rm port.5555.pid rm port.5555.system rm -r port.5555.tpm/ TPM machines kept, in the project: X.5555.serial.log, .stdout, .stderr spread's discard: image-garden discard "$SPREAD_SYSTEM_ADDRESS" $ image-garden discard 127.0.0.1:5555 exit status 1, no message
image-garden discard localhost:5555 reads port.5555.pid, kills QEMU with SIGTERM, removes port.5555.pid, port.5555.system and, for a TPM machine, port.5555.tpm/. The logs in the project stay. spread's discard hook runs the same command; any address not of the form localhost:PORT exits 1 without a message.

image-garden discard localhost:5555 kills the PID, removes port.5555.pid, port.5555.system and port.5555.tpm/. The logs stay. Any other address form exits 1 silently.

To stop it, run image-garden discard with the address allocate printed, localhost:5555. discard takes the port from the address, reads the pid file and sends that process the usual termination signal. Then it removes the pid file and the system file. A machine with a TPM also has a runtime directory, and that goes too. The log files in your project stay, so you can still read the console afterwards. That's what spread runs in its discard hook. An address in any other form, such as 127.0.0.1, makes discard exit with status one, without a word.

Details and sources

Details

  • With no argument discard prints "usage: image-garden discard ADDRESS | -a" and exits 1 (image-garden:1250-1253).
  • For localhost:PORT it reads port.PORT.pid, runs kill PID || true (SIGTERM, the default signal), removes port.PORT.pid and port.PORT.system and calls discard_tpm_run_dir; when the pid file is missing it does nothing and exits 0. Any other address form exits 1 without a message (image-garden:1254-1272). Run: discard 127.0.0.1:5555 exited 1 and printed nothing.
  • The serial, stdout and stderr logs that allocate writes in the project directory, or in .image-garden/, are not touched (image-garden:1157-1162).
  • The machine runs with -snapshot unless QEMU_SNAPSHOT_OPTION was set, so stopping it drops every change made in the guest (image-garden:1154).
  • spread's discard hook in the repository runs exec image-garden discard "$SPREAD_SYSTEM_ADDRESS", as the README does, so discard's exit status is the hook's (spread.yaml:48-59); the craft components discard from an exit trap (snapcraft-vm:268-281).
  • The man page says discard removes the files "then kills its process"; the code kills first (image-garden.1:589-591). The outcome is the same.

03 Discard them all

DISCARD -A a cancelled job · a crashed script · lost track $ image-garden discard -a $XDG_RUNTIME_DIR/image-garden/ port.5555.pid ubuntu-cloud-24.04.x86_64 port.6021.pid fedora-cloud-42.x86_64 port.8317.pid debian-cloud-13.x86_64 each: kill, remove both files, the TPM dir exit 0, even with nothing to stop Every machine you allocated, from any project, not only this one restore-each: for pidfile in …/port.*.pid; do image-garden discard "localhost:$port"; done
When the address is lost, image-garden discard -a goes over every port.*.pid in the runtime directory and kills, removes and cleans up each machine, exiting 0 even with nothing to stop. It stops every machine you allocated, from any project. image-garden's integration suite does the same in restore-each.

image-garden discard -a discards the machine of every port.*.pid in the runtime directory, from every project, and exits 0.

Sometimes you don't have the address any more. A job was cancelled, a script crashed between allocate and discard, or you simply lost track. discard -a walks over every pid file in the runtime directory, in order, and does for each one what discard does for a single address: kill, remove both files, remove a TPM directory. It exits with success even when there was nothing to stop. Be careful where you run it: it stops every machine you allocated, from any project, not only this one. image-garden's own integration suite does the same by hand after each test, discarding each address it finds.

Details and sources

Details

  • discard -a exits 0 when $XDG_RUNTIME_DIR/image-garden does not exist; otherwise, for each port.*.pid in glob order, it kills the PID, removes the pid and system files and the TPM directory, and exits 0 (image-garden:1223-1247).
  • A failed kill prints the shell's error, such as "kill: No such process", and the loop goes on (run).
  • The runtime directory is per user, not per project, so -a reaches every machine your user allocated (derived from image-garden:61-73). Where XDG_RUNTIME_DIR is unset and falls back to /run, as in many CI containers, every user of that /run shares it.
  • image-garden's integration suite discards each localhost:PORT it finds in restore-each (spread.yaml:81-90).

04 What is running

IMAGE-GARDEN PS $ image-garden ps ADDRESS PID RSS(MB) SYSTEM localhost:5555 41872 812.6 ubuntu-cloud-24.04.x86_64 RSS: VmRSS in /proc/41872/status what the guest has touched, not the -m 2048 it was given Only entries whose process still exists port.*.pid, port.*.system image-garden ps no SSH, no network, no question to the guest
image-garden ps prints a header and a row per running machine: ADDRESS localhost:5555, PID 41872, RSS(MB) from /proc/PID/status, and SYSTEM ubuntu-cloud-24.04.x86_64. It lists only machines whose process exists and reads only the files allocate left.

image-garden ps prints ADDRESS, PID, RSS(MB) and SYSTEM for each pid file whose process runs. RSS comes from /proc/PID/status.

To see what's running, use image-garden ps. It prints a header and one row per machine: the address, QEMU's process ID, how much memory that process holds right now, in megabytes, and the system name. The memory column is the resident size, read from the kernel's view of the process, so it shows what the guest has really touched, not the two gigabytes it was given. By default, ps lists only machines whose process still exists. It doesn't ask the guest anything and it doesn't touch the network: it only reads the files allocate left behind.

Details and sources

Details

  • ps always prints the header ADDRESS PID RSS(MB) SYSTEM with printf '%-16s %-8s %-8s %s', and exits 0 with only the header when the runtime directory does not exist (image-garden:1300-1307).
  • A row is printed for each port.*.pid whose PID runs: /proc/PID exists, or kill -0 succeeds (image-garden:1309-1356, image-garden:867-878).
  • RSS is VmRSS from /proc/PID/status, divided by 1024, one decimal (image-garden:880-885). The system name comes from port.PORT.system, or - without one (image-garden:1347-1350).
  • Run: with one live process the output was a header and one row. The PID and RSS on the slide are examples.
  • That QEMU's resident size grows with the memory the guest touches, not with -m, is derived from how Linux counts resident memory.
  • The usage line: image-garden ps [--all] [--stale]; any other option gives "image-garden: unknown option" and exit 1 (image-garden:1276-1299).

05 Stale entries

STALE ENTRIES QEMU, fedora-cloud-42 port.7021.pid stays shut down inside, or crashed $ image-garden ps no row for it $ image-garden ps --all --stale: the last row only ADDRESS PID RSS(MB) SYSTEM localhost:5555 41872 812.6 ubuntu-cloud-24.04.x86_64 localhost:7021 38110 fedora-cloud-42.x86_64 (stale) a pid file without a number: (malformed) $ image-garden discard localhost:7021 …: kill: No such process the files go anyway PIDs get reused: an old stale entry may name another process
A guest shut down from inside or a crashed QEMU leaves port.7021.pid behind. ps hides it; ps --all shows it as (stale) next to the running machine, ps --stale shows only it, and a pid file without a number is (malformed). discard localhost:7021 reports that the kill failed and removes the files. PIDs get reused, so an old stale entry may name another process.

ps --stale lists entries whose process is gone, or whose pid file is malformed; ps --all lists everything. discard removes a stale entry.

Not every machine ends with discard. Shut the guest down from inside, or let QEMU crash, and the process is gone, but its pid file stays. ps hides such an entry. --stale shows only those, marked stale, and --all shows them next to the running ones. A pid file that doesn't hold a number is marked malformed. To clean up, discard the address as usual: the kill fails, it says so, and the files go anyway. One caution: discard trusts the number in the file, and process numbers get reused. An old stale entry may name another of your processes.

Details and sources

Details

  • An entry is (stale) when its PID is a number that does not run, (malformed) when the pid file does not hold a number. Default: running entries only; --stale: stale and malformed ones only; --all: every entry, with the suffix (image-garden:1309-1355).
  • Run: with one live and one dead PID, ps showed one row, ps --all both with (stale) on the dead one and an empty RSS column, ps --stale only the dead one.
  • A guest that powers off ends QEMU, and with it the process; the pid file stays because nothing removes it but discard and allocate's failure paths (image-garden:1186-1190, image-garden:1208-1210).
  • discard does not check that the PID belongs to QEMU before kill (image-garden:1260-1265). PID reuse is derived from how Linux assigns process IDs.

06 Machines with a TPM

TPM MACHINES QEMU swtpm $XDG_RUNTIME_DIR/image-garden/port.5555.tpm/ swtpm.sock swtpm.pid state/ a copy of the TPM state QEMU gone: swtpm exits by itself wait up to 5 s, then kill swtpm rm -rf port.5555.tpm/ the launcher's watcher: rm -rf too so discard retries for 2 s, then fails
A TPM machine's swtpm listens in port.5555.tpm/ with its socket, pid file and a copy of the TPM state. discard stops QEMU; swtpm exits by itself or is killed after 5 seconds; discard removes the directory, retrying for 2 seconds because the launcher's watcher removes it too.

For port.5555.tpm/, discard waits up to 5 s for swtpm to exit, kills it, and removes the directory, retrying for 2 s while the launcher's watcher does the same.

A machine with a TPM has one more thing to clean up. Its launcher started swtpm, the TPM emulator, in a directory of its own, named after the port. When discard has stopped QEMU, swtpm notices the disconnect and exits by itself. discard gives it up to five seconds, then kills it if it's still there. Then it removes the directory, with the copy of the TPM state inside. The launcher's watcher may be removing the same directory at the very same moment, so discard keeps trying for two seconds before it reports a failure.

Details and sources

Details

  • allocate names the directory $XDG_RUNTIME_DIR/image-garden/port.PORT.tpm, exports it as GARDEN_TPM_RUN_DIR and removes a leftover one first (image-garden:825-831, image-garden:1108-1112).
  • discard_tpm_run_dir reads swtpm.pid, waits up to 50 × 0.1 s while swtpm runs and is not a zombie, then kills it; then it tries rm -rf up to 20 times, 0.1 s apart, and a last time that may fail (image-garden:833-865).
  • The launcher's watcher, tail --pid=… -f /dev/null; rm -rf …, removes the directory when swtpm exits (010-qemu-x86_64.mk:203).
  • Only the four +tpm Ubuntu Core systems have such a directory.

07 Where to go from here

ROADS discard and ps image-garden allocate where the files come from The adhoc backend spread's discard hook swtpm per launch the TPM directory Where files live and what you can delete
Four roads from discard and ps: image-garden allocate, spread's adhoc backend, swtpm per launch, and where image-garden keeps its files

Next: allocate, spread's adhoc backend, swtpm per launch, and where image-garden keeps its files.

That's discard and ps: two small files per machine, one command that stops machines by address or all at once, and one that lists them, with stale entries on request. From here, the allocate deck shows where those files come from, and spread's adhoc backend shows the discard hook that calls this for you. The swtpm deck explains the directory a machine with a TPM keeps. And the cache deck shows every other file image-garden leaves around, and which ones you can delete.

Details and sources

Read more

Basis of the statements on this page

Read in the image-garden source at the commit in garden.commit ("code"). The outputs of ps, ps --all, ps --stale and discard were run with the v0.8.0 script, whose ps and discard are unchanged at that commit, against a scratch XDG_RUNTIME_DIR holding one pid file of a live process and one of a dead one; no VM was started. PIDs, ports and RSS values on the slides are examples.

Commands

CommandEffectExit
image-garden discard localhost:PORTkill the PID in port.PORT.pid, remove it, port.PORT.system and port.PORT.tpm/0, also when there is no pid file
image-garden discard -athe same for every port.*.pid0
image-garden discardprints "usage: image-garden discard ADDRESS \-a"1
image-garden discard OTHERnothing1
image-garden psheader and running entries0
image-garden ps --staleheader and stale or malformed entries0
image-garden ps --allheader and every entry, with (stale) or (malformed)0

Sources: image-garden:1221-1358.

Files

File in $XDG_RUNTIME_DIR/image-garden/Written byRemoved by
port.PORT.pidQEMU (-pidfile)discard, allocate when the machine fails to come up
port.PORT.systemallocatediscard, allocate when the machine dies before SSH
port.PORT.tpm/the +tpm launcherthe launcher's watcher, discard, the next allocate on that port
allocate.lockallocatenothing; it is a lock

Sources: image-garden:1070-1080, image-garden:1108-1112, image-garden:1151-1215, 010-qemu-x86_64.mk:173-203.

Example output

$ image-garden ps --all
ADDRESS           PID       RSS(MB)   SYSTEM
localhost:5555    41872     812.6     ubuntu-cloud-24.04.x86_64
localhost:7021    38110               fedora-cloud-42.x86_64  (stale)

Roads from here

RUNTIME · make rules and the CLI

SPREAD · tests

BUILT ON IT · CI integrations