Where image-garden keeps its files

cache, project, runtime for users

Downloads live in the cache, ~/.cache/image-garden/dl for a native or source install, under ~/snap/image-garden/common/cache in the snap and in the /var/cache/image-garden volume in the container; prepared disks, launchers and logs live in the project or .image-garden/; pid files and locks of running machines live in $XDG_RUNTIME_DIR/image-garden. make clean empties the project side and keeps the downloads; distclean removes them too.

On the map: runtime

▶ Watch this deck — 7 scenes, ~5 min

Worth seeing first

01 Three places, three lifetimes

THREE PLACES the cache the project the runtime directory ~/.cache/image-garden/dl/ ubuntu/ noble-…-amd64.img debian/ fedora/ … kept until you remove it large, shared by every project your-project/.image-garden/ ….x86_64.qcow2 ….x86_64.qcow2.log ….x86_64.seed.iso ….x86_64.run kept with the project made for this project /run/user/1000/image-garden/ allocate.lock port.5123.pid port.5123.system empty after a reboot machines running right now which is which: where to look when something is wrong, and what is safe to delete
Three places: the cache in ~/.cache/image-garden/dl holds downloads for every project and is kept until you remove it; the project's .image-garden/ holds disks, launchers, seeds and logs for this project; the runtime directory /run/user/1000/image-garden holds the lock and pid files of running machines and is empty after a reboot

The cache keeps downloads for every project; the project directory keeps what was made for it; the runtime directory tracks running machines and is empty after a reboot.

image-garden writes to three places, and each lives for a different length of time. The cache holds downloaded cloud images. They're large, the same for every project, and kept until you remove them. The project directory, or .image-garden inside it, holds what's made for this project: the prepared disks, the seed ISOs, the launchers and the logs. And the runtime directory holds the small files of machines that are running right now; it starts empty after every reboot. Knowing which is which tells you where to look when something's wrong, and what you can safely delete.

Details and sources

Details

  • The cache is GARDEN_CACHE_DIR, by default $XDG_CACHE_HOME/image-garden or ~/.cache/image-garden; downloads go to GARDEN_DL_DIR, by default $GARDEN_CACHE_DIR/dl (image-garden:20-29).
  • image-garden make builds in .image-garden/ when it exists, else in the current directory (image-garden:982-993).
  • The runtime directory is $XDG_RUNTIME_DIR/image-garden; without XDG_RUNTIME_DIR image-garden uses /run (image-garden:61-73). That /run/user/UID is a tmpfs, empty after a reboot, is how systemd-logind sets it up (derived); the man page calls it the "Directory with per-user runtime files that do not persist across reboots" (image-garden.1:529-538).
  • The user ID 1000 and port 5123 on the slide are examples.

02 Where the cache is, per install

THE CACHE, PER INSTALL package or source ~/.cache/image-garden or $XDG_CACHE_HOME/image-garden when you set it the snap ~/snap/image-garden/common/cache the container /var/cache/image-garden a volume --volume $HOME/.cache/image-garden:/var/cache/image-garden CACHE/dl/ubuntu/ CACHE/dl/debian/ … the downloads GARDEN_CACHE_DIR moves the cache, GARDEN_DL_DIR only the downloads
The cache per install: ~/.cache/image-garden for a native package or source build, or under XDG_CACHE_HOME; ~/snap/image-garden/common/cache for the snap; the /var/cache/image-garden volume in the container, where you mount your own cache; downloads in its dl subdirectory; GARDEN_CACHE_DIR and GARDEN_DL_DIR move them

Native or source: ~/.cache/image-garden. Snap: ~/snap/image-garden/common/cache. Container: /var/cache/image-garden, a volume. Downloads are in dl/; GARDEN_CACHE_DIR and GARDEN_DL_DIR override.

Where's the cache? That depends on how you installed image-garden. From a native package or a source build, it's ~/.cache/image-garden, or the image-garden directory under XDG_CACHE_HOME if you set that. The snap keeps it in its own directory, ~/snap/image-garden/common/cache. In the container it's /var/cache/image-garden, a volume, so mount your own cache there or each container starts empty. Downloads go in its dl subdirectory. GARDEN_CACHE_DIR moves the whole cache, and GARDEN_DL_DIR moves just the downloads.

Details and sources

Details

  • In the snap (SNAP_USER_COMMON set) GARDEN_CACHE_DIR defaults to $SNAP_USER_COMMON/cache; elsewhere to ${XDG_CACHE_HOME:-$HOME/.cache}/image-garden, and with neither HOME nor XDG_CACHE_HOME nor GARDEN_CACHE_DIR the script stops with "set HOME, XDG_CACHE_HOME or GARDEN_CACHE_DIR to say where to keep images" (image-garden:20-29). The makefile sets the same defaults (000-garden.mk:9-12).
  • SNAP_USER_COMMON is ~/snap/image-garden/common for a snap named image-garden (snapd's convention, derived).
  • The container sets ENV GARDEN_CACHE_DIR=/var/cache/image-garden and VOLUME /var/cache/image-garden (Dockerfile:96-97); the README mounts $HOME/.cache/image-garden there (README.md:365-382). Without a mount, docker gives the volume a new anonymous volume per container (derived).
  • The snap's pkgproxy keeps its own cache in $SNAP_COMMON/cache/pkgproxy, apart from this one; see why pkgproxy.
  • GARDEN_CACHE_DIR and GARDEN_DL_DIR are read from the environment by both the script and the makefile; a moved GARDEN_DL_DIR leaves existing overlays pointing at the old place, see rebase.

03 What is in the download cache

INSIDE THE CACHE ~/.cache/image-garden/dl/ debian/ fedora/ opensuse/ … ubuntu/ noble-server-cloudimg-amd64.img noble-server-cloudimg-amd64.img.sha256sums the image its checksum list noble-server-cloudimg-amd64.img.lock empty, held during a download ubuntu-cloud-24.04.x86_64.url ubuntu-cloud-24.04.x86_64.filename written by make ubuntu-cloud-24.04.x86_64.qcow2 -> noble-…-amd64.img the link project-a/.image-garden/….x86_64.qcow2 project-b/.image-garden/….x86_64.qcow2 delete the cache, and these disks can no longer boot
~/.cache/image-garden/dl has a directory per distribution; dl/ubuntu holds noble-server-cloudimg-amd64.img, its .sha256sums list, an empty .lock file, ubuntu-cloud-24.04.x86_64.url and .filename, and the link ubuntu-cloud-24.04.x86_64.qcow2; prepared disks in two projects point at that link, so deleting the cache leaves them unable to boot

dl/ubuntu/ holds the image, its .sha256sums, an empty .lock, ubuntu-cloud-24.04.x86_64.url and .filename, and the link ubuntu-cloud-24.04.x86_64.qcow2 that prepared disks use as their backing file.

Look inside and you'll find one directory per distribution. In the ubuntu one, our system has left six files. The image itself, under the distribution's name. Its checksum list, saved next to it. A lock file, empty, used only while a download runs. The .url and .filename files that make wrote. And the link named after the system, pointing at the image. Your prepared disks, in every project, point at that link. That's what makes the cache shareable, and it's also why deleting it costs more than a download: the disks built on it can no longer boot.

Details and sources

Details

  • Each data file makes $(GARDEN_DL_DIR)/GROUP (rulegen.py:447-450); X.url, X.filename and the link X.qcow2 are written there (rulegen.py:480-536).
  • The helper writes FILE, FILE.sha256sums (or .sha512sums) and FILE.lock in the same directory (image-garden:351-356, image-garden:475-545). During a download there may also be FILE.partial or FILE.sha256sums.partial; see download and verify.
  • Ubuntu Core also keeps the .img.xz download and the converted .qcow2 there (010-qemu.mk:334-347).
  • The launcher of each system starts with a comment that says so: # WARNING: The .qcow2 file refers to a file in DL/GROUP (from image-garden make -n).
  • The README: "Shared across all projects for the same user" (README.md:860-868). The other directories on the slide depend on which systems you made.

04 What allocate adds to your project

WHAT ALLOCATE ADDS your-project/.image-garden/ … about eight files per system … ubuntu-cloud-24.04.x86_64.lock locked while allocate builds ubuntu-cloud-24.04.x86_64.5123.serial.log ubuntu-cloud-24.04.x86_64.5123.stdout.log ubuntu-cloud-24.04.x86_64.5123.stderr.log the serial console QEMU's output QEMU's errors $ image-garden discard localhost:5123 the machine stops, the logs stay ubuntu-cloud-24.04.x86_64.5241.serial.log … $ image-garden make clean removes every log, with the disks and launchers
In your-project/.image-garden, besides about eight files per system, allocate adds ubuntu-cloud-24.04.x86_64.lock while it builds and, for the machine on port 5123, a serial log, a stdout log and a stderr log; image-garden discard stops the machine and keeps the logs; image-garden make clean removes them

allocate adds SYSTEM.lock and, per machine, SYSTEM.PORT.serial.log, .stdout.log and .stderr.log. discard keeps the logs; make clean removes every *.log.

The project side you've met already: about eight files per system, in .image-garden when that directory exists. Machines you allocate add a few more. allocate takes a lock on a file named after the system while it builds, so two allocations don't build the same disk at once. Then each machine writes three logs, named after the system and its port: the serial console, and QEMU's output and its errors. discard stops the machine but leaves the logs, so you can read them after a failed test. They pile up over time, until image-garden make clean removes them with everything else.

Details and sources

Details

  • Unless --premade, allocate runs flock <.image-garden/ or ./>SYSTEM.lock image-garden make SYSTEM.run SYSTEM.qcow2 (image-garden:1058-1064).
  • It starts the launcher with -serial file:SYSTEM.PORT.serial.log and redirects standard output and error to SYSTEM.PORT.stdout.log and SYSTEM.PORT.stderr.log, in .image-garden/ when it exists (image-garden:1153-1163). With --spread, a failed start prints the last line of the stderr log in <FATAL …>.
  • discard kills the process and removes the pid and system files and a TPM run directory, not the logs (image-garden:1221-1274).
  • make clean runs rm -f *.log in the directory make runs in (000-garden.mk:48-50). The lock files are not removed by any rule.
  • A +tpm system also keeps its TPM state in SYSTEM.x86_64.tpm/ next to the disk (README.md:836-837); make clean removes it (050-ubuntu-core.mk:315-319).
  • The ports on the slide are examples.

05 The runtime directory

THE RUNTIME DIRECTORY /run/user/1000/image-garden/ allocate.lock while allocate picks a port port.5123.pid port.5123.system QEMU's process ID the system's name read by ps and discard port.5123.tpm/ a TPM machine's swtpm pkgproxy-unreachable.3f9a… skip that pkgproxy for 5 minutes without XDG_RUNTIME_DIR, as for root in a container: /run/image-garden/
/run/user/1000/image-garden holds allocate.lock while allocate picks a port, port.5123.pid and port.5123.system for each running machine, read by ps and discard, port.5123.tpm for a TPM machine's swtpm, and pkgproxy-unreachable files that skip an unreachable pkgproxy for five minutes; without XDG_RUNTIME_DIR the directory is /run/image-garden

$XDG_RUNTIME_DIR/image-garden/ holds allocate.lock, port.PORT.pid, port.PORT.system, port.PORT.tpm/ and pkgproxy-unreachable.HASH. Without XDG_RUNTIME_DIR it is /run/image-garden/.

The third place is an image-garden directory inside your runtime directory, the one XDG_RUNTIME_DIR names, usually under /run/user. allocate keeps a lock there while it picks a port. For each running machine it writes a pid file and a file with the system's name, both named after the port: that's how ps lists machines and discard finds them. A machine with a TPM gets a directory for its swtpm. And a pkgproxy that couldn't be reached is remembered there for five minutes. Without XDG_RUNTIME_DIR, as for root in a container, image-garden uses /run instead.

Details and sources

Details

  • XDG_RUNTIME_DIR falls back to /run, "in CI/CD systems that run a container runtime and don't use systemd or its PAM module"; in the snap, when the directory snapd names does not exist, image-garden uses /run/snap.$SNAP_INSTANCE_NAME (image-garden:61-73); see the snap's paths.
  • allocate.lock is taken with flock --exclusive -n in a loop while allocate picks a port and starts QEMU (image-garden:1070-1080).
  • QEMU writes port.PORT.pid through -pidfile; allocate writes the system's name to port.PORT.system (image-garden:1153-1166); ps and discard read them (image-garden:1221-1358).
  • port.PORT.tpm holds swtpm's socket and, as the machine runs with -snapshot, its copy of the TPM state (image-garden:825-831). The first boot of a +tpm system runs swtpm in a tpm.XXXX directory there (050-ubuntu-core.mk:84-89).
  • pkgproxy-unreachable.HASH, with the first 16 hex digits of the sha256 of the pkgproxy address, is touched when curl cannot resolve, connect or connect in time; downloads skip that pkgproxy while the file is younger than 300 seconds; sockets are not remembered (image-garden:138-193).
  • The user ID, port and hash on the slide are examples.

06 What you can delete

WHAT YOU CAN DELETE $ image-garden make clean project: disks, launchers, seeds, firmware, logs the next make builds them again from the cache cache: the system links go, the images stay $ image-garden make distclean clean, then the downloads as well each system's image, its .url and .filename Ubuntu Core: .img.xz and the converted .qcow2 next make: download and provision again $ rm -r ~/.cache/image-garden the same, with the checksum lists and locks the runtime directory: leave it to image-garden discard
image-garden make clean removes the project's disks, launchers, seeds, firmware copies and logs, and the system links in the cache, but keeps the images; distclean runs clean, then removes each system's download with its .url and .filename files, and Ubuntu Core's .img.xz and converted .qcow2; deleting the cache by hand also takes the checksum lists and locks; either costs a download and a provisioning boot per system; the runtime directory is left to discard

make clean empties the project and removes the cache links; distclean also removes the downloads with their .url and .filename files. Either way of emptying the cache costs a download and a provisioning boot per system.

So what's safe to delete? In the project, anything: image-garden make clean removes the disks, launchers, seeds, firmware copies and logs, and the next make builds them again from the cache. Clean also removes the system links in the cache, but keeps the images, so nothing is downloaded again. distclean goes further. It runs clean, then removes each system's download with its .url and .filename files, and Ubuntu Core's compressed and converted images. That frees the space, and the next make pays a download and a provisioning boot for each system you use. Deleting the cache by hand does the same, and takes the checksum lists and locks too. And leave the runtime directory to discard.

Details and sources

Details

  • clean:: rules remove *.log (000-garden.mk:48-50); *.seed.iso, *.meta-data, *.user-data (010-qemu.mk:300-302); the firmware copies and *.x86_64.qcow2 with the per-system firmware files (010-qemu-x86_64.mk:332-334, and the same for the other architectures); and per system the cache link $(GARDEN_DL_DIR)/GROUP/X.qcow2, X.qcow2, X.run, X.meta-data and X.user-data (rulegen.py:473-478).
  • distclean:: clean comes first in image-garden.mk (000-garden.mk:52-53). Each system's distclean:: recipe then removes the file the link points to, if the link is still there, and the file named in X.filename, which clean keeps, then the link, X.url and X.filename (rulegen.py:486-492). The README: distclean "also removes downloaded base images from the cache directory" (README.md:547-550).
  • Ubuntu Core's distclean:: removes ubuntu-core-R-*.img.xz and the converted ubuntu-core-R-*.qcow2 from $(GARDEN_DL_DIR)/ubuntu-core/ (050-ubuntu-core.mk:326-328), where they are downloaded (050-ubuntu-core.mk:229-232).
  • Run on a scratch cache with an image-garden.mk built from the commit in garden.commit: image-garden make distclean removed the project files, the image of ubuntu-cloud-24.04.x86_64 with its link, .url and .filename, and Ubuntu Core's .img.xz and converted .qcow2.
  • The downloads are shared by every project of the user, so distclean in one project also takes the backing files of the prepared disks of the others (derived).
  • The checksum lists and .lock files in the cache are removed by no rule; deleting the cache directory by hand removes everything. A prepared disk whose backing file is gone fails to boot from .run; image-garden make sees the missing link, downloads the image again, and the newer image makes it provision the disk again (derived from the rules). After distclean the disks are gone as well, so make provisions them in any case.
  • The README's own advice: the cache "Can be safely deleted to reclaim disk space (use make distclean)" (README.md:860-868).
  • Removing pid files while machines run leaves QEMU running where ps and discard cannot see it (derived); image-garden ps --stale lists entries whose QEMU is gone.

07 Where to go from here

ROADS Where files live Download and verify the files in the cache rebase after the cache moved discard and ps the runtime files Snap confinement paths that move The container mount your cache
Five roads from where files live: download and verify, rebase, discard and ps, the snap's confinement and the container

Next: download and verify, rebase, discard and ps, the snap's paths and the container.

Three places, then: a shared cache of downloads, your project's prepared machines, and a runtime directory that lasts until the next reboot. Short of disk space? Clean a project first; the cache is the part that's slow to get back. The download deck explains every file a download leaves in the cache. Rebase repairs prepared disks after the cache has moved. discard and ps read the runtime files. The snap's confinement deck shows where its paths move. And the container deck shows how to mount your cache, so that each container doesn't start empty.

Details and sources

Read more

Basis of the statements on this page

Read in the image-garden source at the commit in garden.commit ("code"): the image-garden script, mk/*.mk, rulegen.py, the Dockerfile and the README. What make clean and make distclean leave behind was checked by running them with an image-garden.mk built from that tree on a scratch cache ("run"). The snap's SNAP_USER_COMMON path and the lifetime of /run/user/UID follow from how snapd and systemd set them up ("derived").

The three places

PlaceDefaultHoldsLifetime
cache (GARDEN_CACHE_DIR)see the next tabledownloads in dl/, shared by every project of the useruntil you delete it
projectthe current directory, or .image-garden/ when it existsdisks, launchers, seeds, firmware copies, logs, allocate's locksuntil make clean
runtime ($XDG_RUNTIME_DIR/image-garden)/run/user/UID/image-garden, or /run/image-garden without XDG_RUNTIME_DIRallocate's lock, pid and system files, TPM run directories, pkgproxy-unreachable markersuntil reboot

The cache per install

InstallGARDEN_CACHE_DIRDownloads (GARDEN_DL_DIR)
native package or source${XDG_CACHE_HOME:-~/.cache}/image-garden~/.cache/image-garden/dl
snap$SNAP_USER_COMMON/cache~/snap/image-garden/common/cache/dl
container/var/cache/image-garden (a volume)/var/cache/image-garden/dl

Either variable may be set in the environment to move the cache or only the downloads (image-garden:20-29, 000-garden.mk:9-12, Dockerfile:96-97). The snap's pkgproxy has a cache of its own in $SNAP_COMMON/cache/pkgproxy.

Files in the download cache

In dl/GROUP/, for a system X whose download is FILE:

FileMade byRemoved by
FILEthe download helperdistclean
FILE.sha256sums or FILE.sha512sumsthe download helpera failed check
FILE.lockthe download helpernothing
FILE.partial, FILE.sha256sums.partialan unfinished downloada completed download
X.url, X.filenamemakedistclean
X.qcow2, a link to FILEmakeclean, distclean

Files in the project

FileMade byRemoved by
X.qcow2, X.qcow2.log, X.run, X.seed.iso, X.user-data, X.meta-datamakeclean
X.efi-code.qcow2, X.efi-vars.qcow2, efi-code.ARCH.img, efi-vars.ARCH.imgmakeclean
X.x86_64.tpm/ (+tpm systems)the first bootclean
X.lockallocatenothing
X.PORT.serial.log, X.PORT.stdout.log, X.PORT.stderr.logallocateclean (any *.log)

Files in the runtime directory

FileMade byRemoved by
allocate.lockallocatenothing; it is only locked
port.PORT.pidQEMU's -pidfilediscard, a failed allocate
port.PORT.systemallocatediscard
port.PORT.tpm/the launcher of a +tpm systemits watcher, discard
tpm.XXXX/the first boot of a +tpm systemthat boot
pkgproxy-unreachable.HASHa download that cannot reach pkgproxynothing; it is ignored after 300 seconds

Sources: image-garden:138-193, image-garden:825-859, image-garden:1070-1274, 050-ubuntu-core.mk:84-106.

clean and distclean

CommandRemovesKeeps
image-garden make cleanin the project: every file in the project table but the locks; in the cache: the system linksthe downloads, their lists and locks, .url, .filename
image-garden make distcleanwhat clean removes, then each system's download (found through X.filename), .url and .filename, and Ubuntu Core's .img.xz and converted .qcow2the checksum lists and locks

distclean works as the README says (README.md:547-550, rulegen.py:486-492, 050-ubuntu-core.mk:326-328). The downloads are shared by every project, so other projects' prepared disks lose their base too; the next make there downloads what it needs and provisions the disks again (derived).

Roads from here

RUNTIME · make rules and the CLI

SNAP · snap-only features

CONTAINER · OCI image