Where image-garden keeps its files
cache, project, runtime for users
Downloads live in the cache, ~/.cache/image-garden/dl for a native or source install, under ~/snap/image-garden/common/cache in the snap and in the /var/cache/image-garden volume in the container; prepared disks, launchers and logs live in the project or .image-garden/; pid files and locks of running machines live in $XDG_RUNTIME_DIR/image-garden. make clean empties the project side and keeps the downloads; distclean removes them too.
On the map: runtime
▶ Watch this deck — 7 scenes, ~5 min
Worth seeing first
- image-garden make and the other commands — where image-garden make puts the files it builds
01 Three places, three lifetimes
The cache keeps downloads for every project; the project directory keeps what was made for it; the runtime directory tracks running machines and is empty after a reboot.
image-garden writes to three places, and each lives for a different length of time. The cache holds downloaded cloud images. They're large, the same for every project, and kept until you remove them. The project directory, or .image-garden inside it, holds what's made for this project: the prepared disks, the seed ISOs, the launchers and the logs. And the runtime directory holds the small files of machines that are running right now; it starts empty after every reboot. Knowing which is which tells you where to look when something's wrong, and what you can safely delete.
Details and sources
Details
- The cache is
GARDEN_CACHE_DIR, by default$XDG_CACHE_HOME/image-gardenor~/.cache/image-garden; downloads go toGARDEN_DL_DIR, by default$GARDEN_CACHE_DIR/dl(image-garden:20-29). image-garden makebuilds in.image-garden/when it exists, else in the current directory (image-garden:982-993).- The runtime directory is
$XDG_RUNTIME_DIR/image-garden; withoutXDG_RUNTIME_DIRimage-garden uses/run(image-garden:61-73). That/run/user/UIDis a tmpfs, empty after a reboot, is how systemd-logind sets it up (derived); the man page calls it the "Directory with per-user runtime files that do not persist across reboots" (image-garden.1:529-538). - The user ID 1000 and port 5123 on the slide are examples.
Read more
02 Where the cache is, per install
Native or source: ~/.cache/image-garden. Snap: ~/snap/image-garden/common/cache. Container: /var/cache/image-garden, a volume. Downloads are in dl/; GARDEN_CACHE_DIR and GARDEN_DL_DIR override.
Where's the cache? That depends on how you installed image-garden. From a native package or a source build, it's ~/.cache/image-garden, or the image-garden directory under XDG_CACHE_HOME if you set that. The snap keeps it in its own directory, ~/snap/image-garden/common/cache. In the container it's /var/cache/image-garden, a volume, so mount your own cache there or each container starts empty. Downloads go in its dl subdirectory. GARDEN_CACHE_DIR moves the whole cache, and GARDEN_DL_DIR moves just the downloads.
Details and sources
Details
- In the snap (
SNAP_USER_COMMONset)GARDEN_CACHE_DIRdefaults to$SNAP_USER_COMMON/cache; elsewhere to${XDG_CACHE_HOME:-$HOME/.cache}/image-garden, and with neitherHOMEnorXDG_CACHE_HOMEnorGARDEN_CACHE_DIRthe script stops with "set HOME, XDG_CACHE_HOME or GARDEN_CACHE_DIR to say where to keep images" (image-garden:20-29). The makefile sets the same defaults (000-garden.mk:9-12). SNAP_USER_COMMONis~/snap/image-garden/commonfor a snap named image-garden (snapd's convention, derived).- The container sets
ENV GARDEN_CACHE_DIR=/var/cache/image-gardenandVOLUME /var/cache/image-garden(Dockerfile:96-97); the README mounts$HOME/.cache/image-gardenthere (README.md:365-382). Without a mount, docker gives the volume a new anonymous volume per container (derived). - The snap's pkgproxy keeps its own cache in
$SNAP_COMMON/cache/pkgproxy, apart from this one; see why pkgproxy. GARDEN_CACHE_DIRandGARDEN_DL_DIRare read from the environment by both the script and the makefile; a movedGARDEN_DL_DIRleaves existing overlays pointing at the old place, see rebase.
Read more
03 What is in the download cache
dl/ubuntu/ holds the image, its .sha256sums, an empty .lock, ubuntu-cloud-24.04.x86_64.url and .filename, and the link ubuntu-cloud-24.04.x86_64.qcow2 that prepared disks use as their backing file.
Look inside and you'll find one directory per distribution. In the ubuntu one, our system has left six files. The image itself, under the distribution's name. Its checksum list, saved next to it. A lock file, empty, used only while a download runs. The .url and .filename files that make wrote. And the link named after the system, pointing at the image. Your prepared disks, in every project, point at that link. That's what makes the cache shareable, and it's also why deleting it costs more than a download: the disks built on it can no longer boot.
Details and sources
Details
- Each data file makes
$(GARDEN_DL_DIR)/GROUP(rulegen.py:447-450);X.url,X.filenameand the linkX.qcow2are written there (rulegen.py:480-536). - The helper writes
FILE,FILE.sha256sums(or.sha512sums) andFILE.lockin the same directory (image-garden:351-356, image-garden:475-545). During a download there may also beFILE.partialorFILE.sha256sums.partial; see download and verify. - Ubuntu Core also keeps the
.img.xzdownload and the converted.qcow2there (010-qemu.mk:334-347). - The launcher of each system starts with a comment that says so:
# WARNING: The .qcow2 file refers to a file in DL/GROUP(fromimage-garden make -n). - The README: "Shared across all projects for the same user" (README.md:860-868). The other directories on the slide depend on which systems you made.
Read more
04 What allocate adds to your project
allocate adds SYSTEM.lock and, per machine, SYSTEM.PORT.serial.log, .stdout.log and .stderr.log. discard keeps the logs; make clean removes every *.log.
The project side you've met already: about eight files per system, in .image-garden when that directory exists. Machines you allocate add a few more. allocate takes a lock on a file named after the system while it builds, so two allocations don't build the same disk at once. Then each machine writes three logs, named after the system and its port: the serial console, and QEMU's output and its errors. discard stops the machine but leaves the logs, so you can read them after a failed test. They pile up over time, until image-garden make clean removes them with everything else.
Details and sources
Details
- Unless
--premade, allocate runsflock <.image-garden/ or ./>SYSTEM.lock image-garden make SYSTEM.run SYSTEM.qcow2(image-garden:1058-1064). - It starts the launcher with
-serial file:SYSTEM.PORT.serial.logand redirects standard output and error toSYSTEM.PORT.stdout.logandSYSTEM.PORT.stderr.log, in.image-garden/when it exists (image-garden:1153-1163). With--spread, a failed start prints the last line of the stderr log in<FATAL …>. discardkills the process and removes the pid and system files and a TPM run directory, not the logs (image-garden:1221-1274).make cleanrunsrm -f *.login the directory make runs in (000-garden.mk:48-50). The lock files are not removed by any rule.- A +tpm system also keeps its TPM state in
SYSTEM.x86_64.tpm/next to the disk (README.md:836-837);make cleanremoves it (050-ubuntu-core.mk:315-319). - The ports on the slide are examples.
Read more
05 The runtime directory
$XDG_RUNTIME_DIR/image-garden/ holds allocate.lock, port.PORT.pid, port.PORT.system, port.PORT.tpm/ and pkgproxy-unreachable.HASH. Without XDG_RUNTIME_DIR it is /run/image-garden/.
The third place is an image-garden directory inside your runtime directory, the one XDG_RUNTIME_DIR names, usually under /run/user. allocate keeps a lock there while it picks a port. For each running machine it writes a pid file and a file with the system's name, both named after the port: that's how ps lists machines and discard finds them. A machine with a TPM gets a directory for its swtpm. And a pkgproxy that couldn't be reached is remembered there for five minutes. Without XDG_RUNTIME_DIR, as for root in a container, image-garden uses /run instead.
Details and sources
Details
XDG_RUNTIME_DIRfalls back to/run, "in CI/CD systems that run a container runtime and don't use systemd or its PAM module"; in the snap, when the directory snapd names does not exist, image-garden uses/run/snap.$SNAP_INSTANCE_NAME(image-garden:61-73); see the snap's paths.allocate.lockis taken withflock --exclusive -nin a loop while allocate picks a port and starts QEMU (image-garden:1070-1080).- QEMU writes
port.PORT.pidthrough-pidfile; allocate writes the system's name toport.PORT.system(image-garden:1153-1166);psanddiscardread them (image-garden:1221-1358). port.PORT.tpmholds swtpm's socket and, as the machine runs with-snapshot, its copy of the TPM state (image-garden:825-831). The first boot of a +tpm system runs swtpm in atpm.XXXXdirectory there (050-ubuntu-core.mk:84-89).pkgproxy-unreachable.HASH, with the first 16 hex digits of the sha256 of the pkgproxy address, is touched when curl cannot resolve, connect or connect in time; downloads skip that pkgproxy while the file is younger than 300 seconds; sockets are not remembered (image-garden:138-193).- The user ID, port and hash on the slide are examples.
Read more
06 What you can delete
make clean empties the project and removes the cache links; distclean also removes the downloads with their .url and .filename files. Either way of emptying the cache costs a download and a provisioning boot per system.
So what's safe to delete? In the project, anything: image-garden make clean removes the disks, launchers, seeds, firmware copies and logs, and the next make builds them again from the cache. Clean also removes the system links in the cache, but keeps the images, so nothing is downloaded again. distclean goes further. It runs clean, then removes each system's download with its .url and .filename files, and Ubuntu Core's compressed and converted images. That frees the space, and the next make pays a download and a provisioning boot for each system you use. Deleting the cache by hand does the same, and takes the checksum lists and locks too. And leave the runtime directory to discard.
Details and sources
Details
clean::rules remove*.log(000-garden.mk:48-50);*.seed.iso,*.meta-data,*.user-data(010-qemu.mk:300-302); the firmware copies and*.x86_64.qcow2with the per-system firmware files (010-qemu-x86_64.mk:332-334, and the same for the other architectures); and per system the cache link$(GARDEN_DL_DIR)/GROUP/X.qcow2,X.qcow2,X.run,X.meta-dataandX.user-data(rulegen.py:473-478).distclean:: cleancomes first inimage-garden.mk(000-garden.mk:52-53). Each system'sdistclean::recipe then removes the file the link points to, if the link is still there, and the file named inX.filename, whichcleankeeps, then the link,X.urlandX.filename(rulegen.py:486-492). The README: distclean "also removes downloaded base images from the cache directory" (README.md:547-550).- Ubuntu Core's
distclean::removesubuntu-core-R-*.img.xzand the convertedubuntu-core-R-*.qcow2from$(GARDEN_DL_DIR)/ubuntu-core/(050-ubuntu-core.mk:326-328), where they are downloaded (050-ubuntu-core.mk:229-232). - Run on a scratch cache with an
image-garden.mkbuilt from the commit in garden.commit:image-garden make distcleanremoved the project files, the image ofubuntu-cloud-24.04.x86_64with its link,.urland.filename, and Ubuntu Core's.img.xzand converted.qcow2. - The downloads are shared by every project of the user, so
distcleanin one project also takes the backing files of the prepared disks of the others (derived). - The checksum lists and
.lockfiles in the cache are removed by no rule; deleting the cache directory by hand removes everything. A prepared disk whose backing file is gone fails to boot from.run;image-garden makesees the missing link, downloads the image again, and the newer image makes it provision the disk again (derived from the rules). Afterdistcleanthe disks are gone as well, so make provisions them in any case. - The README's own advice: the cache "Can be safely deleted to reclaim disk space (use
make distclean)" (README.md:860-868). - Removing pid files while machines run leaves QEMU running where
psanddiscardcannot see it (derived);image-garden ps --stalelists entries whose QEMU is gone.
Read more
07 Where to go from here
Next: download and verify, rebase, discard and ps, the snap's paths and the container.
Three places, then: a shared cache of downloads, your project's prepared machines, and a runtime directory that lasts until the next reboot. Short of disk space? Clean a project first; the cache is the part that's slow to get back. The download deck explains every file a download leaves in the cache. Rebase repairs prepared disks after the cache has moved. discard and ps read the runtime files. The snap's confinement deck shows where its paths move. And the container deck shows how to mount your cache, so that each container doesn't start empty.
Details and sources
Read more
Basis of the statements on this page
Read in the image-garden source at the commit in garden.commit ("code"): the image-garden script, mk/*.mk, rulegen.py, the Dockerfile and the README. What make clean and make distclean leave behind was checked by running them with an image-garden.mk built from that tree on a scratch cache ("run"). The snap's SNAP_USER_COMMON path and the lifetime of /run/user/UID follow from how snapd and systemd set them up ("derived").
The three places
| Place | Default | Holds | Lifetime |
|---|---|---|---|
cache (GARDEN_CACHE_DIR) | see the next table | downloads in dl/, shared by every project of the user | until you delete it |
| project | the current directory, or .image-garden/ when it exists | disks, launchers, seeds, firmware copies, logs, allocate's locks | until make clean |
runtime ($XDG_RUNTIME_DIR/image-garden) | /run/user/UID/image-garden, or /run/image-garden without XDG_RUNTIME_DIR | allocate's lock, pid and system files, TPM run directories, pkgproxy-unreachable markers | until reboot |
The cache per install
| Install | GARDEN_CACHE_DIR | Downloads (GARDEN_DL_DIR) |
|---|---|---|
| native package or source | ${XDG_CACHE_HOME:-~/.cache}/image-garden | ~/.cache/image-garden/dl |
| snap | $SNAP_USER_COMMON/cache | ~/snap/image-garden/common/cache/dl |
| container | /var/cache/image-garden (a volume) | /var/cache/image-garden/dl |
Either variable may be set in the environment to move the cache or only the downloads (image-garden:20-29, 000-garden.mk:9-12, Dockerfile:96-97). The snap's pkgproxy has a cache of its own in $SNAP_COMMON/cache/pkgproxy.
Files in the download cache
In dl/GROUP/, for a system X whose download is FILE:
| File | Made by | Removed by |
|---|---|---|
FILE | the download helper | distclean |
FILE.sha256sums or FILE.sha512sums | the download helper | a failed check |
FILE.lock | the download helper | nothing |
FILE.partial, FILE.sha256sums.partial | an unfinished download | a completed download |
X.url, X.filename | make | distclean |
X.qcow2, a link to FILE | make | clean, distclean |
Files in the project
| File | Made by | Removed by |
|---|---|---|
X.qcow2, X.qcow2.log, X.run, X.seed.iso, X.user-data, X.meta-data | make | clean |
X.efi-code.qcow2, X.efi-vars.qcow2, efi-code.ARCH.img, efi-vars.ARCH.img | make | clean |
X.x86_64.tpm/ (+tpm systems) | the first boot | clean |
X.lock | allocate | nothing |
X.PORT.serial.log, X.PORT.stdout.log, X.PORT.stderr.log | allocate | clean (any *.log) |
Files in the runtime directory
| File | Made by | Removed by |
|---|---|---|
allocate.lock | allocate | nothing; it is only locked |
port.PORT.pid | QEMU's -pidfile | discard, a failed allocate |
port.PORT.system | allocate | discard |
port.PORT.tpm/ | the launcher of a +tpm system | its watcher, discard |
tpm.XXXX/ | the first boot of a +tpm system | that boot |
pkgproxy-unreachable.HASH | a download that cannot reach pkgproxy | nothing; it is ignored after 300 seconds |
Sources: image-garden:138-193, image-garden:825-859, image-garden:1070-1274, 050-ubuntu-core.mk:84-106.
clean and distclean
| Command | Removes | Keeps |
|---|---|---|
image-garden make clean | in the project: every file in the project table but the locks; in the cache: the system links | the downloads, their lists and locks, .url, .filename |
image-garden make distclean | what clean removes, then each system's download (found through X.filename), .url and .filename, and Ubuntu Core's .img.xz and converted .qcow2 | the checksum lists and locks |
distclean works as the README says (README.md:547-550, rulegen.py:486-492, 050-ubuntu-core.mk:326-328). The downloads are shared by every project, so other projects' prepared disks lose their base too; the next make there downloads what it needs and provisions the disks again (derived).
Roads from here
RUNTIME · make rules and the CLI
- Downloading and checking images — the files a download leaves in the cache
- image-garden rebase — prepared disks after the cache moved
- Stopping and listing VMs — the runtime files that discard and ps read
- image-garden make and the other commands — the project directory and .image-garden/
SNAP · snap-only features
- Living with strict confinement — where the snap's paths move
CONTAINER · OCI image
- Running image-garden in a container — mounting your cache into the container