How the action caches
pkgproxy, prepared images for CI operators
image-garden-action keeps pkgproxy's cache as two GitHub caches, images per system and packages per cache-scope, archived with pkgproxy's extended attributes and saved only when a job added to them. Opt-in, it also caches the prepared image, restoring the time of .image-garden.mk and rebasing the image on this runner's download.
On the map: built on it
▶ Watch this deck — 7 scenes, ~5 min
Worth seeing first
- image-garden-action for GitHub — the job these caches belong to
01 What a runner forgets
A runner starts empty. The action keeps pkgproxy's cache, which holds every download, in the GitHub cache by default; the prepared image only with cache-prepared-images.
A GitHub runner is a fresh machine for every job, so everything the last job fetched is gone. Without help, each job would download the image-garden snap, the cloud image, and every package and snap the guest installs, and then spend minutes on the image's first boot. The action keeps two kinds of things between jobs. Downloads, by default: they all go through pkgproxy, so keeping pkgproxy's cache is enough. And, only if you ask, the prepared image itself. This deck takes them in turn, from where pkgproxy keeps its files to the rebase that makes a cached image usable on a new runner.
Details and sources
Details
- "pkgproxy caches everything this action downloads: snaps and their components on the host, pristine images, and apt, snap and go traffic of the guests" (action.yaml:173-178); "The action uses pkgproxy as its only cache for downloads" (README.md:152-158).
- This holds when the action installs the pkgproxy snap ("snap" mode). With a pkgproxy that already listened on port 3142 ("external" mode, such as setup-pkgproxy's), the action neither restores nor saves a cache: it belongs to that pkgproxy (README.md:229-239).
- "Images prepared from
.image-garden.mkare not part of either: seecache-prepared-images" (README.md:173-174); that input defaults to"false"(action.yaml:18-20).
Read more
02 pkgproxy's cache, as two GitHub caches
files/, the pristine images, is one GitHub cache per runner.arch and garden-system; everything else is another, per cache-scope. A matrix shares one scope and lets one job save it.
pkgproxy keeps its cache in one directory, with the cloud images under files, stored by their checksum. The action splits it in two. The images part is that files directory, saved per runner architecture and garden system, since each system downloads a different image. The packages part is everything else, the snaps, apt and Go downloads, which most systems share. Its key uses cache-scope, which defaults to the system. In a matrix, give every job one scope, let one job save it, and set save-package-cache to false in the others. Then each job restores the shared packages, plus its own image.
Details and sources
Details
pkgproxy-cachesplits/var/snap/pkgproxy/common/cache: "images: the pristine images that image-garden downloads, which pkgproxy keeps by checksum under files/, cached per garden system; packages: everything else: snaps, apt, docker, lxd and go, which jobs of a matrix can share" (pkgproxy-cache:5-12, pkgproxy-cache:37).- Keys:
PREFIX-images-RUNNER_ARCH-GARDEN_SYSTEM-RUN_ID-RUN_ATTEMPTandPREFIX-packages-RUNNER_ARCH-SCOPE-RUN_ID-RUN_ATTEMPT, whereSCOPEiscache-scopeor elsegarden-system; each is restored by the key without the run's part (action.yaml:189-208).PREFIXiscache-key-prefix,image-garden-pkgproxy-v1by default; change it to start empty, as pkgproxy never deletes on its own (README.md:105-110). - The matrix recipe: every job
cache-scope: shared, one canary job saves, the otherssave-package-cache: "false"; "Every job then restores the shared packages and its own images, and saves its own images when they change" (README.md:199-227). X64on the slide is GitHub'srunner.archfor an x86-64 runner (GitHub's naming, not from these trees);sharedis the README's example scope.
03 Restored before pkgproxy starts
The pkgproxy snap is installed and stopped, both archives are unpacked with --xattrs --xattrs-include='user.*', each part is fingerprinted, and pkgproxy starts. actions/cache alone would drop the attributes.
At the start of a job, the action installs the pkgproxy snap and stops it straight away. While it's stopped, it unpacks both archives into pkgproxy's cache directory. Those aren't the plain archives GitHub's cache would make. pkgproxy keeps what it knows of each file in extended attributes, and the tar that GitHub's cache uses drops them. A file that lost them gets downloaded again. So the action writes its own tar files, with the user attributes kept, and GitHub's cache stores them as they are. Then it records a fingerprint of each part, and starts pkgproxy.
Details and sources
Details
- In snap mode the action installs pkgproxy,
snap stop pkgproxy, gives it its settings in onesnap set, then forimagesandpackagesrunssudo tar --xattrs --xattrs-include='user.*' -xf ARCHIVE -C /var/snap/pkgproxy/common/cache, recordspkgproxy-cache fingerprintof each part and the usage table, and starts pkgproxy (action.yaml:231-238, action.yaml:304-322). - "Write that part to ARCHIVE, a tar file that keeps the extended attributes in which pkgproxy keeps what it knows of each file. The tar of actions/cache drops them, and pkgproxy downloads again a file that lost them." (pkgproxy-cache:18-22); the archive commands (pkgproxy-cache:74-86).
- The archives are restored with
actions/cache/restoreinto$RUNNER_TEMP/image-garden-pkgproxy/(action.yaml:193-208). - Why stopped: pkgproxy's configure hook leaves a stopped daemon stopped, so it starts once with everything; one
snap setper setting went over systemd's start limit (action.yaml:235-237, action.yaml:294-303).
Read more
- What pkgproxy gives a garden; the pkgproxy map: pkgproxy.
04 Saved only when it grew
save-cache decides who may save; pkgproxy is stopped; each part's fingerprint, the sorted paths and sizes hashed, is compared with the one from the start; only a changed part is saved, under a key ending in the run ID and attempt.
At the end, the action decides whether to save. The save-cache policy comes first: by default only the default branch saves, failed jobs included. Then it stops pkgproxy, so the archives are consistent, and fingerprints each part again: the sorted list of file paths and sizes, hashed, leaving out pkgproxy's statistics file. If a fingerprint matches the one from the start, the job added nothing, and that part isn't saved. Otherwise it's archived under a new key that ends in the run's number and attempt. The next job restores by the key's prefix, which finds the newest entry.
Details and sources
Details
- The step runs
always()in snap mode.save-cache: alwayssaves;default-branchsaves only whenGITHUB_REFis the default branch, otherwise "only the default branch saves (save-cache: default-branch)";neverdoes not (action.yaml:545-601). "Failed and cancelled jobs save too, as whatever pkgproxy cached is valid." - It stops pkgproxy, with a trap that starts it again, and for each part: skips
packageswhensave-package-cacheis nottrue; comparespkgproxy-cache fingerprintwith the one taken at the start; "Not saving the pkgproxy cache of images: nothing was added to it", or archives the part and marks it for saving (action.yaml:602-620). - The fingerprint is
find … -type f -printf '%P %s\n' | sort | sha256sum, overfiles/for images and over everything else exceptstats.jsonfor packages (pkgproxy-cache:63-73). A file whose size changed counts; one rewritten at the same size does not. - The key ends in
${{ github.run_id }}-${{ github.run_attempt }}, and the restore step'srestore-keysis the key without them (action.yaml:193-208, action.yaml:621-634). That a prefix match restores the most recent cache is actions/cache's behaviour (derived, not in these trees). - After saving, the action looks the keys up again and warns "The pkgproxy cache could not be saved" when they are missing; the outputs
images-cache-savedandpackage-cache-savedreport it (action.yaml:635-656).
Read more
- What is kept between jobs, the job summary's cache table: README.md:143-145.
05 Prepared images and file times
cache-prepared-images: true caches .image-garden/ without logs, keyed by system and hashFiles('.image-garden.mk'). git restore-mtime gives .image-garden.mk its last commit's time, so make keeps the image; it needs fetch-depth: 0.
Then there's the prepared image, the result of that long first boot. With cache-prepared-images set to true, the action also caches your project's .image-garden directory, minus the logs, keyed by the system and a hash of your .image-garden.mk. But make decides by file times, and the image depends, through the user-data and the seed, on your .image-garden.mk. A fresh checkout gives that file the time of the checkout, newer than any cached image, so make would rebuild it. The action sets the file's time back to its last commit, from the git history. That's why the workflow checks out with fetch-depth: 0.
Details and sources
Details
actions/cacheofSUBDIR/.image-gardenand!SUBDIR/.image-garden/*.log, keyimage-garden-img-GARDEN_SYSTEM-HASH, whereHASHishashFiles('SUBDIR/.image-garden.mk')(action.yaml:116-123). The key has no architecture and no image-garden version in it. The README: "This uses a lot of cache storage and scales poorly with the number of systems" (README.md:71-77).- When
.image-garden.mkexists, the action installs git-restore-mtime and runsgit restore-mtime .image-garden.mk, withgit whatchangedaliased togit log --raw --no-mergesfor newer git; when the file is not found in the history it warns "Cannot restore the mtime of .image-garden.mk from git history; check out with fetch-depth: 0" (action.yaml:124-152). This runs whether or not prepared images are cached. - The dependency chain:
SYSTEM.user-datadepends on$(MAKEFILE_LIST)and.image-garden.mk(rulegen.py:467-471);%.seed.isoon the user-data and meta-data (010-qemu.mk:268);SYSTEM.qcow2on the downloaded base image and the seed (rulegen.py:601-602). - The README's workflow checks out with
fetch-depth: 0, "essential for git restore-mtime to work correctly" (README.md:405-432). - Derived, not run:
SYSTEM.qcow2also depends on the downloaded base image in the snap's download directory, outside.image-garden/. On a new runner that image is downloaded again, from pkgproxy, and gets the time of that download, newer than a restored prepared image; image-garden's downloads do not keep the server's file time. If that holds, make provisions the image again even on a cache hit.
06 Rebasing onto this runner's download
After make, image-garden rebase SYSTEM.$(uname -m).qcow2 finds the backing image by name in GARDEN_DL_DIR and runs qemu-img rebase. It runs whenever cache-prepared-images is true.
One more step goes with a cached image. A prepared qcow2 doesn't hold the whole system: it records the path of the cloud image it was made from, and stores only the changes. On a new runner, that cloud image is a fresh download. So after make, the action runs image-garden rebase on the qcow2 file. rebase reads the old backing path, finds a file of the same name in this runner's download directory, and points the image at it. The step runs whenever prepared images are cached, whether the cache was hit or not. And keep the README's warning in mind: this cache is big, and it grows with every system.
Details and sources
Details
- The overlay is made with
qemu-img create -f qcow2 -b $(GARDEN_DL_DIR)/GROUP/SYSTEM.qcow2 -F qcow2, so it names its base by that path (rulegen.py:601-604). - rebase looks for the image in
.image-garden/when that exists, readsfull-backing-filenamefromqemu-img info, finds a file of that base name underGARDEN_DL_DIR, prints "image-garden: re-basing IMAGE on top of NEW" and runsqemu-img rebase -u -b NEW -F qcow2 IMAGE, which rewrites the path without opening the old base; it fails with "cannot find the backing image of …" when the download is not there (image-garden:943-980). - The action runs it after
image-garden make, inspread-subdir, asimage-garden rebase GARDEN_SYSTEM.$(uname -m).qcow2(action.yaml:411-442). - Known discrepancy: image-garden-action rebases even without a cache hit. The step runs whenever
cache-prepared-imagesis true; aTODOsays it should run only on a hit (action.yaml:436-442). On a miss make has just built the image against this runner's download, so the rebase changes nothing. - rebase takes the system name or the
.qcow2file name alike; the action passes the.qcow2form. - The README's warning: "This uses a lot of cache storage and scales poorly with the number of systems" (README.md:71-77).
07 Where to go from here
Next: the GitHub action, why pkgproxy, rebase, the overlay, and GitLab versus GitHub.
So the action keeps downloads by keeping pkgproxy's cache, in two parts, with its extended attributes, saved only when a job added to it. Prepared images are optional, and need the file's commit time and a rebase. Back in the action's own deck, you'll find the rest of the job around these caches. The pkgproxy deck explains why its cache is the one worth keeping. The rebase and overlay decks show what a backing file is. And the comparison deck shows how GitLab jobs keep the same downloads without any CI cache at all.
Details and sources
Read more
Basis of the statements on this page
Read in image-garden-action at 2891c7a ("code"): action.yaml, pkgproxy-cache and the README, and in image-garden at the commit in garden.commit for make's dependencies and rebase. How actions/cache matches restore keys is GitHub's behaviour, not in these trees ("derived"). Nothing was run for this page.
The GitHub caches
| Cache | Path | Key | Restored by | Saved when |
|---|---|---|---|---|
| pkgproxy images | $RUNNER_TEMP/image-garden-pkgproxy/images.tar (files/ of pkgproxy's cache) | PREFIX-images-ARCH-SYSTEM-RUN_ID-RUN_ATTEMPT | PREFIX-images-ARCH-SYSTEM- | snap mode, the policy allows, the fingerprint changed |
| pkgproxy packages | $RUNNER_TEMP/image-garden-pkgproxy/packages.tar (the rest) | PREFIX-packages-ARCH-SCOPE-RUN_ID-RUN_ATTEMPT | PREFIX-packages-ARCH-SCOPE- | as above, and save-package-cache is true |
| prepared images | SUBDIR/.image-garden, without *.log | image-garden-img-SYSTEM-HASH | the exact key | cache-prepared-images is true (actions/cache saves after the job) |
PREFIX is cache-key-prefix; ARCH is runner.arch; SYSTEM is garden-system; SCOPE is cache-scope, or garden-system when empty; HASH is hashFiles('SUBDIR/.image-garden.mk'). Sources: action.yaml:116-123, action.yaml:189-208, action.yaml:545-634.
The cache inputs
| Input | Default | Meaning |
|---|---|---|
save-cache | default-branch | which runs save the pkgproxy cache: default-branch, always or never; anything else fails the job |
save-package-cache | "true" | whether this job saves the packages part; "false" in all but one job of a shared scope |
cache-scope | "" (the garden system) | the scope of the packages part |
cache-key-prefix | image-garden-pkgproxy-v1 | change it to start from an empty cache |
cache-prepared-images | "false" | also cache .image-garden/ and rebase the image |
Source: action.yaml:18-32, action.yaml:96-115.
The cache outputs
| Output | Value |
|---|---|
pkgproxy-mode | snap, or external when a pkgproxy already listened on port 3142 |
images-cache-restored-key, package-cache-restored-key | the key restored from; empty on a cold start and in external mode |
images-cache-saved, package-cache-saved | true when the key exists after saving |
Source: action.yaml:57-72.
The order of the steps
| When | Step |
|---|---|
| start | restore prepared images (opt-in); restore .image-garden.mk's time; detect pkgproxy; restore the two archives |
| setup | install snapd and pkgproxy; stop it; unpack the archives with user.* attributes; fingerprint; start it |
| work | install image-garden; image-garden make; image-garden rebase (opt-in); spread |
| end, always | stop pkgproxy; fingerprint; archive the parts that changed; save; check the keys exist; start pkgproxy again |
| after the job | actions/cache saves the prepared images, when that input is on |
Matrix recipe
jobs: canary: steps: - uses: zyga/image-garden-action@v0 with: garden-system: ubuntu-cloud-24.04 cache-scope: shared test: needs: canary strategy: matrix: garden-system: [debian-cloud-13, fedora-cloud-44] steps: - uses: zyga/image-garden-action@v0 with: garden-system: ${{ matrix.garden-system }} cache-scope: shared save-package-cache: "false"
From README.md:199-227.
Roads from here
RUNTIME · make rules and the CLI
- image-garden rebase — what rebase does to a qcow2 file
- The overlay and the provisioning boot — the overlay and its backing file
BUILT ON IT · CI integrations
- image-garden-action for GitHub — the rest of the job around the caches
- Container on GitLab, snap on GitHub — how GitLab jobs keep the same downloads without a CI cache
PKGPROXY · caching for guests
- What pkgproxy gives a garden — why pkgproxy's cache is the one worth keeping