How the action caches

pkgproxy, prepared images for CI operators

image-garden-action keeps pkgproxy's cache as two GitHub caches, images per system and packages per cache-scope, archived with pkgproxy's extended attributes and saved only when a job added to them. Opt-in, it also caches the prepared image, restoring the time of .image-garden.mk and rebasing the image on this runner's download.

On the map: built on it

▶ Watch this deck — 7 scenes, ~5 min

Worth seeing first

01 What a runner forgets

BETWEEN JOBS a fresh runner, every job the image-garden snap the cloud image the guest's packages and snaps the first boot: minutes the GitHub cache pkgproxy's cache every download, by default the prepared image only if you ask this deck: pkgproxy's two caches, then the prepared image and rebase
Each GitHub job gets a fresh runner that would download the image-garden snap, the cloud image and the guest's packages and snaps and spend minutes on the first boot; the action keeps pkgproxy's cache, which holds every download, in the GitHub cache by default, and the prepared image only when asked

A runner starts empty. The action keeps pkgproxy's cache, which holds every download, in the GitHub cache by default; the prepared image only with cache-prepared-images.

A GitHub runner is a fresh machine for every job, so everything the last job fetched is gone. Without help, each job would download the image-garden snap, the cloud image, and every package and snap the guest installs, and then spend minutes on the image's first boot. The action keeps two kinds of things between jobs. Downloads, by default: they all go through pkgproxy, so keeping pkgproxy's cache is enough. And, only if you ask, the prepared image itself. This deck takes them in turn, from where pkgproxy keeps its files to the rebase that makes a cached image usable on a new runner.

Details and sources

Details

  • "pkgproxy caches everything this action downloads: snaps and their components on the host, pristine images, and apt, snap and go traffic of the guests" (action.yaml:173-178); "The action uses pkgproxy as its only cache for downloads" (README.md:152-158).
  • This holds when the action installs the pkgproxy snap ("snap" mode). With a pkgproxy that already listened on port 3142 ("external" mode, such as setup-pkgproxy's), the action neither restores nor saves a cache: it belongs to that pkgproxy (README.md:229-239).
  • "Images prepared from .image-garden.mk are not part of either: see cache-prepared-images" (README.md:173-174); that input defaults to "false" (action.yaml:18-20).

02 pkgproxy's cache, as two GitHub caches

TWO CACHES /var/snap/pkgproxy/common/cache/ files/ cloud images, by checksum images …-images-X64-debian-cloud-13-… per runner architecture and garden system everything else snaps, apt, Go packages …-packages-X64-SCOPE-… SCOPE: cache-scope, or else the garden system canary: garden-system: ubuntu-cloud-24.04 cache-scope: shared matrix: garden-system: [debian-cloud-13, fedora-cloud-44] cache-scope: shared save-package-cache: "false" each job: the shared packages, plus its own image
pkgproxy's cache directory holds files/, the cloud images by checksum, and everything else; the action archives files/ as the images cache keyed per runner architecture and garden system, and the rest as the packages cache keyed per cache-scope, which defaults to the system; in a matrix every job shares one scope, one job saves it and the others set save-package-cache to false

files/, the pristine images, is one GitHub cache per runner.arch and garden-system; everything else is another, per cache-scope. A matrix shares one scope and lets one job save it.

pkgproxy keeps its cache in one directory, with the cloud images under files, stored by their checksum. The action splits it in two. The images part is that files directory, saved per runner architecture and garden system, since each system downloads a different image. The packages part is everything else, the snaps, apt and Go downloads, which most systems share. Its key uses cache-scope, which defaults to the system. In a matrix, give every job one scope, let one job save it, and set save-package-cache to false in the others. Then each job restores the shared packages, plus its own image.

Details and sources

Details

  • pkgproxy-cache splits /var/snap/pkgproxy/common/cache: "images: the pristine images that image-garden downloads, which pkgproxy keeps by checksum under files/, cached per garden system; packages: everything else: snaps, apt, docker, lxd and go, which jobs of a matrix can share" (pkgproxy-cache:5-12, pkgproxy-cache:37).
  • Keys: PREFIX-images-RUNNER_ARCH-GARDEN_SYSTEM-RUN_ID-RUN_ATTEMPT and PREFIX-packages-RUNNER_ARCH-SCOPE-RUN_ID-RUN_ATTEMPT, where SCOPE is cache-scope or else garden-system; each is restored by the key without the run's part (action.yaml:189-208). PREFIX is cache-key-prefix, image-garden-pkgproxy-v1 by default; change it to start empty, as pkgproxy never deletes on its own (README.md:105-110).
  • The matrix recipe: every job cache-scope: shared, one canary job saves, the others save-package-cache: "false"; "Every job then restores the shared packages and its own images, and saves its own images when they change" (README.md:199-227).
  • X64 on the slide is GitHub's runner.arch for an x86-64 runner (GitHub's naming, not from these trees); shared is the README's example scope.

03 Restored before pkgproxy starts

RESTORE install pkgproxy snap stop pkgproxy tar --xattrs --xattrs-include='user.*' -xf images.tar -C …/common/cache tar --xattrs --xattrs-include='user.*' -xf packages.tar -C …/common/cache a cached file + what pkgproxy knows kept in extended attributes actions/cache's own tar drops them: pkgproxy downloads again so the action writes its own tar files; GitHub's cache stores them as they are pkgproxy-cache fingerprint images snap start pkgproxy
The action installs the pkgproxy snap and stops it, unpacks images.tar and packages.tar into its cache directory with tar keeping the user extended attributes, which actions/cache's own tar would drop and without which pkgproxy downloads again, records a fingerprint of each part, and starts pkgproxy

The pkgproxy snap is installed and stopped, both archives are unpacked with --xattrs --xattrs-include='user.*', each part is fingerprinted, and pkgproxy starts. actions/cache alone would drop the attributes.

At the start of a job, the action installs the pkgproxy snap and stops it straight away. While it's stopped, it unpacks both archives into pkgproxy's cache directory. Those aren't the plain archives GitHub's cache would make. pkgproxy keeps what it knows of each file in extended attributes, and the tar that GitHub's cache uses drops them. A file that lost them gets downloaded again. So the action writes its own tar files, with the user attributes kept, and GitHub's cache stores them as they are. Then it records a fingerprint of each part, and starts pkgproxy.

Details and sources

Details

  • In snap mode the action installs pkgproxy, snap stop pkgproxy, gives it its settings in one snap set, then for images and packages runs sudo tar --xattrs --xattrs-include='user.*' -xf ARCHIVE -C /var/snap/pkgproxy/common/cache, records pkgproxy-cache fingerprint of each part and the usage table, and starts pkgproxy (action.yaml:231-238, action.yaml:304-322).
  • "Write that part to ARCHIVE, a tar file that keeps the extended attributes in which pkgproxy keeps what it knows of each file. The tar of actions/cache drops them, and pkgproxy downloads again a file that lost them." (pkgproxy-cache:18-22); the archive commands (pkgproxy-cache:74-86).
  • The archives are restored with actions/cache/restore into $RUNNER_TEMP/image-garden-pkgproxy/ (action.yaml:193-208).
  • Why stopped: pkgproxy's configure hook leaves a stopped daemon stopped, so it starts once with everything; one snap set per setting went over systemd's start limit (action.yaml:235-237, action.yaml:294-303).

Read more

04 Saved only when it grew

SAVE save-cache: default-branch failed jobs on it save too snap stop pkgproxy consistent archives find … -printf '%P %s\n' | sort | sha256sum (not stats.json) same as at the start Not saving the pkgproxy cache of images: nothing was added to it changed: archive and save …-images-X64-debian-cloud-13-RUN-ATTEMPT the next job restores by prefix: …-images-X64-debian-cloud-13-
At the end of a job: the save-cache policy decides, by default only the default branch saves and failed jobs too; pkgproxy is stopped; each part is fingerprinted as the sorted paths and sizes, hashed, without stats.json; an unchanged fingerprint means nothing is saved; a changed part is saved under a key ending in the run ID and attempt, and the next job restores the newest by prefix

save-cache decides who may save; pkgproxy is stopped; each part's fingerprint, the sorted paths and sizes hashed, is compared with the one from the start; only a changed part is saved, under a key ending in the run ID and attempt.

At the end, the action decides whether to save. The save-cache policy comes first: by default only the default branch saves, failed jobs included. Then it stops pkgproxy, so the archives are consistent, and fingerprints each part again: the sorted list of file paths and sizes, hashed, leaving out pkgproxy's statistics file. If a fingerprint matches the one from the start, the job added nothing, and that part isn't saved. Otherwise it's archived under a new key that ends in the run's number and attempt. The next job restores by the key's prefix, which finds the newest entry.

Details and sources

Details

  • The step runs always() in snap mode. save-cache: always saves; default-branch saves only when GITHUB_REF is the default branch, otherwise "only the default branch saves (save-cache: default-branch)"; never does not (action.yaml:545-601). "Failed and cancelled jobs save too, as whatever pkgproxy cached is valid."
  • It stops pkgproxy, with a trap that starts it again, and for each part: skips packages when save-package-cache is not true; compares pkgproxy-cache fingerprint with the one taken at the start; "Not saving the pkgproxy cache of images: nothing was added to it", or archives the part and marks it for saving (action.yaml:602-620).
  • The fingerprint is find … -type f -printf '%P %s\n' | sort | sha256sum, over files/ for images and over everything else except stats.json for packages (pkgproxy-cache:63-73). A file whose size changed counts; one rewritten at the same size does not.
  • The key ends in ${{ github.run_id }}-${{ github.run_attempt }}, and the restore step's restore-keys is the key without them (action.yaml:193-208, action.yaml:621-634). That a prefix match restores the most recent cache is actions/cache's behaviour (derived, not in these trees).
  • After saving, the action looks the keys up again and warns "The pkgproxy cache could not be saved" when they are missing; the outputs images-cache-saved and package-cache-saved report it (action.yaml:635-656).

Read more

05 Prepared images and file times

PREPARED IMAGES cache-prepared-images: true .image-garden/ !*.log image-garden-img-SYSTEM-hash(.image-garden.mk) SYSTEM.qcow2 .seed.iso .user-data .image-garden.mk make compares file times: a target older than what it depends on is rebuilt fresh checkout: .image-garden.mk is newer than the cached image make would rebuild it git restore-mtime .image-garden.mk the time of its last commit actions/checkout with fetch-depth: 0 otherwise: a warning, and a rebuild
With cache-prepared-images true the action caches .image-garden/ without logs, keyed by the garden system and a hash of .image-garden.mk; the prepared qcow2 depends through the seed and user-data on .image-garden.mk; a fresh checkout gives that file a new time, newer than the cached image, so make would rebuild; the action sets the time back to the file's last commit, which needs fetch-depth 0

cache-prepared-images: true caches .image-garden/ without logs, keyed by system and hashFiles('.image-garden.mk'). git restore-mtime gives .image-garden.mk its last commit's time, so make keeps the image; it needs fetch-depth: 0.

Then there's the prepared image, the result of that long first boot. With cache-prepared-images set to true, the action also caches your project's .image-garden directory, minus the logs, keyed by the system and a hash of your .image-garden.mk. But make decides by file times, and the image depends, through the user-data and the seed, on your .image-garden.mk. A fresh checkout gives that file the time of the checkout, newer than any cached image, so make would rebuild it. The action sets the file's time back to its last commit, from the git history. That's why the workflow checks out with fetch-depth: 0.

Details and sources

Details

  • actions/cache of SUBDIR/.image-garden and !SUBDIR/.image-garden/*.log, key image-garden-img-GARDEN_SYSTEM-HASH, where HASH is hashFiles('SUBDIR/.image-garden.mk') (action.yaml:116-123). The key has no architecture and no image-garden version in it. The README: "This uses a lot of cache storage and scales poorly with the number of systems" (README.md:71-77).
  • When .image-garden.mk exists, the action installs git-restore-mtime and runs git restore-mtime .image-garden.mk, with git whatchanged aliased to git log --raw --no-merges for newer git; when the file is not found in the history it warns "Cannot restore the mtime of .image-garden.mk from git history; check out with fetch-depth: 0" (action.yaml:124-152). This runs whether or not prepared images are cached.
  • The dependency chain: SYSTEM.user-data depends on $(MAKEFILE_LIST) and .image-garden.mk (rulegen.py:467-471); %.seed.iso on the user-data and meta-data (010-qemu.mk:268); SYSTEM.qcow2 on the downloaded base image and the seed (rulegen.py:601-602).
  • The README's workflow checks out with fetch-depth: 0, "essential for git restore-mtime to work correctly" (README.md:405-432).
  • Derived, not run: SYSTEM.qcow2 also depends on the downloaded base image in the snap's download directory, outside .image-garden/. On a new runner that image is downloaded again, from pkgproxy, and gets the time of that download, newer than a restored prepared image; image-garden's downloads do not keep the server's file time. If that holds, make provisions the image again even on a cache hit.

06 Rebasing onto this runner's download

REBASE ubuntu-cloud-24.04.x86_64.qcow2 only the changes backing: …/cache/dl/ubuntu/ubuntu-cloud-24.04.x86_64.qcow2 a new runner: that cloud image is a fresh download $ image-garden rebase ubuntu-cloud-24.04.x86_64.qcow2 image-garden: re-basing …qcow2 on top of …/dl/ubuntu/… qemu-img rebase -u -b NEW -F qcow2 IMAGE runs whenever cache-prepared-images is true, hit or not the README: a lot of cache storage, scales poorly with the number of systems
A prepared ubuntu-cloud-24.04.x86_64.qcow2 holds only changes and records the path of its cloud image; on a new runner that image is a fresh download; after make the action runs image-garden rebase on the qcow2 file, which finds a file of the same name in this runner's download directory and runs qemu-img rebase -u; the step runs whenever prepared images are cached, hit or not; the README warns the cache is big and grows with every system

After make, image-garden rebase SYSTEM.$(uname -m).qcow2 finds the backing image by name in GARDEN_DL_DIR and runs qemu-img rebase. It runs whenever cache-prepared-images is true.

One more step goes with a cached image. A prepared qcow2 doesn't hold the whole system: it records the path of the cloud image it was made from, and stores only the changes. On a new runner, that cloud image is a fresh download. So after make, the action runs image-garden rebase on the qcow2 file. rebase reads the old backing path, finds a file of the same name in this runner's download directory, and points the image at it. The step runs whenever prepared images are cached, whether the cache was hit or not. And keep the README's warning in mind: this cache is big, and it grows with every system.

Details and sources

Details

  • The overlay is made with qemu-img create -f qcow2 -b $(GARDEN_DL_DIR)/GROUP/SYSTEM.qcow2 -F qcow2, so it names its base by that path (rulegen.py:601-604).
  • rebase looks for the image in .image-garden/ when that exists, reads full-backing-filename from qemu-img info, finds a file of that base name under GARDEN_DL_DIR, prints "image-garden: re-basing IMAGE on top of NEW" and runs qemu-img rebase -u -b NEW -F qcow2 IMAGE, which rewrites the path without opening the old base; it fails with "cannot find the backing image of …" when the download is not there (image-garden:943-980).
  • The action runs it after image-garden make, in spread-subdir, as image-garden rebase GARDEN_SYSTEM.$(uname -m).qcow2 (action.yaml:411-442).
  • Known discrepancy: image-garden-action rebases even without a cache hit. The step runs whenever cache-prepared-images is true; a TODO says it should run only on a hit (action.yaml:436-442). On a miss make has just built the image against this runner's download, so the rebase changes nothing.
  • rebase takes the system name or the .qcow2 file name alike; the action passes the .qcow2 form.
  • The README's warning: "This uses a lot of cache storage and scales poorly with the number of systems" (README.md:71-77).

07 Where to go from here

ROADS the action's caches GitHub action the rest of the job Why pkgproxy the cache worth keeping rebase a new backing file The overlay what a backing file is GitLab vs GitHub no CI cache at all
Five roads from the action's caches: image-garden-action, why pkgproxy, rebase, the overlay, and GitLab versus GitHub

Next: the GitHub action, why pkgproxy, rebase, the overlay, and GitLab versus GitHub.

So the action keeps downloads by keeping pkgproxy's cache, in two parts, with its extended attributes, saved only when a job added to it. Prepared images are optional, and need the file's commit time and a rebase. Back in the action's own deck, you'll find the rest of the job around these caches. The pkgproxy deck explains why its cache is the one worth keeping. The rebase and overlay decks show what a backing file is. And the comparison deck shows how GitLab jobs keep the same downloads without any CI cache at all.

Details and sources

Read more

Basis of the statements on this page

Read in image-garden-action at 2891c7a ("code"): action.yaml, pkgproxy-cache and the README, and in image-garden at the commit in garden.commit for make's dependencies and rebase. How actions/cache matches restore keys is GitHub's behaviour, not in these trees ("derived"). Nothing was run for this page.

The GitHub caches

CachePathKeyRestored bySaved when
pkgproxy images$RUNNER_TEMP/image-garden-pkgproxy/images.tar (files/ of pkgproxy's cache)PREFIX-images-ARCH-SYSTEM-RUN_ID-RUN_ATTEMPTPREFIX-images-ARCH-SYSTEM-snap mode, the policy allows, the fingerprint changed
pkgproxy packages$RUNNER_TEMP/image-garden-pkgproxy/packages.tar (the rest)PREFIX-packages-ARCH-SCOPE-RUN_ID-RUN_ATTEMPTPREFIX-packages-ARCH-SCOPE-as above, and save-package-cache is true
prepared imagesSUBDIR/.image-garden, without *.logimage-garden-img-SYSTEM-HASHthe exact keycache-prepared-images is true (actions/cache saves after the job)

PREFIX is cache-key-prefix; ARCH is runner.arch; SYSTEM is garden-system; SCOPE is cache-scope, or garden-system when empty; HASH is hashFiles('SUBDIR/.image-garden.mk'). Sources: action.yaml:116-123, action.yaml:189-208, action.yaml:545-634.

The cache inputs

InputDefaultMeaning
save-cachedefault-branchwhich runs save the pkgproxy cache: default-branch, always or never; anything else fails the job
save-package-cache"true"whether this job saves the packages part; "false" in all but one job of a shared scope
cache-scope"" (the garden system)the scope of the packages part
cache-key-prefiximage-garden-pkgproxy-v1change it to start from an empty cache
cache-prepared-images"false"also cache .image-garden/ and rebase the image

Source: action.yaml:18-32, action.yaml:96-115.

The cache outputs

OutputValue
pkgproxy-modesnap, or external when a pkgproxy already listened on port 3142
images-cache-restored-key, package-cache-restored-keythe key restored from; empty on a cold start and in external mode
images-cache-saved, package-cache-savedtrue when the key exists after saving

Source: action.yaml:57-72.

The order of the steps

WhenStep
startrestore prepared images (opt-in); restore .image-garden.mk's time; detect pkgproxy; restore the two archives
setupinstall snapd and pkgproxy; stop it; unpack the archives with user.* attributes; fingerprint; start it
workinstall image-garden; image-garden make; image-garden rebase (opt-in); spread
end, alwaysstop pkgproxy; fingerprint; archive the parts that changed; save; check the keys exist; start pkgproxy again
after the jobactions/cache saves the prepared images, when that input is on

Matrix recipe

jobs: canary: steps: - uses: zyga/image-garden-action@v0 with: garden-system: ubuntu-cloud-24.04 cache-scope: shared test: needs: canary strategy: matrix: garden-system: [debian-cloud-13, fedora-cloud-44] steps: - uses: zyga/image-garden-action@v0 with: garden-system: ${{ matrix.garden-system }} cache-scope: shared save-package-cache: "false"

From README.md:199-227.

Roads from here

RUNTIME · make rules and the CLI

BUILT ON IT · CI integrations

PKGPROXY · caching for guests