image-garden rebase

moved caches for CI operators

rebase takes a prepared image by system name or .qcow2 file, reads the backing file name it records, finds a file of that name under GARDEN_DL_DIR and runs qemu-img rebase -u onto it, so an image restored from a CI cache uses this machine's download even when the old one is gone.

On the map: runtime

▶ Watch this deck — 6 scenes, ~4 min

Worth seeing first

01 A path inside the image

A MOVED CACHE CI saves prepared images between jobs (example paths) machine A, where make ran ubuntu-cloud-24.04.x86_64.qcow2 /cache-a/dl/ubuntu/…x86_64.qcow2 machine B, a later job ubuntu-cloud-24.04.x86_64.qcow2 /cache-b/dl/ubuntu/…x86_64.qcow2 reads the rest backing: /cache-a/dl/… backing: /cache-a/dl/… /cache-a leads nowhere here image-garden rebase points it at this machine's download
On machine A the prepared ubuntu-cloud-24.04.x86_64.qcow2 holds only its changes and reads the rest from its base, recorded as a full path into /cache-a. Restored on machine B, whose cache is in /cache-b, that path leads nowhere. image-garden rebase points the image at this machine's download.

X.qcow2 records its base as a full path into GARDEN_DL_DIR. image-garden rebase points it at the base in this machine's download directory.

Say your CI saves the prepared images of a project between jobs, to skip the long first boot. The next job restores them, on a different machine, where the download cache lives somewhere else. That breaks something you can't see. A prepared qcow2 holds only the changes of its machine. For everything else, it reads the cloud image under it, and it finds that image by a full path, written into the file when make created it. On the new machine, that path leads nowhere. image-garden rebase is the command that points the image at the cache where it is now.

Details and sources

Details

  • The overlay is made with qemu-img create -f qcow2 -b $(GARDEN_DL_DIR)/GROUP/X.qcow2 -F qcow2, so the path it records is the full path of the system's link in the download directory (rulegen.py:601-604). Run: qemu-img info on such an overlay shows that path as "backing file".
  • The README: rebase "updates the backing file reference in the VM disk image to point to the current system's $GARDEN_DL_DIR", for a CI cache used on another system "where everything is compatible except the effective location of the download directory" (README.md:434-448).
  • GARDEN_DL_DIR is $GARDEN_CACHE_DIR/dl, and the cache directory differs between the snap, a native install and the container (image-garden:20-29).
  • /cache-a and /cache-b on the slides are example paths.

02 What rebase does

STEP BY STEP X.qcow2 in .image-garden/ if any $GARDEN_DL_DIR must exist qemu-img info --output=json full-backing-filename ubuntu-cloud-24.04.x86_64.qcow2 only the file name $ find "$GARDEN_DL_DIR" -name NAME -print -quit the first match image-garden: cannot find the backing image of X.qcow2, NAME, in DL_DIR image-garden: re-basing X.qcow2 on top of NEW exec qemu-img rebase -u -b $GARDEN_DL_DIR/ubuntu/ubuntu-cloud-24.04.x86_64.qcow2 -F qcow2 X.qcow2
rebase finds the image, in .image-garden/ when that exists, checks that GARDEN_DL_DIR exists, reads full-backing-filename with qemu-img info and keeps the file name, finds the first file of that name under GARDEN_DL_DIR, fails when there is none, and otherwise prints re-basing and runs qemu-img rebase -u -b NEW -F qcow2, which only rewrites the path.

rebase reads full-backing-filename from qemu-img info, finds that base name under GARDEN_DL_DIR and runs qemu-img rebase -u -b NEW -F qcow2 X.qcow2.

Here's what rebase does with the image you name. If your project has a .image-garden directory, it looks for the image there. It checks that the download directory exists. Then it asks qemu-img for the image's full backing path, and keeps only the last part, the file name. It searches the download directory for a file with that name, and takes the first one it finds. If there's none, it stops and says so. Otherwise it prints what it's about to do and hands over to qemu-img rebase, in its unsafe mode, which writes the new path into the image and changes nothing else.

Details and sources

Details

  • With an argument count other than one, rebase prints its usage and exits 0 (image-garden:943-950).
  • The image is $GARDEN_PROJECT_DIR/.image-garden/ARG when that directory exists, else ARG, with .qcow2 added when only that file exists; a missing file gives "image-garden: cannot find image at PATH" (image-garden:951-964).
  • A missing download directory gives "image-garden: cannot rebase without downloaded image directory: DIR" (image-garden:965-968).
  • The backing path is cut out of qemu-img info --output=json with grep, cut and sed, without jq; the new base is find "$GARDEN_DL_DIR" -name BASENAME -print -quit, the first match anywhere below the directory (image-garden:969-971).
  • No match gives "image-garden: cannot find the backing image of IMAGE, NAME, in DIR"; otherwise it prints "image-garden: re-basing IMAGE on top of NEW" and execs qemu-img rebase -u -b NEW -F qcow2 IMAGE (image-garden:972-980). Run: the usage, "cannot find image", "cannot rebase without downloaded image directory" and "re-basing" messages, with the script at garden.commit and a host qemu-img.
  • qemu-img is $QEMU_IMG when set, else qemu-img from PATH, for both the info and the rebase call (image-garden:970, image-garden:980), as make's rules use $(QEMU_IMG).
  • The name found is the system's link, such as ubuntu-cloud-24.04.x86_64.qcow2 in dl/ubuntu/; it exists once the system's download rule has run.

Read more

03 A system name or a file

EITHER NAME WORKS ubuntu-cloud-24.04.x86_64 usage text, man page, README ubuntu-cloud-24.04.x86_64.qcow2 image-garden's GitHub action PATH = $GARDEN_PROJECT_DIR/.image-garden/ARG when that directory exists if [ ! -f "$IMAGE_PATH" ] && [ -f "$IMAGE_PATH".qcow2 ]; then IMAGE_PATH="$IMAGE_PATH".qcow2 no such file: add .qcow2 both: .image-garden/ubuntu-cloud-24.04.x86_64.qcow2 $ image-garden rebase debian-cloud-13.x86_64 image-garden: cannot find image at …/.image-garden/debian-cloud-13.x86_64
rebase takes the system name, as the usage text, man page and README write it, or the file name with .qcow2, as image-garden's GitHub action passes it. It looks in .image-garden/ when the project has one, and when no file has exactly that name it adds .qcow2. Both forms find .image-garden/ubuntu-cloud-24.04.x86_64.qcow2. A system never prepared fails with cannot find image.

image-garden rebase ubuntu-cloud-24.04.x86_64 and image-garden rebase ubuntu-cloud-24.04.x86_64.qcow2 both re-base .image-garden/ubuntu-cloud-24.04.x86_64.qcow2.

You can name the image in two ways. The usage text, the manual page and the README give rebase a system name, like the other commands take. image-garden's GitHub action gives it the file name, with .qcow2 on the end. Both work. Inside a project with a .image-garden directory, rebase puts that directory in front of what you typed. If no file has exactly that name, it tries again with .qcow2 added. So the system name and the file name find the same prepared image. A system you never prepared still stops with cannot find image.

Details and sources

Details

  • The usage line says image-garden rebase SYSTEM.ARCH (image-garden:893, image-garden:946); so do the README (README.md:434-440) and the man page (image-garden.1:19). image-garden-action passes SYSTEM.$(uname -m).qcow2 (action.yaml:442).
  • The argument gets $GARDEN_PROJECT_DIR/.image-garden/ in front when that directory exists; when no file has that exact name but one with .qcow2 added does, rebase takes that one (image-garden:951-964). The "cannot find image at PATH" message shows the path without .qcow2.
  • Run: in a project with .image-garden/ubuntu-cloud-24.04.x86_64.qcow2, rebase ubuntu-cloud-24.04.x86_64 re-based it; rebase debian-cloud-13.x86_64 printed "image-garden: cannot find image at …/.image-garden/debian-cloud-13.x86_64" and exited 1. tests/check-rebase.sh runs both forms (tests/check-rebase.sh:21-39).
  • With .image-garden/ present, an argument that already starts with .image-garden/, or an absolute path, gets the directory put in front again and is not found (derived from image-garden:954-956).
  • Without .image-garden/, the argument is a path from the current directory, and the same .qcow2 rule applies (derived from image-garden:953-960).

04 Only the path changes

UNSAFE MODE, ON PURPOSE without -u, safe mode qemu-img rebase -b NEW compares old: /cache-a/dl/ubuntu/… new: /cache-b/dl/ubuntu/… Could not open backing file: Could not open '/cache-a/dl/ubuntu/…' with -u, what rebase runs $ qemu-img rebase -u -b /cache-b/dl/ubuntu/… -F qcow2 X.qcow2 writes only the path old: …/ubuntu-cloud-24.04.x86_64.qcow2 = new: …/ubuntu-cloud-24.04.x86_64.qcow2 Found by its name, the system's verified download: right while both caches hold the same build
Without -u, qemu-img rebase compares the old base with the new one and so opens both; with the old cache gone that would fail with Could not open backing file. rebase runs qemu-img rebase -u, which writes only the new path. It trusts that both bases hold the same data: the new base has the same file name, the system's verified download, so they match as long as both caches fetched the same build.

rebase runs qemu-img rebase -u, which rewrites the backing path without opening the old base, so it works when the old cache is gone.

Why the unsafe mode? Without it, qemu-img rebase compares the old base with the new one, so that the image's contents stay the same. That means it has to open the old base too. On a fresh runner the old cache is gone, the very case rebase is for, and that comparison would stop with could not open backing file. With -u, qemu-img opens nothing it doesn't need and writes only the new path. It trusts that both bases hold the same data, and rebase counts on the name for that: the new base has the same file name, the system's verified download, so the two match as long as both caches fetched the same build.

Details and sources

Details

  • rebase runs qemu-img rebase -u -b NEW -F qcow2 IMAGE; its comment: "Unsafe mode only rewrites the backing file name: the old backing file is usually gone, which is why the image needs rebasing, and the new one is the same download" (image-garden:976-980). The NEWS entry: rebase "works when the old download directory is gone" (NEWS:5-6).
  • Without -u, qemu-img compares the old and the new backing chains and writes the differences into the image, so it opens the old one (qemu-img's documented "safe mode"). A safe-mode rebase with the old directory moved away stopped with "Could not open backing file: Could not open '…/old/dl/ubuntu/ubuntu-cloud-24.04.x86_64.qcow2': No such file or directory" (run with a host qemu-img).
  • Run: with the old download directory removed, rebase re-based the image and qemu-img info showed the new backing file. tests/check-rebase.sh checks the same, for both argument forms (tests/check-rebase.sh:21-39).
  • -u does not compare contents. The new base is found by the file name of the old one, the system's link in dl/GROUP/ (image-garden:970-971); if that link points at a different build than the one the image was made from, the image would read changed data under its own changes (derived). After a restore, image-garden make runs before rebase and decides whether the prepared image is still current: see the caching deck.
  • /cache-a and /cache-b on the slide are example paths.

05 Where it goes in a CI job

IN A CI JOB restore .image-garden/ image-garden make rebase X.qcow2 boot, spread image-garden-action: the same order, rebase when cache-prepared-images is true GitHub runner, job 1 ~/snap/image-garden/common/cache/dl GitHub runner, job 2 ~/snap/image-garden/common/cache/dl same path, same home: rebase changes nothing Your own runners, /cache-a and /cache-b: here rebase matters The GitHub action's caches: their own deck
A CI job restores .image-garden/, runs image-garden make, then image-garden rebase X.qcow2, then boots with spread, as image-garden's GitHub action does. On GitHub every runner keeps the cache at the same path under its home, so rebase changes nothing; it matters when runners keep the cache in different places.

Restore .image-garden/, image-garden make, image-garden rebase X.qcow2, then boot. On GitHub's runners the path does not change.

In CI, the order matters. Restore the prepared images, run image-garden make, so the download is in the cache, then rebase, and only then boot. image-garden's GitHub action does exactly that, after make and before spread. On GitHub, though, the cache sits at the same path on every runner, inside the runner's home directory, so the path in a restored image is already right, and rebase changes nothing. It matters when your runners keep the cache in different places. The caching deck tells the action's side of the story.

Details and sources

Details

  • image-garden-action caches .image-garden/ without its logs when cache-prepared-images is true (action.yaml:116-123), runs image-garden make for the image, launcher, user-data, meta-data and seed (action.yaml:420-435), then image-garden rebase SYSTEM.$(uname -m).qcow2 (action.yaml:436-442), then spread.
  • The action makes ~/snap/image-garden/common/cache/dl, the snap's download directory (action.yaml:416). That every GitHub-hosted runner has the same home directory, so the same path, is derived.
  • Known discrepancy: image-garden-action rebases even without a cache hit. The step runs whenever cache-prepared-images is true; its TODO says it should run only on a hit (action.yaml:436-439).
  • Whether make keeps a restored image depends on file times; the action restores the time of .image-garden.mk for that (action.yaml:124-152). See the caching deck.

06 Where to go from here

ROADS rebase The overlay the image and its base The action's caches restore, then rebase Where files live the download directory
Three roads from image-garden rebase: the overlay and the provisioning boot, the GitHub action's caches, and where image-garden keeps its files

Next: the overlay, the GitHub action's caches, and where image-garden keeps its files.

That's rebase. It reads the base's name from the image, finds that name in your download directory, and has qemu-img write the new path, even when the old cache is long gone. Give it the system name or the file name. From here, the overlay deck shows how the image and its base are made, and the caching deck shows how image-garden's GitHub action restores prepared images and rebases them. The cache deck shows where the download directory is for each way of installing image-garden.

Details and sources

Read more

Basis of the statements on this page

Read in the image-garden source at the commit in garden.commit ("code"). Every message was run: the script at that commit and a host qemu-img on a scratch overlay of an empty qcow2, with the download directory removed or missing, and tests/check-rebase.sh passed. That GitHub runners share one home path, and when an unsafe rebase is correct, are derived.

Usage

image-garden rebase SYSTEM.ARCH
image-garden rebase SYSTEM.ARCH.qcow2

The argument is looked up in .image-garden/ when the project has that directory, otherwise from the current directory. When no file has exactly that name, rebase adds .qcow2 (image-garden:951-964). image-garden-action passes the .qcow2 form.

What it runs

${QEMU_IMG:-qemu-img} info --output=json IMAGE         # full-backing-filename
find "$GARDEN_DL_DIR" -name BASENAME -print -quit      # the first match
${QEMU_IMG:-qemu-img} rebase -u -b NEW -F qcow2 IMAGE  # rewrites the path only

-u does not open the old base, so rebase works after the old download directory is gone (image-garden:969-980). It does not compare the two bases either: the new one is trusted because it has the same file name, the system's link to its verified download.

Messages

MessageCause
Usage: image-garden rebase SYSTEM.ARCH (exit 0)not exactly one argument
image-garden: cannot find image at PATHneither PATH nor PATH.qcow2 exists, for example a system never prepared
image-garden: cannot rebase without downloaded image directory: DIRGARDEN_DL_DIR does not exist
image-garden: cannot find the backing image of IMAGE, NAME, in DIRno file named after the backing file under GARDEN_DL_DIR, or the image has no backing file
image-garden: re-basing IMAGE on top of NEWabout to run qemu-img rebase -u

The project's check

tests/check-rebase.sh makes an overlay on a base in one directory, moves the base to another and removes the first, then runs rebase with demo.x86_64 and with demo.x86_64.qcow2 and checks the new backing file (tests/check-rebase.sh:1-39). make check runs it (Makefile:108-109).

Roads from here

RUNTIME · make rules and the CLI

BUILT ON IT · CI integrations