The overlay and the provisioning boot

64G qcow2, one boot for users for contributors

SYSTEM.qcow2 is a 64G copy-on-write overlay whose backing file is the system's link in the download cache. make boots it once with the cidata seed attached, writes the console to SYSTEM.qcow2.log, and is done when the guest powers off; the overlay is made again, from scratch, when the download, the seed or the firmware copies are newer.

On the map: runtime

▶ Watch this deck — 7 scenes, ~5 min

Worth seeing first

01 A disk of its own, on a shared image

THE OVERLAY noble-server-cloudimg-amd64.img the download, never written to ubuntu-cloud-24.04.x86_64.qcow2 backing file the guest sees one whole disk every write lands in the overlay; unchanged blocks are read from the download ubuntu-cloud-24.04@web.x86_64.qcow2 ubuntu-cloud-24.04@db.x86_64.qcow2 named instances: more overlays, one download
The guest sees one whole disk, ubuntu-cloud-24.04.x86_64.qcow2, an overlay whose backing file is the unchanged download noble-server-cloudimg-amd64.img; writes land in the overlay and unchanged blocks are read from the download; overlays of named instances share the same download

ubuntu-cloud-24.04.x86_64.qcow2 is a copy-on-write overlay: writes go to it, unchanged blocks are read from the download, which stays as it was.

The image you downloaded is a few hundred megabytes, and you don't want to fetch it again each time a test machine changes something. So image-garden doesn't boot it directly. It makes an overlay: a qcow2 file that starts empty and names the download as its backing file. The guest sees one whole disk. Reads of anything unchanged go to the download, and every write lands in the overlay. One download can carry many overlays, one for each system and named instance built on it. This deck looks at that overlay, the one boot that prepares it, the log it leaves, and when it's made again.

Details and sources

Details

  • X.qcow2 is created with qemu-img create -f qcow2 -b $(GARDEN_DL_DIR)/GROUP/X.qcow2 -F qcow2 X.qcow2 $(QEMU_IMG_SIZE) (rulegen.py:601-604); QEMU_IMG_SIZE = 64G (010-qemu.mk:8).
  • The README: "QEMU disk image with internal reference to base image in cache directory. Both files required at runtime." (README.md:824).
  • A named instance's overlay is created on the base system's link in the cache (020-instances.mk:94-98). The instance names on the slide are examples.
  • That qcow2 reads unallocated clusters from the backing file and allocates written clusters in the overlay is how QEMU's qcow2 format works (derived).

02 One command makes the overlay

QEMU-IMG CREATE $ qemu-img create -f qcow2 -b ~/.cache/image-garden/dl/ubuntu/ubuntu-cloud-24.04.x86_64.qcow2 -F qcow2 ubuntu-cloud-24.04.x86_64.qcow2 64G -f qcow2 the overlay's format -b the backing file: the system's link in the cache, by its full path -F qcow2 the backing file's format 64G the size the guest sees $ qemu-img info ubuntu-cloud-24.04.x86_64.qcow2 virtual size: 64 GiB (68719476736 bytes) disk size: 196 KiB grows as the guest writes cloud images usually grow their root file system to fill the disk on first boot
qemu-img create -f qcow2 -b with the full path of the system's link in the cache, -F qcow2, the overlay's name and 64G; qemu-img info then shows a virtual size of 64 GiB and a disk size of 196 KiB; the guest's root file system usually grows to fill the disk on first boot

qemu-img create -f qcow2 -b ~/.cache/image-garden/dl/ubuntu/ubuntu-cloud-24.04.x86_64.qcow2 -F qcow2 ubuntu-cloud-24.04.x86_64.qcow2 64G: 64 GiB virtual, about 200 KiB on disk when made.

The overlay comes from one qemu-img command. It names the format, qcow2, then the backing file. That's the link in the cache named after the system, not the distribution's own file name, and it's written with its full path. The backing file is qcow2 too, and last comes the size, 64 gigabytes. That's the size the guest sees, not what it takes on your disk: the new file is about two hundred kilobytes, and it grows only as the guest writes. Cloud images usually grow their root file system to fill the disk on first boot, so your machine starts with plenty of room.

Details and sources

Details

  • The recipe line is $(QEMU_IMG) create -f qcow2 -b $< -F qcow2 $@ $(QEMU_IMG_SIZE), where $< is $(GARDEN_DL_DIR)/GROUP/X.qcow2, the link, and QEMU_IMG is qemu-img from PATH, required when make starts (rulegen.py:601-604, 010-qemu.mk:3).
  • QEMU_IMG_SIZE = 64G (010-qemu.mk:8). It is a make variable, so image-garden make QEMU_IMG_SIZE=128G X.run would override it (derived from GNU make's rules for command-line variables; not documented).
  • GARDEN_DL_DIR is an absolute path by default, so the overlay records the absolute path of the link; image-garden make -n shows it in full.
  • The sizes on the slide come from running qemu-img create with these options on a small backing file and qemu-img info on the result: 64 GiB virtual, 196 KiB on disk.
  • That Ubuntu's cloud images grow their root partition and file system on first boot is cloud-init's growpart and resizefs defaults (derived); image-garden's user-data does not configure it.

03 What the overlay points at

TWO HOPS the overlay the system's link the downloaded image ubuntu-cloud-24.04.x86_64.qcow2 dl/ubuntu/…x86_64.qcow2 noble-server-cloudimg-amd64.img Ubuntu Core converted once, locked ubuntu-core-24-amd64.img.xz backing file: /home/you/.cache/image-garden/dl/ubuntu/ubuntu-cloud-24.04.x86_64.qcow2 a moved cache: the base is not found any more $ image-garden rebase ubuntu-cloud-24.04.x86_64.qcow2 leave the download as it is: every overlay on it depends on its exact contents
The overlay ubuntu-cloud-24.04.x86_64.qcow2 points at the system's link in the cache, which points at noble-server-cloudimg-amd64.img; Ubuntu Core's .img.xz is converted once, under a lock; the overlay records the link's full path, so a moved cache needs image-garden rebase, and the download must stay as it is

X.qcow2 → DL/ubuntu/X.qcow2 (a link) → noble-server-cloudimg-amd64.img. A moved cache needs image-garden rebase; a changed download breaks every overlay on it.

Follow the backing file and you take two hops. The overlay points at the system's link in the cache, and the link points at the downloaded image. Ubuntu Core's download is compressed, so make first converts it to qcow2 in the cache, once, under a lock, and links the system to that. The overlay records the link's full path. That's why the cache has to stay where it was: move it, as a CI cache restored on another machine might, and the overlay can't find its base. image-garden rebase points it at the new place. And leave the download as it is: every overlay on it depends on its exact contents.

Details and sources

Details

  • The overlay's backing file is $(GARDEN_DL_DIR)/GROUP/X.qcow2, the link made by the download rule with GARDEN_SYMLINK to the downloaded file (rulegen.py:494-514, rulegen.py:601-604).
  • Ubuntu Core's ubuntu-core-R-ARCH.img.xz is converted by %.qcow2: %.img.xz inside GARDEN_LOCKED_RECIPE (010-qemu.mk:304-347), and the system links point at the converted file (050-ubuntu-core.mk:273-289). A second make that waited prints "… was made by another image-garden make". See Ubuntu Core.
  • image-garden rebase, given the system name or the .qcow2 file, reads full-backing-filename from qemu-img info --output=json, finds a file of the same name under $GARDEN_DL_DIR and runs qemu-img rebase -u -b NEW -F qcow2, which rewrites the path without opening the old base; it fails with "cannot find the backing image" when there is none (image-garden:943-980).
  • /home/you on the slide stands for your home directory.
  • A changed backing file changes what every overlay on it reads wherever the overlay holds no data of its own, so the guest's file system no longer matches what it wrote (derived from how qcow2 overlays work).

04 The one boot that prepares it

THE PROVISIONING BOOT set -o pipefail && exec qemu-system-x86_64 -cpu max -machine q35,accel=… -m 2048 -smp … -nographic -drive if=pflash … the launcher's options … -drive file=ubuntu-cloud-24.04.x86_64.seed.iso,…,readonly=true,media=cdrom -device virtio-blk,drive=drive1 2>&1 | tee ubuntu-cloud-24.04.x86_64.qcow2.log the serial console tee your terminal ubuntu-cloud-24.04.x86_64.qcow2.log pipefail: when QEMU fails, the step fails, whatever tee says $ QEMU_MEM_OPTION="-m 4096" image-garden make ubuntu-cloud-24.04.x86_64.qcow2 done when cloud-init powers the guest off and QEMU exits
The provisioning command: set -o pipefail, the launcher's QEMU command with -nographic, plus the seed ISO as a read-only CD-ROM, piped through tee into ubuntu-cloud-24.04.x86_64.qcow2.log; the console goes to your terminal and to the log; QEMU option variables such as QEMU_MEM_OPTION apply; the step ends when the guest powers off

The provisioning boot is the launcher's QEMU command plus -drive file=X.seed.iso,…,readonly=true,media=cdrom, piped through tee X.qcow2.log under set -o pipefail.

Then make boots the overlay, once. The command is the same QEMU command line the launcher uses, with one addition: the seed ISO, attached read-only as a CD-ROM, so cloud-init finds its user-data. There's no window. The guest's serial console comes out on your terminal, and make passes it through tee into the overlay's log. The line runs with pipefail, so a QEMU that fails fails the step. Your QEMU option variables, such as the memory size, apply to this boot too. The step ends when cloud-init powers the guest off and QEMU exits.

Details and sources

Details

  • The recipe: set -o pipefail && $(call QEMU_SYSTEM_X86_64_EFI_CMDLINE,X.qcow2,X.efi-code.qcow2,X.efi-vars.qcow2) -drive file=X.seed.iso,format=raw,id=drive1,if=none,readonly=true,media=cdrom -device virtio-blk,drive=drive1 2>&1 | tee X.qcow2.log, with SHELL=bash for the target (rulegen.py:601-609).
  • Every provisioning boot runs this way, under bash with set -o pipefail: the rules rulegen.py writes, named instances (020-instances.mk:94-103) and Ubuntu Core, with and without a TPM (050-ubuntu-core.mk:37-53). A QEMU that fails to start or exits with an error fails the step, and .DELETE_ON_ERROR removes the half-made overlay.
  • QEMU_SYSTEM_X86_64_EFI_CMDLINE reads ${QEMU_MEM_OPTION=…}, ${QEMU_SMP_OPTION=…}, ${QEMU_DISPLAY_OPTION=…} and the other option variables from the environment, the same names the launcher uses (010-qemu-x86_64.mk:13-26). The default display is -nographic and the default memory -m 2048 (010-qemu.mk:9-13). The command on the slide is abridged; image-garden make -n prints it in full.
  • The seed is attached as a virtio block device with CD-ROM media; cloud-init's NoCloud source finds it by its cidata label (010-qemu.mk:263-278).
  • The guest powers off because the user-data sets power_state (010-qemu.mk:190-195); see provisioning runs once. The +tpm systems also start swtpm for this boot (050-ubuntu-core.mk:70-106).
  • Setting QEMU_MEM_OPTION in the environment of image-garden make is an example; the variables are listed in the .run script.

05 The console log of that boot

THE LOG ubuntu-cloud-24.04.x86_64.qcow2 ubuntu-cloud-24.04.x86_64.qcow2.log … the firmware … … the kernel boots … … cloud-init, stage by stage … Powering off everything the console printed ubuntu-cloud-24.04.x86_64.qcow2 provisioning failed: the disk is deleted, the log stays $ grep -n -i -e error -e warn ubuntu-cloud-24.04.x86_64.qcow2.log | head the next provisioning boot writes it anew: copy it first to compare two runs $ image-garden make clean removes it, with every other log here
Next to ubuntu-cloud-24.04.x86_64.qcow2 sits ubuntu-cloud-24.04.x86_64.qcow2.log with the firmware, kernel and cloud-init output and the final Powering off; when provisioning fails the disk is deleted and the log stays to be searched; the next provisioning boot overwrites it, and image-garden make clean removes every log

ubuntu-cloud-24.04.x86_64.qcow2.log keeps the console of the provisioning boot. It survives a failed boot, is overwritten by the next one, and is removed by make clean.

That log is the only record of provisioning, so it's worth knowing. Next to the disk, the qcow2.log file holds everything the console printed: the firmware, the kernel, each cloud-init stage, and the power-off at the end. When provisioning fails, make deletes the half-made disk, but the log stays, so search it for the first error or a cloud-init warning. Each provisioning boot writes the log anew, so copy it first if you want to compare two runs. And image-garden make clean removes it, along with every other log in the directory.

Details and sources

Details

  • The provisioning boot ends with 2>&1 | tee X.qcow2.log; tee without -a truncates the file, so each boot writes it anew (rulegen.py:609-609).
  • The log is not a make target, so .DELETE_ON_ERROR leaves it when it deletes the failed X.qcow2 (000-garden.mk:4-7).
  • clean:: runs rm -f *.log in the directory make runs in (000-garden.mk:48-50); that also removes the logs allocate writes.
  • "Powering off" is the message of the user-data's power_state (010-qemu.mk:190-195). The other lines on the slide stand for output that depends on the image (derived). The grep command is an example.
  • The README calls it "Serial console output collected during first boot and cloud-init execution" (README.md:825).

06 When the disk is made again

MADE AGAIN ubuntu-cloud-24.04.x86_64.qcow2 missing, or older than an input a new seed: an upgrade, or your .image-garden.mk new firmware copies a new download: an upgrade names a newer image the old overlay: gone an empty one, provisioned from scratch changes made by booting .run without -snapshot go with it # what a machine needs: in .image-garden.mk, in its user-data $ image-garden allocate ubuntu-cloud-24.04.x86_64 # boots with -snapshot
ubuntu-cloud-24.04.x86_64.qcow2 is made again when it is missing or older than one of its inputs: a new seed after an upgrade or an edit of .image-garden.mk, new firmware copies, or a new download; the old overlay is replaced by an empty one and provisioned again, so changes made by booting .run without -snapshot are lost; put them in .image-garden.mk and use allocate

X.qcow2 is remade when the download link, the seed or the firmware copies are newer. The new overlay starts empty: changes made by booting .run without -snapshot are lost.

make prepares the disk again when it's missing, or when something it was made from is newer. That can be a new seed, because image-garden was upgraded or you edited .image-garden.mk. It can be new firmware copies. Or it can be a new download, when an upgrade names a newer image. Then the old overlay is replaced by an empty one and provisioned from scratch. Anything you changed by booting the .run script without -snapshot goes with it. So put what a machine needs into .image-garden.mk rather than into the disk by hand, and let allocate boot it with -snapshot.

Details and sources

Details

  • X.qcow2 depends on $(GARDEN_DL_DIR)/GROUP/X.qcow2, X.seed.iso, X.efi-code.qcow2 and X.efi-vars.qcow2 (rulegen.py:601-602). The seed depends on the user-data and meta-data (010-qemu.mk:268-268), and they on the makefiles and .image-garden.mk (rulegen.py:460-471); the firmware copies on the host firmware (010-qemu-x86_64.mk:316-329).
  • make compares the link's time by following it to the image (derived; checked with GNU make 4.4.1). A refreshed link to the same image does not make the overlay older; a newly downloaded image does. The image is downloaded again only when its file is missing or fails its check, for example when an upgrade changes the URL's file name.
  • qemu-img create writes a new, empty overlay over the old file.
  • .run alone keeps changes in the overlay; allocate adds -snapshot unless QEMU_SNAPSHOT_OPTION says otherwise (image-garden:1153-1154).
  • Run in a scratch tree with make -n: deleting the seed ISO gives the seed, the overlay and the provisioning boot.

07 Where to go from here

ROADS The overlay Provisioning runs once what cloud-init does The target chain what it is made from rebase when the cache moves @instances overlays on one download The .run script boot the result
Five roads from the overlay: provisioning runs once, the target chain, rebase, named instances and the .run script

Next: provisioning runs once, the target chain, rebase, named instances and the .run script.

So the disk is a thin overlay on a shared download, prepared by one logged boot, and made again, from scratch, only when what it's made from changes. First boot explains what cloud-init does in that boot and how it stops itself. The target chain shows the files the overlay is made from. Rebase is the fix when the cache moves. Named instances put several overlays on one download, each with its own provisioning. And the .run script is how you boot the result.

Details and sources

Read more

Basis of the statements on this page

The rules were read in the image-garden source at the commit in garden.commit ("code"). The sizes were taken by running qemu-img create and qemu-img info with the same options on a small backing file, and the rebuild cases by running image-garden make -n on a scratch project ("run"). How qcow2 overlays and GNU make behave in general is marked "derived".

The two commands of step nine

For ubuntu-cloud-24.04.x86_64, with DL the download cache:

qemu-img create -f qcow2 -b DL/ubuntu/ubuntu-cloud-24.04.x86_64.qcow2 -F qcow2 ubuntu-cloud-24.04.x86_64.qcow2 64G

set -o pipefail && exec qemu-system-x86_64 -cpu max -machine q35,accel=… \
    -m 2048 -smp … -nographic \
    -drive if=pflash,format=qcow2,unit=0,file=ubuntu-cloud-24.04.x86_64.efi-code.qcow2,readonly=on \
    -drive if=pflash,format=qcow2,unit=1,file=ubuntu-cloud-24.04.x86_64.efi-vars.qcow2 \
    -drive file=ubuntu-cloud-24.04.x86_64.qcow2,if=none,format=qcow2,id=drive0,media=disk,cache=writeback,discard=unmap \
    -device virtio-blk,drive=drive0,bootindex=0 \
    -device virtio-rng-pci -netdev user,id=netdev0 … \
    -drive file=ubuntu-cloud-24.04.x86_64.seed.iso,format=raw,id=drive1,if=none,readonly=true,media=cdrom \
    -device virtio-blk,drive=drive1 \
    2>&1 | tee ubuntu-cloud-24.04.x86_64.qcow2.log

Abridged from image-garden make -n; the accelerator is kvm:tcg when QEMU knows KVM, else tcg, and the SMP default depends on the host (rulegen.py:601-609, 010-qemu-x86_64.mk:13-26).

Variables

VariableDefaultKindEffect on step nine
QEMU_IMG_SIZE64Gmake variablethe virtual size of the overlay
QEMU_IMGqemu-img on PATHmake variablethe program that creates the overlay
QEMU_MEM_OPTION-m 2048environmentmemory of the provisioning boot
QEMU_SMP_OPTION-smp NenvironmentCPUs of the provisioning boot
QEMU_DISPLAY_OPTION-nographicenvironmentwhere the console goes

The environment variables are the launcher's; see the .run script for the rest.

What makes the overlay again

InputNewer when
the cache link, followed to the imagethe image is downloaded again (a new file name after an upgrade, or a missing or failed file)
X.seed.isothe user-data or meta-data are rewritten: an upgrade of image-garden, an edit of .image-garden.mk
X.efi-code.qcow2, X.efi-vars.qcow2the firmware copies are made again, for example after make clean or newer host firmware
X.qcow2 itselfit is missing, for example after a failed provisioning boot

The launcher is not rewritten by a new overlay; see the target chain.

Files

FileWhat it is
X.qcow2the overlay, about 200 KiB when created, then as large as what the guest wrote
X.qcow2.logthe console of the last provisioning boot
DL/GROUP/X.qcow2the link the overlay names as its backing file
DL/GROUP/<download>the image, read-only for every overlay on it

Roads from here

RUNTIME · make rules and the CLI

FIRST BOOT · cloud-init changes