The overlay and the provisioning boot
64G qcow2, one boot for users for contributors
SYSTEM.qcow2 is a 64G copy-on-write overlay whose backing file is the system's link in the download cache. make boots it once with the cidata seed attached, writes the console to SYSTEM.qcow2.log, and is done when the guest powers off; the overlay is made again, from scratch, when the download, the seed or the firmware copies are newer.
On the map: runtime
▶ Watch this deck — 7 scenes, ~5 min
Worth seeing first
- From a target name to a VM — where the overlay sits among the ten steps
01 A disk of its own, on a shared image
ubuntu-cloud-24.04.x86_64.qcow2 is a copy-on-write overlay: writes go to it, unchanged blocks are read from the download, which stays as it was.
The image you downloaded is a few hundred megabytes, and you don't want to fetch it again each time a test machine changes something. So image-garden doesn't boot it directly. It makes an overlay: a qcow2 file that starts empty and names the download as its backing file. The guest sees one whole disk. Reads of anything unchanged go to the download, and every write lands in the overlay. One download can carry many overlays, one for each system and named instance built on it. This deck looks at that overlay, the one boot that prepares it, the log it leaves, and when it's made again.
Details and sources
Details
X.qcow2is created withqemu-img create -f qcow2 -b $(GARDEN_DL_DIR)/GROUP/X.qcow2 -F qcow2 X.qcow2 $(QEMU_IMG_SIZE)(rulegen.py:601-604);QEMU_IMG_SIZE = 64G(010-qemu.mk:8).- The README: "QEMU disk image with internal reference to base image in cache directory. Both files required at runtime." (README.md:824).
- A named instance's overlay is created on the base system's link in the cache (020-instances.mk:94-98). The instance names on the slide are examples.
- That qcow2 reads unallocated clusters from the backing file and allocates written clusters in the overlay is how QEMU's qcow2 format works (derived).
Read more
02 One command makes the overlay
qemu-img create -f qcow2 -b ~/.cache/image-garden/dl/ubuntu/ubuntu-cloud-24.04.x86_64.qcow2 -F qcow2 ubuntu-cloud-24.04.x86_64.qcow2 64G: 64 GiB virtual, about 200 KiB on disk when made.
The overlay comes from one qemu-img command. It names the format, qcow2, then the backing file. That's the link in the cache named after the system, not the distribution's own file name, and it's written with its full path. The backing file is qcow2 too, and last comes the size, 64 gigabytes. That's the size the guest sees, not what it takes on your disk: the new file is about two hundred kilobytes, and it grows only as the guest writes. Cloud images usually grow their root file system to fill the disk on first boot, so your machine starts with plenty of room.
Details and sources
Details
- The recipe line is
$(QEMU_IMG) create -f qcow2 -b $< -F qcow2 $@ $(QEMU_IMG_SIZE), where$<is$(GARDEN_DL_DIR)/GROUP/X.qcow2, the link, andQEMU_IMGisqemu-imgfromPATH, required when make starts (rulegen.py:601-604, 010-qemu.mk:3). QEMU_IMG_SIZE = 64G(010-qemu.mk:8). It is a make variable, soimage-garden make QEMU_IMG_SIZE=128G X.runwould override it (derived from GNU make's rules for command-line variables; not documented).GARDEN_DL_DIRis an absolute path by default, so the overlay records the absolute path of the link;image-garden make -nshows it in full.- The sizes on the slide come from running
qemu-img createwith these options on a small backing file andqemu-img infoon the result: 64 GiB virtual, 196 KiB on disk. - That Ubuntu's cloud images grow their root partition and file system on first boot is cloud-init's
growpartandresizefsdefaults (derived); image-garden's user-data does not configure it.
Read more
03 What the overlay points at
X.qcow2 → DL/ubuntu/X.qcow2 (a link) → noble-server-cloudimg-amd64.img. A moved cache needs image-garden rebase; a changed download breaks every overlay on it.
Follow the backing file and you take two hops. The overlay points at the system's link in the cache, and the link points at the downloaded image. Ubuntu Core's download is compressed, so make first converts it to qcow2 in the cache, once, under a lock, and links the system to that. The overlay records the link's full path. That's why the cache has to stay where it was: move it, as a CI cache restored on another machine might, and the overlay can't find its base. image-garden rebase points it at the new place. And leave the download as it is: every overlay on it depends on its exact contents.
Details and sources
Details
- The overlay's backing file is
$(GARDEN_DL_DIR)/GROUP/X.qcow2, the link made by the download rule withGARDEN_SYMLINKto the downloaded file (rulegen.py:494-514, rulegen.py:601-604). - Ubuntu Core's
ubuntu-core-R-ARCH.img.xzis converted by%.qcow2: %.img.xzinsideGARDEN_LOCKED_RECIPE(010-qemu.mk:304-347), and the system links point at the converted file (050-ubuntu-core.mk:273-289). A second make that waited prints "… was made by another image-garden make". See Ubuntu Core. image-garden rebase, given the system name or the.qcow2file, readsfull-backing-filenamefromqemu-img info --output=json, finds a file of the same name under$GARDEN_DL_DIRand runsqemu-img rebase -u -b NEW -F qcow2, which rewrites the path without opening the old base; it fails with "cannot find the backing image" when there is none (image-garden:943-980)./home/youon the slide stands for your home directory.- A changed backing file changes what every overlay on it reads wherever the overlay holds no data of its own, so the guest's file system no longer matches what it wrote (derived from how qcow2 overlays work).
Read more
04 The one boot that prepares it
The provisioning boot is the launcher's QEMU command plus -drive file=X.seed.iso,…,readonly=true,media=cdrom, piped through tee X.qcow2.log under set -o pipefail.
Then make boots the overlay, once. The command is the same QEMU command line the launcher uses, with one addition: the seed ISO, attached read-only as a CD-ROM, so cloud-init finds its user-data. There's no window. The guest's serial console comes out on your terminal, and make passes it through tee into the overlay's log. The line runs with pipefail, so a QEMU that fails fails the step. Your QEMU option variables, such as the memory size, apply to this boot too. The step ends when cloud-init powers the guest off and QEMU exits.
Details and sources
Details
- The recipe:
set -o pipefail && $(call QEMU_SYSTEM_X86_64_EFI_CMDLINE,X.qcow2,X.efi-code.qcow2,X.efi-vars.qcow2) -drive file=X.seed.iso,format=raw,id=drive1,if=none,readonly=true,media=cdrom -device virtio-blk,drive=drive1 2>&1 | tee X.qcow2.log, withSHELL=bashfor the target (rulegen.py:601-609). - Every provisioning boot runs this way, under bash with
set -o pipefail: the rules rulegen.py writes, named instances (020-instances.mk:94-103) and Ubuntu Core, with and without a TPM (050-ubuntu-core.mk:37-53). A QEMU that fails to start or exits with an error fails the step, and.DELETE_ON_ERRORremoves the half-made overlay. QEMU_SYSTEM_X86_64_EFI_CMDLINEreads${QEMU_MEM_OPTION=…},${QEMU_SMP_OPTION=…},${QEMU_DISPLAY_OPTION=…}and the other option variables from the environment, the same names the launcher uses (010-qemu-x86_64.mk:13-26). The default display is-nographicand the default memory-m 2048(010-qemu.mk:9-13). The command on the slide is abridged;image-garden make -nprints it in full.- The seed is attached as a virtio block device with CD-ROM media; cloud-init's NoCloud source finds it by its
cidatalabel (010-qemu.mk:263-278). - The guest powers off because the user-data sets
power_state(010-qemu.mk:190-195); see provisioning runs once. The +tpm systems also start swtpm for this boot (050-ubuntu-core.mk:70-106). - Setting
QEMU_MEM_OPTIONin the environment ofimage-garden makeis an example; the variables are listed in the .run script.
Read more
05 The console log of that boot
ubuntu-cloud-24.04.x86_64.qcow2.log keeps the console of the provisioning boot. It survives a failed boot, is overwritten by the next one, and is removed by make clean.
That log is the only record of provisioning, so it's worth knowing. Next to the disk, the qcow2.log file holds everything the console printed: the firmware, the kernel, each cloud-init stage, and the power-off at the end. When provisioning fails, make deletes the half-made disk, but the log stays, so search it for the first error or a cloud-init warning. Each provisioning boot writes the log anew, so copy it first if you want to compare two runs. And image-garden make clean removes it, along with every other log in the directory.
Details and sources
Details
- The provisioning boot ends with
2>&1 | tee X.qcow2.log;teewithout-atruncates the file, so each boot writes it anew (rulegen.py:609-609). - The log is not a make target, so
.DELETE_ON_ERRORleaves it when it deletes the failedX.qcow2(000-garden.mk:4-7). clean::runsrm -f *.login the directory make runs in (000-garden.mk:48-50); that also removes the logsallocatewrites.- "Powering off" is the
messageof the user-data'spower_state(010-qemu.mk:190-195). The other lines on the slide stand for output that depends on the image (derived). Thegrepcommand is an example. - The README calls it "Serial console output collected during first boot and cloud-init execution" (README.md:825).
06 When the disk is made again
X.qcow2 is remade when the download link, the seed or the firmware copies are newer. The new overlay starts empty: changes made by booting .run without -snapshot are lost.
make prepares the disk again when it's missing, or when something it was made from is newer. That can be a new seed, because image-garden was upgraded or you edited .image-garden.mk. It can be new firmware copies. Or it can be a new download, when an upgrade names a newer image. Then the old overlay is replaced by an empty one and provisioned from scratch. Anything you changed by booting the .run script without -snapshot goes with it. So put what a machine needs into .image-garden.mk rather than into the disk by hand, and let allocate boot it with -snapshot.
Details and sources
Details
X.qcow2depends on$(GARDEN_DL_DIR)/GROUP/X.qcow2,X.seed.iso,X.efi-code.qcow2andX.efi-vars.qcow2(rulegen.py:601-602). The seed depends on the user-data and meta-data (010-qemu.mk:268-268), and they on the makefiles and.image-garden.mk(rulegen.py:460-471); the firmware copies on the host firmware (010-qemu-x86_64.mk:316-329).- make compares the link's time by following it to the image (derived; checked with GNU make 4.4.1). A refreshed link to the same image does not make the overlay older; a newly downloaded image does. The image is downloaded again only when its file is missing or fails its check, for example when an upgrade changes the URL's file name.
qemu-img createwrites a new, empty overlay over the old file..runalone keeps changes in the overlay;allocateadds-snapshotunlessQEMU_SNAPSHOT_OPTIONsays otherwise (image-garden:1153-1154).- Run in a scratch tree with
make -n: deleting the seed ISO gives the seed, the overlay and the provisioning boot.
Read more
07 Where to go from here
Next: provisioning runs once, the target chain, rebase, named instances and the .run script.
So the disk is a thin overlay on a shared download, prepared by one logged boot, and made again, from scratch, only when what it's made from changes. First boot explains what cloud-init does in that boot and how it stops itself. The target chain shows the files the overlay is made from. Rebase is the fix when the cache moves. Named instances put several overlays on one download, each with its own provisioning. And the .run script is how you boot the result.
Details and sources
Read more
Basis of the statements on this page
The rules were read in the image-garden source at the commit in garden.commit ("code"). The sizes were taken by running qemu-img create and qemu-img info with the same options on a small backing file, and the rebuild cases by running image-garden make -n on a scratch project ("run"). How qcow2 overlays and GNU make behave in general is marked "derived".
The two commands of step nine
For ubuntu-cloud-24.04.x86_64, with DL the download cache:
qemu-img create -f qcow2 -b DL/ubuntu/ubuntu-cloud-24.04.x86_64.qcow2 -F qcow2 ubuntu-cloud-24.04.x86_64.qcow2 64G
set -o pipefail && exec qemu-system-x86_64 -cpu max -machine q35,accel=… \
-m 2048 -smp … -nographic \
-drive if=pflash,format=qcow2,unit=0,file=ubuntu-cloud-24.04.x86_64.efi-code.qcow2,readonly=on \
-drive if=pflash,format=qcow2,unit=1,file=ubuntu-cloud-24.04.x86_64.efi-vars.qcow2 \
-drive file=ubuntu-cloud-24.04.x86_64.qcow2,if=none,format=qcow2,id=drive0,media=disk,cache=writeback,discard=unmap \
-device virtio-blk,drive=drive0,bootindex=0 \
-device virtio-rng-pci -netdev user,id=netdev0 … \
-drive file=ubuntu-cloud-24.04.x86_64.seed.iso,format=raw,id=drive1,if=none,readonly=true,media=cdrom \
-device virtio-blk,drive=drive1 \
2>&1 | tee ubuntu-cloud-24.04.x86_64.qcow2.log
Abridged from image-garden make -n; the accelerator is kvm:tcg when QEMU knows KVM, else tcg, and the SMP default depends on the host (rulegen.py:601-609, 010-qemu-x86_64.mk:13-26).
Variables
| Variable | Default | Kind | Effect on step nine |
|---|---|---|---|
QEMU_IMG_SIZE | 64G | make variable | the virtual size of the overlay |
QEMU_IMG | qemu-img on PATH | make variable | the program that creates the overlay |
QEMU_MEM_OPTION | -m 2048 | environment | memory of the provisioning boot |
QEMU_SMP_OPTION | -smp N | environment | CPUs of the provisioning boot |
QEMU_DISPLAY_OPTION | -nographic | environment | where the console goes |
The environment variables are the launcher's; see the .run script for the rest.
What makes the overlay again
| Input | Newer when |
|---|---|
| the cache link, followed to the image | the image is downloaded again (a new file name after an upgrade, or a missing or failed file) |
X.seed.iso | the user-data or meta-data are rewritten: an upgrade of image-garden, an edit of .image-garden.mk |
X.efi-code.qcow2, X.efi-vars.qcow2 | the firmware copies are made again, for example after make clean or newer host firmware |
X.qcow2 itself | it is missing, for example after a failed provisioning boot |
The launcher is not rewritten by a new overlay; see the target chain.
Files
| File | What it is |
|---|---|
X.qcow2 | the overlay, about 200 KiB when created, then as large as what the guest wrote |
X.qcow2.log | the console of the last provisioning boot |
DL/GROUP/X.qcow2 | the link the overlay names as its backing file |
DL/GROUP/<download> | the image, read-only for every overlay on it |
Roads from here
RUNTIME · make rules and the CLI
- From a target name to a VM — the files the overlay is made from
- image-garden rebase — pointing an overlay at a cache that moved
- Named instances — several overlays on one download
- The .run script — booting the prepared overlay
- Downloading and checking images — how the image under the overlay is fetched and checked
FIRST BOOT · cloud-init changes
- Provisioning runs once — what cloud-init does in the provisioning boot and how it stops