Changing what cloud-init does

.image-garden.mk, EXTRA_* for users for contributors

A project's .image-garden.mk, next to its tests, is read after image-garden's own make files and changes the cloud-init user-data of every image the project makes. EXTRA_CLOUD_INIT_FILES and EXTRA_CLOUD_INIT_APT_CONFIG are the simplest hooks, slow stable setup belongs here rather than in spread's prepare, and an edit rebuilds the images.

On the map: first boot

▶ Watch this deck — 7 scenes, ~5 min

Part of journeys

01 A file next to your tests

YOUR PROJECT FILE a plain image a cloud image, two users, open SSH run 1 · prepare: install … run 2 · prepare: install … run 3 · prepare: install … paid again on every run .image-garden.mk next to your tests the user-data of every image changes what cloud-init does where it goes · two hooks · cloud-init or prepare · what an edit costs
A plain image with two users and open SSH; installing extras in every test's prepare step is paid on every run; a .image-garden.mk next to the tests changes the user-data of every image instead

A .image-garden.mk in your project changes the cloud-init user-data of the images image-garden makes for it.

The images image-garden makes are deliberately plain: a distribution's cloud image, two users and open SSH. Your tests usually want a bit more, maybe a package or two, a configuration file or an apt setting. You could add those in every test's prepare step, but then every run pays for them again. Instead, put a file called .image-garden.mk next to your tests. image-garden reads it whenever it makes images for your project, and what you write there changes the cloud-init user-data of those images. This deck shows where the file goes, the two simplest hooks, and what an edit costs.

Details and sources

Details

  • The README section "Local cloud-init Templates" describes the file: project-specific dependencies, cloud-init per distribution, release or instance, named instances and first-boot commands (README.md:949-960).
  • mk/999-local.mk includes it: -include $(GARDEN_PROJECT_DIR)/.image-garden.mk (999-local.mk:4-5).
  • The plain image's user-data is the common template: users root and the distribution user, SSH password logins, power off at the end (010-qemu.mk:187-247).

02 Where the file goes and when it is read

WHERE IT GOES my-project/ spread.yaml tests/ .image-garden.mk image-garden make … GARDEN_PROJECT_DIR=$(pwd) the list of systems image-garden's settings .image-garden.mk, if it's there the generated system rules defined after image-garden's: yours wins and before the system rules that use it README: keep to cloud-init templates and hooks
A project directory holds spread.yaml, tests and .image-garden.mk; image-garden make gets the directory as GARDEN_PROJECT_DIR; image-garden.mk reads the list of systems, image-garden's own settings, then .image-garden.mk through its last make file, then the generated system rules; what you define last wins, and the README advises keeping to templates and hooks

image-garden runs make with your project directory; the last of image-garden's own make files includes .image-garden.mk, after image-garden's settings and before the generated system rules.

The file lives in your project directory, the one you run image-garden from. image-garden passes that directory to make, and the very last of its own make files, the local one, includes your file if it's there and quietly skips it if not. Order matters. Your file comes after image-garden's own settings, so anything you define replaces theirs, and before the generated system rules, which use what you defined. It's plain make, so you could do a lot in it. The README's advice is to keep to cloud-init templates and hooks.

Details and sources

Details

  • GARDEN_PROJECT_DIR is the directory image-garden runs in (image-garden:7); image-garden make passes it to make with -f image-garden.mk (image-garden:982-993).
  • mk/999-local.mk is -include $(GARDEN_PROJECT_DIR)/.image-garden.mk; the leading - makes a missing file no error (999-local.mk:4-5).
  • image-garden.mk is data/000-systems.mk, then the sorted mk/*.mk (so 999-local.mk last), then the generated data/050-*.gen.mk (Makefile:48-50). A variable defined in your file therefore replaces one defined in mk/, and is defined before the system rules expand it.
  • The README: "While powerful enough for arbitrary build rules, it's strongly recommended to use only cloud-init user-data templates" (README.md:1061-1070).

03 Two hooks: files and apt

TWO HOOKS #cloud-config ssh_pwauth: true users: … write_files: - path: /root/… runcmd: … EXTRA_CLOUD_INIT_FILES EXTRA_CLOUD_INIT_APT_CONFIG define EXTRA_CLOUD_INIT_FILES - path: /etc/my-test.conf content: | built for $1 $2 endef $1 = ubuntu, $2 = 24.04 the next make writes it into every user-data
The skeleton of the user-data with the apt hook at the top level after ssh_pwauth and the files hook inside write_files; an example definition of EXTRA_CLOUD_INIT_FILES that writes a file and uses the group and release; the next make writes it into every user-data

EXTRA_CLOUD_INIT_FILES adds to write_files; EXTRA_CLOUD_INIT_APT_CONFIG adds a top-level section such as apt:. Both get the group as $1 and the release as $2.

The two simplest hooks are variables image-garden calls while it writes the user-data. EXTRA_CLOUD_INIT_FILES lands inside the write_files list, so it adds files to the image. EXTRA_CLOUD_INIT_APT_CONFIG lands at the top level, so it can add an apt section. Both are called with the group and the release, such as ubuntu and 24.04, so one definition can tell systems apart. Define one with define and endef, write YAML inside, indented the way cloud-init expects, and the next make puts it into every user-data.

Details and sources

Details

  • mk/010-qemu.mk declares both hooks (010-qemu.mk:47-48). The common template calls $(call EXTRA_CLOUD_INIT_APT_CONFIG,$1,$(value 3)) at the top level, right after ssh_pwauth (010-qemu.mk:197-198), and $(call EXTRA_CLOUD_INIT_FILES,$1,$(value 3)) at the end of write_files (010-qemu.mk:236-237). In the template $1 is the group and $(value 3) the release, so inside a hook they arrive as $1 and $2.
  • EXTRA_CLOUD_INIT_FILES starts empty. EXTRA_CLOUD_INIT_APT_CONFIG is already used by image-garden for old Ubuntu releases: see the next slide.
  • The file /etc/my-test.conf and its content are an illustrative example, not taken from image-garden and not run for this map.

Read more

  • cloud-init's own documentation describes write_files and apt.

04 A hook you define replaces image-garden's

REPLACING, NOT ADDING image-garden's apt hook end-of-life Ubuntu: the old releases archive define EXTRA_CLOUD_INIT_APT_CONFIG old releases lose their mirror define EXTRA_CLOUD_INIT_APT_CONFIG apt: conf: | APT::Install-Recommends "false"; $(subst apt:,,$(call UBUNTU_OLD_RELEASES_APT_CONFIG,$1,$2)) endef both survive redefine a hook: keep what was there
image-garden's apt hook points end-of-life Ubuntu at the old releases archive; a careless redefinition cuts it; image-garden's own .image-garden.mk turns off recommended packages and calls the old releases configuration inside the same apt section, so both survive

image-garden defines EXTRA_CLOUD_INIT_APT_CONFIG for end-of-life Ubuntu releases. Its own .image-garden.mk redefines it and calls UBUNTU_OLD_RELEASES_APT_CONFIG inside, keeping both.

One catch: defining a hook replaces it, it doesn't add to it. And image-garden already uses the apt hook itself. For Ubuntu releases past their end of life, it points apt at the old releases archive. Redefine the hook carelessly and those releases lose their mirror. image-garden's own file shows the way. It turns off recommended packages, and inside the same apt section it calls the old releases configuration it replaced, so both survive. When you redefine a hook, keep what was there.

Details and sources

Details

  • mk/050-ubuntu.mk lists the end-of-life releases that need old-releases.ubuntu.com (050-ubuntu.mk:12), defines UBUNTU_OLD_RELEASES_APT_CONFIG (050-ubuntu.mk:15-24) and sets EXTRA_CLOUD_INIT_APT_CONFIG to call it (050-ubuntu.mk:26-28).
  • image-garden's own .image-garden.mk redefines the hook: APT::Install-Recommends "false" and the old-releases configuration with its leading apt: removed by $(subst apt:,,…), so there is one apt: map (.image-garden.mk:5-14). Its comment: QEMU with its recommends meant 319 packages on Ubuntu 26.04 instead of 81.
  • That the later definition replaces the earlier one is make's rule for variables; see where the file goes.

05 cloud-init or spread's prepare

CLOUD-INIT OR PREPARE allocate: a fresh -snapshot of the prepared image, every time spread.yaml prepare: / prepare-each: every run, every machine cloud-init in .image-garden.mk once, in the provisioning boot packages, toolchains services, boot settings building your project per-test state image-garden's own tests: packages: make, xorriso, curl, qemu-utils, qemu-system-… prepare: | make; sudo make install
Two columns: cloud-init in .image-garden.mk runs once at make time and holds packages, toolchains, services and boot settings; spread's prepare runs on every run and every machine and holds building the project and per-test state; image-garden's own tests install make, xorriso, curl and QEMU through cloud-init and build and install image-garden in prepare

Slow, stable setup goes in cloud-init: it runs once, when make builds the image. What depends on the code under test stays in prepare:, which runs on every fresh -snapshot machine.

So what belongs in this file, and what in spread's prepare? Think about how a test machine starts: allocate boots a fresh snapshot of the prepared image, every time. Whatever prepare or prepare-each installs is paid again on every run and on every machine. cloud-init here runs once, in the provisioning boot, and every allocate starts with it done. So put slow, stable setup in cloud-init: packages, toolchains, services, boot settings. Keep in prepare what depends on the code under test, like building your project, and per-test state. image-garden's own tests do exactly that.

Details and sources

Details

  • allocate starts the .run script with -snapshot unless QEMU_SNAPSHOT_OPTION says otherwise, so each machine starts from the prepared image and its changes are thrown away (image-garden:1153-1163).
  • The provisioning boot, where cloud-init runs, is part of making X.qcow2 (rulegen.py:593-610); it runs again only when the image is rebuilt.
  • image-garden's own project splits the work this way: its .image-garden.mk installs python3, make, whois, xorriso, curl, qemu-utils and QEMU for a nested VM through cloud-init (.image-garden.mk:43-58), while its spread.yaml top-level prepare: builds and installs image-garden itself (spread.yaml:73-77).
  • Spread behaviour, not image-garden's: spread runs a project's prepare: on each machine it allocates, a suite's prepare: before that suite, and prepare-each: before every task. With fresh machines per run, all of them are paid on every run.
  • The trade-off: an edit to .image-garden.mk rebuilds the images (next slide). For the cost over many runs drawn as two timelines, see paid for once.

06 An edit rebuilds the images

AN EDIT REBUILDS .image-garden.mk ubuntu-cloud-24.04 user-data debian-cloud-13 user-data fedora-cloud-42 user-data edited: newer than every user-data user-data seed ISO prepared disk provisioning boot for every system of the project, not only the one you meant downloaded images: kept one slow make afterwards
An edited .image-garden.mk is newer than the user-data of every system; make writes the user-data, the seed ISO and the prepared disk again, with a provisioning boot, for every system of the project; the downloaded images are kept

Each .user-data depends on .image-garden.mk. An edit remakes the user-data, the seed and the prepared disk of every system in the project; downloads are kept.

That's the trade-off. Every system's user-data depends on your file. Change it, and make sees each user-data is older than the file it came from, so it writes the user-data again, then the seed, then the prepared disk, which means another provisioning boot. That happens for every system your project makes, not just the one you meant, because each user-data depends on the whole file. The downloaded cloud images stay as they are. So edit it when you mean to, and expect one slow make afterwards.

Details and sources

Details

  • The generated rule for each system: X.user-data: $(MAKEFILE_LIST) $(wildcard $(GARDEN_PROJECT_DIR)/.image-garden.mk) (rulegen.py:467-471); named instances have the same dependency (020-instances.mk:75-76). Because the file is included, it is also in $(MAKEFILE_LIST), which the meta-data and .run rules depend on.
  • X.qcow2 depends on the download, X.seed.iso and the firmware copies, and its recipe creates the overlay and runs the provisioning boot (rulegen.py:593-610); the seed is made from the user-data and meta-data (010-qemu.mk:268-279).
  • The downloaded image and its symlink in the cache do not depend on .image-garden.mk, so an edit downloads nothing.
  • The system names on the slide are examples of a project that uses three systems.

07 Where to go from here

ROADS Your own changes Picking a template five names, first wins Named instances more systems, one download One base, many VMs profiles built once Once, then off why provisioning runs once The seed ISO how the user-data gets in
Roads from this deck: picking a template, named instances, several systems from one base image, provisioning runs once, and the seed ISO

Next: how a template is chosen, named instances, and several systems from one base image.

So: one file next to your tests, included after image-garden's own make files, with two hooks for files and apt, slow setup baked in once instead of in prepare, and a rebuild whenever the file changes. Hooks are small. For anything bigger, a template replaces the whole user-data of a group, a release or an instance, and the next deck shows how one is chosen. Named instances give you more systems from one download. And the profiles deck puts it all together.

Details and sources

Roads from here

RUNTIME · make rules and the CLI

SPREAD · tests

FIRST BOOT · cloud-init changes