Several systems from one base image

profiles you pay for once for users for CI operators

With define-instance and a cloud-init template in .image-garden.mk, one downloaded image becomes several prepared systems, such as openSUSE booted with AppArmor instead of SELinux, or a machine with heavy software already installed; the work is done once by make, not by every test run.

On the map: first boot

▶ Watch this deck — 9 scenes, ~6 min

Worth seeing first

Part of journeys

  • Your own systems with .image-garden.mk, step 4 of 4 · previous: Named instances

01 Install it once, not on every run

PROFILES openSUSE, as it ships: SELinux openSUSE, booted with AppArmor compiler, database, nested QEMU in every test's prepare step: run 1 install … minutes … test run 2 install … minutes … test run 3 install … minutes … test on every run, on every machine .image-garden.mk: extra systems from one base image, each with a profile built once by make, used by every run
Three systems a project needs: openSUSE with SELinux, openSUSE with AppArmor, and a machine with a compiler, a database and nested QEMU. Installing in every test run costs minutes on every run; .image-garden.mk builds them once from one base image

A project's .image-garden.mk can define extra systems from one base image, each with its own cloud-init profile, so their setup is done once instead of in every test run.

Say your tests need openSUSE twice: once as it ships, with SELinux, and once booted with AppArmor instead. Another suite needs a compiler, a database and a nested QEMU, which take minutes to install. You could do all of that in each test's prepare step, and pay for it on every run, on every machine. image-garden lets you do it once instead. A few lines in a file called .image-garden.mk describe extra systems built from the same base image, each with its own cloud-init profile. This deck builds two of them and shows when the work is done, and when it's done again.

Details and sources

Details

  • image-garden make includes the project's .image-garden.mk through -include $(GARDEN_PROJECT_DIR)/.image-garden.mk (999-local.mk:5). The README lists what it is for: project dependencies, cloud-init per distribution, release or named instance, named instances, and extra first-boot commands (README.md:949-965).
  • The README's own case for it: pre-installed dependencies speed up test iteration and reduce network use, and changes invalidate existing images (README.md:1053-1060).
  • The openSUSE and SELinux/AppArmor pairing is this map's example; image-garden's source has no such profile.

02 A named instance shares the download

A NAMED INSTANCE the verified openSUSE download, opensuse-cloud-16.0.x86_64 $(eval $(call define-instance,opensuse-cloud-16.0,apparmor)) opensuse-cloud-16.0@apparmor.x86_64 own user-data · seed · overlay · .run overlay with the AppArmor profile overlay of the base system opensuse-cloud-16.0.x86_64 two systems, one download, each with its own changes on top
One define-instance line in .image-garden.mk makes opensuse-cloud-16.0@apparmor.x86_64 with its own user-data, seed, overlay and .run script; both it and the base system's overlay sit on the one verified openSUSE download

$(eval $(call define-instance,opensuse-cloud-16.0,apparmor)) makes opensuse-cloud-16.0@apparmor.x86_64: its own user-data, seed, overlay and .run, on the base system's download.

Start with the base system, opensuse-cloud-16.0 on x86. One line in the .image-garden.mk file, a call to define-instance with the base name and the word apparmor, creates a new system. Its name is the base name with at apparmor before the architecture. It gets its own user-data, its own seed, its own overlay and its own .run script. What it doesn't get is its own download: the overlay sits on the same verified openSUSE image as the base system. So two systems cost one download, and each keeps its own changes on top.

Details and sources

Details

  • define-instance takes a base system, with or without the architecture, and an instance name; without an architecture it uses GARDEN_ARCH (020-instances.mk:4-28). The result for this example is opensuse-cloud-16.0@apparmor.x86_64.
  • The instance's overlay is created on the base system's download, $(GARDEN_DL_DIR)/<group>/<base>.qcow2, with qemu-img create -f qcow2 -b (020-instances.mk:96-98).
  • It inherits the base system's group, release and boot type from data/000-systems.mk (020-instances.mk:47-51). The base used here is [opensuse-cloud-16.0.x86_64] (opensuse.ini:50-55).
  • The base must be a system defined in data/*.ini, which is every system but Ubuntu Core: its systems are made by rules of their own and have no instances. define-instance with an Ubuntu Core system lists the instance in list-systems, but make has no rule for it (020-instances.mk:9-10, README.md:976-979).
  • The hostname follows base-release-instance, for example ubuntu-24-04-webserver (README.md:988-989).

03 The instance gets its own template

THE TEMPLATE most specific first, for opensuse-cloud-16.0@apparmor 1 OPENSUSE_16.0@apparmor_CLOUD_INIT_USER_DATA_TEMPLATE 2 OPENSUSE@apparmor_CLOUD_INIT_USER_DATA_TEMPLATE 3 OPENSUSE_16.0_CLOUD_INIT_USER_DATA_TEMPLATE 4 OPENSUSE_CLOUD_INIT_USER_DATA_TEMPLATE 5 CLOUD_INIT_USER_DATA_TEMPLATE, the common one define OPENSUSE_16.0@… $(CLOUD_INIT_USER_DATA_TEMPLATE) users, SSH, power-off runcmd: … ends the common - your command endef your lines join runcmd
The template lookup for opensuse-cloud-16.0@apparmor, most specific first: OPENSUSE_16.0@apparmor, OPENSUSE@apparmor, OPENSUSE_16.0, OPENSUSE, then the common CLOUD_INIT_USER_DATA_TEMPLATE; a template starts by expanding the common one, which ends with runcmd, and appends commands

The most specific template wins: OPENSUSE_16.0@apparmor_CLOUD_INIT_USER_DATA_TEMPLATE first, CLOUD_INIT_USER_DATA_TEMPLATE last. A template starts with $(CLOUD_INIT_USER_DATA_TEMPLATE) and appends to its runcmd:.

Next, the profile itself. image-garden picks a user-data template by name, most specific first. For our instance it looks for a variable that joins the group in capitals, the release and the instance, followed by the words cloud-init user-data template. If that's missing, it tries the group with the instance, then the group with the release, then the group alone, and finally the common template. Start your template by expanding the common one, so you keep the users, the SSH settings and the power-off. The common template ends with a runcmd list, so the lines you add after it are more commands to run.

Details and sources

Details

  • The picker tries, in order, GROUP_RELEASE@INSTANCE_, GROUP@INSTANCE_, GROUP_RELEASE_, GROUP_, each followed by CLOUD_INIT_USER_DATA_TEMPLATE, then CLOUD_INIT_USER_DATA_TEMPLATE itself (010-qemu.mk:293-298). For an instance the group is passed in capitals and the instance name as given (020-instances.mk:75), so the first name for this example is OPENSUSE_16.0@apparmor_CLOUD_INIT_USER_DATA_TEMPLATE. The README documents the same order (README.md:1046-1051).
  • CLOUD_INIT_USER_DATA_TEMPLATE is the common template, BASE_CLOUD_INIT_USER_DATA_TEMPLATE (010-qemu.mk:249). It sets the users, SSH and power_state, and ends with runcmd: (010-qemu.mk:187-247).
  • The README asks templates to include the common one and says it ends with runcmd, so commands can be appended as a YAML list (README.md:1061-1070).
  • Appended commands run after the common template's last command, touch /etc/cloud/cloud-init.disabled (010-qemu.mk:246). That file stops cloud-init on later boots, not the current one.

04 Changing how the system boots

CHANGING HOW IT BOOTS - zypper --non-interactive install -t pattern apparmor - sed -i 's/security=selinux selinux=1/security=apparmor/' … - update-bootloader --refresh illustrative: an AppArmor profile for openSUSE provisioning boot SELinux still in charge the kernel picks its security module when it starts power off next boot: AppArmor the first one tests see opensuse-cloud-16.0, from the same download, still boots with SELinux
During the provisioning boot SELinux is active while runcmd installs AppArmor and adds security=apparmor to the kernel command line; cloud-init powers off; the next boot, the first one tests see, comes up with AppArmor; the base system still boots with SELinux

Boot settings changed during provisioning apply from the next boot, the first one your tests see. The base system keeps the distribution's defaults.

In the AppArmor profile, the commands install the AppArmor packages and change the kernel command line, so the kernel starts AppArmor rather than SELinux. They run during the provisioning boot, while SELinux is still in charge. That's fine: the kernel picks its security module when it starts, so the new setting takes effect on the next boot. And the next boot comes straight after: cloud-init powers the machine off, make is done, and the first time you or your tests start this system, it comes up with AppArmor. The base system, built from the same image, still boots with SELinux.

Details and sources

Details

  • The three commands on the slide are illustrative, not taken from image-garden and not run for this map. Check the AppArmor pattern name and the bootloader tool against the openSUSE release you use.
  • cloud-init powers the machine off once it is done (power_state, 010-qemu.mk:190-196), so a kernel command line written during provisioning is first used by the next boot, the first boot of the prepared image.
  • image-garden's own example of a boot-time change is in the spread demo: Alpine's template sets GRUB_TIMEOUT=0 and regenerates the grub configuration (.image-garden.mk:6-14).

05 Heavy software, already installed

HEAVY SOFTWARE define UBUNTU_CLOUD_INIT_USER_DATA_TEMPLATE $(CLOUD_INIT_USER_DATA_TEMPLATE) packages: - make - qemu-utils - $(GUEST_QEMU_SYSTEM) - $(GUEST_UEFI_FIRMWARE) endef image-garden's own .image-garden.mk, shortened APT::Install-Recommends "false"; 81 packages instead of 319 ubuntu-cloud-24.04@mc packages: mc, in the spread demo installed while provisioning and kept in the prepared image
A template with a packages list after the common template: image-garden's own file installs QEMU, firmware and tools for nested VMs without recommends, 81 packages instead of 319 on Ubuntu 26.04; the spread demo's mc instance adds Midnight Commander; the packages are installed during provisioning and end up in the prepared image

A packages: list in a template installs software during the provisioning boot. image-garden's own .image-garden.mk installs QEMU and firmware for nested VMs, without recommends.

Software works the same way. A packages list after the common template installs whatever your tests need: compilers, a database, or, as image-garden's own tests do, QEMU and its firmware for a VM inside the VM. image-garden's own file also turns off recommended packages, because QEMU with its recommends meant 319 packages on Ubuntu 26.04 instead of 81. The spread demo has a smaller example: an mc instance of Ubuntu 24.04 that only adds Midnight Commander. Whatever the list, the packages are fetched and installed during the provisioning boot, and they end up in the prepared image.

Details and sources

Details

  • image-garden's own .image-garden.mk turns off recommended packages in an EXTRA_CLOUD_INIT_APT_CONFIG; its comment counts 319 packages with recommends and 81 without, for QEMU on Ubuntu 26.04 (.image-garden.mk:5-14).
  • Its Ubuntu template installs python3, make, whois, xorriso, curl, qemu-utils, a qemu-system-* for the guest's architecture and its UEFI firmware, for nested VMs (.image-garden.mk:33-55); Debian uses the same template (.image-garden.mk:58).
  • The spread demo defines ubuntu-cloud-24.04@mc and installs mc in it (.image-garden.mk:16-23).
  • On Debian and Ubuntu, packages come through pkgproxy when one is configured, because its apt setup runs in bootcmd; see what pkgproxy adds to user-data.

06 Paid for once, used by every run

PAID ONCE without a profile boot install tests boot install tests boot install tests with a profile make: provisioning boot installs once prepared overlay allocate · tests allocate · tests allocate · tests -snapshot: a test's changes are thrown away, the image stays ten runs or a hundred: the install happened once
Two timelines: without a profile each of three runs installs packages before testing; with a profile make installs once in the provisioning boot, and each allocate starts from the prepared overlay with -snapshot and goes straight to testing

The provisioning boot pays the install cost once, at make time. Each allocate starts from the prepared overlay with -snapshot, so runs neither repeat the install nor change the image.

Here's what that buys you. Without a profile, every test run boots the plain image and installs the same packages again, on every machine, before the first test can start. With a profile, the cost moves to make. The provisioning boot installs everything once, and the result is the prepared overlay. After that, image-garden allocate starts each machine from that overlay in snapshot mode, so whatever a test changes is thrown away, and the next run starts from the same prepared state. Ten runs or a hundred, and the install happened once.

Details and sources

Details

  • The instance's .qcow2 rule creates the overlay and runs the provisioning boot once; make does not run it again while the overlay is newer than its inputs (020-instances.mk:96-103).
  • allocate runs the .run script with -snapshot unless QEMU_SNAPSHOT_OPTION says otherwise, so writes go to a temporary file and the prepared overlay is not changed (image-garden:1153-1163).
  • The three-runs drawing is a schematic, not a measurement.

07 When the image is built again

BUILT AGAIN .image-garden.mk edited: newer user-data seed ISO new overlay new provisioning boot verified download: kept every system's user-data depends on the file: opensuse-cloud-16.0 opensuse-cloud-16.0@apparmor ubuntu-cloud-24.04 … all rebuilt CI that caches prepared images restores the file's modification time
Editing .image-garden.mk makes every system's user-data out of date; make writes the user-data, the seed ISO and a new overlay with a new provisioning boot, for every system made in the project; the verified download is kept

Editing .image-garden.mk makes every system's user-data, seed and overlay out of date. The download is kept; the provisioning boot runs again.

Once doesn't mean never. The user-data of every system depends on your .image-garden.mk file. Edit the file, and make sees that the user-data is older. It writes it again, packs a new seed, and builds a new overlay with a new provisioning boot. The verified download stays where it is, so only the provisioning is paid again. Because every system in the project depends on the same file, a change rebuilds all of the images you've made there, not only the one whose template you touched. That's also why CI jobs that cache prepared images restore the file's modification time.

Details and sources

Details

  • An instance's user-data depends on $(MAKEFILE_LIST) and $(wildcard $(GARDEN_PROJECT_DIR)/.image-garden.mk) (020-instances.mk:75-76); the generated rules give every other system the same dependency (rulegen.py:467-468).
  • The seed ISO is made from the user-data (010-qemu.mk:268-279), and the overlay depends on the seed ISO (020-instances.mk:96), so a newer .image-garden.mk reaches the overlay and its provisioning boot.
  • The download is a separate target, not affected by the file.
  • The GitHub action restores the file's modification time with git-restore-mtime for this reason; see how the action caches.

08 Reaching the instance from spread

FROM SPREAD systems: - opensuse-cloud-16.0: - opensuse-cloud-16.0-at-apparmor: username: opensuse spread.yaml: no @ in a spread name SYSTEM="${SYSTEM/-at-/@}" … allocate --spread "$SYSTEM.$ARCH" opensuse-cloud-16.0@apparmor.x86_64 user and password: the base system's, opensuse / opensuse listed by its .run script's --query tests on openSUSE, SELinux tests on openSUSE, AppArmor side by side, from one download
In spread.yaml the system is opensuse-cloud-16.0-at-apparmor with user opensuse and password opensuse; the allocate script turns -at- into @ and adds the architecture, asking image-garden for opensuse-cloud-16.0@apparmor.x86_64; the user and password are the base system's, and the instance's .run script lists them with --query; tests run on both openSUSE systems from one download

In spread.yaml the instance is opensuse-cloud-16.0-at-apparmor; allocate maps -at- to @ and adds the architecture. The credentials are the base system's, opensuse/opensuse, and --query lists them.

spread can run tests on these systems too, with one catch. spread system names can't carry an at sign, so the instance is written with -at- in spread.yaml: opensuse-cloud-16.0-at-apparmor. The allocate script of the garden backend turns -at- back into an at sign, adds the architecture, and asks image-garden for that system. The user name and password are the base system's, because the profile starts from the common template, and the instance's .run script lists them under --query, like any other. Your tests then run on both openSUSE systems, side by side, from one download.

Details and sources

Details

  • The README asks for + to be written -plus- and @ to be written -at- in spread system names, and shows the allocate lines that map them back (README.md:720-732). image-garden's own spread.yaml does the same and adds the architecture (spread.yaml:45-47).
  • The instance has the base system's users, root/root and the group's user. Its launcher's --query lists them, as a base system's does: the rule passes root distro and the group (020-instances.mk:89) to the preamble that prints system.users.0 and system.users.1 (010-qemu.mk:80-88). spread itself still reads them from spread.yaml.

09 Where to go from here

ROADS One base, many VMs Named instances what define-instance makes Picking a template the full lookup order Your own changes files and apt settings Names in spread -plus- and -at- allocate each machine from the overlay Once, then off provisioning runs once
Six roads from this deck: named instances, the template lookup, your own changes, spread names, allocate, and provisioning runs once

Next: named instances, the template lookup order, the simpler hooks, spread names, allocate and the provisioning boot.

So a profile is a few lines of make and cloud-init that turn one download into as many prepared systems as your tests need, each paid for once. From here, the named instances deck shows everything define-instance creates. The template deck has the full lookup order, and the deck on your own changes covers the simpler hooks for files and apt settings. To run these systems under spread, the naming deck explains dash plus and dash at. And allocate, with the once-only provisioning boot, is where the saved time shows up.

Details and sources

Details

  • A profile in one line: define-instance for the system, a template named after it that starts with $(CLOUD_INIT_USER_DATA_TEMPLATE), and make pays the provisioning once.

Read more

Basis of the statements on this page

The mechanics, define-instance, the template lookup and the rebuild rules, were read in the image-garden source at the commit this map describes ("code"). The template names for the examples follow from the lookup rule ("derived"). The openSUSE AppArmor commands are illustrative: they are not part of image-garden and were not run for this map.

A complete .image-garden.mk

Put it next to your tests, where you run image-garden make or spread.

# Two systems from one openSUSE download: the base system,
# opensuse-cloud-16.0.x86_64, boots with SELinux as it ships;
# this instance boots with AppArmor.
$(eval $(call define-instance,opensuse-cloud-16.0,apparmor))

# Illustrative: check the pattern and the bootloader tool for your release.
define OPENSUSE_16.0@apparmor_CLOUD_INIT_USER_DATA_TEMPLATE
$(CLOUD_INIT_USER_DATA_TEMPLATE)
- zypper --non-interactive install -t pattern apparmor
- sed -i 's/security=selinux selinux=1/security=apparmor/' /etc/default/grub
- update-bootloader --refresh
endef

# Heavy software, installed once into every Ubuntu image of the project.
define UBUNTU_CLOUD_INIT_USER_DATA_TEMPLATE
$(CLOUD_INIT_USER_DATA_TEMPLATE)
packages:
- build-essential
- postgresql
- qemu-system-x86
endef
  • The first line of each template expands the common template: the users, SSH settings, pkgproxy setup and power-off.
  • The common template ends with runcmd:, so lines starting with - right after it are more commands.
  • packages: starts a new top-level key after the commands.
  • Editing this file rebuilds the user-data, seed and overlay of every system made in this directory.

The template names for these examples

OrderVariable nameUsed by
1OPENSUSE_16.0@apparmor_CLOUD_INIT_USER_DATA_TEMPLATEopensuse-cloud-16.0@apparmor only
2OPENSUSE@apparmor_CLOUD_INIT_USER_DATA_TEMPLATEevery @apparmor instance of openSUSE
3OPENSUSE_16.0_CLOUD_INIT_USER_DATA_TEMPLATEopenSUSE 16.0 without a more specific template
4OPENSUSE_CLOUD_INIT_USER_DATA_TEMPLATEevery openSUSE system without a more specific template
5CLOUD_INIT_USER_DATA_TEMPLATEthe common template, for everything else

The same order with UBUNTU, UBUNTU_24.04 and an instance such as @mc applies to Ubuntu: image-garden's spread demo defines UBUNTU_24.04@mc_CLOUD_INIT_USER_DATA_TEMPLATE.

The systems in spread.yaml

Garden namespread nameUser and password
opensuse-cloud-16.0.x86_64opensuse-cloud-16.0opensuse / opensuse
opensuse-cloud-16.0@apparmor.x86_64opensuse-cloud-16.0-at-apparmoropensuse / opensuse

The allocate script maps -at- back to @ and adds .$(uname -m). See names in spread. Each instance's .run script lists the same users under --query.

Which systems can have instances

Every system defined in data/*.ini. Ubuntu Core's systems are made by rules of their own and have no instances: define-instance with one lists the instance in list-systems, but make has no rule for it (README.md:976-979).

Roads from here

RUNTIME · make rules and the CLI

SPREAD · tests

FIRST BOOT · cloud-init changes