Provisioning runs once

power off, cloud-init off for everyone

The provisioning boot ends with cloud-init powering the VM off, which ends make's recipe, and its last runcmd command creates /etc/cloud/cloud-init.disabled, so every later boot, from the .run script or allocate, starts as prepared without provisioning again.

On the map: first boot

▶ Watch this deck — 7 scenes, ~5 min

Worth seeing first

01 How does make know it's done?

ONE PROVISIONING BOOT make QEMU the guest: cloud-init at work make waits, and can't see inside the guest the guest powers itself off QEMU exits · the recipe ends · make writes the .run script then many more boots the .run script allocate must not set it up again 1 power off at the end 2 cloud-init switched off afterwards
make starts QEMU, which boots the guest while cloud-init works; make cannot see inside, so the guest powers itself off, QEMU exits, the recipe ends and make writes the .run script; later boots from .run and allocate must not set the image up again; the deck covers the power-off and the switch that keeps cloud-init off

make waits for QEMU to exit. The guest ends the provisioning boot by powering off, and switches cloud-init off for every later boot.

Making the image means booting it once, and make has to wait for that boot to finish. But how would it know cloud-init is done? It can't look inside the guest. The answer is simple: the guest switches itself off. When it powers off, QEMU exits, the recipe ends, and make moves on to the .run script. That leaves a second question. The image will boot many more times, from the run script and from allocate, and those boots must not set it up again. This deck shows both halves: the power-off at the end, and the switch that keeps cloud-init quiet afterwards.

Details and sources

Details

  • The image rule creates the overlay, then runs QEMU in the foreground with the seed attached, piped through tee X.qcow2.log; the recipe ends when QEMU exits (rulegen.py:601-609). The .run rule is ordered after the image, through | X.qcow2 when GARDEN_RUN_REQUIRES_QCOW2 is set, which it is by default (rulegen.py:594, 010-qemu.mk:14).
  • The two halves are the power_state block (010-qemu.mk:190-195) and touch /etc/cloud/cloud-init.disabled (010-qemu.mk:246).
  • Derived: QEMU exits when the guest powers off because the command line has no -no-shutdown.

02 Power off at the end

POWER OFF power_state: delay: now mode: poweroff message: Powering off timeout: 0 condition: true runcmd: every command has run cloud-init powers the guest off QEMU exits make: the recipe is done ubuntu-cloud-24.04.x86_64.qcow2.log the console, through tee the first place to look when it goes wrong
The power_state block: delay now, mode poweroff, message Powering off, timeout 0, condition true; after every runcmd command cloud-init powers the guest off, QEMU exits and make's recipe is done; the console went through tee into ubuntu-cloud-24.04.x86_64.qcow2.log, the first place to look when it goes wrong

power_state: {delay: now, mode: poweroff, message: Powering off, timeout: 0, condition: true}. The console goes to ubuntu-cloud-24.04.x86_64.qcow2.log.

The first half is a power state block near the top of the user-data. It asks for a power-off, with no delay, the message Powering off, no timeout, and a condition that's simply true. cloud-init acts on it at the very end, after every command in the runcmd list has run. The guest shuts down, QEMU exits, and make's recipe is done. Everything the console printed along the way went through tee into a log next to the image, so for ubuntu-cloud-24.04.x86_64 that's its .qcow2.log. It's the first place to look when a provisioning boot goes wrong.

Details and sources

Details

  • The block in the common template: power_state: with delay: now, mode: poweroff, message: Powering off, timeout: 0, condition: true (010-qemu.mk:190-195). The Ubuntu Core template has the same block (050-ubuntu-core.mk:135-140).
  • The provisioning command is set -o pipefail && <QEMU command> … 2>&1 | tee X.qcow2.log (rulegen.py:606-609); the display defaults to -nographic, so the serial console reaches the log (010-qemu.mk:13).
  • Derived, from cloud-init's behaviour: the power-state module runs at the end of cloud-init's final stage, after the runcmd script, and condition: true means it acts whatever happened before.
  • Amazon Linux 2 does not act on power_state, and its template powers off by hand: see what differs per distribution.

03 Then cloud-init is switched off

SWITCHED OFF runcmd: - … - touch /etc/cloud/cloud-init.disabled cloud-init.disabled an empty file cloud-init, a later boot finds the file, does nothing this boot: the list goes on, then power off later boots: same users, host name and SSH settings, no second setup
The last runcmd command touches /etc/cloud/cloud-init.disabled; on every later boot cloud-init finds the file and does nothing; the provisioning boot itself still finishes its list and powers off; later boots keep the users, host name and SSH settings without a second round of setup

The last runcmd command is touch /etc/cloud/cloud-init.disabled. It stops cloud-init on every later boot, not on this one.

The second half is the last command in the runcmd list. It creates an empty file, cloud-init.disabled, in cloud-init's config directory. When cloud-init starts and finds that file, it does nothing at all. The file is written during the provisioning boot, but it only matters on the next one, so this boot still finishes its list and powers off. From then on, the image boots as prepared: the same users, the same host name, the same SSH settings, and no second round of setup.

Details and sources

Details

  • touch /etc/cloud/cloud-init.disabled is the last command of the common template's runcmd list, after pkgproxy's late setup (010-qemu.mk:238-247).
  • Derived, from cloud-init's behaviour: at boot, cloud-init's systemd generator looks for /etc/cloud/cloud-init.disabled and, when it exists, keeps cloud-init's services from running. A file created while cloud-init runs does not stop that run.
  • The Ubuntu Core template has no such command (050-ubuntu-core.mk:131-177); see Ubuntu Core images.
  • Commands that a project template adds after $(CLOUD_INIT_USER_DATA_TEMPLATE) run after this one, in the same boot; see how a user-data template is chosen.

04 Every later boot

LATER BOOTS ./….run the prepared overlay no seed, cloud-init off a login prompt in seconds changes are written here, and stay allocate .run -snapshot starts prepared changes dropped at discard provisioning again only when make prepares the image again for example: an edit to .image-garden.mk, a new user-data
The .run script boots the overlay with no seed and cloud-init off, and changes stay in the overlay; allocate starts the same script with -snapshot, so each machine starts prepared and its changes are dropped at discard; provisioning repeats only when make prepares the image again, for example after an edit to .image-garden.mk changes the user-data

.run keeps changes in the overlay; allocate adds -snapshot and keeps none. Neither provisions again; a new user-data makes make prepare the image again.

So what happens on the boots that follow? The .run script boots the prepared overlay with no seed attached, cloud-init stays off, and you get a login prompt in seconds. Changes you make there are written to the overlay, and they stay. allocate starts the same script with QEMU's snapshot option, so each test machine begins in the prepared state, and its changes are thrown away when it's discarded. Either way, provisioning doesn't repeat. It happens again only when make prepares the image again, for example after you edit your .image-garden.mk, which changes the user-data.

Details and sources

Details

  • The .run script attaches only the overlay, as the boot disk (010-qemu-x86_64.mk:59-108); writes go to X.qcow2, the base image stays as downloaded.
  • allocate runs the script with -snapshot unless QEMU_SNAPSHOT_OPTION says otherwise, in the background with a pid file (image-garden:1153-1163). Derived, from QEMU's behaviour: with -snapshot writes go to a temporary file and are lost when QEMU exits.
  • The user-data depends on the make files and .image-garden.mk (rulegen.py:467-471); the image depends on the seed made from it (rulegen.py:601-604), so a new user-data means a new overlay and a new provisioning boot.
  • "In seconds" depends on the host and on KVM; under TCG emulation a boot takes longer.

05 What runs when in the provisioning boot for contributors

WHAT RUNS WHEN one provisioning boot, top to bottom bootcmd pkgproxy: hosts line, probe, apt setup write_files, packages files; a template's packages runcmd, the common part hosts · SSH · apt · pkgproxy · touch your template's lines after the touch: fine, it stops later boots power_state the machine switches off
Inside the provisioning boot, in order: bootcmd with pkgproxy's hosts line and apt setup; write_files and a template's packages; the common runcmd list with hosts, SSH, apt, pkgproxy's late setup and the touch of cloud-init.disabled; a template's own runcmd lines, after the touch; and finally power_state switches the machine off

bootcmd, then write_files and packages, then runcmd in order, with template lines after the touch, then power_state.

When you write a template, the order inside this boot matters. bootcmd runs first, early on: that's where pkgproxy's hosts line and apt setup go, so that cloud-init's own package installs already use it. Then files are written, and any packages your template lists are installed. Then the runcmd list runs top to bottom: hosts, SSH, apt, pkgproxy's late setup, and the touch that disables cloud-init. Lines your template adds after the common one come after that touch, which is fine, because the file only stops later boots. Last of all, the power state switches the machine off.

Details and sources

Details

  • The comment above pkgproxy's parts: PKGPROXY_BOOTCMD_SETUP "runs from bootcmd, early in the first boot, so that cloud-init's own package installs (packages: in a custom template) already go through pkgproxy", and PKGPROXY_RUNCMD_SETUP "runs from runcmd, at the end of the first boot", because the snap script restarts snapd (010-qemu.mk:108-126). The two blocks are at 010-qemu.mk:156-168.
  • The common runcmd list: hosts, SSH snippet, PermitRootLogin sed, apt sed, pkgproxy's late setup, then touch /etc/cloud/cloud-init.disabled (010-qemu.mk:238-246).
  • A template that starts with $(CLOUD_INIT_USER_DATA_TEMPLATE) and adds dash lines extends that list after the touch; Amazon Linux 2's shutdown --poweroff now is such a line (050-amazonlinux-2.mk:4-8).
  • Derived, from cloud-init's behaviour: bootcmd runs in an early stage on every boot cloud-init runs; write_files runs before the package module; the package module runs before the runcmd script; the power-state module runs last.

06 When the boot doesn't end, or ends badly

WHEN IT GOES WRONG the guest never powers off make waits: no time limit a template without the common one · a system that ignores power_state ….qcow2.log watch it: tee writes as it goes a command fails condition: true, power off anyway QEMU exits cleanly, make calls it done so read the log for errors Ctrl-C, or QEMU fails make deletes the half-made image the next run boots it again
If the guest never powers off, because a template left out the common one or the system ignores power_state, make waits with no time limit; the .qcow2.log shows the console as it happens; a failed command still powers off and make treats the image as done, so read the log; an interrupted make or a failed QEMU deletes the half-made image and the next run boots again

No timeout: a guest that never powers off keeps make waiting. A failed command still powers off and counts as success. An interrupted or failed boot deletes X.qcow2.

What if the boot never ends? make sets no time limit on it. If the guest never powers off, because a template left out the common one or the system ignores the power state, make simply waits. Watch the console log, which tee writes as it goes. A command that fails doesn't stop the power-off either. The condition is true, so the machine shuts down anyway, QEMU exits cleanly, and make calls the image done. So read the log for errors. And if you interrupt make, or QEMU itself fails, make deletes the half-made image, and the next run starts the provisioning boot over.

Details and sources

Details

  • The provisioning command has no timeout: set -o pipefail && <QEMU command> … 2>&1 | tee X.qcow2.log (rulegen.py:606-609). With pipefail, a QEMU that exits non-zero fails the recipe even though tee succeeds.
  • .DELETE_ON_ERROR: removes a target whose recipe failed (000-garden.mk:4-7); The pattern .PRECIOUS: %.qcow2 %.img (010-qemu.mk:334) is meant for the converted downloads; it does not protect the overlay, whose rule is explicit. Run for this map with GNU make 4.4.1 on a two-line makefile of the same shape: a failing explicit x.qcow2 rule had its target deleted. On an interrupt GNU make deletes a target that is not precious (derived).
  • condition: true in power_state (010-qemu.mk:190-195). Derived, from cloud-init's behaviour: a failing runcmd command does not stop the rest of the list or the power-off, so QEMU exits with status 0.
  • Leaving out the common template removes power_state: see start from the common template. Amazon Linux 2 ignores power_state and has its own shutdown: see what differs per distribution.
  • The Ubuntu Core rules run the same way, under bash with set -o pipefail (050-ubuntu-core.mk:37-53, 050-ubuntu-core.mk:77-106), so a QEMU that fails there fails the build too.

07 Where to go from here

ROADS Provisioning once The NoCloud seed what the boot reads Picking a template your own setup The hooks files and apt settings allocate a fresh snapshot per test Per-distribution quirks powering off by hand The serial console reading the log
Roads from this deck: the NoCloud seed, how a user-data template is chosen, the hooks in .image-garden.mk, allocate, what differs per distribution, and the serial console

Next: the seed, templates and hooks, allocate, the per-distribution quirks, and the serial console.

One boot that powers itself off, and a file that keeps cloud-init quiet from then on. The seed deck shows what that boot reads and how it gets there. To add your own setup to it, start with how a template is chosen, or with the simpler hooks. allocate is where the prepared image earns its keep, a fresh snapshot for every test. The quirks deck has the system that can't power itself off. And the console deck helps when you're reading that log.

Details and sources

Read more

Basis of the statements on this page

Read in rulegen.py, mk/000-garden.mk, mk/010-qemu.mk and image-garden of image-garden at the commit this map describes ("code"). When cloud-init runs each part is cloud-init's behaviour ("derived"). One run: a GNU make 4.4.1 test that a failing explicit %.qcow2 target is deleted under .DELETE_ON_ERROR despite the pattern .PRECIOUS.

The provisioning boot, in order

StepFromNotes
bootcmdpkgproxy's part (010-qemu.mk:156-162)only when a pkgproxy is reachable from guests
write_files, users, packagesthe templatepackages: only in a project template
runcmd, the common part010-qemu.mk:238-246ends with touch /etc/cloud/cloud-init.disabled
runcmd, a template's linesthe project or a quirkafter the touch; Amazon Linux 2's shutdown --poweroff now
power_state010-qemu.mk:190-195poweroff, delay: now, condition: true

After the boot

Seedcloud-initChanges kept
provisioning boot (make X.qcow2)attachedruns oncein X.qcow2
./X.runnot attachedoffin X.qcow2
image-garden allocatenot attachedoffno: -snapshot

When it goes wrong

What happensWhat make doesWhat to do
the guest never powers offwaits; there is no timeoutread X.qcow2.log; check that the template starts from the common one
a runcmd command failsfinishes; the image counts as made (derived)read X.qcow2.log; rm X.qcow2 and make again
QEMU exits non-zerofails the recipe (set -o pipefail) and deletes X.qcow2 (000-garden.mk:4-7)fix the cause and make again
make is interrupteddeletes X.qcow2 (derived)make again

Roads from here

RUNTIME · make rules and the CLI

FIRST BOOT · cloud-init changes