Try, trying, done

boot mechanism for Ubuntu Core developers for snapd developers

A new kernel or base is booted once under a status that moves from try to trying. The bootloader and the initramfs each check it and snapd clears it after a successful boot. If the status is still trying on the next boot the old revision is used and snapd reverts the snap.

On the map: ubuntu core boot

▶ Watch this deck — 8 scenes, ~6 min

01 A new kernel that nobody can test first

TRY pc-kernel_2901.snapknown goodpc-kernel_2950.snapnew kernel_status=nothing pending trysnapd: boot the new one, once tryingboot code: I have started it snapd, after a good boot: it worked still trying at the next boot: it did not work
Two kernel snaps, the known good pc-kernel_2901.snap and the new pc-kernel_2950.snap, above a status strip with three values: empty, try and trying. snapd empties trying after a good boot; trying seen again at the next boot means the old kernel boots

A try boot uses one status with three values: empty, try and trying. snapd sets try, the boot code sets trying and starts the new kernel, and snapd clears it after a successful boot.

The gateway has just downloaded a new kernel. If that kernel can't boot this board, who's going to notice, and who's going to fix it? So the new kernel gets exactly one attempt, and a status with three values carries that attempt across the reboot. Empty means nothing is pending. snapd writes try. The boot code changes it to trying just before it starts the new kernel. After a good boot, snapd empties it again. Think of trying as a note on the door that says "gone in". If the note's still there next time, nobody came back out, so the old kernel boots.

02 Setting up the try, in a safe order

BEFORE THE REBOOT 1 · trust itmodeenvcurrent_kernels=pc-kernel_2901.snap,pc-kernel_2950.snap 2 · reseal the keysubuntu-data and ubuntu-saveTPMsealed for both kernels 3 · point at itubuntu-boottry-kernel.efi -> pc-kernel_2950.snap/kernel.efi 4 · arm itgrubenvkernel_status=try a power cuthere bootsthe old kernelas if nothinghappened
Four writes before the reboot, in order: add pc-kernel_2950.snap to current_kernels in modeenv, reseal the keys in the TPM for both kernels, create the try-kernel.efi symlink, and set kernel_status=try in grubenv. A power cut after any of the first three boots the old kernel

Order of writes for a kernel try on GRUB: add the kernel to current_kernels in modeenv, reseal, create try-kernel.efi, then set kernel_status=try.

Before the gateway reboots, snapd has four things to write. Think about what happens if the power fails between any two of them. First, the new kernel joins current_kernels, the list of trusted kernels in modeenv, and modeenv is written. Second, the disk keys are resealed, so either kernel can unlock the data. Only then does snapd create the try-kernel.efi symlink. And last of all, it sets the kernel_status to try. Until that final write, GRUB has no reason to look at the new kernel. By the time it does, everything that kernel needs is already in place.

03 The bootloader flips try to trying

FIRST JUDGE if [ "$kernel_status" = "try" ]set kernel_status="trying"save_env kernel_statusset kernel=try-kernel.efisave trying, then boottry-kernel.efi elif [ "$kernel_status" = "trying" ]set kernel_status=""; save_env ...the last try failed: clear, boot kernel.efi elif [ -n "$kernel_status" ]set kernel_status=""; save_env ...invalid: clear it, boot kernel.efi finothing pending: kernel=kernel.efiboot kernel.efi other boards flip try to trying in a script, in C or in the initramfs; only GRUB clears an invalid value
The three branches of the managed grub.cfg: on try it saves trying and boots try-kernel.efi; on trying it clears the status and boots kernel.efi; any other value is invalid and is cleared; with nothing pending kernel.efi boots. Other boards flip try to trying too; only GRUB clears an invalid value

GRUB turns try into trying, saves it and boots try-kernel.efi. Finding trying at boot means the previous attempt failed: the status is cleared and kernel.efi boots.

Now the gateway reboots, and the first judge of the new kernel is a few lines of GRUB script snapd wrote. GRUB reads the kernel_status. If it says try, GRUB changes it to trying, saves it, and only then starts try-kernel.efi. Suppose the new kernel hangs and the board restarts. GRUB finds trying, concludes that nobody cleared it, empties the status and boots the old kernel. Any other value is invalid and is emptied too. With nothing pending, the old kernel boots. Other boards flip try to trying in a script, in C or in the initramfs; only GRUB clears an invalid value.

04 The initramfs is the second judge

SECOND JUDGE snap-bootstrap, in the new kernel status is trying?yes: the try kernel in current_kernels?yes: mount it status wrong or try snap missingreboot not trusted, a fallback existsreboot not trusted, no fallbackstopafter a reboot, GRUB finds trying and boots the old kernel
snap-bootstrap in the new kernel checks that the status is trying and that the kernel is in current_kernels before mounting it; a wrong status or missing try snap reboots, an untrusted kernel with a fallback reboots, and an untrusted kernel without a fallback stops the boot

In the initramfs, snap-bootstrap expects trying for a try kernel and requires the kernel to be in current_kernels. Otherwise it reboots, or fails when there is no fallback.

The new kernel started. That's one test passed, but the gateway isn't taking its word for it yet. Inside it, snap-bootstrap chooses which kernel snap to mount. With a try kernel, it expects the status to be trying, and the kernel must be in current_kernels, the trusted list in modeenv. If both hold, the boot goes on. A wrong status, or a try snap that's missing, and it reboots, so GRUB's trying branch falls back. An untrusted kernel with an older one to fall back to also means a reboot. With no fallback, it stops. So a kernel that reaches snapd has passed two separate checks.

05 snapd says done

DONE snapd startsensureBootOkMarkBootSuccessfulrun mode only · once per boot 1kernel_status=grubenv 2kernel.efi -> pc-kernel_2950.snap/kernel.efiubuntu-boot3remove try-kernel.efiubuntu-boot 4current_kernels=pc-kernel_2950.snapmodeenv the same commit also settles the base, boot assets, command line, recovery systems and model success means snapd started, not that your apps are healthy
snapd starts, ensureBootOk calls MarkBootSuccessful once per boot in run mode, which clears kernel_status, points kernel.efi at pc-kernel_2950.snap, removes try-kernel.efi and trims current_kernels, in that order

MarkBootSuccessful runs once per boot in run mode. For the kernel it clears kernel_status, repoints kernel.efi, removes try-kernel.efi and trims current_kernels, in that order.

The gateway is up on the new kernel and snapd has started. So when, exactly, does the try become permanent? Right now. In run mode, the device manager marks the boot successful, once per boot. For the kernel that's four writes. First it empties the kernel_status, so a restart from here on is an ordinary boot. Then it points kernel.efi at the new kernel, and removes the try symlink. Last, current_kernels is trimmed to the one in use. The same update settles the base, boot assets, command line, recovery systems and model. Success means snapd started. It isn't a health check of your apps.

06 When the try fails

ROLLBACK booted: pc-kernel_2901.snap state: pc-kernel 2950 is current UpdateBootRevisionsbooted revision differs from state: revert to the booted one $ snap changesID Status Spawn Ready Summary14 Error today today Refresh "pc-kernel" snap15 Doing today - Update kernel and core snap revisions state: pc-kernel 2901 is currentthe old kernel already boots, so no new try is set up
After a failed try the device booted pc-kernel_2901.snap while snapd's state says revision 2950 is current; UpdateBootRevisions sees the difference. snap changes shows the refresh, change 14, in Error and a new change 15, Update kernel and core snap revisions, which reverts the state to 2901

After a failed try the old kernel is running. UpdateBootRevisions compares the booted revision with snapd's state and queues a change, "Update kernel and core snap revisions", that reverts the snap. The refresh change ends in Error: "cannot finish pc-kernel installation, there was a rollback across reboot".

Let's rerun that boot, and this time the new kernel hangs halfway. The gateway restarts. What does snapd find when it comes back? GRUB saw trying and booted the old kernel, 2901. But snapd's state still says 2950 is current. Right after marking the boot, snapd compares the revision that actually booted with its state, for the kernel and the base. In snap changes the refresh ends in Error, and a new change, "Update kernel and core snap revisions", puts the old one back. The old kernel is already what boots, so the revert doesn't set up another try.

07 The base is tried without the bootloader

BASE bootloader: loads kernels, not root file systems 1 · snapdbase=core24_1587.snaptry_base=core24_1612.snapbase_status=try 2 · initramfsbase=core24_1587.snaptry_base=core24_1612.snapbase_status=tryingroot = core24_1612.snap 3 · snapd, after a good bootbase=core24_1612.snaptry_base=base_status= initramfs, next boot, still tryingbase=core24_1587.snaptry_base=core24_1612.snapbase_status=root = core24_1587.snap the gadget has no try; its boot files are tracked by hash
Four states of modeenv during a base try: snapd sets try_base and base_status=try; the initramfs writes trying and mounts core24_1612.snap; after a good boot snapd makes it the base; if the initramfs finds trying it resets the status and mounts core24_1587.snap. A bootloader loads kernels, not root file systems. The gadget has no try; its boot files are tracked by hash

A base try is recorded in modeenv: try_base and base_status. The initramfs changes try to trying and writes modeenv; finding trying, it resets the status and uses the old base.

A few weeks later the gateway gets a new base. Same idea, same three words, but GRUB can't help this time. A bootloader loads kernels, not root file systems, so it can't judge a base. So snapd writes the try into modeenv: try_base is the new core twenty-four, and base_status is try. The initramfs reads try, writes trying back, and mounts the new base as the root. After a good boot, snapd promotes it. If the initramfs meets trying, it resets the status and uses the old base. The gadget has no try; its update keeps the old and new hashes of its boot files instead.

08 Where to go from here

ROADS Try, trying, doneReboots and snap rebootbetween try and trying GRUB on Ubuntu Coresymlinks and the fallback reboot Bootloaderswho flips the status elsewhere Sealed keysthe reseal before the try Remove and revertsnap revert Ubuntu Coreback to the overview
Roads from Try, trying, done: reboots, GRUB on Ubuntu Core, bootloaders, sealed keys and resealing, removing and reverting, and the Ubuntu Core overview

Roads: reboots, GRUB, the other bootloaders, resealing, revert and the overview.

Three words carry a kernel and a base across a reboot. Here's where the pieces around them are explained. The reboots deck covers the restart between try and trying, and what waits for it. The GRUB deck shows the symlinks, and why a kernel that won't load costs an extra reboot. The bootloaders deck says who flips the status on other boards. Sealing explains the reseal before the try. The remove and revert deck has revert as a command you type. And the lighthouse takes you back to Ubuntu Core.

Read more

Statuses

ValueWritten byMeaning
(empty)snapd after a successful boot; boot code after a failed onenothing is being tried
trysnapd, before the rebootboot the try kernel or base once
tryingthe bootloader (kernel) or the initramfs (base)the try has been started; if seen again at boot, it failed

Where the state lives

WhatKernel on GRUBKernel on other bootloadersBase
statuskernel_status in the ubuntu-boot grubenvkernel_status in the bootloader environmentbase_status in modeenv
currentthe kernel.efi symlinksnap_kernelbase in modeenv
candidatethe try-kernel.efi symlinksnap_try_kerneltry_base in modeenv
trusted listcurrent_kernels in modeenvcurrent_kernels in modeenvnot used

Who flips try to trying on each bootloader is listed in Bootloaders.

modeenv

The file is /var/lib/snapd/modeenv: key=value lines without a section header. Unknown keys are preserved when it is rewritten.

KeyMeaning
moderun, install, recover or factory-reset
recovery_systemseed label; used on first boot to take snapd from the seed
current_recovery_systemsrecovery systems that are tested or being tried
good_recovery_systemsrecovery systems that passed their test
basethe base snap file in use
try_basethe base being tried
base_statusempty, try or trying
gadgetthe active gadget snap file
current_kernelsthe kernels the initramfs will accept
model, grade, model_sign_key_id, classicthe device's model
try_model, try_grade, try_model_sign_key_idthe candidate model during a remodel
boot_flagsflags for the next boot
current_trusted_boot_assetsasset name to one or two hashes
current_trusted_recovery_boot_assetsthe same for the ubuntu-seed bootloader
current_kernel_command_linesone or two expected command lines

Order of writes

Setting up a kernel try (GRUB):

1. add the kernel to current_kernels and write modeenv; 2. reseal; 3. create try-kernel.efi; 4. set kernel_status=try.

Committing a successful boot:

1. set kernel_status to empty; 2. repoint kernel.efi; 3. remove try-kernel.efi; 4. trim current_kernels.

On bootloaders that keep the kernel in variables, each of the two is one write of the environment.

Initramfs verdicts for a try kernel

SituationResult
status is trying, try kernel present and trustedboot continues with the try kernel
status is wrong, or the try snap is missingreboot
chosen kernel not in current_kernels, a fallback existsreboot
chosen kernel not in current_kernels, no fallbackfailure

After a failed try

snapd marks the boot successful for the kernel that did boot, then compares the booted kernel and base revisions with its state. Where they differ it queues the change "Update kernel and core snap revisions", which reverts the snap to the booted revision. The booted revision is already current for the bootloader, so the revert does not set up another try.

The refresh change itself ends in Error with "cannot finish pc-kernel installation, there was a rollback across reboot" (overlord/snapstate/snapstate.go:396-397). In snap changes the gateway shows both: the refresh in Error, and the "Update kernel and core snap revisions" change after it.

A boot unlocked with the recovery key is still marked successful (FIXME at overlord/devicestate/devicemgr.go:1335-1337).

Ubuntu Core 16 and 18

There is no modeenv. One bootloader environment holds snap_mode (the status), snap_core and snap_try_core, snap_kernel and snap_try_kernel, and the boot script of the gadget does the flip.

Further reading

Roads from here

RUNNING SNAPS

UBUNTU CORE · boot

UBUNTU CORE · bootloaders

UBUNTU CORE · full disk encryption