Try, trying, done
boot mechanism for Ubuntu Core developers for snapd developers
A new kernel or base is booted once under a status that moves from try to trying. The bootloader and the initramfs each check it and snapd clears it after a successful boot. If the status is still trying on the next boot the old revision is used and snapd reverts the snap.
On the map: ubuntu core boot
01 A new kernel that nobody can test first
A try boot uses one status with three values: empty, try and trying. snapd sets try, the boot code sets trying and starts the new kernel, and snapd clears it after a successful boot.
The gateway has just downloaded a new kernel. If that kernel can't boot this board, who's going to notice, and who's going to fix it? So the new kernel gets exactly one attempt, and a status with three values carries that attempt across the reboot. Empty means nothing is pending. snapd writes try. The boot code changes it to trying just before it starts the new kernel. After a good boot, snapd empties it again. Think of trying as a note on the door that says "gone in". If the note's still there next time, nobody came back out, so the old kernel boots.
02 Setting up the try, in a safe order
Order of writes for a kernel try on GRUB: add the kernel to current_kernels in modeenv, reseal, create try-kernel.efi, then set kernel_status=try.
Before the gateway reboots, snapd has four things to write. Think about what happens if the power fails between any two of them. First, the new kernel joins current_kernels, the list of trusted kernels in modeenv, and modeenv is written. Second, the disk keys are resealed, so either kernel can unlock the data. Only then does snapd create the try-kernel.efi symlink. And last of all, it sets the kernel_status to try. Until that final write, GRUB has no reason to look at the new kernel. By the time it does, everything that kernel needs is already in place.
03 The bootloader flips try to trying
GRUB turns try into trying, saves it and boots try-kernel.efi. Finding trying at boot means the previous attempt failed: the status is cleared and kernel.efi boots.
Now the gateway reboots, and the first judge of the new kernel is a few lines of GRUB script snapd wrote. GRUB reads the kernel_status. If it says try, GRUB changes it to trying, saves it, and only then starts try-kernel.efi. Suppose the new kernel hangs and the board restarts. GRUB finds trying, concludes that nobody cleared it, empties the status and boots the old kernel. Any other value is invalid and is emptied too. With nothing pending, the old kernel boots. Other boards flip try to trying in a script, in C or in the initramfs; only GRUB clears an invalid value.
04 The initramfs is the second judge
In the initramfs, snap-bootstrap expects trying for a try kernel and requires the kernel to be in current_kernels. Otherwise it reboots, or fails when there is no fallback.
The new kernel started. That's one test passed, but the gateway isn't taking its word for it yet. Inside it, snap-bootstrap chooses which kernel snap to mount. With a try kernel, it expects the status to be trying, and the kernel must be in current_kernels, the trusted list in modeenv. If both hold, the boot goes on. A wrong status, or a try snap that's missing, and it reboots, so GRUB's trying branch falls back. An untrusted kernel with an older one to fall back to also means a reboot. With no fallback, it stops. So a kernel that reaches snapd has passed two separate checks.
05 snapd says done
MarkBootSuccessful runs once per boot in run mode. For the kernel it clears kernel_status, repoints kernel.efi, removes try-kernel.efi and trims current_kernels, in that order.
The gateway is up on the new kernel and snapd has started. So when, exactly, does the try become permanent? Right now. In run mode, the device manager marks the boot successful, once per boot. For the kernel that's four writes. First it empties the kernel_status, so a restart from here on is an ordinary boot. Then it points kernel.efi at the new kernel, and removes the try symlink. Last, current_kernels is trimmed to the one in use. The same update settles the base, boot assets, command line, recovery systems and model. Success means snapd started. It isn't a health check of your apps.
06 When the try fails
After a failed try the old kernel is running. UpdateBootRevisions compares the booted revision with snapd's state and queues a change, "Update kernel and core snap revisions", that reverts the snap. The refresh change ends in Error: "cannot finish pc-kernel installation, there was a rollback across reboot".
Let's rerun that boot, and this time the new kernel hangs halfway. The gateway restarts. What does snapd find when it comes back? GRUB saw trying and booted the old kernel, 2901. But snapd's state still says 2950 is current. Right after marking the boot, snapd compares the revision that actually booted with its state, for the kernel and the base. In snap changes the refresh ends in Error, and a new change, "Update kernel and core snap revisions", puts the old one back. The old kernel is already what boots, so the revert doesn't set up another try.
07 The base is tried without the bootloader
A base try is recorded in modeenv: try_base and base_status. The initramfs changes try to trying and writes modeenv; finding trying, it resets the status and uses the old base.
A few weeks later the gateway gets a new base. Same idea, same three words, but GRUB can't help this time. A bootloader loads kernels, not root file systems, so it can't judge a base. So snapd writes the try into modeenv: try_base is the new core twenty-four, and base_status is try. The initramfs reads try, writes trying back, and mounts the new base as the root. After a good boot, snapd promotes it. If the initramfs meets trying, it resets the status and uses the old base. The gadget has no try; its update keeps the old and new hashes of its boot files instead.
08 Where to go from here
Roads: reboots, GRUB, the other bootloaders, resealing, revert and the overview.
Three words carry a kernel and a base across a reboot. Here's where the pieces around them are explained. The reboots deck covers the restart between try and trying, and what waits for it. The GRUB deck shows the symlinks, and why a kernel that won't load costs an extra reboot. The bootloaders deck says who flips the status on other boards. Sealing explains the reseal before the try. The remove and revert deck has revert as a command you type. And the lighthouse takes you back to Ubuntu Core.
Read more
Statuses
| Value | Written by | Meaning |
|---|---|---|
| (empty) | snapd after a successful boot; boot code after a failed one | nothing is being tried |
try | snapd, before the reboot | boot the try kernel or base once |
trying | the bootloader (kernel) or the initramfs (base) | the try has been started; if seen again at boot, it failed |
Where the state lives
| What | Kernel on GRUB | Kernel on other bootloaders | Base |
|---|---|---|---|
| status | kernel_status in the ubuntu-boot grubenv | kernel_status in the bootloader environment | base_status in modeenv |
| current | the kernel.efi symlink | snap_kernel | base in modeenv |
| candidate | the try-kernel.efi symlink | snap_try_kernel | try_base in modeenv |
| trusted list | current_kernels in modeenv | current_kernels in modeenv | not used |
Who flips try to trying on each bootloader is listed in Bootloaders.
modeenv
The file is /var/lib/snapd/modeenv: key=value lines without a section header. Unknown keys are preserved when it is rewritten.
| Key | Meaning |
|---|---|
mode | run, install, recover or factory-reset |
recovery_system | seed label; used on first boot to take snapd from the seed |
current_recovery_systems | recovery systems that are tested or being tried |
good_recovery_systems | recovery systems that passed their test |
base | the base snap file in use |
try_base | the base being tried |
base_status | empty, try or trying |
gadget | the active gadget snap file |
current_kernels | the kernels the initramfs will accept |
model, grade, model_sign_key_id, classic | the device's model |
try_model, try_grade, try_model_sign_key_id | the candidate model during a remodel |
boot_flags | flags for the next boot |
current_trusted_boot_assets | asset name to one or two hashes |
current_trusted_recovery_boot_assets | the same for the ubuntu-seed bootloader |
current_kernel_command_lines | one or two expected command lines |
Order of writes
Setting up a kernel try (GRUB):
1. add the kernel to current_kernels and write modeenv; 2. reseal; 3. create try-kernel.efi; 4. set kernel_status=try.
Committing a successful boot:
1. set kernel_status to empty; 2. repoint kernel.efi; 3. remove try-kernel.efi; 4. trim current_kernels.
On bootloaders that keep the kernel in variables, each of the two is one write of the environment.
Initramfs verdicts for a try kernel
| Situation | Result |
|---|---|
status is trying, try kernel present and trusted | boot continues with the try kernel |
| status is wrong, or the try snap is missing | reboot |
chosen kernel not in current_kernels, a fallback exists | reboot |
chosen kernel not in current_kernels, no fallback | failure |
After a failed try
snapd marks the boot successful for the kernel that did boot, then compares the booted kernel and base revisions with its state. Where they differ it queues the change "Update kernel and core snap revisions", which reverts the snap to the booted revision. The booted revision is already current for the bootloader, so the revert does not set up another try.
The refresh change itself ends in Error with "cannot finish pc-kernel installation, there was a rollback across reboot" (overlord/snapstate/snapstate.go:396-397). In snap changes the gateway shows both: the refresh in Error, and the "Update kernel and core snap revisions" change after it.
A boot unlocked with the recovery key is still marked successful (FIXME at overlord/devicestate/devicemgr.go:1335-1337).
Ubuntu Core 16 and 18
There is no modeenv. One bootloader environment holds snap_mode (the status), snap_core and snap_try_core, snap_kernel and snap_try_kernel, and the boot script of the gadget does the flip.
Further reading
- Remodel essential snaps: shows a modeenv file and describes the test boot of a new base and kernel.
- Snaps in Ubuntu Core: the requirement that updates survive broken and interrupted attempts.
- Manage updates:
snap changesandsnap revert, the visible side of a rollback. - Calculate partition sizes: why ubuntu-boot needs room for one extra kernel.
Roads from here
RUNNING SNAPS
- Remove and revert — revert as a user command
UBUNTU CORE · boot
- Reboots and snap reboot — when the reboot for a try happens and what waits for it
- From power-on to snapd — where modeenv is read during boot
- Ubuntu Core: how a device boots and recovers — back to the Ubuntu Core overview
UBUNTU CORE · bootloaders
- GRUB on Ubuntu Core — the symlinks and the fallback entry on GRUB
- Bootloaders: one interface, six implementations — who flips the status on each bootloader
UBUNTU CORE · full disk encryption
- Sealed keys, boot chains and resealing — why the keys are resealed before the try