From power-on to snapd

boot mechanism for Ubuntu Core developers for administrators

A run-mode boot passes through the bootloader on ubuntu-seed and the one on ubuntu-boot and then the kernel whose initramfs runs snap-bootstrap. It reads the mode from the kernel command line and mounts the four partitions under /run/mnt and unlocks the data and picks the base and kernel from modeenv before the real system starts.

On the map: ubuntu core boot

▶ Watch this deck — 8 scenes, ~6 min

01 A relay with one message

POWER-ON passed along the chain: which mode to start · which snaps to use firmwareshimoutside snapd; it knows only their filesseed GRUBubuntu-seed boot GRUBubuntu-boot kernel.efikernel + initramfs in one file snap-bootstrapmounts the disk under /run/mnt systemd, then snapdthe real system
The run-mode boot chain of a GRUB device: firmware, shim, GRUB on ubuntu-seed, GRUB on ubuntu-boot, kernel.efi with its initramfs, snap-bootstrap mounting under /run/mnt, then systemd and snapd

Run-mode boot on a GRUB device: firmware, shim, GRUB on ubuntu-seed, GRUB on ubuntu-boot, kernel.efi, snap-bootstrap in the initramfs, then the system and snapd. What the firmware and shim do is outside snapd, which only knows where their files are.

The power has just come back at the customer's site, and the gateway is starting on its own. Let's watch what it does before snapd is running. It's a relay, and the runners pass on two things: which mode to start, and which snaps to use. The firmware starts shim. Shim starts a GRUB on the seed partition, and that GRUB hands over to a second one on the boot partition. The second GRUB starts kernel.efi: a kernel and its initramfs in one file. Inside that initramfs, snap-bootstrap puts the disk together under /run/mnt. Then the real system starts, and snapd with it.

02 Two GRUBs, one variable

STAGE ONE AND TWO ubuntu-seed ubuntu-boot EFI/ubuntu/grubenvsnapd_recovery_mode=run mode = run?unset: installany other mode: a recovery system's kernel run: chainloadEFI/boot/grubx64.efi EFI/ubuntu/grubenvkernel_status=picks kernel.efi or try-kernel.efi snapd_recovery_mode=runkernel.efi the mode is fixed text in the second grub.cfg; every run boot starts on ubuntu-seed
The GRUB on ubuntu-seed reads snapd_recovery_mode from its grubenv; unset means install, other modes boot a recovery system's kernel, and run chainloads the GRUB on ubuntu-boot, which reads kernel_status and starts kernel.efi with snapd_recovery_mode=run

The GRUB on ubuntu-seed reads snapd_recovery_mode from its grubenv; unset means install. For run it chainloads the GRUB on ubuntu-boot, which starts kernel.efi with snapd_recovery_mode=run.

Something people don't expect: the gateway runs two GRUBs on every boot, and the first one lives on the recovery partition. That GRUB reads two variables from its environment file: the mode and the recovery system. If the mode isn't set, it assumes install: that's how a blank device installs itself. Any other mode boots a kernel from a recovery system. For run, it chainloads the GRUB binary on the boot partition. That second GRUB reads its own environment, picks kernel.efi or the try kernel, and starts it with snapd_recovery_mode=run, fixed text in its config. Even the most ordinary boot passes through the seed.

03 The mode arrives on the command line

INITRAMFS $ cat /proc/cmdlinesnapd_recovery_mode=run console=ttyS0,115200n8 console=tty1 panic=-1 snap-initramfs-mounts.servicesnap-bootstrap initramfs-mounts runinstallrecoverfactory-resetcloudimg-rootfs empty value: installunknown value: errora system label in run mode: ignored any failure: emergency.target, then a reboot after 10 sdangerous on the command line: no reboot, the board stays where it stopped
The kernel command line carries snapd_recovery_mode=run; snap-initramfs-mounts.service runs snap-bootstrap initramfs-mounts, which switches on the mode: run, install, recover, factory-reset or cloudimg-rootfs; failure leads to the emergency target and a reboot after ten seconds unless dangerous is set

snap-initramfs-mounts.service runs snap-bootstrap initramfs-mounts, which takes the mode from snapd_recovery_mode on the kernel command line. On failure the initramfs reboots after ten seconds. With dangerous on the command line it stays where it stopped, so a developer can look around.

The kernel is up, but there's no root file system yet. All the initramfs knows about what to do next is one argument on the command line. A systemd unit runs snap-bootstrap initramfs-mounts, and early on it reads snapd_recovery_mode. An empty value means install. A value it doesn't know is an error. And in run mode, a recovery system label is ignored. If anything fails, the unit drops to the emergency target, and a second unit reboots the gateway ten seconds later. On a device nobody visits, starting again beats sitting stuck.

04 Four partitions, mounted in order

/RUN/MNT the disk mounted, in this order ubuntu-seed ubuntu-boot ubuntu-save ubuntu-data ubuntu-boot1 /run/mnt/ubuntu-bootfsck, then mount ubuntu-bootmodel read and measured ubuntu-seed2 /run/mnt/ubuntu-seednosuid, nodev, noexec ubuntu-datasealed key, from ubuntu-boot3 /run/mnt/dataubuntu-data, unlocked ubuntu-savekey: a file inside ubuntu-data4 /run/mnt/ubuntu-saveunlocked with that key file data and save carry matching markers; a mismatch stops the boot
The disk's four partitions and the /run/mnt mount points they get in run mode, in order: ubuntu-boot with fsck and the model measured, ubuntu-seed with nosuid, nodev and noexec, ubuntu-data unlocked with the sealed key from ubuntu-boot, and ubuntu-save unlocked with a key file from ubuntu-data

Run mode mounts ubuntu-boot, measures the model, mounts ubuntu-seed, unlocks ubuntu-data with the sealed key from ubuntu-boot, then unlocks ubuntu-save with the key stored in ubuntu-data and checks that the two belong together.

Now the gateway has to put its disk together, and the order matters, because each partition holds what the next one needs. First the boot partition, checked with fsck and mounted at /run/mnt/ubuntu-boot. snap-bootstrap reads the model file from it and measures it into the TPM before using it. The seed comes next, mounted so nothing on it can be executed. Then data, unlocked with the sealed key from the boot partition, and mounted at /run/mnt/data. Then save, whose key is an ordinary file inside data. Last, the markers on data and save are compared, so a save partition from another install is refused.

05 The model is checked by the lock

TRUST model file on ubuntu-bootnot encryptednot verified yet measureTPM unsealubuntu-datalockedthe key was sealed against the expected model, among other thingsa different model: a different measurement, and no key ubuntu-dataunlockedverified without encryption: no such checkdata opens anyway, and the model is marked verified all the same
The model file on ubuntu-boot is measured into the TPM; the TPM unseals the key for ubuntu-data only if the measurement matches; only then is the model counted as verified; without encryption there is no such check

The run-mode model file is measured before use and counted as verified only once ubuntu-data has been unlocked, because the key was sealed against the expected model. Without encryption no cryptographic check happens at this step.

Did you notice that the gateway read its model from a partition that isn't encrypted? So why is it allowed to believe it? Because of the order. snap-bootstrap measured that file into the TPM before using it. The data key was sealed against the expected model, among other things. A different model gives a different measurement, and the TPM keeps the key. Only after data opens does the code mark the model as verified, and it won't finish the boot without a verified model. On a device without encryption, nothing checks the model at this point. How the key is sealed is a story for the encryption decks.

06 modeenv picks the snaps

WHICH SNAPS $ cat /var/lib/snapd/modeenvmode=runbase=core24_1587.snapgadget=pc_211.snapcurrent_kernels=pc-kernel_2901.snap… core24_1587.snapsysroot.mountthe future root pc_211.snapmounted/run/mnt/gadget pc-kernel_2901.snapmountedunless a drivers tree exists current_kernels=pc-kernel_2901.snapthe booted kernel must be listed herethe try mechanism builds on this snapd, from the seedfirst boot only, while modeenvnames a recovery system
snap-bootstrap reads modeenv: the base core24_1587.snap becomes the root through sysroot.mount, the gadget pc_211.snap is mounted, the kernel pc-kernel_2901.snap is mounted unless a drivers tree exists and must be listed in current_kernels, and snapd comes from the seed on first boot only

snap-bootstrap reads modeenv and selects the base, gadget and kernel. The base becomes the root through a generated sysroot.mount; snapd is taken from the seed only on first boot, while modeenv still names a recovery system.

The data partition is open. It holds plenty of snap files, and the gateway needs three of them to become a system. The modeenv file is how snapd talks to the next boot. snap-bootstrap reads it and picks the base, the gadget and the kernel. The base isn't mounted directly: it's written out as sysroot.mount, the unit for the future root. The gadget is mounted, and so is the kernel. The kernel must be one of those listed here, which is what the try deck builds on. And only on the first boot does snapd come from the seed.

07 Handing over to the real system

PIVOT in the initramfs in the running system /run/mnt/datasysroot-writable.mount/writable a script, the-modeenv, adds binds to the system's fstab/run/mnt/ubuntu-boot/EFI/ubuntubind/boot/grub /run/mnt/ubuntu-seedbind/boot/efi switch rootsystemdsnapd still mounted under /run/mnt:ubuntu-boot ubuntu-seed data ubuntu-save boot marked successfulsee: try and rollback
Before switching root the initramfs binds /run/mnt/data as /writable and adds fstab binds of /run/mnt/ubuntu-boot/EFI/ubuntu at /boot/grub and /run/mnt/ubuntu-seed at /boot/efi; systemd switches root and starts snapd, which marks the boot successful; the partitions stay mounted under /run/mnt

The initramfs binds /run/mnt/data as the writable area and adds binds such as /boot/grub (from ubuntu-boot) and /boot/efi (ubuntu-seed), then systemd switches root. The mounts under /run/mnt remain.

Everything is mounted, but it's all still inside the initramfs. The last step is to hand it over without dropping anything. snap-bootstrap reloads systemd, so the new mount units count. The data partition is bound in as writable. A small script then adds bind mounts to the system's fstab. That's why, on the running gateway, /boot/grub is really a folder on the boot partition, and /boot/efi is the seed. Then systemd switches root in its usual way, and the partitions stay visible under /run/mnt. Once snapd is up, it marks the boot as successful, and that's where the try and rollback deck begins.

08 Where to go from here

ROADS From power-on to snapd The boot modesinstall, recover, reset Try and rollbacka new kernel or base GRUB on Ubuntu Corethe two stagesOther bootloadersbefore the kernel Unlocking at bootthe keysThe command linethe rest of the arguments First bootwhat the seed addsUbuntu Corethe overview
Roads from the boot path: the boot modes, try and rollback, GRUB, other bootloaders, unlocking at boot, the kernel command line, first boot and the Ubuntu Core overview

Roads: the other modes, try and rollback, GRUB, other bootloaders, unlocking, the command line, first boot and the Ubuntu Core overview.

That was an ordinary morning. The interesting days are the ones where something on this path is new or broken. The modes deck follows the same path through install, recover and factory-reset. Try and rollback shows how a new kernel or base gets its first boot. The GRUB deck takes the two stages apart, and the bootloaders deck shows what changes before the kernel on other boards. Unlocking at boot has the keys you saw in passing. The command line deck says where the rest of the kernel's arguments come from. The seed deck adds first boot. And the overview takes you back to Ubuntu Core.

Read more

Run-mode steps in snap-bootstrap

1. Mount ubuntu-boot, with fsck. 2. Read device/model from ubuntu-boot and measure it. 3. Mount ubuntu-seed with nosuid, nodev, noexec and fsck. On classic systems with modes it is optional. 4. Update the bootloader's kernel status from the initramfs. Only piboot needs this. 5. Unlock ubuntu-data with the sealed key from ubuntu-boot (a recovery key is accepted) and mount it at /run/mnt/data. On core26 and later the key data is kept in LUKS2 tokens instead of the key file (boot/makebootable.go:398-447). 6. Mark the model as verified. 7. Unlock ubuntu-save with the key file stored in ubuntu-data and compare the pairing markers of data and save. 8. Write unlocked.json. 9. Read modeenv and select the base, gadget and kernel. The selected kernel must be listed in current_kernels. 10. Write sysroot.mount for the base (24+ initramfs). 11. Mount the gadget at /run/mnt/gadget, and the kernel at /run/mnt/kernel unless a drivers tree exists. 12. On first boot only, while modeenv still names a recovery system, take snapd from the seed. 13. daemon-reload and start initrd-root-fs.target again.

Without encryption, step 5 mounts ubuntu-data directly and step 6 still marks the model as verified; no cryptographic check of the model happens in run mode.

The /run/mnt map

PathWhat it is
/run/mnt/ubuntu-bootthe ubuntu-boot partition
/run/mnt/ubuntu-seedthe ubuntu-seed partition
/run/mnt/ubuntu-savethe ubuntu-save partition
/run/mnt/dataubuntu-data in run mode; a tmpfs in install, recover and factory-reset
/run/mnt/host/ubuntu-datathe real ubuntu-data in recover mode
/run/mnt/ubuntu-datathe install target in install mode
/run/mnt/gadgetthe gadget snap
/run/mnt/kernelthe kernel snap, when there is no drivers tree
/run/mnt/ubuntu-boot/device/fdethe sealed run key
/run/mnt/ubuntu-seed/device/fdethe sealed fallback keys

The root file system directory is /run/mnt/data/system-data on Ubuntu Core and /run/mnt/data on a classic system in run mode.

Binds in the running system (run mode)

SourceAppears as
/run/mnt/data/writable
/run/mnt/host/host (every mode)
/run/mnt/ubuntu-boot/EFI/ubuntu/boot/grub (GRUB)
/run/mnt/ubuntu-seed/boot/efi (GRUB)
/run/mnt/ubuntu-boot/uboot/ubuntu/boot/uboot (U-Boot)
/run/mnt/ubuntu-seed/piboot/ubuntu/boot/piboot (Raspberry Pi)

How the mode is read

Kernel command lineResult
snapd_recovery_mode=runrun mode; a snapd_recovery_system label is ignored
snapd_recovery_mode= (empty)install mode
snapd_recovery_mode=install without a system labelerror
an unknown modeerror
a system label without a modeerror
neither argument"cannot detect mode nor recovery system to use"

The two GRUB stages

StageReads from its grubenvStarts
GRUB on ubuntu-seedsnapd_recovery_mode, snapd_recovery_systemfor run, EFI/boot/grubx64.efi on ubuntu-boot; otherwise kernel.efi from a recovery system's kernel snap
GRUB on ubuntu-bootkernel_status, snapd_extra_cmdline_args, snapd_full_cmdline_argskernel.efi or try-kernel.efi with snapd_recovery_mode=run

On x86 the static arguments of the run GRUB are console=ttyS0,115200n8 console=tty1 panic=-1.

for Ubuntu Core developers for snapd developers

Initramfs units

UnitRole
snap-initramfs-mounts.serviceruns snap-bootstrap initramfs-mounts; after sysinit.target, before initrd-root-device.target; bound to dev-disk-snapd-disk.device; failure goes to emergency.target
udev rule 90-ubuntu-core-partitions.rulescalls snap-bootstrap scan-disk to create /dev/disk/snapd/*
emergency-reboot.servicesleeps 10 s and reboots unless dangerous is on the command line
sysroot-writable.mountbinds /run/mnt/data
populate-writable.serviceruns handle-writable-paths, the-modeenv and extra-paths

The boot disk is found from the partition UUID that the firmware reports for the booted kernel, with the file system label as the fallback.

Further reading

Roads from here

UBUNTU CORE · boot

UBUNTU CORE · bootloaders

UBUNTU CORE · install and recovery

UBUNTU CORE · full disk encryption