From power-on to snapd
boot mechanism for Ubuntu Core developers for administrators
A run-mode boot passes through the bootloader on ubuntu-seed and the one on ubuntu-boot and then the kernel whose initramfs runs snap-bootstrap. It reads the mode from the kernel command line and mounts the four partitions under /run/mnt and unlocks the data and picks the base and kernel from modeenv before the real system starts.
On the map: ubuntu core boot
01 A relay with one message
Run-mode boot on a GRUB device: firmware, shim, GRUB on ubuntu-seed, GRUB on ubuntu-boot, kernel.efi, snap-bootstrap in the initramfs, then the system and snapd. What the firmware and shim do is outside snapd, which only knows where their files are.
The power has just come back at the customer's site, and the gateway is starting on its own. Let's watch what it does before snapd is running. It's a relay, and the runners pass on two things: which mode to start, and which snaps to use. The firmware starts shim. Shim starts a GRUB on the seed partition, and that GRUB hands over to a second one on the boot partition. The second GRUB starts kernel.efi: a kernel and its initramfs in one file. Inside that initramfs, snap-bootstrap puts the disk together under /run/mnt. Then the real system starts, and snapd with it.
02 Two GRUBs, one variable
The GRUB on ubuntu-seed reads snapd_recovery_mode from its grubenv; unset means install. For run it chainloads the GRUB on ubuntu-boot, which starts kernel.efi with snapd_recovery_mode=run.
Something people don't expect: the gateway runs two GRUBs on every boot, and the first one lives on the recovery partition. That GRUB reads two variables from its environment file: the mode and the recovery system. If the mode isn't set, it assumes install: that's how a blank device installs itself. Any other mode boots a kernel from a recovery system. For run, it chainloads the GRUB binary on the boot partition. That second GRUB reads its own environment, picks kernel.efi or the try kernel, and starts it with snapd_recovery_mode=run, fixed text in its config. Even the most ordinary boot passes through the seed.
03 The mode arrives on the command line
snap-initramfs-mounts.service runs snap-bootstrap initramfs-mounts, which takes the mode from snapd_recovery_mode on the kernel command line. On failure the initramfs reboots after ten seconds. With dangerous on the command line it stays where it stopped, so a developer can look around.
The kernel is up, but there's no root file system yet. All the initramfs knows about what to do next is one argument on the command line. A systemd unit runs snap-bootstrap initramfs-mounts, and early on it reads snapd_recovery_mode. An empty value means install. A value it doesn't know is an error. And in run mode, a recovery system label is ignored. If anything fails, the unit drops to the emergency target, and a second unit reboots the gateway ten seconds later. On a device nobody visits, starting again beats sitting stuck.
04 Four partitions, mounted in order
Run mode mounts ubuntu-boot, measures the model, mounts ubuntu-seed, unlocks ubuntu-data with the sealed key from ubuntu-boot, then unlocks ubuntu-save with the key stored in ubuntu-data and checks that the two belong together.
Now the gateway has to put its disk together, and the order matters, because each partition holds what the next one needs. First the boot partition, checked with fsck and mounted at /run/mnt/ubuntu-boot. snap-bootstrap reads the model file from it and measures it into the TPM before using it. The seed comes next, mounted so nothing on it can be executed. Then data, unlocked with the sealed key from the boot partition, and mounted at /run/mnt/data. Then save, whose key is an ordinary file inside data. Last, the markers on data and save are compared, so a save partition from another install is refused.
05 The model is checked by the lock
The run-mode model file is measured before use and counted as verified only once ubuntu-data has been unlocked, because the key was sealed against the expected model. Without encryption no cryptographic check happens at this step.
Did you notice that the gateway read its model from a partition that isn't encrypted? So why is it allowed to believe it? Because of the order. snap-bootstrap measured that file into the TPM before using it. The data key was sealed against the expected model, among other things. A different model gives a different measurement, and the TPM keeps the key. Only after data opens does the code mark the model as verified, and it won't finish the boot without a verified model. On a device without encryption, nothing checks the model at this point. How the key is sealed is a story for the encryption decks.
06 modeenv picks the snaps
snap-bootstrap reads modeenv and selects the base, gadget and kernel. The base becomes the root through a generated sysroot.mount; snapd is taken from the seed only on first boot, while modeenv still names a recovery system.
The data partition is open. It holds plenty of snap files, and the gateway needs three of them to become a system. The modeenv file is how snapd talks to the next boot. snap-bootstrap reads it and picks the base, the gadget and the kernel. The base isn't mounted directly: it's written out as sysroot.mount, the unit for the future root. The gadget is mounted, and so is the kernel. The kernel must be one of those listed here, which is what the try deck builds on. And only on the first boot does snapd come from the seed.
07 Handing over to the real system
The initramfs binds /run/mnt/data as the writable area and adds binds such as /boot/grub (from ubuntu-boot) and /boot/efi (ubuntu-seed), then systemd switches root. The mounts under /run/mnt remain.
Everything is mounted, but it's all still inside the initramfs. The last step is to hand it over without dropping anything. snap-bootstrap reloads systemd, so the new mount units count. The data partition is bound in as writable. A small script then adds bind mounts to the system's fstab. That's why, on the running gateway, /boot/grub is really a folder on the boot partition, and /boot/efi is the seed. Then systemd switches root in its usual way, and the partitions stay visible under /run/mnt. Once snapd is up, it marks the boot as successful, and that's where the try and rollback deck begins.
08 Where to go from here
Roads: the other modes, try and rollback, GRUB, other bootloaders, unlocking, the command line, first boot and the Ubuntu Core overview.
That was an ordinary morning. The interesting days are the ones where something on this path is new or broken. The modes deck follows the same path through install, recover and factory-reset. Try and rollback shows how a new kernel or base gets its first boot. The GRUB deck takes the two stages apart, and the bootloaders deck shows what changes before the kernel on other boards. Unlocking at boot has the keys you saw in passing. The command line deck says where the rest of the kernel's arguments come from. The seed deck adds first boot. And the overview takes you back to Ubuntu Core.
Read more
Run-mode steps in snap-bootstrap
1. Mount ubuntu-boot, with fsck. 2. Read device/model from ubuntu-boot and measure it. 3. Mount ubuntu-seed with nosuid, nodev, noexec and fsck. On classic systems with modes it is optional. 4. Update the bootloader's kernel status from the initramfs. Only piboot needs this. 5. Unlock ubuntu-data with the sealed key from ubuntu-boot (a recovery key is accepted) and mount it at /run/mnt/data. On core26 and later the key data is kept in LUKS2 tokens instead of the key file (boot/makebootable.go:398-447). 6. Mark the model as verified. 7. Unlock ubuntu-save with the key file stored in ubuntu-data and compare the pairing markers of data and save. 8. Write unlocked.json. 9. Read modeenv and select the base, gadget and kernel. The selected kernel must be listed in current_kernels. 10. Write sysroot.mount for the base (24+ initramfs). 11. Mount the gadget at /run/mnt/gadget, and the kernel at /run/mnt/kernel unless a drivers tree exists. 12. On first boot only, while modeenv still names a recovery system, take snapd from the seed. 13. daemon-reload and start initrd-root-fs.target again.
Without encryption, step 5 mounts ubuntu-data directly and step 6 still marks the model as verified; no cryptographic check of the model happens in run mode.
The /run/mnt map
| Path | What it is |
|---|---|
/run/mnt/ubuntu-boot | the ubuntu-boot partition |
/run/mnt/ubuntu-seed | the ubuntu-seed partition |
/run/mnt/ubuntu-save | the ubuntu-save partition |
/run/mnt/data | ubuntu-data in run mode; a tmpfs in install, recover and factory-reset |
/run/mnt/host/ubuntu-data | the real ubuntu-data in recover mode |
/run/mnt/ubuntu-data | the install target in install mode |
/run/mnt/gadget | the gadget snap |
/run/mnt/kernel | the kernel snap, when there is no drivers tree |
/run/mnt/ubuntu-boot/device/fde | the sealed run key |
/run/mnt/ubuntu-seed/device/fde | the sealed fallback keys |
The root file system directory is /run/mnt/data/system-data on Ubuntu Core and /run/mnt/data on a classic system in run mode.
Binds in the running system (run mode)
| Source | Appears as |
|---|---|
/run/mnt/data | /writable |
/run/mnt/host | /host (every mode) |
/run/mnt/ubuntu-boot/EFI/ubuntu | /boot/grub (GRUB) |
/run/mnt/ubuntu-seed | /boot/efi (GRUB) |
/run/mnt/ubuntu-boot/uboot/ubuntu | /boot/uboot (U-Boot) |
/run/mnt/ubuntu-seed/piboot/ubuntu | /boot/piboot (Raspberry Pi) |
How the mode is read
| Kernel command line | Result |
|---|---|
snapd_recovery_mode=run | run mode; a snapd_recovery_system label is ignored |
snapd_recovery_mode= (empty) | install mode |
snapd_recovery_mode=install without a system label | error |
| an unknown mode | error |
| a system label without a mode | error |
| neither argument | "cannot detect mode nor recovery system to use" |
The two GRUB stages
| Stage | Reads from its grubenv | Starts |
|---|---|---|
| GRUB on ubuntu-seed | snapd_recovery_mode, snapd_recovery_system | for run, EFI/boot/grubx64.efi on ubuntu-boot; otherwise kernel.efi from a recovery system's kernel snap |
| GRUB on ubuntu-boot | kernel_status, snapd_extra_cmdline_args, snapd_full_cmdline_args | kernel.efi or try-kernel.efi with snapd_recovery_mode=run |
On x86 the static arguments of the run GRUB are console=ttyS0,115200n8 console=tty1 panic=-1.
for Ubuntu Core developers for snapd developers
Initramfs units
| Unit | Role |
|---|---|
snap-initramfs-mounts.service | runs snap-bootstrap initramfs-mounts; after sysinit.target, before initrd-root-device.target; bound to dev-disk-snapd-disk.device; failure goes to emergency.target |
udev rule 90-ubuntu-core-partitions.rules | calls snap-bootstrap scan-disk to create /dev/disk/snapd/* |
emergency-reboot.service | sleeps 10 s and reboots unless dangerous is on the command line |
sysroot-writable.mount | binds /run/mnt/data |
populate-writable.service | runs handle-writable-paths, the-modeenv and extra-paths |
The boot disk is found from the partition UUID that the firmware reports for the booted kernel, with the file system label as the fallback.
Further reading
- Inside Ubuntu Core: the responsibilities of snap-bootstrap, in order.
- Storage layout: which bootloader stage lives on which partition.
- Full disk encryption: the verify, measure and measure-the-model sequence.
- Kernel boot parameters:
snapd.debugandsnapd_system_disk, for watching a boot. - Test on QEMU: QEMU command lines, with and without an emulated TPM.
Roads from here
UBUNTU CORE · boot
- Run, install, recover, factory reset — what the same path does in install recover and factory reset
- Try, trying, done — how the kernel and base named in modeenv are tried
- The kernel command line — where the rest of the command line comes from
- Ubuntu Core: how a device boots and recovers — back to the Ubuntu Core overview
UBUNTU CORE · bootloaders
- GRUB on Ubuntu Core — the two GRUB stages in detail
- Bootloaders: one interface, six implementations — the same hand-over on other bootloaders
UBUNTU CORE · install and recovery
- ubuntu-seed and first boot — what first boot adds to this path
UBUNTU CORE · full disk encryption
- Unlocking at boot — how ubuntu-data and ubuntu-save are unlocked