Bootloaders: one interface, six implementations

the shared mechanism for Ubuntu Core developers for snapd developers

snapd talks to every bootloader through one small interface for reading and writing boot variables and placing kernels. Six implementations are probed in a fixed order. Only GRUB has trusted assets and a managed configuration and command line which is why only GRUB supports TPM-backed encryption.

On the map: ubuntu core bootloaders

▶ Watch this deck — 8 scenes, ~6 min

01 Three things snapd needs from any bootloader

THE CONTRACT gateway · GRUBPi 4 · pibootArm board · U-BootDragonboard · lkone snapd, four kinds of boot code, one kernel refresh remember variablesGetBootVarsSetBootVars hold a kernelExtractKernelAssetsRemoveKernelAssets be set up from a gadgetInstallBootConfigRequiredByGadget Bootloader: those six, plus Name and Present, eight methods in all everything else is optional, and the options are where the bootloaders differgrub · piboot · uboot · ubootpart · lk · androidboot
Four boards, a GRUB gateway, a Raspberry Pi with piboot, an Arm board with U-Boot and a Dragonboard with lk, all need the same three things from snapd's Bootloader interface: remember variables, hold a kernel, and be set up from a gadget; with Name and Present that is eight methods, and everything else is optional

The Bootloader interface has eight methods: get and set boot variables, extract and remove kernel assets, install the boot config, name, presence and whether a gadget requires it. Optional interfaces describe everything else.

Put four boards on the bench: an x86 gateway, a Raspberry Pi, an Arm board with U-Boot and a Dragonboard. Their boot code has nothing in common, and snapd has to refresh a kernel on all four. So what does snapd need from a bootloader? Not much. It needs it to remember a few named variables. It needs to put a kernel's boot files where the bootloader can load them, and take them away again. And it needs to set the bootloader up from the gadget. Add a name and a presence check, and that's eight methods. Everything else is optional, and that's where the six implementations part ways.

02 Which bootloader is this?

FINDING IT ask the disk ask the gadget bootloader.Find1 ubootpart2 uboot3 grub4 androidboot5 lk6 piboot 3 grubPresent: yesthe first yes wins present but broken: "found but not usable",the search stops; a later one does not win bootloader.ForGadgetgadget pcgrub.confempty marker file gadget.yaml bootloader:marker filegrubgrub.confu-bootuboot.confandroid-bootandroidboot.conflklk.confpibootpiboot.conf ubootpart has no marker; gadget.yaml saysbootloader: u-bootand a structure with role: system-boot-state
On the left, bootloader.Find asks ubootpart, uboot, grub, androidboot, lk and piboot in order whether they are present and grub answers yes first; a broken one stops the search. On the right, bootloader.ForGadget finds the marker file grub.conf in the gadget; the gadget.yaml values grub, u-boot, android-boot, lk and piboot map to the markers grub.conf, uboot.conf, androidboot.conf, lk.conf and piboot.conf, and ubootpart has no marker

Find probes ubootpart, uboot, grub, androidboot, lk and piboot in that order and takes the first one present. ForGadget looks for a marker file <name>.conf in the gadget; ubootpart is recognised from gadget.yaml.

snapd on the gateway has no setting that says this is a GRUB machine. It works it out each time. On a running device it asks each implementation whether it's present, in a fixed order, and the first yes wins. ubootpart is asked before U-Boot on purpose. If one is present but broken, the search stops with an error, so a later one can't win. When snapd starts from a gadget, to build an image or install, it looks for an empty marker file named after the implementation. It's uboot.conf, without the hyphen gadget.yaml uses. ubootpart has no marker; gadget.yaml alone identifies it.

03 Three roles

ROLES acme-gateway disk ubuntu-seed ubuntu-boot ubuntu-save · ubuntu-data recoverychooses the modeand the recovery system run-modechooses the kernel set up when the image is builtcreated in install moderefused at image time UC16 and UC18one bootloadersole $ snap debug boot-varsthe run-mode variables$ snap debug boot-vars --root-dir /run/mnt/ubuntu-seedthe recovery variables
The gateway's disk: the bootloader on ubuntu-seed has the recovery role and chooses the mode and recovery system; the one on ubuntu-boot has the run-mode role and chooses the kernel; the recovery one is set up at image build time, the run one in install mode; UC16 and UC18 had one bootloader in the sole role; snap debug boot-vars prints either set of variables

Roles: recovery on ubuntu-seed, run-mode on ubuntu-boot, and the sole bootloader of UC16/18. The run bootloader is created in install mode, not at image build time.

The gateway has two bootloaders, and they don't do the same job. The one on ubuntu-seed has the recovery role. It picks the mode and the recovery system. The one on ubuntu-boot has the run-mode role, and it decides which kernel to start. The recovery one is put in place when the image is built. The run one only appears in install mode, and the code refuses to set it up at image time. Older releases had a single bootloader, in the sole role. And snap debug boot-vars prints the run variables, or the recovery ones when you point it at the seed.

04 A dozen variables carry everything

BOOTENV recovery bootloader · ubuntu-seed run-mode bootloader · ubuntu-boot snapd_recovery_modesnapd_recovery_systemwhat to boot next try_recovery_systemrecovery_system_statusa recovery system on trial kernel_statusempty, try or trying snap_kernelsnap_try_kernelwhich kernels, on bootloaders without symlinks snapd_extra_cmdline_argssnapd_full_cmdline_argsthe kernel command line, GRUB only GRUB only, in each recovery system's own environment:snapd_recovery_kernel
Two panels of boot variables. Recovery: snapd_recovery_mode and snapd_recovery_system say what to boot next; try_recovery_system and recovery_system_status cover a recovery system on trial. Run: kernel_status is empty, try or trying; snap_kernel and snap_try_kernel name the kernels on bootloaders without symlinks; snapd_extra_cmdline_args and snapd_full_cmdline_args carry the command line on GRUB only. On GRUB each recovery system also has snapd_recovery_kernel

Recovery variables: snapd_recovery_mode, snapd_recovery_system, try_recovery_system, recovery_system_status. Run variables: kernel_status, snap_kernel, snap_try_kernel and the two command-line variables.

If you had to tell a bootloader what to do next using only a handful of words, which would you pick? snapd picked about a dozen. On the recovery side there's the mode, and the label of the recovery system to boot. Two more are used while a new recovery system is on trial. On the run side there's the kernel_status, which moves through empty, try and trying. on-bootloaders without symlinks, two more name the current and the try kernel. Two carry the kernel command line, and only GRUB uses them. On GRUB, each recovery system also names its kernel. The full table is below the deck.

05 Two ways to hold a kernel

KERNEL STATE GRUB · symlinksthe gatewaykernel.efi -> pc-kernel_2901.snap/kernel.efi 1 try-kernel.efi -> pc-kernel_2950.snap/…2 kernel_status=try separate writes, in a careful order everyone else · variablesPi 4, Arm board, Dragonboardsnap_kernel=pi-kernel_812.snap snap_try_kernel=pi-kernel_830.snapkernel_status=tryoneone SetBootVars call after a good boot, one more write:snap_kernel=pi-kernel_830.snapsnap_try_kernel= kernel_status="the ordering here doesn't matter, as the only actual state we mutate is writing the bootloader env vars"
Left, GRUB on the gateway: kernel.efi and try-kernel.efi are symlinks into the pc-kernel snaps, written first, then kernel_status=try, as separate writes in a careful order. Right, every other bootloader: snap_kernel, snap_try_kernel and kernel_status=try are set in one SetBootVars call, and after a good boot one more write sets snap_kernel and clears the other two

GRUB keeps the current and try kernels as symlinks (kernel.efi, try-kernel.efi) plus kernel_status. Other bootloaders use snap_kernel, snap_try_kernel and kernel_status, set in a single write. snapd picks the strategy by checking for ExtractedRunKernelImageBootloader.

Back to the bench. The same kernel refresh is about to land on all four boards, and snapd records it in one of two ways. GRUB keeps the current and the try kernel as two symlinks next to grub.cfg, and the status as a variable. So on the gateway a refresh is separate writes, and their order matters: the try symlink goes first, then the status. Every other bootloader keeps all three facts as variables. Setting up the try is one write of the environment, and so is committing it. The code notes that ordering doesn't matter there, because there's only one mutation.

06 Who flips try to trying?

THE FLIP gateway · GRUBsnapd's own grub.cfgpart of snapd Arm board · U-Bootthe gadget's boot script, boot.scrfrom the gadget Dragonboard · lka C function in the lk forkin the bootloader Pi 4 · pibootsnap-bootstrap, in the initramfspart of snapd kernel_status=tryset trying, boot the try kernelkernel_status=tryingthe try failed: clear it, boot the current kernelGRUB only: anything elseclearedon every board, the initramfs check of the try kernel runs on top ubootpart and androidboot: no boot-side code was found, so this deck does not say
Who changes kernel_status from try to trying on each board: snapd's own grub.cfg on the GRUB gateway, the gadget's boot.scr on the U-Boot Arm board, a C function in the lk fork on the Dragonboard, and snap-bootstrap in the initramfs on the Pi; every board flips try to trying and treats trying at boot as a failure, and only GRUB also clears an invalid value; for ubootpart and androidboot no boot-side code was found

GRUB: snapd's managed grub.cfg. U-Boot: the gadget's boot.scr. lk: C code in the bootloader. piboot: snap-bootstrap in the initramfs. Every board flips try to trying and treats trying at boot as a failure; only GRUB also clears an invalid value. The initramfs check of the try kernel runs on top on every board.

Someone has to change try into trying before the new kernel starts. On our four boards it's four different someones. On the gateway, it's the grub.cfg that snapd itself ships. On the Arm board, it's the gadget's boot.scr script. On the Dragonboard, it's C code compiled into the bootloader. The Pi's firmware can't run a script or write a variable, so snapd marks the try on the kernel command line, and snap-bootstrap flips it from the initramfs. Wherever it lives, try becomes trying, and trying at boot means it failed. For ubootpart and Android boot, no boot-side code was found.

07 Trusted assets, managed assets, and why only GRUB

ASSETS trusted assetshim · grubx64.efiexact content matters; snapd records its hash current_trusted_boot_assets={"grubx64.efi":["5d0e…","a71c…"]}two hashes while an update is in flight /var/lib/snapd/boot-assets/grub/grubx64.efi-<hash>a copy of every trusted version; trimmed after a good boot managed assetgrub.cfgsnapd writes it; a gadget update leaves it alone TrustedAssetsBootloadergrubyespibootnoubootnoubootpartnolknoandroidbootno boot chains for TPM sealinga managed kernel command lineGRUB only
A trusted asset such as shim or grubx64.efi has its hash recorded in modeenv, with two hashes while an update is in flight, and a copy of every version under /var/lib/snapd/boot-assets; a managed asset such as grub.cfg is written by snapd and left alone by gadget updates; only grub implements TrustedAssetsBootloader, so boot chains for TPM sealing and a managed command line are GRUB only

Trusted assets are boot binaries whose hashes snapd tracks in modeenv and caches under /var/lib/snapd/boot-assets; managed assets are files snapd writes itself. Only GRUB is a TrustedAssetsBootloader, so only GRUB has boot chains and a managed command line.

Three of the four boards can't do TPM-backed disk encryption. Even with a TPM, the bootloader would decide. A trusted asset is a boot binary whose exact content matters, like shim or GRUB. When a gadget refresh replaces one, snapd records old and new hashes in modeenv, keeps a copy of each version, and after a good boot drops the hash it no longer needs. A managed asset is a file snapd writes itself, like grub.cfg, and a gadget update leaves it alone. Only GRUB has either. So only on GRUB can snapd describe the chain for sealing keys to the TPM, and manage the command line.

08 Where to go from here

ROADS BootloadersGRUBtwo stages, managed config Raspberry Pipiboot and tryboot U-Bootboot.sel or a raw partition Little Kerneland Android boot Try, trying, doneand rollbackKernel command linemanaged on GRUBSealed keysboot chains, resealing Ubuntu Corethe overview
Roads from bootloaders: GRUB on Ubuntu Core, the Raspberry Pi with piboot and tryboot, U-Boot, Little Kernel and Android boot, then try and rollback, the kernel command line, sealed keys and boot chains, and the Ubuntu Core overview

Roads: one deck per bootloader, then try and rollback, the command line, boot chains and the overview.

Each board on the bench has its own deck, so pick the one on your desk. The GRUB deck has two stages and a configuration that snapd manages. The Raspberry Pi deck is about a bootloader that can't write anything itself. The U-Boot deck has a script from the gadget and an environment that can live in a raw partition. The Little Kernel deck has no file system at all. Then three decks follow the mechanisms: try, trying and rollback; the kernel command line; and sealed keys with their boot chains. Or go back to the Ubuntu Core overview.

Read more

The implementations

snapd namegadget.yaml bootloader:Marker in the gadgetUsed on
grubgrubgrub.confx86-64 and arm64 UEFI
pibootpiboot (UC20 onwards)piboot.confRaspberry Pi, firmware loads the kernel
ubootu-bootuboot.confU-Boot with the environment in a file
ubootpartu-boot plus a structure with role: system-boot-statenoneU-Boot with the environment in a raw partition
lklklk.confLittle Kernel (aboot) on Qualcomm boards
androidbootandroid-bootandroidboot.confAndroid boot; not usable as a UC20+ recovery bootloader

Probe order at runtime: ubootpart, uboot, grub, androidboot, lk, piboot. ForGadget walks the same list, so a gadget with uboot.conf and a system-boot-state structure gets ubootpart.

Optional interfaces

InterfaceMeaninggrubpibootubootubootpartlkandroidboot
RecoveryAwareBootloaderper-recovery-system environmentyesnonononono
ExtractedRecoveryKernelImageBootloaderrecovery kernels are unpackednoyesyesyesimage time onlyno
ExtractedRunKernelImageBootloaderrun kernels as symlinksyesnonononono
TrustedAssetsBootloadertrusted assets, managed config, command line, boot chainsyesnonononono
UefiBootloaderEFI boot entriesyes (recovery role)nonononono
NotScriptableBootloaderthe initramfs flips the statusnoyesnononono
RebootBootloaderneeds an argument to the reboot callnoyesnononono
RecoveryBootConfigBootloaderconfig can be rebuilt after a gadget updatenoyesnononono

A recovery bootloader must unpack recovery kernels or be recovery-aware; otherwise: "cannot use %s bootloader: does not support recovery systems".

Boot variables

VariableRoleMeaning
snapd_recovery_moderecoverymode of the next boot; unset means install
snapd_recovery_systemrecoverylabel of the recovery system to use
snapd_recovery_kernelper recovery system, GRUB onlythe kernel snap of that system; other bootloaders unpack the kernel into the system directory
try_recovery_systemrecoverylabel of a recovery system being tested
recovery_system_statusrecoverytry, then tried
snapd_good_recovery_systemsrecovery, GRUB onlysystems that passed their test
snapd_boot_flagsrecoveryflags for the next boot
kernel_statusrunempty, try or trying
snap_kernelruncurrent kernel, on bootloaders without symlinks
snap_try_kernelrunkernel being tried, on bootloaders without symlinks
snapd_extra_cmdline_argsrun and per recovery system, GRUB onlyarguments appended to the static command line
snapd_full_cmdline_argsrun and per recovery system, GRUB onlya complete replacement of the static command line
snap_mode, snap_core, snap_try_coresole (UC16/18)status and base of the older scheme

snap debug boot-vars prints them; --root-dir /run/mnt/ubuntu-seed selects the recovery bootloader.

Who changes try to trying

BootloaderWhoWhere
GRUB, UC20+snapd's managed grub.cfgpart of snapd
GRUB, UC16/18the gadget's grub.cfg (snap_mode)pc gadget
pibootsnap-bootstrap in the initramfspart of snapd
U-Boot, UC20the gadget's boot.scrbuilt from flash-kernel's Raspberry Pi boot script
U-Boot, UC16/18snappy_boot in the gadget's built-in environment (snap_mode)pi gadget
ubootparta gadget boot scriptnot determined: no reference script was found
lk, UC20a C function in the lk forksnap_get_target_boot_params
lk, UC16/18a patch to aboot in the gadget (snap_mode)dragonboard gadget
androidbootnot determinedno boot code was found

Where the environment is kept, and how it is protected

BootloaderStorageIntegrity
GRUBgrubenv, a 1024-byte filenone
pibootpiboot.conf, U-Boot environment formatCRC32
ubootboot.sel or uboot.env fileCRC32
ubootpartraw partition ubuntu-boot-stateCRC32 and a second copy
lkraw partitions snapbootsel, snaprecoveryselCRC32 and a backup partition
androidbootandroidboot.env, a text filenone

Trusted and managed assets

  • Trusted assets are named in current_trusted_boot_assets (run bootloader) and current_trusted_recovery_boot_assets (recovery bootloader) in modeenv, each with one or two SHA3-384 hashes.
  • Every trusted version is copied to /var/lib/snapd/boot-assets/<bootloader>/<name>-<hash>.
  • After a successful boot the list is trimmed to the hash that booted. An unexpected hash is an error only when the device is encrypted.
  • A managed asset is replaced only when snapd's built-in copy has a higher Snapd-Boot-Config-Edition.

Further reading

Roads from here

UBUNTU CORE · boot

UBUNTU CORE · bootloaders

UBUNTU CORE · install and recovery

UBUNTU CORE · full disk encryption