Ubuntu Core: how a device boots and recovers

overview for everyone for Ubuntu Core developers for administrators

An Ubuntu Core device has four partitions and boots in one of a few modes. This deck follows one device from installation through refreshes that can roll back to recovery and factory reset and points to the deck for each step.

On the map: ubuntu core boot

▶ Watch this deck — 8 scenes, ~6 min

01 A device nobody visits

UBUNTU CORE the gatewayin a locked cabinet, no keyboard kernelgadgetbasesnapdappsthe whole system is snaps 1 · an update that fails rolls back 2 · a spare system stays on the disk 3 · the data opens only on this device this deck: the disk, the modes, one boot, one lifetime
A gateway in a locked cabinet with no keyboard. Everything on it is a snap: kernel, gadget, base, snapd and apps. Three properties follow: an update that fails rolls back, a spare system stays on the disk, and the data opens only on this device

On Ubuntu Core the kernel, gadget, base, snapd and apps are all snaps. The boot design gives three properties: updates that roll back, a recovery system kept on disk, and optional encryption tied to the device.

Picture a small gateway bolted inside a cabinet at a customer's site. Nobody's going to walk up to it with a keyboard for years, so if an update goes wrong, nobody's coming. That's why everything on it is a snap that snapd-controls: the kernel, the gadget with the boot assets and device set-up, the base, snapd itself, and the apps. Three promises follow. A failed kernel or base update rolls back by itself. A spare system stays on the disk to repair or rebuild the device. And with encryption, the data opens only on this device. The rest of this deck shows where each promise comes from.

02 Four partitions, four jobs

THE DISK ubuntu-seedrecovery bootloaderrecovery systems ubuntu-bootrun bootloaderkernels ubuntu-savedevice identitysurvives a factory reset ubuntu-datasnaps, system data,your data encryptedalways readable: read before any unlock snapd writes to ubuntu-seed rarely:to set the next boot mode and to add recovery systems
A disk split into four partitions: ubuntu-seed with the recovery bootloader and recovery systems, ubuntu-boot with the run bootloader and kernels, ubuntu-save with the device identity that survives a factory reset, and ubuntu-data with everything else. Only ubuntu-save and ubuntu-data are encrypted; seed and boot stay readable, and snapd writes to seed rarely

ubuntu-seed: recovery bootloader and recovery systems. ubuntu-boot: run bootloader and kernels. ubuntu-save: device identity, kept across a factory reset. ubuntu-data: everything else. Only save and data are encrypted.

Open the gateway's disk and you find four partitions. Each one exists because of what has to survive what. ubuntu-seed holds the first bootloader and the recovery systems, each a complete set of snaps plus the device's model. ubuntu-boot holds the second bootloader and the kernels. ubuntu-save keeps what must outlive a factory-reset, like the device's identity. ubuntu-data holds everything else. With encryption on, only save and data are encrypted, because the device has to read seed and boot before it unlocks anything. And seed isn't read-only: snapd writes there now and then, to set the next boot mode or add a recovery system.

03 One device, several modes

MODES runruns from ubuntu-dataerases nothing recovera recovery system,in memoryerases nothing factory-resetrebuilds boot and datakeeps ubuntu-save installbuilds the device froma recovery systemerases save too $ sudo snap reboot --recoverReboot into "recover" mode.or, at the device itself:hold the 1 key on a keyboardwhile it starts(console-conf's chooser menu) a fifth mode, cloudimg-rootfs, exists only for confidential virtual machines in a cloud
Four modes as cards. run runs from ubuntu-data and erases nothing. recover runs a recovery system in memory and erases nothing. factory-reset rebuilds boot and data and keeps ubuntu-save. install builds the device from a recovery system and erases save too. snap reboot --recover picks the mode; holding the 1 key on a keyboard while it starts opens console-conf's chooser menu. A fifth mode, cloudimg-rootfs, exists for confidential cloud VMs

run: the normal system. recover: a temporary system in memory; data untouched. factory-reset: boot and data rebuilt, ubuntu-save kept. install: everything rebuilt, ubuntu-save erased. snap reboot or the chooser picks the mode. The 1-key menu is console-conf's chooser, so it needs console-conf on the device and a keyboard attached.

Most days the gateway starts in run mode and you don't think about it. The other modes are for the bad days. Recover starts a temporary system in memory, from a recovery system, and leaves your data alone, so you can look at what went wrong. factory-reset rebuilds boot and data but keeps save, so the device keeps its identity. Install builds everything again and erases save too. You pick a mode with snap reboot, or, if someone's standing there with a keyboard, by holding the 1 key while it starts. There's a fifth mode as well, but it's only for confidential virtual machines in a cloud.

04 From power-on to snapd

ONE BOOT firmwarepower on recoverybootloaderon ubuntu-seedmode = run runbootloaderon ubuntu-bootpicks the kernel kernel +initramfssnap-bootstrapunlocks and mounts snapdmarks the boot good any other mode: a recovery system's kernelfrom ubuntu-seed; ubuntu-boot is skippeddrawn as GRUB runs it; the bootloader decks show how the others differ
A chain of five stages: firmware, the recovery bootloader on ubuntu-seed which reads the mode, the run bootloader on ubuntu-boot which picks the kernel, the kernel and initramfs where snap-bootstrap unlocks and mounts, and snapd which marks the boot good. In any other mode the recovery bootloader starts a recovery system's kernel from ubuntu-seed instead

Firmware, recovery bootloader on ubuntu-seed, run bootloader on ubuntu-boot, the kernel with snap-bootstrap in its initramfs, then snapd. In any mode other than run the first bootloader starts a recovery system's kernel from ubuntu-seed.

Press the power button and the gateway runs a relay. Five runners, and the mode travels along with them. The firmware starts the recovery bootloader on ubuntu-seed, which reads the mode. For run mode it hands over to the run bootloader on ubuntu-boot, and that one picks the kernel. Inside the kernel's initramfs, snap-bootstrap unlocks the encrypted partitions, mounts everything and picks the base. Then the real system starts, and snapd confirms that the boot worked. In any other mode the first bootloader skips ubuntu-boot and starts a recovery system's kernel from seed instead.

05 The life of a device

ONE LIFETIME imagea model + snaps,on ubuntu-seed installpartitions made,keys sealed first bootsnaps seeded runrefresh · try · keep or roll back recoverthen back to runfactory resetback to first boot, identity kept remodela new model,same device
A timeline of one device: image, install, first boot, run with a loop of refresh, try, keep or roll back, and remodel. Below run, recover returns to run and factory reset returns to first boot with the identity kept

Image, install, first boot, run with refreshes that are tried and kept or rolled back, recover or factory reset when needed, and remodel to change what the device is.

Let's follow the gateway from the factory to the day it's retired. An image is a model and a set of snaps, written to ubuntu-seed. The first start is install mode: it makes the other partitions and, with encryption, seals the keys. First boot then seeds the snaps. After that the gateway lives in run mode, where each kernel or base refresh is tried once and kept only if that boot succeeds. On a bad day, recover lets you look around, and factory-reset starts over while keeping the identity. And a remodel moves the device to a new model without reinstalling it.

06 Three questions you can ask the device for users for administrators

ASK THE DEVICE $ snap modelbrand Acme Corp (acme)model Acme Gateway (acme-gateway)grade securedserial 0b7c7a2e-...what this device is $ snap recoveryLabel Brand Model Notes20260301 acme acme-gateway current20261112 acme acme-gateway default-recoveryits recovery systemson ubuntu-seed, by labelcurrent: the last one seeded $ sudo snap reboot --recoverReboot into "recover" mode.restarts at onceinto the default recovery system recover mode lasts one boot: the next restart returns to run mode
A terminal on acme-gateway. snap model prints brand, model, grade and serial. snap recovery lists 20260301 as current, the system last seeded, and 20261112 as default-recovery. sudo snap reboot --recover prints Reboot into recover mode and restarts on the default recovery system. Recover mode lasts one boot

snap model shows the device's model, snap recovery lists recovery systems, and snap reboot --recover restarts into recover mode, on the default recovery system unless a label is given. The next restart returns to run mode.

You don't need to open the cabinet to learn what the gateway is. Three commands tell you most of it. Snap model prints the brand, the model, and the serial that names this one device. Snap recovery lists the recovery systems on ubuntu-seed, by label. Current marks the one it was last seeded from, here the one it was installed from. The newer one, which you made the default, is marked default recovery. Then sudo snap reboot --recover, restarts into recover mode at once, using the default system unless you name another. Recover mode lasts one boot, so the next restart brings you back to run mode.

07 Data that opens only on this device

ENCRYPTION ubuntu-dataubuntu-saveencryptedencrypted bootloaderskernelcommand linemodelmeasured during this boot TPMchecks the match key releasedubuntu-dataholds the key for saveubuntu-save kernel, changedTPMno key releasedthe recovery key is the way in when snapd changes one of these pieces itself, it reseals the key first
ubuntu-data and ubuntu-save are encrypted. The bootloaders, kernel, command line and model are measured during boot; the TPM releases the key for ubuntu-data only if they match. ubuntu-data holds the key for ubuntu-save. If a piece was changed behind snapd's back the key stays sealed and the recovery key is the way in. snapd reseals before changing a piece itself

ubuntu-data and ubuntu-save can be encrypted. The key is unsealed only when the measured boot chain and model match; ubuntu-save's key is stored inside ubuntu-data. The full disk encryption district explains sealing, unlocking and recovery keys.

Suppose someone pulls the disk out of the gateway. What stops them from just reading it? If the hardware can do it and the model doesn't say otherwise, save and data are encrypted. The key for data is sealed so the TPM releases it only if the bootloaders, the kernel, the command line and the model measured during this boot match what snapd expected. Data opens first, and it holds the key for save. If something in that chain was changed behind snapd's back, the key stays sealed, and the recovery key is the way in. When snapd changes one of those pieces itself, it reseals first.

08 Where to go from here

ROADS Ubuntu Core BOOTFrom power-onto snapdmodestry and rollbackrebootskernel command line BOOTLOADERSOne interface,six implementationsGRUBRaspberry PiU-BootLittle Kernel INSTALL AND RECOVERYModel, gadgetand kernelseedinstallrecovery systemsfactory reset and remodel FULL DISK ENCRYPTIONWhat is encryptedand who decidessealingunlockinghookskeys
Four districts around Ubuntu Core. Boot starts at From power-on to snapd. Bootloaders starts at One interface, six implementations. Install and recovery starts at Model, gadget and kernel. Full disk encryption starts at What is encrypted and who decides

Four districts: boot, bootloaders, install and recovery, full disk encryption. Each starts with an overview deck and follows the life of a device.

That's the whole of Ubuntu Core from the outside. Now pick a door. For the boot itself, start with the walk from power-on to snapd, then the other modes, try and rollback, reboots and the kernel command line. The bootloader district starts with what snapd asks of any bootloader, then has a deck each for GRUB, the Raspberry Pi, U-Boot and Little Kernel. Install and recovery starts with the model and follows the device's life: the seed, install, recovery systems, factory-reset and remodel. Encryption starts with what's encrypted and who decides.

Read more

The four partitions

PartitionHoldsEncrypted
ubuntu-seedrecovery bootloader, its boot variables, recovery systems, and the snap files they shareno
ubuntu-bootrun bootloader, its boot variables, unpacked kernels, the sealed run keyno
ubuntu-savedevice key and serial backup, per-snap save data, encryption statewhen encryption is on
ubuntu-datasnaps, snapd state, system and user data, the key for ubuntu-savewhen encryption is on

An encrypted partition carries the label with -enc appended, for example ubuntu-data-enc. ubuntu-seed is mounted writable in run mode: snapd writes the next boot mode and new recovery systems there.

The modes

ModeStarted fromRoot file systemWhat is erased
runubuntu-boot kernelubuntu-datanothing
installa recovery systemtemporary, in memory (or installed and run in one boot)boot, data and save
recovera recovery systemtemporary, in memorynothing
factory-reseta recovery systemtemporary, in memoryboot and data; save and seed are kept
cloudimg-rootfsAzure confidential VM imagethe image's data partitionnothing

Recover mode lasts one boot: the next restart returns to run mode.

for Ubuntu Core developers for administrators for snapd developers

Where things are

PartitionGadget roleOn disk
ubuntu-seedsystem-seedrecovery systems under systems/<label>/, the shared snaps/ pool, the recovery bootloader's variables
ubuntu-bootsystem-bootthe run bootloader's variables, unpacked kernels, the sealed run key
ubuntu-savesystem-savedevice key and serial backup, per-snap save data
ubuntu-datasystem-datasnaps, snapd state, the key for ubuntu-save

The mode is the kernel argument snapd_recovery_mode; the recovery system is snapd_recovery_system. A present but empty mode means install. The modeenv is the file through which snapd and the initramfs agree on the mode, the base, the trusted kernels, the trusted boot assets and the kernel command lines.

Words used across the area

  • Recovery system: a set of snaps with a model assertion and snap assertions on ubuntu-seed, named by a label such as 20260301, from which the device can be installed, recovered or reset.
  • Try boot: one boot with a new kernel or base. snapd keeps it only after that boot succeeds.
  • snap-bootstrap: the program in the initramfs that finds the disk, unlocks and mounts the partitions and selects the snaps to boot.
  • Boot variables (bootenv): the few variables snapd stores where the bootloader can read them.

Commands

CommandWhat it shows or does
snap modelthe device's brand, model and serial, and its grade and storage safety when the model sets them
snap recoveryrecovery systems: Label, Brand, Model, Notes (current, default-recovery)
snap reboota plain reboot
snap reboot --recover [label]reboot into recover mode
snap reboot --factory-reset [label]reboot into factory-reset mode
snap reboot --install [label]reboot into install mode; erases the device
snap reboot --runreboot into run mode (current system only)

In snap recovery, current marks the system the device was last seeded from: the one it was installed or factory reset from, or remodelled to. default-recovery marks the system chosen as the default, for example when it was created with "mark default". Without a label, --recover, --factory-reset and --install use the default recovery system, or the current one if no default is set.

At the device itself, holding the 1 key on a keyboard for two seconds, while the trigger service waits for it at start-up (up to ten seconds), opens a menu of recovery systems and modes. The menu is console-conf's recovery chooser: it runs in the booted system, needs an input device, and needs console-conf on the device.

Further reading

Roads from here

START HERE

  • snapd — back to the overview

UBUNTU CORE · boot

UBUNTU CORE · bootloaders

UBUNTU CORE · install and recovery

UBUNTU CORE · full disk encryption