Run, install, recover, factory reset

boot mechanism for administrators for Ubuntu Core developers

In install and recover and factory-reset modes the initramfs boots a recovery system from ubuntu-seed into a temporary file system. Recover mode tries each partition in turn and records what it could unlock and copies credentials from the installed system only when it can trust it.

On the map: ubuntu core boot

▶ Watch this deck — 8 scenes, ~6 min

01 The same disk, a different system

MODES acme-gateway diskubuntu-seedubuntu-bootubuntu-saveubuntu-dataerrors run: the installed system a recovery system, running from a tmpfspicked by its label, for example 20260301installrecoverfactory-reset kernel command linesnapd_recovery_mode=recover snapd_recovery_system=20260301 in a model, a snap's modes list may say ephemeral: all three of these
The gateway's disk: ubuntu-data feeds run mode, the installed system; ubuntu-seed feeds a recovery system running from a tmpfs in install, recover or factory-reset mode, chosen by two kernel command line arguments

Install, recover and factory-reset boot a recovery system from ubuntu-seed into a tmpfs. The mode and the system label arrive as snapd_recovery_mode and snapd_recovery_system. In a model, ephemeral means every mode except run.

The gateway's data partition has started throwing errors. Rebooting it into the same system would just show you the same errors, so let's start it as something else. Run mode boots the installed system, from data. The other three modes, install, recover and factory-reset, don't start it at all. They start a recovery system from the seed partition, picked by its label, and run it from a temporary file system in memory. Which mode, and which label, arrive as two arguments on the kernel command line. And when a model lists the modes a snap should be present in, the word ephemeral covers all three.

02 What the three modes share

COMMON START 1 · mount ubuntu-seedubuntu-seed 2 · load the system, verify its snapssystems/20260301/snapdkernelbasegadgetmetadata checked against assertions 3 · measure its model, mark it verifiedTPMmodel measured, verified 4 · mount kernel and gadget; base is rootthe temporary systemkernelgadgetbase (root) 5 · a tmpfs where ubuntu-data would be/run/mnt/data (tmpfs)skipped when the initramfs installs
Five steps shared by install, recover and factory-reset: mount ubuntu-seed, load the recovery system and verify its four essential snaps, measure the model into the TPM, mount kernel and gadget with the base as root, and put a tmpfs where ubuntu-data would be

Install, recover and factory-reset all mount ubuntu-seed, load and verify the chosen recovery system, measure its model, mount its kernel, gadget and base, and put a tmpfs at /run/mnt/data. Installing in the initramfs skips the tmpfs.

Whichever of the three modes you picked, the first minute looks the same. snap-bootstrap mounts the seed partition first, because everything else comes from it. It loads the recovery system named by the label and verifies the metadata of its four essential snaps: snapd, kernel, base and gadget. The model comes from the seed too, with its signatures, so it's measured into the TPM and counted as verified straight away. In run mode that waits until data unlocks. Then the kernel and gadget are mounted, and the base becomes the root. Where data would be, there's an empty tmpfs, unless the initramfs is about to install.

03 "Install: the initramfs side"

INSTALL install mode, after the common startan empty snapd_recovery_mode also means install snapd will install/run/mnt/data (tmpfs) a mount unit for the snapd snapgadget defaults applied, cloud-init kept outmodeenv: mode=install the temporary system startssnapd installs, then rebootsacme-gateway takes this lane the initramfs installsthe installation runs herenext boot set to runmode = runcontinues as a run bootno reboot in between which lane, and why: the Install mode deck
Install mode after the common start splits in two lanes: either the initramfs mounts a tmpfs, applies gadget defaults, writes a short modeenv and starts a temporary system where snapd installs, or the initramfs installs by itself, sets the next boot to run and continues as a run boot

In install mode the initramfs either mounts a tmpfs, applies gadget defaults and starts a temporary system for snapd to install from, or runs the installation itself, sets the next boot to run and continues as a run boot. The Install mode deck explains the choice.

A gateway straight from the factory has no data partition at all. An empty mode on the command line counts as install. After the common start, there are two lanes. In the first, a tmpfs goes where data would be, the snapd snap gets a mount unit, the gadget's defaults are applied with cloud-init kept out, and a short modeenv says install. Then the temporary system starts, and snapd installs. Our gateway goes this way. In the second, snap-bootstrap installs by itself, sets the next boot to run, and carries straight on as a run boot. The install deck explains the choice.

04 "Recover: look without touching"

RECOVER temporary system (tmpfs)the recovery system, as first installed the installed data/run/mnt/host/ubuntu-data mounted nosuid, no fsck user accounts and SSH keysnetwork config and machine-idcopied only if data is trusted: unlocked, and paired with ubuntu-save not trusted: nothing copied, safe defaults insteadconsole set-up switched off, so nobody at the keyboard can create a user snapd_recovery_mode=runalready set for the next boot, before the temporary system starts
Recover mode: the temporary system runs from a tmpfs; the installed data is mounted at /run/mnt/host/ubuntu-data; accounts, SSH keys, network configuration and machine-id are copied only when the data is trusted, otherwise safe defaults are written; the next boot is set to run mode

Recover mode mounts the installed data at /run/mnt/host/ubuntu-data. If that data is trusted it copies accounts, SSH keys and network configuration into the temporary system; otherwise it writes safe defaults. The next boot is always run mode.

Recover mode is the one you'll use when the gateway is sick. You get a clean temporary system, with the real data to one side, at /run/mnt/host/ubuntu-data, where you can look at it. It's mounted nosuid, and not checked first, since it could be corrupted. If that data is trusted, meaning it unlocked and is paired with save, your accounts, ssh-keys, network settings and machine-id are copied in, so you can log in as before. If not, nothing is copied, and safe defaults switch off the console set-up. And before the temporary system starts, the next boot is already set back to run.

05 Try everything, write down what worked

DEGRADED run mode: a failure stops the bootrecover mode: log it and keep going mount ubuntu-bootunlock ubuntu-datarun keyfallback keyoffered in one call mount ubuntu-dataunlock ubuntu-savekey from datafallback keyonly if the first fails mount ubuntu-savea failed step is logged, and the next one still runs unlocked.jsonalwaysdegraded.jsonwhen anything left the first choice: another key, a partition not mounted, an error data and save markers differ: ubuntu-data is recorded asmounted-untrusted
Recover mode walks five steps: mount ubuntu-boot, unlock ubuntu-data with the run key or the fallback key, mount ubuntu-data, unlock ubuntu-save with the key from ubuntu-data or its fallback key, mount ubuntu-save; it writes unlocked.json always and degraded.json when anything left the first choice

Recover mode walks: mount boot, unlock data (run key or fallback key), mount data, unlock save (key from data, then fallback), mount save. Anything off the first choice makes the boot degraded; the outcome is written to unlocked.json and degraded.json.

On our gateway the data partition is the suspect. When a partition won't open, run mode stops the boot. Recover mode is for broken devices, so it keeps going. It mounts boot. It unlocks data, with the run key or the fallback key, and mounts it. It unlocks save with the key kept in data, or failing that, save's fallback key. Then it mounts save. A failed step is logged, and the ladder moves on. Anything off the first choice makes the boot degraded. unlocked.json is always written, degraded.json only then. And if data and save carry different markers, data is recorded as untrusted.

06 "Factory reset: only ubuntu-save matters"

FACTORY RESET ubuntu-seedubuntu-bootubuntu-saveubuntu-dataafter the common start ubuntu-bootwill be wipednot looked forubuntu-datawill be wipednot looked for ubuntu-savefallback key, from ubuntu-seedsave's run key lives in ubuntu-data unlocked, mounted/run/mnt/ubuntu-save unlocked.jsononly · console set-up disabled · short modeenv · temporary system starts then snapd resets the device and keeps its identity on ubuntu-save
Factory-reset mode: ubuntu-boot and ubuntu-data are not examined because they will be wiped; ubuntu-save is unlocked with the fallback key from ubuntu-seed and mounted; unlocked.json is written, the console set-up is disabled and snapd performs the reset

Factory-reset mode unlocks only ubuntu-save, with the fallback key, because ubuntu-boot and ubuntu-data are rebuilt. It writes unlocked.json and starts a temporary system in which snapd performs the reset.

Suppose recover mode brought bad news, and the data can't be saved. factory-reset is next, and its start is surprisingly short. After the common start, snap-bootstrap doesn't even look for boot or data. They'll be wiped anyway, as the code puts it, and they may be the broken part. It goes straight to save and opens it with the fallback key from seed, because save's run key lives inside the data being thrown away. It writes unlocked.json, switches off the console set-up, writes a short modeenv and starts the temporary system. There, snapd does the reset; the factory-reset deck has that part.

07 How snapd and snaps know the mode for Ubuntu Core developers for snapd developers

INSIDE THE MODE ephemeral modeenvmode=recoverrecovery_system=20260301base=core24_1587.snapgadget=pc_211.snapmodel=acme/acme-gatewaygrade=secured six keys; no kernels or boot assets, unlike run mode snapd reads the mode and acts on itfor example: start the install or factory reset change $ snapctl system-modesystem-mode: recoverseed-loaded: truestorage-encrypted: managedfrom any snap's hook or app
The ephemeral modeenv has six keys: mode, recovery_system, base, gadget, model and grade; snapctl system-mode reports recover, seed-loaded true and storage-encrypted managed

Ephemeral modes write a short modeenv: mode, recovery system, base, gadget, model and grade. A snap can ask with snapctl system-mode.

Inside the temporary system, how does anything know this isn't an ordinary boot? The initramfs wrote it a modeenv, a short one: the mode, the recovery system label, the base, the gadget, the model and the grade. No kernel list, no boot assets; those belong to run mode. snapd reads the mode and acts on it, for example by starting the install or factory-reset change. A snap's hook or app can ask too, with snapctl system-mode. On our gateway it says recover, seed-loaded, and storage-encrypted: managed.

08 Where to go from here

ROADS Run, install, recover, reset Reboots and snap reboothow you get into a mode Recovery systemswhat these modes boot Install modethe installation itselfFactory resetubuntu-save and remodel Unlocking at bootthe keys on the ladder From power-on to snapdthe run-mode path Ubuntu Corethe overview
Roads from the boot modes: reboots and snap reboot, recovery systems, install mode, factory reset and remodel, unlocking at boot, the boot path from power-on to snapd, and the Ubuntu Core overview

Roads: requesting a mode, recovery systems, install, factory reset, unlocking, the run-mode path and the overview.

You now know what each mode does in its first minute. Each one continues in a deck of its own. To get the gateway into a mode, with snap reboot or the boot menu, go to reboots. Recovery systems covers the labels on seed that these modes boot. Install mode follows the installation itself. The factory-reset deck shows what a reset keeps on save, and how the device stays itself. Unlocking at boot has the keys from the ladder. The boot path deck is the run mode path that these modes branch from. And the overview puts the whole of Ubuntu Core on one page.

Read more

What each mode mounts

Modeubuntu-seedubuntu-bootubuntu-dataubuntu-save/run/mnt/data
runmountedmountedunlocked and mountedunlocked and mountedubuntu-data
installmountednot mounted; created by the installationnot mounted; created by the installationnot mounted; created by the installationtmpfs, or the new data when installing in the initramfs
recovermountedmounted if possibleat /run/mnt/host/ubuntu-data if possiblemounted if possibletmpfs
factory-resetmountednot examinednot examinedunlocked with the fallback keytmpfs

In install, recover and factory-reset modes the kernel and gadget of the recovery system are mounted from ubuntu-seed and its base becomes the root. The mode comes from snapd_recovery_mode on the kernel command line; an empty value means install. The label comes from snapd_recovery_system.

Recover mode: the order of steps

1. Mount ubuntu-boot, with fsck. 2. Unlock ubuntu-data: one call offers the run key and the fallback key from ubuntu-seed; a recovery key is accepted. 3. Mount ubuntu-data at /run/mnt/host/ubuntu-data, nosuid, without fsck. 4. Unlock ubuntu-save with the key stored in ubuntu-data. 5. Unlock ubuntu-save with the fallback key, if step 4 was not possible or failed. 6. Mount ubuntu-save.

A step that fails is logged and the next possible step runs; if ubuntu-data cannot be unlocked or mounted, the walk goes straight to step 5. Unencrypted devices take a branch that opens ubuntu-save directly, and ubuntu-save may be absent there. While a new recovery system is being tried, fallback keys are not used and the boot always ends in a reboot. Which key is which is explained in Unlocking at boot.

When a recover boot is "degraded"

  • ubuntu-save is not mounted (on an unencrypted device: not mounted and not absent);
  • a key other than the run key was used;
  • ubuntu-boot or ubuntu-data is not mounted;
  • any error was logged.

If the pairing markers of ubuntu-data and ubuntu-save differ, data is recorded as mounted-untrusted.

unlocked.json and degraded.json

Both are written to snap-bootstrap's run directory. degraded.json is written only in recover mode and only when the boot is degraded. Factory-reset mode writes only unlocked.json.

KeyFields
ubuntu-bootmount-state, mount-location
ubuntu-datamount-state, mount-location, unlock-state, unlock-key
ubuntu-savemount-state, mount-location, unlock-state, unlock-key
statethe activation state reported by the unlocking library; may be absent

unlock-key is run, fallback or recovery. The mount fields are not provided in run mode.

What recover mode copies from trusted data

  • user accounts (extrausers), SSH host keys, sudoers.d, and each user's .ssh, .profile and snap login;
  • the user records in snapd's state;
  • netplan configuration and machine-id;
  • the saved clock of the time synchronisation service.

When data is not trusted nothing is copied, and console-conf is marked complete so that it does not run. Factory-reset mode marks console-conf complete as well.

for Ubuntu Core developers for snapd developers

The ephemeral modeenv

KeyExample
moderecover
recovery_system20260301
basecore24_1587.snap
gadgetpc_211.snap
modelacme/acme-gateway
gradesecured

for Ubuntu Core developers for snapd developers

cloudimg-rootfs

A fifth mode, used only by Azure confidential VM images. snap-bootstrap mounts the EFI system partition as ubuntu-seed (with an optional EFI/ubuntu/manifest.json), provisions the TPM, mounts the partitions, writes sysroot.mount from /run/mnt/data, unmounts the EFI system partition and uses a built-in generic model instead of a model assertion (cmd/snap-bootstrap/cmd_initramfs_mounts_cvm.go). Unlocking at boot covers it.

Further reading

Roads from here

UBUNTU CORE · boot

UBUNTU CORE · install and recovery

UBUNTU CORE · full disk encryption