Run, install, recover, factory reset
boot mechanism for administrators for Ubuntu Core developers
In install and recover and factory-reset modes the initramfs boots a recovery system from ubuntu-seed into a temporary file system. Recover mode tries each partition in turn and records what it could unlock and copies credentials from the installed system only when it can trust it.
On the map: ubuntu core boot
01 The same disk, a different system
Install, recover and factory-reset boot a recovery system from ubuntu-seed into a tmpfs. The mode and the system label arrive as snapd_recovery_mode and snapd_recovery_system. In a model, ephemeral means every mode except run.
The gateway's data partition has started throwing errors. Rebooting it into the same system would just show you the same errors, so let's start it as something else. Run mode boots the installed system, from data. The other three modes, install, recover and factory-reset, don't start it at all. They start a recovery system from the seed partition, picked by its label, and run it from a temporary file system in memory. Which mode, and which label, arrive as two arguments on the kernel command line. And when a model lists the modes a snap should be present in, the word ephemeral covers all three.
02 What the three modes share
Install, recover and factory-reset all mount ubuntu-seed, load and verify the chosen recovery system, measure its model, mount its kernel, gadget and base, and put a tmpfs at /run/mnt/data. Installing in the initramfs skips the tmpfs.
Whichever of the three modes you picked, the first minute looks the same. snap-bootstrap mounts the seed partition first, because everything else comes from it. It loads the recovery system named by the label and verifies the metadata of its four essential snaps: snapd, kernel, base and gadget. The model comes from the seed too, with its signatures, so it's measured into the TPM and counted as verified straight away. In run mode that waits until data unlocks. Then the kernel and gadget are mounted, and the base becomes the root. Where data would be, there's an empty tmpfs, unless the initramfs is about to install.
03 "Install: the initramfs side"
In install mode the initramfs either mounts a tmpfs, applies gadget defaults and starts a temporary system for snapd to install from, or runs the installation itself, sets the next boot to run and continues as a run boot. The Install mode deck explains the choice.
A gateway straight from the factory has no data partition at all. An empty mode on the command line counts as install. After the common start, there are two lanes. In the first, a tmpfs goes where data would be, the snapd snap gets a mount unit, the gadget's defaults are applied with cloud-init kept out, and a short modeenv says install. Then the temporary system starts, and snapd installs. Our gateway goes this way. In the second, snap-bootstrap installs by itself, sets the next boot to run, and carries straight on as a run boot. The install deck explains the choice.
04 "Recover: look without touching"
Recover mode mounts the installed data at /run/mnt/host/ubuntu-data. If that data is trusted it copies accounts, SSH keys and network configuration into the temporary system; otherwise it writes safe defaults. The next boot is always run mode.
Recover mode is the one you'll use when the gateway is sick. You get a clean temporary system, with the real data to one side, at /run/mnt/host/ubuntu-data, where you can look at it. It's mounted nosuid, and not checked first, since it could be corrupted. If that data is trusted, meaning it unlocked and is paired with save, your accounts, ssh-keys, network settings and machine-id are copied in, so you can log in as before. If not, nothing is copied, and safe defaults switch off the console set-up. And before the temporary system starts, the next boot is already set back to run.
05 Try everything, write down what worked
Recover mode walks: mount boot, unlock data (run key or fallback key), mount data, unlock save (key from data, then fallback), mount save. Anything off the first choice makes the boot degraded; the outcome is written to unlocked.json and degraded.json.
On our gateway the data partition is the suspect. When a partition won't open, run mode stops the boot. Recover mode is for broken devices, so it keeps going. It mounts boot. It unlocks data, with the run key or the fallback key, and mounts it. It unlocks save with the key kept in data, or failing that, save's fallback key. Then it mounts save. A failed step is logged, and the ladder moves on. Anything off the first choice makes the boot degraded. unlocked.json is always written, degraded.json only then. And if data and save carry different markers, data is recorded as untrusted.
06 "Factory reset: only ubuntu-save matters"
Factory-reset mode unlocks only ubuntu-save, with the fallback key, because ubuntu-boot and ubuntu-data are rebuilt. It writes unlocked.json and starts a temporary system in which snapd performs the reset.
Suppose recover mode brought bad news, and the data can't be saved. factory-reset is next, and its start is surprisingly short. After the common start, snap-bootstrap doesn't even look for boot or data. They'll be wiped anyway, as the code puts it, and they may be the broken part. It goes straight to save and opens it with the fallback key from seed, because save's run key lives inside the data being thrown away. It writes unlocked.json, switches off the console set-up, writes a short modeenv and starts the temporary system. There, snapd does the reset; the factory-reset deck has that part.
07 How snapd and snaps know the mode for Ubuntu Core developers for snapd developers
Ephemeral modes write a short modeenv: mode, recovery system, base, gadget, model and grade. A snap can ask with snapctl system-mode.
Inside the temporary system, how does anything know this isn't an ordinary boot? The initramfs wrote it a modeenv, a short one: the mode, the recovery system label, the base, the gadget, the model and the grade. No kernel list, no boot assets; those belong to run mode. snapd reads the mode and acts on it, for example by starting the install or factory-reset change. A snap's hook or app can ask too, with snapctl system-mode. On our gateway it says recover, seed-loaded, and storage-encrypted: managed.
08 Where to go from here
Roads: requesting a mode, recovery systems, install, factory reset, unlocking, the run-mode path and the overview.
You now know what each mode does in its first minute. Each one continues in a deck of its own. To get the gateway into a mode, with snap reboot or the boot menu, go to reboots. Recovery systems covers the labels on seed that these modes boot. Install mode follows the installation itself. The factory-reset deck shows what a reset keeps on save, and how the device stays itself. Unlocking at boot has the keys from the ladder. The boot path deck is the run mode path that these modes branch from. And the overview puts the whole of Ubuntu Core on one page.
Read more
What each mode mounts
| Mode | ubuntu-seed | ubuntu-boot | ubuntu-data | ubuntu-save | /run/mnt/data |
|---|---|---|---|---|---|
| run | mounted | mounted | unlocked and mounted | unlocked and mounted | ubuntu-data |
| install | mounted | not mounted; created by the installation | not mounted; created by the installation | not mounted; created by the installation | tmpfs, or the new data when installing in the initramfs |
| recover | mounted | mounted if possible | at /run/mnt/host/ubuntu-data if possible | mounted if possible | tmpfs |
| factory-reset | mounted | not examined | not examined | unlocked with the fallback key | tmpfs |
In install, recover and factory-reset modes the kernel and gadget of the recovery system are mounted from ubuntu-seed and its base becomes the root. The mode comes from snapd_recovery_mode on the kernel command line; an empty value means install. The label comes from snapd_recovery_system.
Recover mode: the order of steps
1. Mount ubuntu-boot, with fsck. 2. Unlock ubuntu-data: one call offers the run key and the fallback key from ubuntu-seed; a recovery key is accepted. 3. Mount ubuntu-data at /run/mnt/host/ubuntu-data, nosuid, without fsck. 4. Unlock ubuntu-save with the key stored in ubuntu-data. 5. Unlock ubuntu-save with the fallback key, if step 4 was not possible or failed. 6. Mount ubuntu-save.
A step that fails is logged and the next possible step runs; if ubuntu-data cannot be unlocked or mounted, the walk goes straight to step 5. Unencrypted devices take a branch that opens ubuntu-save directly, and ubuntu-save may be absent there. While a new recovery system is being tried, fallback keys are not used and the boot always ends in a reboot. Which key is which is explained in Unlocking at boot.
When a recover boot is "degraded"
- ubuntu-save is not mounted (on an unencrypted device: not mounted and not absent);
- a key other than the run key was used;
- ubuntu-boot or ubuntu-data is not mounted;
- any error was logged.
If the pairing markers of ubuntu-data and ubuntu-save differ, data is recorded as mounted-untrusted.
unlocked.json and degraded.json
Both are written to snap-bootstrap's run directory. degraded.json is written only in recover mode and only when the boot is degraded. Factory-reset mode writes only unlocked.json.
| Key | Fields |
|---|---|
ubuntu-boot | mount-state, mount-location |
ubuntu-data | mount-state, mount-location, unlock-state, unlock-key |
ubuntu-save | mount-state, mount-location, unlock-state, unlock-key |
state | the activation state reported by the unlocking library; may be absent |
unlock-key is run, fallback or recovery. The mount fields are not provided in run mode.
What recover mode copies from trusted data
- user accounts (extrausers), SSH host keys,
sudoers.d, and each user's.ssh,.profileand snap login; - the user records in snapd's state;
- netplan configuration and
machine-id; - the saved clock of the time synchronisation service.
When data is not trusted nothing is copied, and console-conf is marked complete so that it does not run. Factory-reset mode marks console-conf complete as well.
for Ubuntu Core developers for snapd developers
The ephemeral modeenv
| Key | Example |
|---|---|
mode | recover |
recovery_system | 20260301 |
base | core24_1587.snap |
gadget | pc_211.snap |
model | acme/acme-gateway |
grade | secured |
for Ubuntu Core developers for snapd developers
cloudimg-rootfs
A fifth mode, used only by Azure confidential VM images. snap-bootstrap mounts the EFI system partition as ubuntu-seed (with an optional EFI/ubuntu/manifest.json), provisions the TPM, mounts the partitions, writes sysroot.mount from /run/mnt/data, unmounts the EFI system partition and uses a built-in generic model instead of a model assertion (cmd/snap-bootstrap/cmd_initramfs_mounts_cvm.go). Unlocking at boot covers it.
Further reading
- Recovery modes: what each mode keeps and erases, from the owner's point of view.
- Use a recovery mode: the three ways into a mode, with a picture of the menu.
- How installation works: install mode step by step and the
install-devicehook. - model assertion: the
modeslist per snap and theephemeralalias. - Use snapctl:
snapctl system-mode.
Roads from here
UBUNTU CORE · boot
- From power-on to snapd — the run-mode path these modes branch from
- Reboots and snap reboot — how a mode is requested with snap reboot or the chooser
- Ubuntu Core: how a device boots and recovers — back to the Ubuntu Core overview
UBUNTU CORE · install and recovery
- Install mode — why install runs in the initramfs or in snapd and what it does
- Recovery systems — the systems on ubuntu-seed that these modes boot
- Factory reset, ubuntu-save and remodel — what factory reset keeps and rebuilds
UBUNTU CORE · full disk encryption
- Unlocking at boot — which key opens which partition in each mode