Recovery systems
device lifecycle for administrators for Ubuntu Core developers
A recovery system is a labelled set of snaps with a model and assertions on ubuntu-seed from which a device can be recovered or reset or reinstalled. snap recovery lists them. A new system is written to the seed then tried with one reboot into recover mode and promoted only if that boot unlocked the disk without help. The current and the default and the last system cannot be removed.
On the map: ubuntu core install and recovery
01 A spare system on the seed
snap recovery lists the recovery systems on ubuntu-seed with their label, brand, model and notes: current (the system the device was last seeded from) and default-recovery. New systems are created, tried and promoted before they can be relied on.
One morning your gateway doesn't come up. The data partition is damaged, or an update went badly. What's left to boot from? A recovery system. It lives on the seed partition, apart from your everyday system: a model, its assertions, and the snaps to start from scratch. Snap recovery lists them, each with a label, usually a date. Current marks the system the device was last seeded from: installed or reset from, or remodelled to. Default recovery is used when you don't name one. A device starts with one, and you can add more. Each new one is tested before it's trusted: create, try, promote, make default, remove.
02 What you can boot it into
Systems that are not current offer Install, Recover and Factory reset; the current system adds Run normally. snap reboot --recover, --factory-reset or --install without a label uses the default recovery system when one is set. What each mode does is in the next deck along.
So you have two systems on the seed. What can you do with each? Pick a system and a mode. Any system can be booted to recover, to factory-reset or to install. Running normally is only offered for the current one. From a shell, it's snap reboot with the mode and the label, and the device restarts straight away. Leave the label out, and for those three modes snapd uses the default recovery system if one is set, otherwise the current one. Mind the last column. Install on a device that's already installed is a reinstall: it wipes the lot, the save partition included.
03 "Current and good: two lists" for administrators for Ubuntu Core developers for snapd developers
current_recovery_systems are tested or being tried, and the run key is sealed for them; good_recovery_systems are tested, and the fallback keys are sealed for them too. snapd_good_recovery_systems in the seed's bootloader environment lists the recovery capable systems.
On an encrypted gateway, "is this recovery system allowed?" isn't a policy question. It's about keys. The disk keys are sealed against what the device expects to boot, so a system snapd hasn't sealed for can't unlock anything. That's why modeenv keeps two lists. Current holds systems that are tested or being tried, and the run key is sealed for them. Good holds the systems that passed. The fallback keys, the ones recover mode uses when the run key won't do, are sealed for these. A third list, in the seed's bootloader environment, records which systems are recovery capable. A system gets from current to good by being tried.
04 Creating a system for administrators for Ubuntu Core developers for snapd developers
POST /v2/systems with action: create takes label, test-system, mark-default, validation-sets and offline; snap debug api can send it. The change runs create-recovery-system and finalize-recovery-system; new files are recorded in snapd-new-file-log for undo.
Eight months on, the gateway's newest recovery system still holds July's snaps. There's no snap command to make a new one; you ask snapd over its API, usually through a management agent. You give a label, and say whether to test it and make it the default. Snapd builds it from what the device has now: every snap the model requires, or that's installed. Anything missing is downloaded, or handed over as files when offline. The create task writes it onto the seed, logging every new file, so a failure removes exactly those. With testing on, it ends by rebooting into the new system. Otherwise it's promoted on the spot.
05 The try boot for administrators for Ubuntu Core developers for snapd developers
The try sets try_recovery_system and recovery_system_status=try, reseals and boots the new system in recover mode. The initramfs disallows the fallback keys, forces the next boot to run mode and sets the status to tried only if data and save unlocked without degradation and the installed system's state file is readable.
How do you test a recovery system? You can't wait for a disaster, so snapd rehearses one. It adds the new label to the current list, reseals the run key, and sets two bootloader variables: the system on trial, and a status, try. The gateway reboots into recover mode on the new system. The initramfs sees it booted the system on trial, so it refuses the fallback keys and points the next boot back at run mode. A hang ends in the normal system. If data and save both unlock with the run key, nothing degraded, the status becomes tried. Either way it reboots. Nobody gets a login prompt.
06 "Back in run mode: promotion" for administrators for Ubuntu Core developers for snapd developers
finalize-recovery-system promotes a tried system to good_recovery_systems, marks it recovery capable and, with mark-default, makes it the default. If the try failed the task errors and the undo removes the files listed in snapd-new-file-log.
The gateway's back in run mode. Did the new system pass? Snapd reads the status the try boot left, notes a pass in its own state, and clears the bootloader variables. Then the finalize task runs. On a pass, it adds the label to the good list, which reseals the fallback keys, marks it recovery capable, and, since you asked, makes it the default. Run snap recovery again: the new system is the default, and the March one is still current, because nothing has reseeded the device since. On a fail, the change is undone. The logged files go, and the old system is still there, still good.
07 Removing a system for administrators for Ubuntu Core developers for snapd developers
POST /v2/systems/<label> with action: remove refuses the current, the default and the last recovery system. It drops the label from modeenv, reseals, then deletes the system and the asserted snaps no other system uses.
The seed partition is small, and old recovery systems hold on to old kernels. Say you made a system in July and don't need it any more. It's another API action, remove, on its label. Snapd refuses three cases, each a way to strand the device: the current system, the default, and the last one left. Otherwise it drops the label from modeenv and reseals, so the keys stop trusting it, and only then deletes files. Snap files are shared between systems, so only the ones no other system uses go. The list is saved in the task first, so a power cut halfway doesn't leave it guessing.
08 Where to go from here
Roads to the modes, factory reset and remodel, the seed, reboots, sealed keys, recovery keys and Little Kernel.
You can list recovery systems, add one safely and remove one. So what happens when you actually boot one? The modes deck shows what recover and factory-reset mount, and the deck after it shows what they copy, wipe and keep. The seed deck opens a recovery system up, file by file. Snap reboot and the boot-time chooser have their own deck. If "sealed for" went past too quickly, the sealing deck takes it apart, and snap recovery's one other flag, --show-keys, is covered with the recovery keys. Little Kernel boards can't add systems at run time at all.
Read more
Commands
| Command | What it does |
|---|---|
snap recovery | lists Label, Brand, Model and Notes (current, default-recovery, or -) |
snap recovery --show-keys | prints the recovery key, see Recovery keys |
snap reboot --recover [<label>] | restart into recover mode |
snap reboot --factory-reset [<label>] | restart into factory-reset mode |
snap reboot --install [<label>] | restart into install mode: a reinstall, which also erases ubuntu-save |
snap reboot --run [<label>] | restart into run mode; only for the current system |
current marks the system the device was most recently seeded from: the one it was installed from, or later the one a factory reset, a remodel or a seed refresh seeded it from. It is not always the system the device was first installed from.
Without a label, --recover, --factory-reset and --install use the default recovery system if one is set and the current system otherwise; --run uses the current system. snap reboot is refused from install and factory-reset mode, and for a system that is still being seeded. See Reboots and snap reboot.
A recovery system whose seed cannot be loaded is left out of the list; the only trace is a line in snapd's log.
What each system offers
| System | Actions |
|---|---|
| any system that is not current | Install, Recover, Factory reset |
| the current system, in run mode | Reinstall, Recover, Factory reset, Run normally |
| the current system, in recover mode | Reinstall, Factory reset, Run normally |
for administrators for Ubuntu Core developers for snapd developers
API
| Request | Body | Effect |
|---|---|---|
GET /v2/systems | the list behind snap recovery | |
POST /v2/systems | {"action": "create", ...} | change "Create new recovery system with label \"<label>\"" |
POST /v2/systems/<label> | {"action": "remove"} | change "Remove recovery system with label \"<label>\"" |
POST /v2/systems/<label> | {"action": "reboot", "mode": "<mode>"} | what snap reboot sends |
There is no snap subcommand that creates or removes a recovery system. A device management agent sends these requests, or an administrator can with snap debug api -X POST -H 'Content-Type: application/json' /v2/systems and the body on standard input.
Fields of create:
| Field | Meaning |
|---|---|
label | the new system's label; lowercase letters, digits and single dashes |
validation-sets | validation sets the new system's snaps are checked against |
test-system | reboot into the new system to try it before promoting it |
mark-default | make it the default recovery system once promoted |
offline | do not contact the store; use installed snaps only, or send snap files as a multipart form |
Creating a system is exclusive with other changes and needs a seeded device. The new system covers every snap and component that the model or a validation set requires, or that is installed.
for Ubuntu Core developers for snapd developers
Where a system's status is kept
| Where | Name | Meaning |
|---|---|---|
| modeenv | current_recovery_systems | tested or being tried; the run key is sealed for these |
| modeenv | good_recovery_systems | tested; the fallback keys are sealed for these too |
| modeenv | try_model | set during a try when the new system's model differs |
| bootloader environment (seed) | try_recovery_system | the label on trial |
| bootloader environment (seed) | recovery_system_status | try, then tried on success: ubuntu-data and ubuntu-save unlocked with the run key, nothing degraded, and the installed system's state file readable |
| bootloader environment (seed) | snapd_good_recovery_systems | recovery capable systems, newest last so the bootloader can pick the last entry |
| snapd state | tried-systems | labels whose try boot succeeded, until finalize consumes them |
| snapd state | default-recovery-system | the system used when no label is given |
| snapd state | seeded-systems | systems this device was seeded from, newest first; the newest entry is the current system |
for Ubuntu Core developers for snapd developers
The tasks
| Task | What it does |
|---|---|
create-recovery-system | writes the seed for the label on ubuntu-seed and makes it bootable; logs new files in systems/<label>/snapd-new-file-log; untested systems are promoted here; tested ones are set up for a try boot and the device restarts |
finalize-recovery-system | promotes the tried label to good_recovery_systems, marks it recovery capable and optionally makes it the default. If the try did not succeed it fails with "cannot promote recovery system \"<label>\": system has not been successfully tried", or during a remodel with "tried recovery system \"<label>\" failed" |
remove-recovery-system | refuses the current, default and last system; drops the label from modeenv and reseals; deletes the system and the asserted snaps no other system uses |
Limits by bootloader: Little Kernel cannot create recovery systems at run time. A bootloader that is neither recovery-aware nor able to extract a recovery kernel fails with "does not support recovery systems".
for Ubuntu Core developers for snapd developers
Seed refresh (experimental)
The seed-refresh feature is work in progress and only enabled in snapd's development mode. With it, a refresh of the model's snaps also creates a tested, default recovery system and evicts older systems created the same way.
Further reading
- Use a recovery mode:
snap recoveryandsnap rebootin practice. The page says recover is only for the current system and that no label means the current system; the code offers recover for every system and prefers the default recovery system. - Create a recovery system from the API: the create and remove actions with JSON and multipart examples.
- Recovery modes: what a recovery system is used for.
- snapd REST API: the
/v2/systemsendpoints. - Validation sets: the assertion that can pin the revisions of a new recovery system.
Roads from here
RUNNING SNAPS
- The overlord — changes and tasks and undo
UBUNTU CORE · boot
- Run, install, recover, factory reset — the modes a recovery system can be booted in
- Reboots and snap reboot — snap reboot and the chooser and what they write
- Ubuntu Core: how a device boots and recovers — back to the Ubuntu Core overview
UBUNTU CORE · bootloaders
- Little Kernel and Android boot — the bootloader that cannot add systems at run time
UBUNTU CORE · install and recovery
- Factory reset, ubuntu-save and remodel — what recover mode and factory reset do once you boot a system
- ubuntu-seed and first boot — how a system is laid out on ubuntu-seed and how it is checked
UBUNTU CORE · full disk encryption
- Sealed keys, boot chains and resealing — why a new system means resealing the keys
- Recovery keys, passphrases, DBX and repair — snap recovery --show-keys