Recovery systems

device lifecycle for administrators for Ubuntu Core developers

A recovery system is a labelled set of snaps with a model and assertions on ubuntu-seed from which a device can be recovered or reset or reinstalled. snap recovery lists them. A new system is written to the seed then tried with one reboot into recover mode and promoted only if that boot unlocked the disk without help. The current and the default and the last system cannot be removed.

On the map: ubuntu core install and recovery

▶ Watch this deck — 8 scenes, ~6 min

01 A spare system on the seed

WHAT YOU BOOT WHEN NOTHING ELSE BOOTS ubuntu-seed20260301modelassertionssnaps $ snap recoveryLabel Brand Model Notes20260301 acme acme-gateway current,default-recovery current: the system this device was last seeded fromdefault-recovery: used when you don't name one 2026071520260715 acme acme-gateway - the life of a recovery systemcreatetrypromotedefaultremove
snap recovery lists one recovery system, 20260301, marked current and default-recovery; it sits on ubuntu-seed as a model, assertions and snaps; a second system 20260715 is added; a recovery system goes through create, try, promote, default and remove

snap recovery lists the recovery systems on ubuntu-seed with their label, brand, model and notes: current (the system the device was last seeded from) and default-recovery. New systems are created, tried and promoted before they can be relied on.

One morning your gateway doesn't come up. The data partition is damaged, or an update went badly. What's left to boot from? A recovery system. It lives on the seed partition, apart from your everyday system: a model, its assertions, and the snaps to start from scratch. Snap recovery lists them, each with a label, usually a date. Current marks the system the device was last seeded from: installed or reset from, or remodelled to. Default recovery is used when you don't name one. A device starts with one, and you can add more. Each new one is tested before it's trusted: create, try, promote, make default, remove.

02 What you can boot it into

A SYSTEM AND A MODE runrecoverfactory resetinstall20260301current, default-recovery20260715 run: current system only $ sudo snap reboot --recover 20260715Reboot into "20260715" "recover" mode. $ sudo snap reboot --recoverReboot into "recover" mode.no label: 20260301the default recovery system,or the current one if none is set install erases everything, ubuntu-save included
A grid of two recovery systems and four modes: 20260301, current and default-recovery, offers run, recover, factory reset and install; 20260715 offers all but run. snap reboot with a mode and label restarts into it; without a label it uses the default recovery system. Install erases everything, ubuntu-save included

Systems that are not current offer Install, Recover and Factory reset; the current system adds Run normally. snap reboot --recover, --factory-reset or --install without a label uses the default recovery system when one is set. What each mode does is in the next deck along.

So you have two systems on the seed. What can you do with each? Pick a system and a mode. Any system can be booted to recover, to factory-reset or to install. Running normally is only offered for the current one. From a shell, it's snap reboot with the mode and the label, and the device restarts straight away. Leave the label out, and for those three modes snapd uses the default recovery system if one is set, otherwise the current one. Mind the last column. Install on a device that's already installed is a reinstall: it wipes the lot, the save partition included.

03 "Current and good: two lists" for administrators for Ubuntu Core developers for snapd developers

HOW MUCH IS A SYSTEM TRUSTED modeenv current_recovery_systems=20260301,20260715,20261112 run keysealed for every current system good_recovery_systems=20260301,20260715 fallback keyswhat recover mode falls back to:sealed for good systems only bootloader environment (seed)snapd_good_recovery_systems=20260301,20260715recovery capable systems,newest last
While 20261112 is being tried, modeenv lists current_recovery_systems 20260301, 20260715 and 20261112, for which the run key is sealed, and good_recovery_systems 20260301 and 20260715, for which the fallback keys are sealed; the seed bootloader environment lists snapd_good_recovery_systems 20260301 and 20260715

current_recovery_systems are tested or being tried, and the run key is sealed for them; good_recovery_systems are tested, and the fallback keys are sealed for them too. snapd_good_recovery_systems in the seed's bootloader environment lists the recovery capable systems.

On an encrypted gateway, "is this recovery system allowed?" isn't a policy question. It's about keys. The disk keys are sealed against what the device expects to boot, so a system snapd hasn't sealed for can't unlock anything. That's why modeenv keeps two lists. Current holds systems that are tested or being tried, and the run key is sealed for them. Good holds the systems that passed. The fallback keys, the ones recover mode uses when the run key won't do, are sealed for these. A third list, in the seed's bootloader environment, records which systems are recovery capable. A system gets from current to good by being tried.

04 Creating a system for administrators for Ubuntu Core developers for snapd developers

ONE REQUEST, ONE CHANGE POST /v2/systems{"action": "create", "label": "20261112","test-system": true, "mark-default": true} change: Create new recovery system with label "20261112"download what's missingcreate-recovery-systemfinalize-recovery-systemself-contained: every snap the model requires, or that is installedoffline: anything missing must be supplied as snap files ubuntu-seed2026030120260715snap filesnew20261112 snapd-new-file-logevery new file is listed:failure or undo removes them test-system: true → reboot to try itfalse → promoted at once
A POST to /v2/systems with action create, label 20261112, test-system and mark-default starts a change: download what is missing, create-recovery-system, finalize-recovery-system. create-recovery-system writes 20261112 onto ubuntu-seed next to 20260301 and 20260715, sharing snap files, and logs new files in snapd-new-file-log. With test-system it reboots to try the system; without, it is promoted at once

POST /v2/systems with action: create takes label, test-system, mark-default, validation-sets and offline; snap debug api can send it. The change runs create-recovery-system and finalize-recovery-system; new files are recorded in snapd-new-file-log for undo.

Eight months on, the gateway's newest recovery system still holds July's snaps. There's no snap command to make a new one; you ask snapd over its API, usually through a management agent. You give a label, and say whether to test it and make it the default. Snapd builds it from what the device has now: every snap the model requires, or that's installed. Anything missing is downloaded, or handed over as files when offline. The create task writes it onto the seed, logging every new file, so a failure removes exactly those. With testing on, it ends by rebooting into the new system. Otherwise it's promoted on the spot.

05 The try boot for administrators for Ubuntu Core developers for snapd developers

ONE REBOOT, NO SESSION run moderecover mode (try)run mode again add 20261112 to currentresealbootloader environment (seed)try_recovery_system=20261112recovery_system_status=tryrecovery_system_statustry next boot: recover, 20261112 initramfsbooted label = try labelfallback keys: not allowednext boot forced back to run ubuntu-data unlockedubuntu-save unlockedwith the run key, nothing degraded tried snapd starts againreads the status: next scenecould not unlock: status stays try, reboots to run anyway
Three stages. In run mode snapd adds 20261112 to the current list, reseals, and sets try_recovery_system and recovery_system_status=try. The next boot is recover mode on 20261112, where the initramfs disallows fallback keys and forces the next boot to run; ubuntu-data and ubuntu-save unlock with the run key and the status becomes tried. Then the device reboots to run mode. If unlocking fails, the status stays try and the device still reboots to run

The try sets try_recovery_system and recovery_system_status=try, reseals and boots the new system in recover mode. The initramfs disallows the fallback keys, forces the next boot to run mode and sets the status to tried only if data and save unlocked without degradation and the installed system's state file is readable.

How do you test a recovery system? You can't wait for a disaster, so snapd rehearses one. It adds the new label to the current list, reseals the run key, and sets two bootloader variables: the system on trial, and a status, try. The gateway reboots into recover mode on the new system. The initramfs sees it booted the system on trial, so it refuses the fallback keys and points the next boot back at run mode. A hang ends in the normal system. If data and save both unlock with the run key, nothing degraded, the status becomes tried. Either way it reboots. Nobody gets a login prompt.

06 "Back in run mode: promotion" for administrators for Ubuntu Core developers for snapd developers

PASS OR FAIL snapd (run mode)reads the status, then clearsit from the bootloader environment tried+ tried-systemsfinalize-recovery-system good_recovery_systemsfallback keys resealedrecovery capablemade defaultmark-default $ snap recoveryLabel Brand Model Notes20260301 acme acme-gateway current20260715 acme acme-gateway -20261112 acme acme-gateway default-recovery tryfinalize fails: cannot promote recovery system "20261112":system has not been successfully triedchange undone: logged files removed, label dropped from modeenv
snapd in run mode reads the status and clears it. If it is tried, 20261112 is added to tried-systems and finalize-recovery-system adds it to the good list, marks it recovery capable and makes it the default; snap recovery then shows 20260301 as current, 20260715 with no notes and 20261112 as default-recovery. If the status is still try, finalize fails and the change is undone

finalize-recovery-system promotes a tried system to good_recovery_systems, marks it recovery capable and, with mark-default, makes it the default. If the try failed the task errors and the undo removes the files listed in snapd-new-file-log.

The gateway's back in run mode. Did the new system pass? Snapd reads the status the try boot left, notes a pass in its own state, and clears the bootloader variables. Then the finalize task runs. On a pass, it adds the label to the good list, which reseals the fallback keys, marks it recovery capable, and, since you asked, makes it the default. Run snap recovery again: the new system is the default, and the March one is still current, because nothing has reseeded the device since. On a fail, the change is undone. The logged files go, and the old system is still there, still good.

07 Removing a system for administrators for Ubuntu Core developers for snapd developers

THREE REFUSALS POST /v2/systems/20260715 {"action": "remove"}a system made in July, neither current nor defaultubuntu-seed20260301 (current)2026071520261112 (default)snap filesbasegadgetkernelkernel cannot remove current recovery systemcannot remove default recovery systemcannot remove last recovery system otherwise:label dropped from modeenvkeys resealedfiles only snap files no other system uses are deleted;the others stay for 20260301 and 20261112 the list of files to delete is saved in the task, so an interrupted removal can finish
A remove request for 20260715. snapd refuses to remove the current, the default or the last recovery system. Otherwise it drops the label from modeenv, reseals, then deletes the system and the snap files only it used, keeping files shared with 20260301 and 20261112. The list of files is saved in the task first

POST /v2/systems/<label> with action: remove refuses the current, the default and the last recovery system. It drops the label from modeenv, reseals, then deletes the system and the asserted snaps no other system uses.

The seed partition is small, and old recovery systems hold on to old kernels. Say you made a system in July and don't need it any more. It's another API action, remove, on its label. Snapd refuses three cases, each a way to strand the device: the current system, the default, and the last one left. Otherwise it drops the label from modeenv and reseals, so the keys stop trusting it, and only then deletes files. Snap files are shared between systems, so only the ones no other system uses go. The list is saved in the task first, so a power cut halfway doesn't leave it guessing.

08 Where to go from here

ROADS Recovery systems The four modeswhat each mode mountsFactory reset and remodelwhat is copied, wiped, kept ubuntu-seed and first boota system, file by file snap reboot and the chooserpicking a system and mode Sealed keyswhat "sealed for" meansRecovery keyssnap recovery --show-keys Little Kernelno new systems at run time
Roads from recovery systems: the four modes and then factory reset and remodel, ubuntu-seed and first boot, snap reboot and the chooser, sealed keys, recovery keys, and Little Kernel

Roads to the modes, factory reset and remodel, the seed, reboots, sealed keys, recovery keys and Little Kernel.

You can list recovery systems, add one safely and remove one. So what happens when you actually boot one? The modes deck shows what recover and factory-reset mount, and the deck after it shows what they copy, wipe and keep. The seed deck opens a recovery system up, file by file. Snap reboot and the boot-time chooser have their own deck. If "sealed for" went past too quickly, the sealing deck takes it apart, and snap recovery's one other flag, --show-keys, is covered with the recovery keys. Little Kernel boards can't add systems at run time at all.

Read more

Commands

CommandWhat it does
snap recoverylists Label, Brand, Model and Notes (current, default-recovery, or -)
snap recovery --show-keysprints the recovery key, see Recovery keys
snap reboot --recover [<label>]restart into recover mode
snap reboot --factory-reset [<label>]restart into factory-reset mode
snap reboot --install [<label>]restart into install mode: a reinstall, which also erases ubuntu-save
snap reboot --run [<label>]restart into run mode; only for the current system

current marks the system the device was most recently seeded from: the one it was installed from, or later the one a factory reset, a remodel or a seed refresh seeded it from. It is not always the system the device was first installed from.

Without a label, --recover, --factory-reset and --install use the default recovery system if one is set and the current system otherwise; --run uses the current system. snap reboot is refused from install and factory-reset mode, and for a system that is still being seeded. See Reboots and snap reboot.

A recovery system whose seed cannot be loaded is left out of the list; the only trace is a line in snapd's log.

What each system offers

SystemActions
any system that is not currentInstall, Recover, Factory reset
the current system, in run modeReinstall, Recover, Factory reset, Run normally
the current system, in recover modeReinstall, Factory reset, Run normally

for administrators for Ubuntu Core developers for snapd developers

API

RequestBodyEffect
GET /v2/systemsthe list behind snap recovery
POST /v2/systems{"action": "create", ...}change "Create new recovery system with label \"<label>\""
POST /v2/systems/<label>{"action": "remove"}change "Remove recovery system with label \"<label>\""
POST /v2/systems/<label>{"action": "reboot", "mode": "<mode>"}what snap reboot sends

There is no snap subcommand that creates or removes a recovery system. A device management agent sends these requests, or an administrator can with snap debug api -X POST -H 'Content-Type: application/json' /v2/systems and the body on standard input.

Fields of create:

FieldMeaning
labelthe new system's label; lowercase letters, digits and single dashes
validation-setsvalidation sets the new system's snaps are checked against
test-systemreboot into the new system to try it before promoting it
mark-defaultmake it the default recovery system once promoted
offlinedo not contact the store; use installed snaps only, or send snap files as a multipart form

Creating a system is exclusive with other changes and needs a seeded device. The new system covers every snap and component that the model or a validation set requires, or that is installed.

for Ubuntu Core developers for snapd developers

Where a system's status is kept

WhereNameMeaning
modeenvcurrent_recovery_systemstested or being tried; the run key is sealed for these
modeenvgood_recovery_systemstested; the fallback keys are sealed for these too
modeenvtry_modelset during a try when the new system's model differs
bootloader environment (seed)try_recovery_systemthe label on trial
bootloader environment (seed)recovery_system_statustry, then tried on success: ubuntu-data and ubuntu-save unlocked with the run key, nothing degraded, and the installed system's state file readable
bootloader environment (seed)snapd_good_recovery_systemsrecovery capable systems, newest last so the bootloader can pick the last entry
snapd statetried-systemslabels whose try boot succeeded, until finalize consumes them
snapd statedefault-recovery-systemthe system used when no label is given
snapd stateseeded-systemssystems this device was seeded from, newest first; the newest entry is the current system

for Ubuntu Core developers for snapd developers

The tasks

TaskWhat it does
create-recovery-systemwrites the seed for the label on ubuntu-seed and makes it bootable; logs new files in systems/<label>/snapd-new-file-log; untested systems are promoted here; tested ones are set up for a try boot and the device restarts
finalize-recovery-systempromotes the tried label to good_recovery_systems, marks it recovery capable and optionally makes it the default. If the try did not succeed it fails with "cannot promote recovery system \"<label>\": system has not been successfully tried", or during a remodel with "tried recovery system \"<label>\" failed"
remove-recovery-systemrefuses the current, default and last system; drops the label from modeenv and reseals; deletes the system and the asserted snaps no other system uses

Limits by bootloader: Little Kernel cannot create recovery systems at run time. A bootloader that is neither recovery-aware nor able to extract a recovery kernel fails with "does not support recovery systems".

for Ubuntu Core developers for snapd developers

Seed refresh (experimental)

The seed-refresh feature is work in progress and only enabled in snapd's development mode. With it, a refresh of the model's snaps also creates a tested, default recovery system and evicts older systems created the same way.

Further reading

  • Use a recovery mode: snap recovery and snap reboot in practice. The page says recover is only for the current system and that no label means the current system; the code offers recover for every system and prefers the default recovery system.
  • Create a recovery system from the API: the create and remove actions with JSON and multipart examples.
  • Recovery modes: what a recovery system is used for.
  • snapd REST API: the /v2/systems endpoints.
  • Validation sets: the assertion that can pin the revisions of a new recovery system.

Roads from here

RUNNING SNAPS

UBUNTU CORE · boot

UBUNTU CORE · bootloaders

UBUNTU CORE · install and recovery

UBUNTU CORE · full disk encryption