Factory reset, ubuntu-save and remodel

device lifecycle for administrators for Ubuntu Core developers

Recover mode runs a temporary system and borrows users and network settings from the installed one without changing it. Factory reset recreates ubuntu-boot and ubuntu-data and keeps ubuntu-seed and ubuntu-save. The device key and serial live on ubuntu-save so the device keeps its identity. A remodel replaces the model on a running device. It first creates and tries a recovery system then installs snaps and sets the model last.

On the map: ubuntu core install and recovery

▶ Watch this deck — 8 scenes, ~6 min

01 How far back do you want to go

WHAT SURVIVES ubuntu-seedubuntu-bootubuntu-saveubuntu-data recovera temporary systemkeptkeptkeptkept factory resetsnap reboot --factory-resetkeptrecreatedkeptrecreated reinstallinstall mode, againkeptrecreatedrecreatedrecreated the difference betweena reset and a reinstallthe device's identity remodelthe other direction: nothing is wiped, the running device becomes a different model
Which of the four partitions survive recover mode, a factory reset and a reinstall: recover mode keeps all four, factory reset recreates ubuntu-boot and ubuntu-data, a reinstall also recreates ubuntu-save; a remodel changes the model in place

Recover mode replaces nothing on disk. Factory reset recreates ubuntu-boot and ubuntu-data and keeps ubuntu-seed and ubuntu-save. Install mode recreates ubuntu-save too, so the device loses its identity. A remodel changes the model in place.

Something's wrong with the gateway, and the question is how far back you'll go. There are three depths. Recover mode replaces nothing: it starts a temporary system so you can look around and repair. A factory-reset recreates ubuntu-boot and ubuntu-data, and keeps ubuntu-seed and ubuntu-save. A reinstall, install mode on a device that's already installed, recreates ubuntu-save too. That row is the difference: save is where the device keeps its identity, so a reset gateway comes back as the same device and a reinstalled one doesn't. A remodel goes the other way: nothing is wiped, but the device becomes a different model.

02 Recover mode borrows from the installed system

A CLEAN SYSTEM THAT KNOWS YOU $ sudo snap reboot --recoverReboot into "recover" mode.installed systemon ubuntu-datarecover modetemporary, in memoryno users, new host keys is ubuntu-data the partnerof this device's ubuntu-save? yes: copied inuser accountsSSH host keys and users' SSH keysnetwork configuration and machine-idthe clocksame DHCP addressas in run mode no: nothing is copied, and console-conf is switched off so nobody at the console can create a user any reboot from here → run modethe installed system is only read, not changed
Recover mode, a temporary system in memory, copies user accounts, SSH keys, network configuration and machine-id and the clock from the installed system only when ubuntu-data is the partner of ubuntu-save; otherwise nothing is copied and console-conf is switched off; any reboot returns to run mode

In recover mode the initramfs copies user accounts, SSH host keys and users' SSH keys, the network configuration, the machine-id and the clock from the installed system, but only when ubuntu-data is trusted as the partner of ubuntu-save. Otherwise console-conf is disabled. The next boot is always set to run mode.

You reboot the gateway into recover mode from across the country. Can you still SSH in? Recover mode is a fresh system built in memory, with no users and new host keys. The initramfs first checks: is ubuntu-data the partner of this device's ubuntu-save? If it is, it copies the user accounts, SSH host keys and users' ssh-keys, the network configuration and machine-id, for the same address, and the clock. If not, nothing is copied and console-conf is switched off, so nobody at the console can create a user. The installed system is only read, and the next boot is set to run mode.

03 Factory reset, step by step for administrators for Ubuntu Core developers for snapd developers

PERFORM FACTORY RESET $ sudo snap reboot --factory-resetReboot into "factory-reset" mode. ubuntu-seedubuntu-bootnot even looked atubuntu-saveubuntu-datanot even looked atinitramfsunlocks only ubuntu-save,with the fallback key change: Perform factory resetfactory-reset-run-systeminstall-device hookif the gadget has onerestart-system-to-run-mode encryption changed since install? → cannot perform factory reset using different encryption ubuntu-bootrecreatedubuntu-datarecreatedkeptkept model + serial copied into the new ubuntu-data
snap reboot --factory-reset; the initramfs unlocks only ubuntu-save; the change Perform factory reset runs factory-reset-run-system, the install-device hook if any and restart-system-to-run-mode; a changed encryption state is refused; ubuntu-boot and ubuntu-data are recreated, ubuntu-seed and ubuntu-save kept, and the model and serial copied from ubuntu-save

The change "Perform factory reset" runs factory-reset-run-system, the gadget's install-device hook if present, and restart-system-to-run-mode. Only ubuntu-boot and ubuntu-data are recreated, the encryption state may not change, and the model and serial are restored from ubuntu-save.

Recover mode didn't help. The gateway's data is beyond saving, and you want it back the way it left the factory, without a trip to the site. So: snap reboot --factory-reset. The initramfs doesn't even look at ubuntu-data; it only unlocks ubuntu-save, with the fallback key. Then, just like install mode, a temporary system seeds and runs one change. Its main task first checks that encryption hasn't changed: a gateway installed encrypted is reset encrypted, or not at all. It recreates ubuntu-boot and ubuntu-data and leaves seed and save alone. Finally it copies the model and serial from save into the new data partition.

04 What lives on ubuntu-save for administrators for Ubuntu Core developers for snapd developers

THE PARTITION THAT REMEMBERS ubuntu-savemounted at /var/lib/snapd/save in run mode$ ls /var/lib/snapd/savedevice snap ... device/the device key pair a backup assertion databasethe serial and its model, written at registration snap/<snap>/per-snap data, Ubuntu Core only$SNAP_SAVE_DATA disk mappingfor gadget updatesencryption stateand the pairing markerinstall-time kernel argskeyboard layout, for one the only writable partition a factory reset keepskey pair + serialsurvive a reset:the same identitycomes back
ubuntu-save, mounted at /var/lib/snapd/save, holds, under device/, the device key pair and a backup assertion database with the serial and model, and per-snap data exposed as SNAP_SAVE_DATA, and bookkeeping: disk mapping, encryption state and install-time kernel arguments

ubuntu-save holds device/, with the device key pair and a backup assertion database holding the serial and model, plus per-snap directories exposed as SNAP_SAVE_DATA, the disk mapping, the encryption state and install-time kernel arguments. It is mounted at /var/lib/snapd/save in run mode and survives a factory reset.

We keep saying ubuntu-save survives. What's actually on it? In run mode it's mounted at /var/lib/snapd/save, so you can look. First, the device's key pair. Next to it sits a small backup assertion database with the serial and the model, written when the gateway registered. Each snap gets a directory too, handed to it as SNAP_SAVE_DATA, for whatever it wants to keep across a reset. Then bookkeeping: where the partitions are, the encryption state, and kernel arguments chosen at install, like the keyboard layout for typing a passphrase. It's small, and it's the only writable partition a factory-reset leaves alone.

05 The first boot after a reset for administrators for Ubuntu Core developers for snapd developers

THE SAME DEVICE COMES BACK first run boot after the reset factory-reset marker on ubuntu-datarecords which ubuntu-save key is the new onekeysverify the markerswitch to the new keydelete the old keysremove the markerthe key details: recovery keys and repair identitya restored serial whose device keyis still on ubuntu-save? yes: keep serial and key, no registration# before the reset$ snap model --serialserial: 0b7c7a2e-...# after the reset$ snap model --serialserial: 0b7c7a2e-... no: new keys, register as a new device
The first run boot after a factory reset: snapd verifies the reset marker, switches to the new ubuntu-save key, deletes the old keys and removes the marker; then it finds a restored serial whose device key is still on ubuntu-save and keeps it, so the serial is the same before and after

On the first run boot after a factory reset snapd verifies the reset marker, switches to the new ubuntu-save key, deletes the old keys and removes the marker. If a restored serial matches a device key still on ubuntu-save, the device keeps its serial without registering again.

The gateway boots its new run system for the first time, with two loose ends waiting. Keys first. During the reset, old and new keys for ubuntu-save existed side by side, and a marker on the new data partition records which is new. Snapd checks it, switches to the new key, deletes the old ones and removes the marker. Then identity. A fresh system would make a key and ask for a serial. But the reset copied the old serial, and its private key is still on save. Snapd carries on as the same device: no registration, no new serial. Only if nothing matches does it start over.

06 "Remodel: what may change"

A NEW MODEL, IN PLACE $ sudo snap remodel acme-gateway-rev4.modelNew model acme-gateway-rev4.model set same brand and model, newer revisionupdateRefresh model assertion from revision 3 to 4 a different storestore switchsame brand and model a different brand or model namere-registrationa new serial firstRemodel device to acme/acme-gateway-pro (0) refuseda different grade · a different architecture · classic ↔ core · pre-UC20 → UC20 · an older revision # offline: local files, no store$ sudo snap remodel \--snap ./pc-kernel_3010.snap \--assertion ./pc-kernel.assert \acme-gateway-rev4.modelfiles imply --offline; offline cannot re-registeror --offline alone: use only what is already installed
snap remodel with a new model file; three kinds of remodel: update for a newer revision of the same model, store switch for a different store, re-registration for a different brand or model; refused changes; an offline remodel from local files

snap remodel <model file> is an update (newer revision), a store switch or a re-registration (different brand or model). A remodel cannot change grade, architecture or series, or go to an older revision. --offline, --snap and --assertion avoid the store, and offline a remodel cannot re-register.

A year on, acme wants every gateway on a new kernel with one more required snap. Nobody will reflash them, so the model has to change in place. That's a remodel: snap remodel and the new signed model file. Snapd first sorts it. Same brand and model, higher revision: an update. A different store: a store switch. A different brand or model name means the device becomes something else, so it gets a new serial first. Refused outright: the grade, the architecture, classic to core or back, from before UC20, or an older revision. It works offline too, from files you hand it, but then it can't re-register.

07 A remodel, in order for administrators for Ubuntu Core developers for snapd developers

THE MODEL CHANGES LAST 1 · downloadeverythingnothing changes yet 2 · recovery systemcreate-recovery-system↻ try bootfinalize-recovery-system20270310, for the new modelalways tested 3 · installthe snapskernel, base, gadget:reboots as needed 4 · lastset-modelvalidation sets swappednew model addedrecovery system made defaultkeys resealed for the new model 20260301 and other systems of the old model: still on ubuntu-seed, no longer usable $ snap modelmodel acme-gateway$ snap recovery20261112 acme acme-gateway -20270310 acme acme-gateway current,default-recoveryolder systems are still listed, before these
A remodel in four stages: download everything, create and try a recovery system for the new model, install the snaps, then set-model, which swaps validation sets, adds the model, makes the recovery system the default and reseals the keys; the old model's recovery systems are no longer usable but still listed; snap recovery shows 20270310, created by the remodel, as current and default-recovery, and 20261112 with no notes

Remodel order: downloads, then create-recovery-system and finalize-recovery-system (always tested), then installs, then set-model. Setting the model reseals for the new model, after which recovery systems of the old model are no longer usable; they are not removed.

A remodel can mean several reboots and many downloads; the order keeps a half-finished one from stranding the gateway. First, every download: if the network drops, nothing has changed. Second, a recovery system for the new model, created and tried with a real boot, always: a device shouldn't become a model it can't recover as. Third, the installs, with the usual reboots for kernel, base or gadget. Last, one task: set-model. It swaps the validation sets, adds the new model, makes the new recovery system the default and reseals the keys. After that the old model's recovery systems are no longer usable. Deleting them is up to you.

08 Where to go from here

ROADS Reset and remodel Recovery systemswhat these modes boot fromThe modeswhat the initramfs mounts Rebootssnap reboot and the chooser Recovery keys and repairkey rotation after a resetSealed keysthe reseal for a new model The modelwhat a model may containThe serialand the device key Install modethe reinstall that takes save too
Roads from reset and remodel: recovery systems and the modes, snap reboot and the chooser, recovery keys and sealed keys, the model, the serial, and install mode

Roads to recovery systems, the modes, reboots, recovery keys and sealed keys, the model, the serial and install mode.

That's the end of the line for this district: a gateway recovered, reset, or turned into a different model. If you skipped them, the recovery systems and modes decks come before this one, and the reboots deck shows how snap reboot and the chooser get you into each mode. The key handling this deck kept at arm's length, after a reset and during a remodel, is in the encryption decks. The model deck says what a model may contain, and the assertions district has the serial. And install mode is where a reinstall goes: the one option that doesn't keep ubuntu-save.

Read more

Which partitions survive

ubuntu-seedubuntu-bootubuntu-saveubuntu-data
Recover modekeptkeptkeptkept (read only when trusted)
Factory resetkeptrecreatedkeptrecreated
Install mode (reinstall)keptrecreatedrecreatedrecreated
Remodela recovery system is addedupdatedkeptkept

Commands

CommandEffect
snap reboot --recover [<label>]restart into recover mode; any later reboot returns to run mode
snap reboot --factory-reset [<label>]restart and perform a factory reset
snap reboot --install [<label>]reinstall; ubuntu-save is erased too
snap model --serialcheck that the serial is the same after a reset
snap remodel <model file>remodel using the store
snap remodel --offline <model file>remodel using only installed snaps and assertions
snap remodel --snap <file> --assertion <file> <model file>remodel from local files; implies offline, and every needed snap and assertion must be supplied or the remodel fails

Without a label the three snap reboot modes use the default recovery system; see Recovery systems.

for administrators for Ubuntu Core developers for snapd developers

What recover mode copies from the installed system

Copied only when ubuntu-data is trusted, that is, when its marker equals the one on ubuntu-save:

  • user accounts (the extrausers database);
  • SSH host keys and each user's .ssh directory;
  • netplan configuration and machine-id, so the device keeps its DHCP address;
  • the saved clock.

Otherwise safe defaults are written that prevent console-conf from offering to create a user. On hybrid classic systems root and the members of the admin and sudo groups are imported. In recover mode snapd does not mount ubuntu-save at /var/lib/snapd/save.

for administrators for Ubuntu Core developers for snapd developers

The factory reset change

Change: "Perform factory reset".

TaskSummary
factory-reset-run-systemPerform factory reset of the system
hook install-deviceonly if the gadget has it
restart-system-to-run-modeEnsure next boot to run mode

factory-reset-run-system, in order:

1. Refuse if the encryption state differs from the installed one: "cannot perform factory reset using different encryption". 2. The TPM and key steps before partitioning. 3. Recreate only the system-boot and system-data structures. 4. The key cleanup on ubuntu-save. 5. Copy into the new assertion database the model and the one serial whose device key still exists under save/device. 6. Write the marker var/lib/snapd/device/factory-reset, which records the new ubuntu-save fallback key. This works when key data is kept in files, as on the core24 gateway. On token-based systems (core26 and later) snapd 13075b6544 has no key file to record and the reset fails at this step (suspected bug F6).

Steps 2, 4 and 6 are told key by key in "Factory reset, key by key" in Recovery keys, passphrases, DBX and repair.

On the next run boot: verify the marker, rotate the save key, delete the old keys, remove the marker; then keep the serial and key if they match, or generate new keys.

for administrators for Ubuntu Core developers for snapd developers

What is on ubuntu-save

Mounted at /var/lib/snapd/save in run mode.

ContentPurpose
device/the device key pair, and a backup assertion database with the serial and its model, written at registration
snap/<snap>/per-snap data, exposed to the snap as SNAP_SAVE_DATA (Ubuntu Core only)
disk mappingwhere the gadget's structures are on disk, for gadget updates
encryption statethe data/save pairing marker, the pre-install check result
install-time kernel command line fragmentsfor example the keyboard layout for passphrase entry

What a remodel refuses

ChangeMessage
before seeding finishedcannot remodel until fully seeded
before the device has a serial, unless the remodel is an offline updatecannot remodel without a serial
an older revision of the same modelcannot remodel to older revision ... than last revision ... known to the device
a different seriescannot remodel to different series yet
from a classic model without modescannot remodel from classic (non-hybrid) model
classic to core or backcannot remodel across classic and non-classic models
pre-UC20 to UC20 or latercannot remodel from pre-UC20 to UC20+ models
a different gradecannot remodel from grade ... to grade ...
a different architecturecannot remodel to different architectures yet
core to basescannot remodel from core to bases yet
a different brand or model while offlinecannot remodel offline to different brand ID / model yet
a gadget with another base than the model'scannot remodel with gadget snap that has a different base than the model

for administrators for Ubuntu Core developers for snapd developers

The three kinds of remodel

KindWhenExtra work
updatesame brand and modelnone; works offline even without a serial
store switcha different storea session with the new store
re-registrationa different brand or modelrequest-serial then prepare-remodeling, which adds the remaining tasks to the running change

for Ubuntu Core developers for snapd developers

Remodel order and set-model

1. All downloads. 2. create-recovery-system and finalize-recovery-system, always with a try boot. The label is the date, with -<n> appended if taken. Skipped when the gadget has no system-seed role. 3. The installs. 4. set-model: clears stale required flags, swaps the model's validation sets (with rollback), adds the model assertion, promotes the recovery system; then reseals for the new model and rewrites the model file on ubuntu-boot, records the seeded system, sets the default recovery system and marks it recovery capable.

During a remodel finalize-recovery-system only checks that the try succeeded; promotion happens in set-model. An error in the final step of set-model is logged and the task still completes, because the changes made there cannot be undone. Old recovery system directories are not deleted.

While the model is being switched, modeenv holds try_model and the keys are sealed for both models; afterwards for the new model only. See Sealed keys, boot chains and resealing.

Further reading

  • Recovery modes: factory reset versus install mode and what happens to ubuntu-save. It says recover mode lets you log in with prior credentials; that holds only when the data partition is trusted.
  • Storage layout: the ubuntu-save section and SNAP_SAVE_DATA.
  • Environment variables: SNAP_SAVE_DATA from the snap developer's side.
  • Remodeling: when a remodel is possible and the brand, store and model matrix. It does not mention that the grade cannot change.
  • Remodel essential snaps: the phases of a remodel that changes base, kernel or gadget.
  • Upgrade Ubuntu Core: a release upgrade done as a remodel.

Roads from here

RUNNING SNAPS

ASSERTIONS · signed statements snapd trusts

UBUNTU CORE · boot

UBUNTU CORE · install and recovery

UBUNTU CORE · full disk encryption