Factory reset, ubuntu-save and remodel
device lifecycle for administrators for Ubuntu Core developers
Recover mode runs a temporary system and borrows users and network settings from the installed one without changing it. Factory reset recreates ubuntu-boot and ubuntu-data and keeps ubuntu-seed and ubuntu-save. The device key and serial live on ubuntu-save so the device keeps its identity. A remodel replaces the model on a running device. It first creates and tries a recovery system then installs snaps and sets the model last.
On the map: ubuntu core install and recovery
01 How far back do you want to go
Recover mode replaces nothing on disk. Factory reset recreates ubuntu-boot and ubuntu-data and keeps ubuntu-seed and ubuntu-save. Install mode recreates ubuntu-save too, so the device loses its identity. A remodel changes the model in place.
Something's wrong with the gateway, and the question is how far back you'll go. There are three depths. Recover mode replaces nothing: it starts a temporary system so you can look around and repair. A factory-reset recreates ubuntu-boot and ubuntu-data, and keeps ubuntu-seed and ubuntu-save. A reinstall, install mode on a device that's already installed, recreates ubuntu-save too. That row is the difference: save is where the device keeps its identity, so a reset gateway comes back as the same device and a reinstalled one doesn't. A remodel goes the other way: nothing is wiped, but the device becomes a different model.
02 Recover mode borrows from the installed system
In recover mode the initramfs copies user accounts, SSH host keys and users' SSH keys, the network configuration, the machine-id and the clock from the installed system, but only when ubuntu-data is trusted as the partner of ubuntu-save. Otherwise console-conf is disabled. The next boot is always set to run mode.
You reboot the gateway into recover mode from across the country. Can you still SSH in? Recover mode is a fresh system built in memory, with no users and new host keys. The initramfs first checks: is ubuntu-data the partner of this device's ubuntu-save? If it is, it copies the user accounts, SSH host keys and users' ssh-keys, the network configuration and machine-id, for the same address, and the clock. If not, nothing is copied and console-conf is switched off, so nobody at the console can create a user. The installed system is only read, and the next boot is set to run mode.
03 Factory reset, step by step for administrators for Ubuntu Core developers for snapd developers
The change "Perform factory reset" runs factory-reset-run-system, the gadget's install-device hook if present, and restart-system-to-run-mode. Only ubuntu-boot and ubuntu-data are recreated, the encryption state may not change, and the model and serial are restored from ubuntu-save.
Recover mode didn't help. The gateway's data is beyond saving, and you want it back the way it left the factory, without a trip to the site. So: snap reboot --factory-reset. The initramfs doesn't even look at ubuntu-data; it only unlocks ubuntu-save, with the fallback key. Then, just like install mode, a temporary system seeds and runs one change. Its main task first checks that encryption hasn't changed: a gateway installed encrypted is reset encrypted, or not at all. It recreates ubuntu-boot and ubuntu-data and leaves seed and save alone. Finally it copies the model and serial from save into the new data partition.
04 What lives on ubuntu-save for administrators for Ubuntu Core developers for snapd developers
ubuntu-save holds device/, with the device key pair and a backup assertion database holding the serial and model, plus per-snap directories exposed as SNAP_SAVE_DATA, the disk mapping, the encryption state and install-time kernel arguments. It is mounted at /var/lib/snapd/save in run mode and survives a factory reset.
We keep saying ubuntu-save survives. What's actually on it? In run mode it's mounted at /var/lib/snapd/save, so you can look. First, the device's key pair. Next to it sits a small backup assertion database with the serial and the model, written when the gateway registered. Each snap gets a directory too, handed to it as SNAP_SAVE_DATA, for whatever it wants to keep across a reset. Then bookkeeping: where the partitions are, the encryption state, and kernel arguments chosen at install, like the keyboard layout for typing a passphrase. It's small, and it's the only writable partition a factory-reset leaves alone.
05 The first boot after a reset for administrators for Ubuntu Core developers for snapd developers
On the first run boot after a factory reset snapd verifies the reset marker, switches to the new ubuntu-save key, deletes the old keys and removes the marker. If a restored serial matches a device key still on ubuntu-save, the device keeps its serial without registering again.
The gateway boots its new run system for the first time, with two loose ends waiting. Keys first. During the reset, old and new keys for ubuntu-save existed side by side, and a marker on the new data partition records which is new. Snapd checks it, switches to the new key, deletes the old ones and removes the marker. Then identity. A fresh system would make a key and ask for a serial. But the reset copied the old serial, and its private key is still on save. Snapd carries on as the same device: no registration, no new serial. Only if nothing matches does it start over.
06 "Remodel: what may change"
snap remodel <model file> is an update (newer revision), a store switch or a re-registration (different brand or model). A remodel cannot change grade, architecture or series, or go to an older revision. --offline, --snap and --assertion avoid the store, and offline a remodel cannot re-register.
A year on, acme wants every gateway on a new kernel with one more required snap. Nobody will reflash them, so the model has to change in place. That's a remodel: snap remodel and the new signed model file. Snapd first sorts it. Same brand and model, higher revision: an update. A different store: a store switch. A different brand or model name means the device becomes something else, so it gets a new serial first. Refused outright: the grade, the architecture, classic to core or back, from before UC20, or an older revision. It works offline too, from files you hand it, but then it can't re-register.
07 A remodel, in order for administrators for Ubuntu Core developers for snapd developers
Remodel order: downloads, then create-recovery-system and finalize-recovery-system (always tested), then installs, then set-model. Setting the model reseals for the new model, after which recovery systems of the old model are no longer usable; they are not removed.
A remodel can mean several reboots and many downloads; the order keeps a half-finished one from stranding the gateway. First, every download: if the network drops, nothing has changed. Second, a recovery system for the new model, created and tried with a real boot, always: a device shouldn't become a model it can't recover as. Third, the installs, with the usual reboots for kernel, base or gadget. Last, one task: set-model. It swaps the validation sets, adds the new model, makes the new recovery system the default and reseals the keys. After that the old model's recovery systems are no longer usable. Deleting them is up to you.
08 Where to go from here
Roads to recovery systems, the modes, reboots, recovery keys and sealed keys, the model, the serial and install mode.
That's the end of the line for this district: a gateway recovered, reset, or turned into a different model. If you skipped them, the recovery systems and modes decks come before this one, and the reboots deck shows how snap reboot and the chooser get you into each mode. The key handling this deck kept at arm's length, after a reset and during a remodel, is in the encryption decks. The model deck says what a model may contain, and the assertions district has the serial. And install mode is where a reinstall goes: the one option that doesn't keep ubuntu-save.
Read more
Which partitions survive
| ubuntu-seed | ubuntu-boot | ubuntu-save | ubuntu-data | |
|---|---|---|---|---|
| Recover mode | kept | kept | kept | kept (read only when trusted) |
| Factory reset | kept | recreated | kept | recreated |
| Install mode (reinstall) | kept | recreated | recreated | recreated |
| Remodel | a recovery system is added | updated | kept | kept |
Commands
| Command | Effect |
|---|---|
snap reboot --recover [<label>] | restart into recover mode; any later reboot returns to run mode |
snap reboot --factory-reset [<label>] | restart and perform a factory reset |
snap reboot --install [<label>] | reinstall; ubuntu-save is erased too |
snap model --serial | check that the serial is the same after a reset |
snap remodel <model file> | remodel using the store |
snap remodel --offline <model file> | remodel using only installed snaps and assertions |
snap remodel --snap <file> --assertion <file> <model file> | remodel from local files; implies offline, and every needed snap and assertion must be supplied or the remodel fails |
Without a label the three snap reboot modes use the default recovery system; see Recovery systems.
for administrators for Ubuntu Core developers for snapd developers
What recover mode copies from the installed system
Copied only when ubuntu-data is trusted, that is, when its marker equals the one on ubuntu-save:
- user accounts (the extrausers database);
- SSH host keys and each user's
.sshdirectory; - netplan configuration and
machine-id, so the device keeps its DHCP address; - the saved clock.
Otherwise safe defaults are written that prevent console-conf from offering to create a user. On hybrid classic systems root and the members of the admin and sudo groups are imported. In recover mode snapd does not mount ubuntu-save at /var/lib/snapd/save.
for administrators for Ubuntu Core developers for snapd developers
The factory reset change
Change: "Perform factory reset".
| Task | Summary |
|---|---|
factory-reset-run-system | Perform factory reset of the system |
hook install-device | only if the gadget has it |
restart-system-to-run-mode | Ensure next boot to run mode |
factory-reset-run-system, in order:
1. Refuse if the encryption state differs from the installed one: "cannot perform factory reset using different encryption". 2. The TPM and key steps before partitioning. 3. Recreate only the system-boot and system-data structures. 4. The key cleanup on ubuntu-save. 5. Copy into the new assertion database the model and the one serial whose device key still exists under save/device. 6. Write the marker var/lib/snapd/device/factory-reset, which records the new ubuntu-save fallback key. This works when key data is kept in files, as on the core24 gateway. On token-based systems (core26 and later) snapd 13075b6544 has no key file to record and the reset fails at this step (suspected bug F6).
Steps 2, 4 and 6 are told key by key in "Factory reset, key by key" in Recovery keys, passphrases, DBX and repair.
On the next run boot: verify the marker, rotate the save key, delete the old keys, remove the marker; then keep the serial and key if they match, or generate new keys.
for administrators for Ubuntu Core developers for snapd developers
What is on ubuntu-save
Mounted at /var/lib/snapd/save in run mode.
| Content | Purpose |
|---|---|
device/ | the device key pair, and a backup assertion database with the serial and its model, written at registration |
snap/<snap>/ | per-snap data, exposed to the snap as SNAP_SAVE_DATA (Ubuntu Core only) |
| disk mapping | where the gadget's structures are on disk, for gadget updates |
| encryption state | the data/save pairing marker, the pre-install check result |
| install-time kernel command line fragments | for example the keyboard layout for passphrase entry |
What a remodel refuses
| Change | Message |
|---|---|
| before seeding finished | cannot remodel until fully seeded |
| before the device has a serial, unless the remodel is an offline update | cannot remodel without a serial |
| an older revision of the same model | cannot remodel to older revision ... than last revision ... known to the device |
| a different series | cannot remodel to different series yet |
| from a classic model without modes | cannot remodel from classic (non-hybrid) model |
| classic to core or back | cannot remodel across classic and non-classic models |
| pre-UC20 to UC20 or later | cannot remodel from pre-UC20 to UC20+ models |
| a different grade | cannot remodel from grade ... to grade ... |
| a different architecture | cannot remodel to different architectures yet |
| core to bases | cannot remodel from core to bases yet |
| a different brand or model while offline | cannot remodel offline to different brand ID / model yet |
| a gadget with another base than the model's | cannot remodel with gadget snap that has a different base than the model |
for administrators for Ubuntu Core developers for snapd developers
The three kinds of remodel
| Kind | When | Extra work |
|---|---|---|
| update | same brand and model | none; works offline even without a serial |
| store switch | a different store | a session with the new store |
| re-registration | a different brand or model | request-serial then prepare-remodeling, which adds the remaining tasks to the running change |
for Ubuntu Core developers for snapd developers
Remodel order and set-model
1. All downloads. 2. create-recovery-system and finalize-recovery-system, always with a try boot. The label is the date, with -<n> appended if taken. Skipped when the gadget has no system-seed role. 3. The installs. 4. set-model: clears stale required flags, swaps the model's validation sets (with rollback), adds the model assertion, promotes the recovery system; then reseals for the new model and rewrites the model file on ubuntu-boot, records the seeded system, sets the default recovery system and marks it recovery capable.
During a remodel finalize-recovery-system only checks that the try succeeded; promotion happens in set-model. An error in the final step of set-model is logged and the task still completes, because the changes made there cannot be undone. Old recovery system directories are not deleted.
While the model is being switched, modeenv holds try_model and the keys are sealed for both models; afterwards for the new model only. See Sealed keys, boot chains and resealing.
Further reading
- Recovery modes: factory reset versus install mode and what happens to ubuntu-save. It says recover mode lets you log in with prior credentials; that holds only when the data partition is trusted.
- Storage layout: the ubuntu-save section and
SNAP_SAVE_DATA. - Environment variables:
SNAP_SAVE_DATAfrom the snap developer's side. - Remodeling: when a remodel is possible and the brand, store and model matrix. It does not mention that the grade cannot change.
- Remodel essential snaps: the phases of a remodel that changes base, kernel or gadget.
- Upgrade Ubuntu Core: a release upgrade done as a remodel.
Roads from here
RUNNING SNAPS
- The overlord — changes and tasks and restarts
ASSERTIONS · signed statements snapd trusts
- Assertions: devices — the serial assertion and the device key
UBUNTU CORE · boot
- Run, install, recover, factory reset — what the initramfs mounts and unlocks in recover and factory-reset mode
- Reboots and snap reboot — snap reboot and the chooser that get you into these modes
- Ubuntu Core: how a device boots and recovers — back to the Ubuntu Core overview
UBUNTU CORE · install and recovery
- Recovery systems — listing and creating and trying the systems these modes boot from
- Model, gadget and kernel — the model that a remodel replaces and what may differ
- Install mode — the reinstall that erases ubuntu-save as well
UBUNTU CORE · full disk encryption
- Recovery keys, passphrases, DBX and repair — key rotation after a factory reset and recovery keys
- Sealed keys, boot chains and resealing — the reseal a remodel performs for the new model