State format patches
state migration for snapd developers
state.json carries a format version of two numbers called patch-level and patch-sublevel. When snapd starts it compares them with its own and runs the missing patch functions before any manager exists. A new level stops an older snapd from starting on that state and a new sublevel does not.
On the map: inside snapd district
01 A new snapd meets an old state
The state records its own format version. A starting snapd compares it with the version it implements and runs the missing patches in order before any other part of snapd reads the state.
A gateway has sat in a warehouse for years. You plug it in, it refreshes snapd, and the new snapd opens a state file written by a snapd years older. Why doesn't it choke? The file holds the installed snaps, their connections, even unfinished changes, but the way that data is laid out has changed since. So the state carries a format version. This one says six point one; the new snapd expects six point three. First it runs the missing migrations, in order, each a small function called a patch. When the numbers match, the rest of snapd starts, and none of it has to know the old layouts.
02 Two numbers: level and sublevel for snapd developers
patch-level and patch-sublevel are stored in the state; the code implements level 6, sublevel 3. Per the code comment, "rollback below x.0 is not possible" and "sublevel patches > 0 do not prevent rollbacks".
You need to change how the state stores something. The first question isn't about code: will snapd still start on a gateway that reverts it tomorrow? The state holds two keys, patch-level and patch-sublevel, and today's snapd implements six and three. Each level is a list of patch functions: the first moves the state to that level, the rest are its sublevels. A new level changes the layout so older code can't read it, and an older snapd refuses the state. A sublevel fixes data older code can live with, so a revert still works. Regenerating service units took a level. Making channels valid took a sublevel.
03 Before any manager exists for snapd developers
loadState calls patch.Init for a new state and patch.Apply for an existing one, before the managers other than the restart manager are created. Each patch runs under the state lock and the level and sublevel are set only after it succeeds; releasing the lock writes both to disk together. A failing patch stops snapd from starting.
If the managers read a half-migrated state, each of them would need to know both layouts. So the patches go first, before there's anyone to confuse. At start, the overlord loads the state. No file yet? patch.Init stamps the new state with today's numbers, and there's nothing to migrate. A file? It's read, then patch.Apply runs. Each patch runs under the state lock, and the two numbers move only once it returns without error. Only then are the managers created. If a patch fails, the overlord isn't built and snapd doesn't start, so a patch has to cope with whatever old data it finds.
04 A patch rewrites work in flight for snapd developers
patch3 adds a start-snap-services task after every unfinished link-snap and a stop-snap-services task before every unfinished unlink-snap or unlink-current-snap. Patches migrate changes in flight as well as stored data.
Here's what catches people out. The state isn't only a record of what is. It also holds half-finished work, planned by the old snapd. Patch three is the clean example. Its snapd has separate tasks to start and stop a snap's services, and expects them in every refresh. A refresh saved by the snapd before it has none. So the patch walks every task that isn't finished. After each link-snap it adds start-snap-services, which waits for it. Before each unlink it adds stop-snap-services, and the unlink waits for that. Finished tasks are left alone. Later patches do the same kind of thing.
05 Sublevel patches run again for snapd developers
At level 6, if patch-sublevel-last-version is missing or differs from the running snapd version, patch-sublevel is reset to 0 and patches 6.1 to 6.3 are applied again. Sublevel patches therefore have to be safe to repeat.
You'd expect a patch to run once per device, ever. At level six that's wrong, and it matters when you write one. The state also records which snapd version last patched it. Say the gateway ran 2.77, and now 2.78 starts. The versions differ, so snapd sets the sublevel back to zero and applies all three sublevel patches again. Then it records 2.78. Why? A gateway that went back to an older snapd for a while may hold data that the newer fixes had already corrected. Which means a sublevel patch must look before it changes anything, and leave correct data alone.
06 Going back to an older snapd for snapd developers for administrators for Ubuntu Core developers
Same level, lower sublevel: the older snapd starts and the stored sublevel ends at its own. A higher level in the state: cannot downgrade: snapd is too old for the current system state (patch level N) and snapd does not start.
A field engineer reverts snapd on the gateway. Does the older one still understand the newer one's state? That depends on which number moved. The state says six point three. A snapd that implements six point one has the same level, so it starts, and the sublevel ends at its own, one. What patches two and three wrote stays in the file, and the old code lives with it. A snapd that implements only level five refuses: cannot downgrade, snapd is too old for the current system state. It doesn't start. That line in the journal means the installed snapd is older than the one that last wrote the state.
07 Where to go from here
Roads: the overlord and its state, restarts of snapd, snap debug state, services, and the REST API.
Patches are the first thing a new snapd does, and the last thing most people think about. Here's where to look for what comes before and after. The overlord deck shows what the state holds, changes and tasks included, which is what the patches rewrite. The restart deck shows the moment a new snapd takes over from the old one. Snap debug has the command that reads a state file offline. The services deck covers the unit files that patch five regenerated. And the REST API deck takes you back to the front of the daemon.
Read more
Keys in the state
| Key | Meaning |
|---|---|
patch-level | the level the state has been migrated to; missing reads as 0 |
patch-sublevel | the sublevel within that level; missing reads as 0 |
patch-sublevel-last-version | the snapd version that last applied patches |
patch-sublevel-reset | an older marker; cleared when the sublevel is reset |
The code at this commit implements level 6, sublevel 3.
Every patch
| Level | What it changes |
|---|---|
| 1 | adds the snap type and the current revision to each snap's state |
| 2 | moves the snap name of tasks into their side info and fills in missing names in each snap's sequence |
| 3 | adds start-snap-services after each unfinished link-snap and stop-snap-services before each unfinished unlink-snap or unlink-current-snap |
| 4 | marks in-progress reverts with a revert flag; renames a field of link-snap tasks; adds a cleanup task to in-progress changes that copy snap data |
| 5 | regenerates the service unit files of every active snap with services, stopping and starting them around it |
| 6.0 | turns snap flags from integers into a structure of booleans |
| 6.1 | adds static plug and slot attributes to connections that lack them, read from the current snap information |
| 6.2 | makes sure snapd snaps have the snapd type, for old snapd snap releases |
| 6.3 | makes every stored channel valid, in snap states and in unfinished tasks |
What Apply does, in order
- Read the two numbers.
- State level above the code's:
cannot downgrade: snapd is too old for the current system state (patch level N). - State at level 6 with a sublevel above 0, and the recorded snapd version missing or different from the running one: set the sublevel to 0.
- Level and sublevel equal the code's: done.
- Same level, state sublevel above the code's: store the code's sublevel and stop.
- Apply the remaining sublevel patches of the state's level.
- For each higher level, log
Patching system state from level N to N+1and apply its patches from the first. A level with no patches givescannot upgrade: snapd is too new for the current system state (patch level N). - Record the running snapd version.
A failing patch is reported as cannot patch system state to level N, sublevel M: ....
Going back
| State compared with the older snapd | Result |
|---|---|
| same level, higher sublevel | snapd starts; the stored sublevel ends at the older snapd's own; data written by the newer sublevel patches stays |
| higher level | cannot downgrade: ...; snapd does not start |
With the code read here, an older snapd of a different version first resets the sublevel to 0 and re-applies its own sublevel patches; the step that only lowers the sublevel is reached when the recorded version matches. How a device recovers after the refusal was not traced in the code.
for snapd developers
Writing a patch
These follow from the code paths above; they are not written down as rules in the tree.
- Use a sublevel when the previous release can still work with the result. A new level stops an older snapd from starting.
- A sublevel patch runs again after every change of snapd version, so check before changing and leave correct data alone.
- Unfinished changes are part of the state. If the new code expects different tasks or task data, migrate the tasks that are not ready.
- A patch that returns an error keeps snapd from starting.
A state created new is stamped with the current level and sublevel but not with a snapd version, so the first restart of the same snapd applies the level 6 sublevel patches once more. No wrong result is known from this; it costs one pass over the state.
No spread test mentions the patch keys; the patches are covered by unit tests in overlord/patch/.
Further reading
No published page explains state patches. The nearest pages:
- Fix common issues:
snap debug state /var/lib/snapd/state.json, the one published use of the state file. - Snapd release process: release cadence, the context for upgrades and reverts of snapd.
Roads from here
RUNNING SNAPS
- The overlord — the state file and the tasks that patches rewrite
- Services — the service units that patch 5 regenerated
TOOLS
- snap debug: tools for looking inside — snap debug state for reading a state file offline
INSIDE SNAPD · the daemon itself
- How snapd restarts and re-executes — the restart that puts a new snapd in front of an old state
- The snapd REST API — back to the daemon's front door