State format patches

state migration for snapd developers

state.json carries a format version of two numbers called patch-level and patch-sublevel. When snapd starts it compares them with its own and runs the missing patch functions before any manager exists. A new level stops an older snapd from starting on that state and a new sublevel does not.

On the map: inside snapd district

▶ Watch this deck — 7 scenes, ~5 min

01 A new snapd meets an old state

STATE PATCHES saved stateformat 6.1written by an old snapd new snapdexpects 6.3 6.26.3patches, in order format 6.3 now the rest of snapd starts
A saved state at format 6.1, written by an old snapd, and a new snapd that expects 6.3. Two patches, 6.2 and 6.3, climb from one to the other; the state ends at format 6.3 and then the rest of snapd starts

The state records its own format version. A starting snapd compares it with the version it implements and runs the missing patches in order before any other part of snapd reads the state.

A gateway has sat in a warehouse for years. You plug it in, it refreshes snapd, and the new snapd opens a state file written by a snapd years older. Why doesn't it choke? The file holds the installed snaps, their connections, even unfinished changes, but the way that data is laid out has changed since. So the state carries a format version. This one says six point one; the new snapd expects six point three. First it runs the missing migrations, in order, each a small function called a patch. When the numbers match, the rest of snapd starts, and none of it has to know the old layouts.

02 Two numbers: level and sublevel for snapd developers

LEVEL AND SUBLEVEL "patch-level": 6"patch-sublevel": 3inside state.json 123456 .1.2.3 an older snapd will not start past this an older snapd still starts regenerate service unitsflags become a struct of boolsmake every channel valid
The two keys in state.json, patch-level 6 and patch-sublevel 3, above a number line with levels 1 to 6 and sublevels .1 to .3. Levels are a one-way step that an older snapd will not start past; sublevels go both ways. Examples: level 5 regenerates service units, level 6 turns flags into a struct of bools, sublevel 6.3 makes every channel valid

patch-level and patch-sublevel are stored in the state; the code implements level 6, sublevel 3. Per the code comment, "rollback below x.0 is not possible" and "sublevel patches > 0 do not prevent rollbacks".

You need to change how the state stores something. The first question isn't about code: will snapd still start on a gateway that reverts it tomorrow? The state holds two keys, patch-level and patch-sublevel, and today's snapd implements six and three. Each level is a list of patch functions: the first moves the state to that level, the rest are its sublevels. A new level changes the layout so older code can't read it, and an older snapd refuses the state. A sublevel fixes data older code can live with, so a revert still works. Regenerating service units took a level. Making channels valid took a sublevel.

03 Before any manager exists for snapd developers

AT START state.json exists? nopatch.Initnew state, stamped 6.3 yesread the filepatch.Apply lockrun one patchwrite the two numbersthe numbers move only after a patch returns without error managers createdthen the loop starts cannot patch system state to level 6, sublevel 2: ...snapd does not start
Start-up of snapd: if state.json does not exist, patch.Init stamps a new state with 6.3; if it exists, the file is read and patch.Apply runs each patch under the lock and writes the two numbers after it; then the managers are created. A failing patch stops snapd from starting

loadState calls patch.Init for a new state and patch.Apply for an existing one, before the managers other than the restart manager are created. Each patch runs under the state lock and the level and sublevel are set only after it succeeds; releasing the lock writes both to disk together. A failing patch stops snapd from starting.

If the managers read a half-migrated state, each of them would need to know both layouts. So the patches go first, before there's anyone to confuse. At start, the overlord loads the state. No file yet? patch.Init stamps the new state with today's numbers, and there's nothing to migrate. A file? It's read, then patch.Apply runs. Each patch runs under the state lock, and the two numbers move only once it returns without error. Only then are the managers created. If a patch fails, the overlord isn't built and snapd doesn't start, so a patch has to cope with whatever old data it finds.

04 A patch rewrites work in flight for snapd developers

PATCH 3 a refresh saved by the old snapdbeforeunlink-current-snap...link-snapDoDo the new snapd has separate tasks for services afterunlink-current-snap...link-snapstart-snap-serviceswaits for stop-snap-serviceswaits for link-snapDoneready tasks are left alone
Patch 3 on a refresh saved by the old snapd: before, unlink-current-snap and link-snap with status Do; after, a stop-snap-services task that unlink-current-snap waits for, and a start-snap-services task that waits for link-snap. A link-snap task that is already Done is left alone

patch3 adds a start-snap-services task after every unfinished link-snap and a stop-snap-services task before every unfinished unlink-snap or unlink-current-snap. Patches migrate changes in flight as well as stored data.

Here's what catches people out. The state isn't only a record of what is. It also holds half-finished work, planned by the old snapd. Patch three is the clean example. Its snapd has separate tasks to start and stop a snap's services, and expects them in every refresh. A refresh saved by the snapd before it has none. So the patch walks every task that isn't finished. After each link-snap it adds start-snap-services, which waits for it. Before each unlink it adds stop-snap-services, and the unlink waits for that. Finished tasks are left alone. Later patches do the same kind of thing.

05 Sublevel patches run again for snapd developers

A DIFFERENT VERSION "patch-level": 6"patch-sublevel": 3"patch-sublevel-last-version": "2.77" snapd 2.78 starts2.77 is not 2.78 "patch-sublevel": 0 6.16.26.3all three, again "patch-sublevel-last-version": "2.78" a sublevel patch must be safe to run twice
The state holds patch-level 6, patch-sublevel 3 and patch-sublevel-last-version 2.77. snapd 2.78 starts, sees a different version, sets patch-sublevel to 0, applies 6.1, 6.2 and 6.3 again and records 2.78. A sublevel patch must be safe to run twice

At level 6, if patch-sublevel-last-version is missing or differs from the running snapd version, patch-sublevel is reset to 0 and patches 6.1 to 6.3 are applied again. Sublevel patches therefore have to be safe to repeat.

You'd expect a patch to run once per device, ever. At level six that's wrong, and it matters when you write one. The state also records which snapd version last patched it. Say the gateway ran 2.77, and now 2.78 starts. The versions differ, so snapd sets the sublevel back to zero and applies all three sublevel patches again. Then it records 2.78. Why? A gateway that went back to an older snapd for a while may hold data that the newer fixes had already corrected. Which means a sublevel patch must look before it changes anything, and leave correct data alone.

06 Going back to an older snapd for snapd developers for administrators for Ubuntu Core developers

DOWNGRADE state on the gatewaylevel 6, sublevel 3 older snapd: implements 6.1 starts"patch-sublevel": 1what 6.2 and 6.3 wrote stays in the file older snapd: implements level 5 cannot downgrade: snapd is too old for thecurrent system state (patch level 6) snapd does not start; look for this line in the journal
A state at level 6, sublevel 3 meets two older snapds. One implements 6.1: it starts, patch-sublevel becomes 1 and what 6.2 and 6.3 wrote stays. One implements level 5: it fails with cannot downgrade: snapd is too old for the current system state (patch level 6) and does not start

Same level, lower sublevel: the older snapd starts and the stored sublevel ends at its own. A higher level in the state: cannot downgrade: snapd is too old for the current system state (patch level N) and snapd does not start.

A field engineer reverts snapd on the gateway. Does the older one still understand the newer one's state? That depends on which number moved. The state says six point three. A snapd that implements six point one has the same level, so it starts, and the sublevel ends at its own, one. What patches two and three wrote stays in the file, and the old code lives with it. A snapd that implements only level five refuses: cannot downgrade, snapd is too old for the current system state. It doesn't start. That line in the journal means the installed snapd is older than the one that last wrote the state.

07 Where to go from here

ROADS State patches The overlordwhat is being patched Restarts and re-executionwhy a new snapd reads it snap debuglook at a state yourself Servicesunits patch 5 regenerated The REST APIback to the front door
Roads from State format patches: the overlord for what is being patched, snapd restarts for why a new snapd is reading an old state, snap debug for looking at a state yourself, and nearby services and the REST API

Roads: the overlord and its state, restarts of snapd, snap debug state, services, and the REST API.

Patches are the first thing a new snapd does, and the last thing most people think about. Here's where to look for what comes before and after. The overlord deck shows what the state holds, changes and tasks included, which is what the patches rewrite. The restart deck shows the moment a new snapd takes over from the old one. Snap debug has the command that reads a state file offline. The services deck covers the unit files that patch five regenerated. And the REST API deck takes you back to the front of the daemon.

Read more

Keys in the state

KeyMeaning
patch-levelthe level the state has been migrated to; missing reads as 0
patch-sublevelthe sublevel within that level; missing reads as 0
patch-sublevel-last-versionthe snapd version that last applied patches
patch-sublevel-resetan older marker; cleared when the sublevel is reset

The code at this commit implements level 6, sublevel 3.

Every patch

LevelWhat it changes
1adds the snap type and the current revision to each snap's state
2moves the snap name of tasks into their side info and fills in missing names in each snap's sequence
3adds start-snap-services after each unfinished link-snap and stop-snap-services before each unfinished unlink-snap or unlink-current-snap
4marks in-progress reverts with a revert flag; renames a field of link-snap tasks; adds a cleanup task to in-progress changes that copy snap data
5regenerates the service unit files of every active snap with services, stopping and starting them around it
6.0turns snap flags from integers into a structure of booleans
6.1adds static plug and slot attributes to connections that lack them, read from the current snap information
6.2makes sure snapd snaps have the snapd type, for old snapd snap releases
6.3makes every stored channel valid, in snap states and in unfinished tasks

What Apply does, in order

  • Read the two numbers.
  • State level above the code's: cannot downgrade: snapd is too old for the current system state (patch level N).
  • State at level 6 with a sublevel above 0, and the recorded snapd version missing or different from the running one: set the sublevel to 0.
  • Level and sublevel equal the code's: done.
  • Same level, state sublevel above the code's: store the code's sublevel and stop.
  • Apply the remaining sublevel patches of the state's level.
  • For each higher level, log Patching system state from level N to N+1 and apply its patches from the first. A level with no patches gives cannot upgrade: snapd is too new for the current system state (patch level N).
  • Record the running snapd version.

A failing patch is reported as cannot patch system state to level N, sublevel M: ....

Going back

State compared with the older snapdResult
same level, higher sublevelsnapd starts; the stored sublevel ends at the older snapd's own; data written by the newer sublevel patches stays
higher levelcannot downgrade: ...; snapd does not start

With the code read here, an older snapd of a different version first resets the sublevel to 0 and re-applies its own sublevel patches; the step that only lowers the sublevel is reached when the recorded version matches. How a device recovers after the refusal was not traced in the code.

for snapd developers

Writing a patch

These follow from the code paths above; they are not written down as rules in the tree.

  • Use a sublevel when the previous release can still work with the result. A new level stops an older snapd from starting.
  • A sublevel patch runs again after every change of snapd version, so check before changing and leave correct data alone.
  • Unfinished changes are part of the state. If the new code expects different tasks or task data, migrate the tasks that are not ready.
  • A patch that returns an error keeps snapd from starting.

A state created new is stamped with the current level and sublevel but not with a snapd version, so the first restart of the same snapd applies the level 6 sublevel patches once more. No wrong result is known from this; it costs one pass over the state.

No spread test mentions the patch keys; the patches are covered by unit tests in overlord/patch/.

Further reading

No published page explains state patches. The nearest pages:

Roads from here

RUNNING SNAPS

  • The overlord — the state file and the tasks that patches rewrite
  • Services — the service units that patch 5 regenerated

TOOLS

INSIDE SNAPD · the daemon itself