The overlord

changes, tasks, state for snapd developers for administrators

Every operation snapd performs is a change made of tasks. Changes are saved in state.json and run by one task runner that calls handlers owned by managers such as snapstate and ifacestate. A failed task undoes the tasks before it within its lane. snap changes and snap tasks show the result and snap abort asks for the undo.

On the map: the town hall

▶ Watch this deck — 8 scenes, ~6 min

01 Every command is a change

CHANGES AND TASKS $ sudo snap install acme-agent the SSH session drops snapdchange 27the command only asked;now it only watches 27 Install "acme-agent" snapDownload snap "acme-agent" (42) from channel "stable"Fetch and check assertions for snap "acme-agent" (42)Mount snap "acme-agent" (42)the change lives in snapd Make snap "acme-agent" (42) available to the systemStart snap "acme-agent" (42) servicesRun configure hook of "acme-agent" snap if presentand about ten more tasks $ snap changesID Status Spawn Ready Summary27 Done today at 10:02 UTC today at 10:03 UTC Install "acme-agent" snap
sudo snap install acme-agent sends a request to snapd and gets back change 27; the SSH session drops but the change carries on; change 27, Install acme-agent snap, holds tasks: download, fetch and check assertions, mount, make available, start services, run configure hook, and about ten more; snap changes later lists change 27 as Done

A snap command asks snapd for an operation and gets back the id of a change. The change is a set of tasks that snapd runs by itself. snap changes lists changes; snap tasks 27 lists the tasks of one.

You start an install over SSH, and the connection drops halfway through. Is the gateway now half installed? No, because the command wasn't doing the work. It sent a request, snapd answered with the number of a change, 27, and from then on the command only watched. The change is the operation: an ordered set of small steps called tasks, each with a plain summary. Download, check its assertions, mount, make it available, start the services, and about ten more. Snapd runs them whether anyone's watching or not. Snap changes lists recent changes with their status, and snap tasks 27 shows the steps of this one.

02 When a task fails

DO AND UNDO $ snap tasks 27Status SummaryDoneDownload snap "acme-agent" (42) from channel "stable"DoneFetch and check assertions for snap "acme-agent" (42)DoneMount snap "acme-agent" (42)DoneMake snap "acme-agent" (42) available to the systemDoingStart snap "acme-agent" (42) servicesDoRun configure hook of "acme-agent" snap if presentDoRun health check of "acme-agent" snapseven of the rows; the Spawn and Ready columns are left outDo → Doing → Done Error HoldHoldDo → Holdnot started, and now it won't be UndoneUndoneUndoneDone → Undo → Undoing → Undonelast finished, first undone row 2 stays Done:checking assertions changed nothing,so there is nothing to undo error: cannot perform the following tasks:- Start snap "acme-agent" (42) services (...)change 27: ErrorError on a change usually means: back where you started. The task list says exactly.
snap tasks 27 with seven rows: download, fetch and check assertions, mount and make available are Done, start services is Doing, configure hook and health check are Do. Start services becomes Error, the last two become Hold, make available, mount and download become Undone, and the assertions check stays Done because it has nothing to undo. The command ends with error: cannot perform the following tasks, and change 27 reports Error

A failed task becomes Error. Tasks not yet started become Hold; finished tasks go Undo, Undoing, Undone in reverse order. A kind without an undo handler stays Done. The change reports Error.

Same install, but this time the service refuses to start. Over a dozen steps are done, four shown here, and two haven't begun. What does snapd do with them? First, the failed task is marked error. The two that hadn't started go to hold, so they won't run. The finished ones are undone, last first: the snap is made unavailable, unmounted, and its download removed. One row still says done. Checking the assertions changed nothing, so there's nothing to undo. The change reports error, and its message names the task that failed. So error usually means you're back where you started, and the task list tells you exactly.

03 One file remembers everything for snapd developers for administrators

STATE.JSON /var/lib/snapd/state.json"data": {...}"changes": {...}"tasks": {...}plus warnings, notices and the id counters state in memoryLock()every read and write needs the lock Unlock() after a modification:the whole file is written again, atomically mount-snap Doingsnapd stops snapd starts againreads state.json oncea task at Doing, no handler running: run it again so a handler must be safe to run twicea snapd refresh, which restarts snapd in the middle of its change, is the everyday case
/var/lib/snapd/state.json holds data, changes and tasks; the state in memory has one lock; every Unlock after a modification rewrites the whole file atomically; snapd stops while mount-snap is Doing; the next snapd reads the file once and runs the mount-snap handler again, so handlers must be safe to run twice

state.json holds data, changes and tasks. It is rewritten atomically on every Unlock that follows a modification. After a restart a task found in Doing has its handler run again, so handlers must be idempotent.

I said the change lives in snapd. More exactly, in one file, and that's why a change can outlive the process that started it. state.json holds it all: installed snaps and connections under data, every change and task with its status. In memory, one lock guards it, and touching the state without it panics. Release it after a modification, and the whole file is written again, atomically. Say snapd stops mid-mount. The next snapd reads the file once, finds a task at doing with nothing running it, and runs its handler again. So handlers must be safe to run twice. A snapd refresh is the everyday case.

04 Managers, the loop and the runner for snapd developers

THE ENSURE LOOP one Ensure passa pass every 5 minutes, or sooner when EnsureBefore asks hookstatesnapstateifacestatedevicestateand the restfourteen managersrestart runs firsta manager's Ensure does quick checks; slow work becomes a change TaskRunneradded lastvalidate-snap Doneprerequisites Donemount-snap Do mount-snap Doingevery task it waits for is Donerunner.AddHandler("mount-snap", m.doMountSnap, m.undoMountSnap)registered by snapstate handlerown goroutinethe handler takes the state lock itself, and lets go of it around slow downloads and mounts when a task finishes and others wait on it: EnsureBefore(0), the next pass at once
The Ensure loop calls fourteen managers, restart first, among them hookstate, snapstate, ifacestate and devicestate, then the TaskRunner, every 5 minutes or sooner. The runner starts mount-snap once validate-snap and prerequisites are Done, using the handler snapstate registered with AddHandler; the handler runs in its own goroutine and its return asks for the next pass with EnsureBefore(0)

Overlord calls Ensure on fourteen managers and then on the shared TaskRunner, every 5 minutes or as soon as EnsureBefore asks. The runner starts a Do task when all its wait tasks are Done, in a goroutine of its own.

Who picks up a task and runs it? A loop, a list of managers and one runner. Each pass, the overlord calls ensure on each of fourteen managers: hookstate, snapstate, ifacestate, devicestate and the rest. Ensure does quick checks, like whether an auto-refresh is due; slow work becomes a change. The task runner goes last. It starts each task whose wait tasks are all done, with the handlers its manager registered: do, and usually undo. The handler gets its own goroutine and takes the state lock itself. Passes come every five minutes, but finishing a task asks for the next one at once.

05 "Lanes: how far a failure spreads" for snapd developers for administrators

LANES $ sudo snap install acme-agent acme-tools28 Install snaps "acme-agent", "acme-tools" lane 1acme-agentdownload-snapDonemount-snapDonelink-snapDonestart-snap-servicesDone lane 2acme-toolsdownload-snapDonemount-snapDonelink-snapErrorstart-snap-servicesDo UndoneUndoneHoldacme-agent stays installed: its lane was not aborted change 28: Error, although one snap was installed $ sudo snap install --transaction=all-snaps acme-agent acme-toolsone laneUndoneUndoneUndoneUndoneUndoneUndoneErrorHoldacme-agentacme-toolsall or nothing
sudo snap install acme-agent acme-tools makes change 28 with two lanes. Lane 1 holds acme-agent's download-snap, mount-snap, link-snap and start-snap-services, all Done. In lane 2, acme-tools' link-snap fails: its download and mount are Undone and its start-snap-services is Hold, while acme-agent stays installed and the change reports Error. With --transaction=all-snaps both snaps share one lane and everything is undone

A failed task aborts its lanes, not the whole change. With the default --transaction=per-snap each snap has its own lane; with --transaction=all-snaps they share one. A change with any Error task reports Error.

You install two snaps with one command, and the second one fails. Should the first come out again? You get to choose, and the mechanism is the lane: a group of tasks that fail together. By default each snap in the command gets its own lane, so a failure in acme-tools undoes acme-tools only, and acme-agent stays installed. The change still reports error, because one of its tasks failed, so read the task list before deciding nothing happened. With --transaction=all-snaps, both share one lane, and a failure anywhere undoes both. An auto-refresh of many snaps works the same way, one lane per snap.

06 One change per snap at a time

CONFLICTS change 27 · DoingInstall "acme-agent" snapacme-agent $ sudo snap remove acme-agent its tasks say whichsnap they touchnew changerefusederror: snap "acme-agent" has "install-snap" change in progress change 27 · Donenothing is queued: once change 27 is Done, the same command is accepted $ sudo snap install acme-toolschange 28 · DoingInstall "acme-tools" snapa different snap: both changes run a few changes, such as a remodel, refuse every other change until they finish
Change 27, Install acme-agent snap, is Doing. sudo snap remove acme-agent is refused before a change is made, with error: snap acme-agent has install-snap change in progress. Nothing is queued; once change 27 is Done the command is accepted. sudo snap install acme-tools makes change 28, which runs alongside. A remodel refuses every other change until it finishes

Creating a change fails with snap "acme-agent" has "install-snap" change in progress when an unfinished change has a task that affects that snap. The request is refused, not queued. Changes on other snaps run in parallel.

The install of acme-agent is still running, and someone else types snap remove acme-agent. What should happen? Two changes pulling one snap opposite ways would be hard to undo, so snapd won't start the second. Before creating a change, snapd asks every task of every unfinished change which snaps it affects. If one is yours, the error names the snap and the kind of change in the way. Nothing's queued; try again once the first change is done. A change on another snap isn't affected and runs alongside. And a few changes that touch the whole device, a remodel for instance, refuse everything else until they finish.

07 Abort, waiting and expiry for administrators for snapd developers

HOW A CHANGE ENDS change 27 · Doingspawned+3 days $ sudo snap abort 27returns at once: the change is flagged,and the undo runs in the next passes on the next passes, each task:Do → HoldDoing → Aborttold to stopDone → Undo Ctrl-C in a waiting snap install: abortsCtrl-C in snap watch: does not $ sudo snap abort 27error: cannot abort change 27 with nothing pendingonce the change is finished unfinished: aborted automaticallyunless it is waiting for a rebootWait a finished changeready+24 hours: removed from the listat most 500 are kept
sudo snap abort 27 flags the change and returns; on the next passes Do becomes Hold, Doing becomes Abort and Done becomes Undo. Ctrl-C in a waiting snap install aborts, in snap watch it does not. Aborting a finished change fails with cannot abort change 27 with nothing pending. On a time line, a change still unfinished 3 days after it was spawned is aborted automatically, unless it is waiting for a reboot with a Wait task; a finished change is removed from the list after 24 hours, and at most 500 are kept

snap abort 27 flags the change and returns; the undo happens in later passes. An unfinished change is aborted automatically 3 days after it was spawned, unless it waits for a system restart. Finished changes are pruned after 24 hours, and beyond 500.

Most changes finish in seconds. This one has sat at doing for an hour, on a download that won't come. What now? Snap abort 27 doesn't kill anything. It flags the change and returns; the runner does the rest: unstarted tasks are held, the running one is told to stop, and finished ones are undone. Snap tasks shows the undo. Ctrl-C in a waiting snap install does the same; in snap watch it doesn't. Once a change is finished, abort is refused. Left alone, a change unfinished three days after it began is aborted, unless it's waiting for a reboot. Finished changes stay listed for a day.

08 Where to go from here

ROADS The overlord Refreshthe busiest change, a lane per snap Remove and reverttwo more kinds of changeReboots and snap reboota task that waits for a reboot Hooksthe hook tasks of an installHow interfaces are implementedconnections and security profilesThe storethe download and validate tasks snap debugstate and timings of a change snapdback to the overview
Roads from the overlord: refresh, remove and revert, reboots and snap reboot, hooks, how interfaces are implemented, the store, snap debug, and back to the snapd overview

Roads: refresh, remove and revert, reboots, hooks, interfaces, the store, the snap debug tools and the overview.

Once you can read a change, the rest of this district gets easier, because almost everything in it is a change of one kind or another. Refresh is the busiest, with a lane per snap. Remove and revert are changes too. On Ubuntu Core a change can stop halfway and wait for a reboot. Hooks, interface connections and downloads are all tasks you've now seen from the outside, and their own decks show the inside. And when you need to look into a change more closely than snap tasks allows, the snap debug deck has the tools.

Read more

Commands

CommandWhat it shows or does
snap changes [<snap>]recent changes: ID Status Spawn Ready Summary; - under Ready means unfinished
snap tasks <id> (also snap change <id>)the tasks of one change: Status Spawn Ready Summary, a percentage on a task in progress, then each task's log
snap tasks --last=installthe most recent change of that type; also on abort and watch
snap watch <id>waits for a change, also through a reboot wait; Ctrl-C does not abort it
snap abort <id>asks for an unfinished change to be undone; returns at once
snap install --no-wait ...prints the change id and returns
snap debug state <file>reads a copy of state.json without the daemon; see snap debug

--last accepts install, refresh, remove, revert, enable, disable, switch, connect, disconnect, configure, try, and other change kinds such as auto-refresh written in full. A trailing ? makes "none found" a success.

Ctrl-C in a command that waits for its change, such as snap install, asks snapd to abort the change. A dropped connection does not: the change carries on.

Statuses

StatusMeaningFinished?
Dowaiting to runno
Doingthe handler is runningno
Donefinishedyes
Abortrunning, and told to stopno
Undofinished earlier, waiting to be undoneno
Undoingthe undo handler is runningno
Undoneundoneyes
Holdwill not runyes
Errorfailedyes
Waitwaiting for something outside snapd, such as a rebootno

A change shows the first of these statuses that any of its tasks has: Abort, Undoing, Undo, Doing, Do, Wait, Error, Undone, Done, Hold. So a change shows Error when any task is in Error, even if every other task was undone. Wait is shown only when every task that could still run is blocked behind a waiting task. The error text of a change is cannot perform the following tasks: followed by one line per failed task. A task log keeps its last 10 lines.

What an abort does to each task

Task wasBecomes
DoHold
DoingAbort. When the handler returns nil: Undo, then Undoing and Undone, or Done if its kind has no undo handler. When it asks to be retried or to wait: Undo, or Hold if its kind has no undo handler. When it returns an error: Error
DoneUndo, then Undoing, then Undone; back to Done if its kind has no undo handler
Waittreated as the status it was waiting to reach

The same rules apply when a task fails, to the lanes of the failed task. The failed task itself is not undone by the runner: its handler is expected to clean up before returning the error. So an aborted or failed change can contain Done and Hold tasks, and "aborted" does not always mean "fully reverted".

Limits and timers

WhatValue
Ensure passevery 5 minutes, or earlier on request
prune passevery 10 minutes
finished changes kept24 hours, and at most 500
unfinished change aborted3 days after it was spawned, unless it waits for a system restart
state write retryevery 3 seconds for up to 5 minutes, then snapd panics
wait for the state lock file at startup to 1 minute
client polling a changeevery 100 milliseconds

for snapd developers for administrators

state.json

KeyHolds
datathe managers' own entries, for example the installed snaps and the connections
changesevery change: id, kind, summary, status, task-ids, spawn-time, ready-time, data
tasksevery task: id, kind, summary, status, wait-tasks, halt-tasks, lanes, log, progress, data, change, the times
warnings, noticespending warnings and notices
last-change-id, last-task-id, last-lane-id, last-notice-idthe counters

The file is /var/lib/snapd/state.json, mode 0600, replaced atomically. Beside it is /var/lib/snapd/state.lock, taken by snapd at start. Statuses are stored as numbers: Hold 1, Do 2, Doing 3, Done 4, Abort 5, Undo 6, Undoing 7, Undone 8, Error 9, Wait 10.

for snapd developers

Managers, in the order Ensure calls them

restart, hookstate, snapstate, servicestate, assertstate, ifacestate, fdestate, devicestate, clusterstate, cmdstate, snapshotstate, confdbstate, certstate, devicemgmtstate, and then the task runner. configstate and healthstate are initialised but are not managers. A manager's Ensure must not do long-running work; that belongs in changes and tasks.

for snapd developers

Rules for a task handler

  • Register it with AddHandler(kind, do, undo); undo may be nil.
  • Start with st.Lock(); defer st.Unlock() and release the lock only around slow I/O or network calls.
  • It can run again after a restart, so external effects must be idempotent. State writes that are not must be made together with Task.SetStatus before the lock is released.
  • Return *Retry to be run again, optionally after a delay; return *Wait when the task needs something outside snapd; any other error fails the task and aborts its lanes.
  • A blocked predicate (AddBlocked) delays a task of an accepted change; a conflict check refuses a new change. They are different tools.
  • A task kind with no handler, left by another version of snapd, is logged as no handler for task "<kind>", task ignored and completes as Done.

for snapd developers for administrators

Lanes

FlagLanes
--transaction=per-snap (the default)one per snap
--transaction=all-snapsone for all the snaps of the command

A task of an aborted lane is spared when it is also in another lane whose tasks are all still live. A task that joined no lane is in lane 0.

for snapd developers for administrators

Conflicts

SituationMessage
an unfinished change affects the snapsnap "<name>" has "<kind>" change in progress
the same, kind not knownsnap "<name>" has changes in progress
a remodel is runningremodeling in progress, no other changes allowed until this is done
a change that must run alone meets othersother changes in progress (conflicting change "<kind>"), change "<kind>" not allowed until they are done

The API answers with HTTP 409 and the error kind snap-change-conflict. Changes that block every other change while unfinished: transition-ubuntu-core, transition-to-snapd-snap, remodel, create-recovery-system, remove-recovery-system, a downgrade of snapd, and a refresh of the seed. Changes that are ignored by the check: finished ones, and the kinds get-confdb, set-confdb, pre-download and become-operational.

A task says which snaps it affects through its snap setup, or through a function registered for its kind (connect, disconnect, quota-control and others) or for an attribute it carries (hook-setup, service-action, snapshot-setup). A task that matches none affects no snap.

for snapd developers for administrators

Restarts inside a change

RestartWhat the task does
of snapdthe handler asks to be retried while the restart is pending (Waiting for automatic snapd restart...); the new process runs it again
of the systemthe task goes to Wait with the current boot id; after a boot with a different id it moves on
of the system, on a classic systemno automatic reboot: the change waits until someone reboots

See Reboots for the delay, the fallback and what happens when the reboot does not occur.

The tasks of an install

Task kindSummary shown by snap tasks
prerequisitesEnsure prerequisites for "acme-agent" are available
download-snapDownload snap "acme-agent" (42) from channel "stable"
validate-snapFetch and check assertions for snap "acme-agent" (42)
prerequisitesWait until prerequisites for "acme-agent" are available
mount-snapMount snap "acme-agent" (42)
copy-snap-dataCopy snap "acme-agent" data
setup-profilesPrepare snap "acme-agent" (42) for security profile setup
link-snapMake snap "acme-agent" (42) available to the system
auto-connectAutomatically connect eligible plugs and slots of snap "acme-agent"
set-auto-aliasesSet automatic aliases for snap "acme-agent"
setup-aliasesSetup snap "acme-agent" aliases
run-hookRun install hook of "acme-agent" snap if present
run-hookRun default-configure hook of "acme-agent" snap if present
start-snap-servicesStart snap "acme-agent" (42) services
run-hookRun configure hook of "acme-agent" snap if present
run-hookRun health check of "acme-agent" snap

auto-connect adds further tasks to the change while it runs: one connect per connection, the interface hooks, and a final setup-profiles. prerequisites and validate-snap have no undo handler. See Hooks and the interface implementation.

Further reading

Roads from here

START HERE

  • snapd — back to the overview

GETTING SNAPS

  • The store — the download and validate tasks at the start of an install

INTERFACES AND CONFINEMENT

RUNNING SNAPS

HOOKS · code snapd runs inside a snap

TOOLS

UBUNTU CORE · boot

INSIDE SNAPD · the daemon itself