The overlord
changes, tasks, state for snapd developers for administrators
Every operation snapd performs is a change made of tasks. Changes are saved in state.json and run by one task runner that calls handlers owned by managers such as snapstate and ifacestate. A failed task undoes the tasks before it within its lane. snap changes and snap tasks show the result and snap abort asks for the undo.
On the map: the town hall
01 Every command is a change
A snap command asks snapd for an operation and gets back the id of a change. The change is a set of tasks that snapd runs by itself. snap changes lists changes; snap tasks 27 lists the tasks of one.
You start an install over SSH, and the connection drops halfway through. Is the gateway now half installed? No, because the command wasn't doing the work. It sent a request, snapd answered with the number of a change, 27, and from then on the command only watched. The change is the operation: an ordered set of small steps called tasks, each with a plain summary. Download, check its assertions, mount, make it available, start the services, and about ten more. Snapd runs them whether anyone's watching or not. Snap changes lists recent changes with their status, and snap tasks 27 shows the steps of this one.
02 When a task fails
A failed task becomes Error. Tasks not yet started become Hold; finished tasks go Undo, Undoing, Undone in reverse order. A kind without an undo handler stays Done. The change reports Error.
Same install, but this time the service refuses to start. Over a dozen steps are done, four shown here, and two haven't begun. What does snapd do with them? First, the failed task is marked error. The two that hadn't started go to hold, so they won't run. The finished ones are undone, last first: the snap is made unavailable, unmounted, and its download removed. One row still says done. Checking the assertions changed nothing, so there's nothing to undo. The change reports error, and its message names the task that failed. So error usually means you're back where you started, and the task list tells you exactly.
03 One file remembers everything for snapd developers for administrators
state.json holds data, changes and tasks. It is rewritten atomically on every Unlock that follows a modification. After a restart a task found in Doing has its handler run again, so handlers must be idempotent.
I said the change lives in snapd. More exactly, in one file, and that's why a change can outlive the process that started it. state.json holds it all: installed snaps and connections under data, every change and task with its status. In memory, one lock guards it, and touching the state without it panics. Release it after a modification, and the whole file is written again, atomically. Say snapd stops mid-mount. The next snapd reads the file once, finds a task at doing with nothing running it, and runs its handler again. So handlers must be safe to run twice. A snapd refresh is the everyday case.
04 Managers, the loop and the runner for snapd developers
Overlord calls Ensure on fourteen managers and then on the shared TaskRunner, every 5 minutes or as soon as EnsureBefore asks. The runner starts a Do task when all its wait tasks are Done, in a goroutine of its own.
Who picks up a task and runs it? A loop, a list of managers and one runner. Each pass, the overlord calls ensure on each of fourteen managers: hookstate, snapstate, ifacestate, devicestate and the rest. Ensure does quick checks, like whether an auto-refresh is due; slow work becomes a change. The task runner goes last. It starts each task whose wait tasks are all done, with the handlers its manager registered: do, and usually undo. The handler gets its own goroutine and takes the state lock itself. Passes come every five minutes, but finishing a task asks for the next one at once.
05 "Lanes: how far a failure spreads" for snapd developers for administrators
A failed task aborts its lanes, not the whole change. With the default --transaction=per-snap each snap has its own lane; with --transaction=all-snaps they share one. A change with any Error task reports Error.
You install two snaps with one command, and the second one fails. Should the first come out again? You get to choose, and the mechanism is the lane: a group of tasks that fail together. By default each snap in the command gets its own lane, so a failure in acme-tools undoes acme-tools only, and acme-agent stays installed. The change still reports error, because one of its tasks failed, so read the task list before deciding nothing happened. With --transaction=all-snaps, both share one lane, and a failure anywhere undoes both. An auto-refresh of many snaps works the same way, one lane per snap.
06 One change per snap at a time
Creating a change fails with snap "acme-agent" has "install-snap" change in progress when an unfinished change has a task that affects that snap. The request is refused, not queued. Changes on other snaps run in parallel.
The install of acme-agent is still running, and someone else types snap remove acme-agent. What should happen? Two changes pulling one snap opposite ways would be hard to undo, so snapd won't start the second. Before creating a change, snapd asks every task of every unfinished change which snaps it affects. If one is yours, the error names the snap and the kind of change in the way. Nothing's queued; try again once the first change is done. A change on another snap isn't affected and runs alongside. And a few changes that touch the whole device, a remodel for instance, refuse everything else until they finish.
07 Abort, waiting and expiry for administrators for snapd developers
snap abort 27 flags the change and returns; the undo happens in later passes. An unfinished change is aborted automatically 3 days after it was spawned, unless it waits for a system restart. Finished changes are pruned after 24 hours, and beyond 500.
Most changes finish in seconds. This one has sat at doing for an hour, on a download that won't come. What now? Snap abort 27 doesn't kill anything. It flags the change and returns; the runner does the rest: unstarted tasks are held, the running one is told to stop, and finished ones are undone. Snap tasks shows the undo. Ctrl-C in a waiting snap install does the same; in snap watch it doesn't. Once a change is finished, abort is refused. Left alone, a change unfinished three days after it began is aborted, unless it's waiting for a reboot. Finished changes stay listed for a day.
08 Where to go from here
Roads: refresh, remove and revert, reboots, hooks, interfaces, the store, the snap debug tools and the overview.
Once you can read a change, the rest of this district gets easier, because almost everything in it is a change of one kind or another. Refresh is the busiest, with a lane per snap. Remove and revert are changes too. On Ubuntu Core a change can stop halfway and wait for a reboot. Hooks, interface connections and downloads are all tasks you've now seen from the outside, and their own decks show the inside. And when you need to look into a change more closely than snap tasks allows, the snap debug deck has the tools.
Read more
Commands
| Command | What it shows or does |
|---|---|
snap changes [<snap>] | recent changes: ID Status Spawn Ready Summary; - under Ready means unfinished |
snap tasks <id> (also snap change <id>) | the tasks of one change: Status Spawn Ready Summary, a percentage on a task in progress, then each task's log |
snap tasks --last=install | the most recent change of that type; also on abort and watch |
snap watch <id> | waits for a change, also through a reboot wait; Ctrl-C does not abort it |
snap abort <id> | asks for an unfinished change to be undone; returns at once |
snap install --no-wait ... | prints the change id and returns |
snap debug state <file> | reads a copy of state.json without the daemon; see snap debug |
--last accepts install, refresh, remove, revert, enable, disable, switch, connect, disconnect, configure, try, and other change kinds such as auto-refresh written in full. A trailing ? makes "none found" a success.
Ctrl-C in a command that waits for its change, such as snap install, asks snapd to abort the change. A dropped connection does not: the change carries on.
Statuses
| Status | Meaning | Finished? |
|---|---|---|
Do | waiting to run | no |
Doing | the handler is running | no |
Done | finished | yes |
Abort | running, and told to stop | no |
Undo | finished earlier, waiting to be undone | no |
Undoing | the undo handler is running | no |
Undone | undone | yes |
Hold | will not run | yes |
Error | failed | yes |
Wait | waiting for something outside snapd, such as a reboot | no |
A change shows the first of these statuses that any of its tasks has: Abort, Undoing, Undo, Doing, Do, Wait, Error, Undone, Done, Hold. So a change shows Error when any task is in Error, even if every other task was undone. Wait is shown only when every task that could still run is blocked behind a waiting task. The error text of a change is cannot perform the following tasks: followed by one line per failed task. A task log keeps its last 10 lines.
What an abort does to each task
| Task was | Becomes |
|---|---|
Do | Hold |
Doing | Abort. When the handler returns nil: Undo, then Undoing and Undone, or Done if its kind has no undo handler. When it asks to be retried or to wait: Undo, or Hold if its kind has no undo handler. When it returns an error: Error |
Done | Undo, then Undoing, then Undone; back to Done if its kind has no undo handler |
Wait | treated as the status it was waiting to reach |
The same rules apply when a task fails, to the lanes of the failed task. The failed task itself is not undone by the runner: its handler is expected to clean up before returning the error. So an aborted or failed change can contain Done and Hold tasks, and "aborted" does not always mean "fully reverted".
Limits and timers
| What | Value |
|---|---|
| Ensure pass | every 5 minutes, or earlier on request |
| prune pass | every 10 minutes |
| finished changes kept | 24 hours, and at most 500 |
| unfinished change aborted | 3 days after it was spawned, unless it waits for a system restart |
| state write retry | every 3 seconds for up to 5 minutes, then snapd panics |
| wait for the state lock file at start | up to 1 minute |
| client polling a change | every 100 milliseconds |
for snapd developers for administrators
state.json
| Key | Holds |
|---|---|
data | the managers' own entries, for example the installed snaps and the connections |
changes | every change: id, kind, summary, status, task-ids, spawn-time, ready-time, data |
tasks | every task: id, kind, summary, status, wait-tasks, halt-tasks, lanes, log, progress, data, change, the times |
warnings, notices | pending warnings and notices |
last-change-id, last-task-id, last-lane-id, last-notice-id | the counters |
The file is /var/lib/snapd/state.json, mode 0600, replaced atomically. Beside it is /var/lib/snapd/state.lock, taken by snapd at start. Statuses are stored as numbers: Hold 1, Do 2, Doing 3, Done 4, Abort 5, Undo 6, Undoing 7, Undone 8, Error 9, Wait 10.
for snapd developers
Managers, in the order Ensure calls them
restart, hookstate, snapstate, servicestate, assertstate, ifacestate, fdestate, devicestate, clusterstate, cmdstate, snapshotstate, confdbstate, certstate, devicemgmtstate, and then the task runner. configstate and healthstate are initialised but are not managers. A manager's Ensure must not do long-running work; that belongs in changes and tasks.
for snapd developers
Rules for a task handler
- Register it with
AddHandler(kind, do, undo);undomay be nil. - Start with
st.Lock(); defer st.Unlock()and release the lock only around slow I/O or network calls. - It can run again after a restart, so external effects must be idempotent. State writes that are not must be made together with
Task.SetStatusbefore the lock is released. - Return
*Retryto be run again, optionally after a delay; return*Waitwhen the task needs something outside snapd; any other error fails the task and aborts its lanes. - A blocked predicate (
AddBlocked) delays a task of an accepted change; a conflict check refuses a new change. They are different tools. - A task kind with no handler, left by another version of snapd, is logged as
no handler for task "<kind>", task ignoredand completes asDone.
for snapd developers for administrators
Lanes
| Flag | Lanes |
|---|---|
--transaction=per-snap (the default) | one per snap |
--transaction=all-snaps | one for all the snaps of the command |
A task of an aborted lane is spared when it is also in another lane whose tasks are all still live. A task that joined no lane is in lane 0.
for snapd developers for administrators
Conflicts
| Situation | Message |
|---|---|
| an unfinished change affects the snap | snap "<name>" has "<kind>" change in progress |
| the same, kind not known | snap "<name>" has changes in progress |
| a remodel is running | remodeling in progress, no other changes allowed until this is done |
| a change that must run alone meets others | other changes in progress (conflicting change "<kind>"), change "<kind>" not allowed until they are done |
The API answers with HTTP 409 and the error kind snap-change-conflict. Changes that block every other change while unfinished: transition-ubuntu-core, transition-to-snapd-snap, remodel, create-recovery-system, remove-recovery-system, a downgrade of snapd, and a refresh of the seed. Changes that are ignored by the check: finished ones, and the kinds get-confdb, set-confdb, pre-download and become-operational.
A task says which snaps it affects through its snap setup, or through a function registered for its kind (connect, disconnect, quota-control and others) or for an attribute it carries (hook-setup, service-action, snapshot-setup). A task that matches none affects no snap.
for snapd developers for administrators
Restarts inside a change
| Restart | What the task does |
|---|---|
| of snapd | the handler asks to be retried while the restart is pending (Waiting for automatic snapd restart...); the new process runs it again |
| of the system | the task goes to Wait with the current boot id; after a boot with a different id it moves on |
| of the system, on a classic system | no automatic reboot: the change waits until someone reboots |
See Reboots for the delay, the fallback and what happens when the reboot does not occur.
The tasks of an install
| Task kind | Summary shown by snap tasks |
|---|---|
prerequisites | Ensure prerequisites for "acme-agent" are available |
download-snap | Download snap "acme-agent" (42) from channel "stable" |
validate-snap | Fetch and check assertions for snap "acme-agent" (42) |
prerequisites | Wait until prerequisites for "acme-agent" are available |
mount-snap | Mount snap "acme-agent" (42) |
copy-snap-data | Copy snap "acme-agent" data |
setup-profiles | Prepare snap "acme-agent" (42) for security profile setup |
link-snap | Make snap "acme-agent" (42) available to the system |
auto-connect | Automatically connect eligible plugs and slots of snap "acme-agent" |
set-auto-aliases | Set automatic aliases for snap "acme-agent" |
setup-aliases | Setup snap "acme-agent" aliases |
run-hook | Run install hook of "acme-agent" snap if present |
run-hook | Run default-configure hook of "acme-agent" snap if present |
start-snap-services | Start snap "acme-agent" (42) services |
run-hook | Run configure hook of "acme-agent" snap if present |
run-hook | Run health check of "acme-agent" snap |
auto-connect adds further tasks to the change while it runs: one connect per connection, the interface hooks, and a final setup-profiles. prerequisites and validate-snap have no undo handler. See Hooks and the interface implementation.
Further reading
- Manage updates: the section "Monitor changes" has sample
snap changesandsnap tasksoutput. - Transactional updates:
--transaction=all-snapsandper-snapfrom the user's side. - Use the REST API: the asynchronous response that carries a change id.
- Error responses: the error kinds the daemon returns, including
snap-change-conflict.
Roads from here
START HERE
- snapd — back to the overview
GETTING SNAPS
- The store — the download and validate tasks at the start of an install
INTERFACES AND CONFINEMENT
- How interfaces are implemented — connect and setup-profiles seen from the interface side
RUNNING SNAPS
- Refresh — the refresh change and its lanes per snap
- Remove and revert — the remove and revert changes
HOOKS · code snapd runs inside a snap
- Hooks: install, configure, refresh, remove — the hook tasks inside an install or refresh
TOOLS
- snap debug: tools for looking inside — snap debug state and timings for looking inside a change
UBUNTU CORE · boot
- Reboots and snap reboot — a task that waits for a reboot
INSIDE SNAPD · the daemon itself
- The snapd REST API — how commands reach snapd