How snapd restarts and re-executes

daemon lifecycle for snapd developers for administrators

snapd is refreshed as a snap while it is running. On classic Debian and Ubuntu the packaged binaries re-execute into a snapd snap that is not older. A snapd refresh restarts the daemon in the middle of its own change. snap-failure starts the previous snapd when the new one cannot run. With no snaps installed the daemon exits and socket activation starts it again.

On the map: inside snapd district

▶ Watch this deck — 8 scenes, ~6 min

01 A program that replaces itself

THE DAEMON'S LIFE snapd, old revisionnew snapd snap restartsnapd, new revisionsame change, picked up where it stopped snapd, previous revisionif the new one cannot start snapd from the packagesnapd from the snapclassic Ubuntu and Debian snapd exitsno snaps: wait for a connection
snapd's life on one time line: the old revision runs until a new snapd snap arrives and restarts into the new revision in the same change; a dashed path leads back to the previous revision if the new one cannot start; on classic Ubuntu and Debian a packaged snapd hands over to the snapd from the snap; with no snaps the daemon exits and waits for a connection

snapd refreshes itself. It restarts into the new revision during its own change, falls back to the previous revision if the new one fails, re-executes from the package into the snap on classic systems, and exits when it has nothing to manage.

Every snap on your machine is refreshed by snapd. So who refreshes snapd? It does, and it has to do it without sawing off the branch it's sitting on. That takes four things. The running daemon has to stop in the middle of its own change and let the new one carry on. If the new one can't start, something that isn't snapd has to bring the previous one back. On a classic system, a packaged snapd has to hand over to the copy in the snap. And when there's nothing to manage, the daemon can leave and be started again on demand.

02 The packaged binary hands over for snapd developers for administrators

RE-EXEC $ systemctl start snapd/usr/lib/snapd/snapdfrom the package, 2.76 /snap/snapd/current → 101usr/lib/snapd/snapdsame path in the snap? usr/lib/snapd/infoVERSION=2.78vs 2.76 execsame arguments,same environment also: snap, snapd-apparmor,snap-gpio-helperfallback: /snap/core/current usr/lib/snapd/snapd/proc/<pid>/exe → /snap/snapd/101/usr/lib/snapd/snapdthe running daemon is the copy from the snap
Re-execution on the laptop: systemctl starts the packaged snapd 2.76 from /usr/lib/snapd/snapd; it finds the same path inside /snap/snapd/current, reads VERSION=2.78 from usr/lib/snapd/info, and because that copy is not older it calls exec on it with the same arguments and environment; /proc shows the daemon's executable under /snap/snapd/101

A packaged snapd or snap looks for itself under /snap/snapd/current, then /snap/core/current, reads usr/lib/snapd/info there and, unless that version is older, calls exec on the copy in the snap with the same arguments and environment.

On the laptop there are two snapds: one from the Ubuntu archive and one inside the snapd snap. Only one should do the work. Here's how they settle it at every start. The packaged binary asks: am I running from a snap? No. Is there a copy of me at the same path under /snap/snapd/current, or failing that, in the core snap? Yes. What version does its info file give? 2.78, and I'm 2.76. That copy isn't older, so I replace myself with it, same arguments, same environment. The snap command does this too, and so do two helpers. In proc, the daemon's executable is in the snap.

03 When it stays where it is for snapd developers for administrators for Ubuntu Core developers

SNAP_REEXEC packaged binaryexec into snapstay: keep running the packaged binary SNAP_REEXEC=0disabled by user not classic, or notlike debian or ubuntuSNAP_REEXEC=1 no such binary inthe snapd or core snap snap older thanthe packageSNAP_REEXEC=force for the daemon:a drop-in for snapd.service# /etc/systemd/system/snapd.service.d/reexec.conf[Service]Environment=SNAP_REEXEC=0
Four gates between the packaged binary and exec into the snap, each with an exit to stay: SNAP_REEXEC=0 disables it; not classic or not like debian or ubuntu, bypassed by SNAP_REEXEC=1; no such binary in the snapd or core snap; the snap is older than the package, bypassed by SNAP_REEXEC=force; and a systemd drop-in that sets SNAP_REEXEC=0 for snapd.service

SNAP_REEXEC=0 keeps the packaged binary. Unset, re-execution happens only on classic systems like Debian or Ubuntu; a true value enables it elsewhere, and force skips the version comparison. On Ubuntu Core and other unsupported systems it is off unless set to a true value.

You've rebuilt snapd with your fix, started it, and nothing changed. Before you doubt the fix, check which snapd is running. Yours probably stepped aside. The switch is one environment variable, SNAP_REEXEC. Set it to zero and the binary stays put. Unset means on, but only where it's supported: classic systems like Debian or Ubuntu. On Ubuntu Core, like the gateway, and on other distributions, it's off unless you set a true value. With no copy in the snapd or core snap, it stays. A snap older than the package is skipped, unless you say force. For the daemon, set it in a drop-in for the service.

04 A refresh that restarts its own worker for snapd developers for administrators

REFRESHING SNAPD $ snap refresh snapddownloadmount…link-snap Requested daemon restart(snapd snap). auto-connect…Waiting for automaticsnapd restart...retry snapd, revision 101stops; systemd starts it again snapd, revision 102reads the state,finds the changeauto-connect… on Ubuntu Core: snapd's own units rewritten by the new code
snap refresh snapd on the gateway: tasks run up to link-snap, which logs Requested daemon restart (snapd snap); auto-connect logs Waiting for automatic snapd restart and retries; below, snapd revision 101 stops and systemd starts it again as revision 102, which continues the waiting tasks; on Ubuntu Core it also rewrites snapd's own units

link-snap for the snapd snap requests restart-daemon. Later tasks log Waiting for automatic snapd restart... and retry until the new process is running; it then continues the same change and, on Ubuntu Core, regenerates snapd's own units.

Run snap refresh snapd on the gateway and watch the change. Somewhere in the middle, the program running the change is replaced. The change doesn't notice. Everything goes as usual up to link-snap, which makes revision 102 current. That task ends by asking for a daemon restart, and says so in its log. The next task, auto-connect, sees the restart pending, logs that it's waiting, and asks to be retried. Revision 101 stops. Systemd starts the service again, from 102 this time. It reads the state, finds the change, and the waiting task goes ahead. On Ubuntu Core, 102 also rewrites snapd's own units with its new code.

05 How the daemon stops for snapd developers

STOP restart requestedHandleRestartwritemaintenance.json shut down overlordclose the snap socket 3 s later: closethe main socketthen up to 25 s for requests in flight then, by typerestart-daemonexit; systemd starts snapd againrestart-system…reboot, halt or power offsee Reboots restart-socketexit 42; wait for a connection stop-daemonjust stopimage preseeding
The stop sequence: restart requested through HandleRestart, write maintenance.json, shut down the overlord and close the snap socket, three seconds later close the main socket and allow 25 seconds for requests in flight; then four endings by type: restart-daemon exits and systemd starts snapd again, the system types reboot, restart-socket exits with status 42 and waits for a connection, stop-daemon just stops for image preseeding

Daemon.Stop writes the maintenance notice, shuts down the overlord, closes the snap socket, closes the main socket about 3 s later and waits up to 25 s for requests. The restart type decides the ending: restart-daemon, a system restart, restart-socket (exit 42) or stop-daemon.

Restart snapd. Inside it's a small ceremony, and it's the same whether the daemon comes back in a second, after a reboot, or not at all. A task, or the idle monitor, asks for a restart. First, a maintenance notice for clients. It waits for the requester to release the state, shuts the overlord down and closes the snap socket. Three seconds later it closes the main socket, and gives requests in flight up to twenty five seconds. Only now does the type matter. restart-daemon exits, and systemd starts snapd again. The system types reboot. restart-socket exits with status forty two. stop-daemon just stops; preseeding uses it.

06 "snap-failure: the way back" for snapd developers for administrators for Ubuntu Core developers

SNAP-FAILURE snapd.servicerevision 102the new snapd fails to start OnFailure=snapd.failure.servicesnap-failure snapd /var/lib/snapd/sequence/snapd.jsoncurrent 102, previous 101 /snap/snapd/current102101snapd.socket stopped first snapd 101SNAPD_REVERT_TO_REV=101started with systemd-run there was a snapd rollback across the restartchange undonethen the socket andthe service are restarted
snap-failure on the gateway: snapd.service with revision 102 fails; OnFailure= starts snapd.failure.service, which runs snap-failure snapd; it reads the previous revision 101 from /var/lib/snapd/sequence/snapd.json, stops snapd.socket, points /snap/snapd/current at 101 instead of 102, and starts snapd 101 with systemd-run and SNAPD_REVERT_TO_REV=101; the waiting task fails with there was a snapd rollback across the restart and the change is undone; then the socket and the service are restarted

snapd.service has OnFailure=snapd.failure.service, which runs snap-failure snapd. It finds the previous revision in the sequence file, points /snap/snapd/current at it and runs that snapd with SNAPD_REVERT_TO_REV set; the old snapd undoes the refresh.

Now the bad day. Revision 102 of snapd reaches the gateway, and it won't start. The only program that knows how to roll a snap back is the one that's broken. So the snapd service names a unit to run when it fails. That unit runs snap-failure, a small program separate from snapd. In the sequence file, the revision before the current one is 101. It stops the socket, points the current link at 101, and starts it with SNAPD_REVERT_TO_REV set. The old snapd finds the half-done refresh, the waiting task fails, and the change is undone. Then the socket and service are restarted.

07 "Nothing to do: exit and wait" for snapd developers for administrators

IDLE EXIT a container with no snapssnapdsnapd.socketsnapd-snap.socket 5 s since startno change in flightno open connection the loop has run oncezero snaps installedseededbusy? ask again later, the wait doubling to about 5 min restart-socket(snapd-idle)exit status 42snapd SuccessExitStatus=42RestartPreventExitStatus=42systemd keepslistening $ snap listsnapda connection starts snapd again
Idle exit in a container with no snaps: a checklist of 5 seconds since start, no change in flight, no open connection, the loop has run once, zero snaps installed and seeded; snapd requests restart-socket with reason snapd-idle and exits with status 42, which snapd.service treats as success and does not restart; systemd keeps listening on snapd.socket and snapd-snap.socket, and the next snap list starts snapd again

When no snaps are installed, the device is seeded, no change or connection is open and 5 s have passed, snapd requests restart-socket and exits with status 42. snapd.service treats 42 as success and does not restart; a connection to either socket starts snapd again.

Look at a fresh container: snapd is installed, the sockets are there, and no snapd is running. It left on purpose. A daemon with nothing to manage only uses memory. A monitor checks: five seconds since start, every change finished, no open connection, the main loop has run once, zero snaps installed, the device seeded. If anything's busy, it tries again later, doubling the wait up to about five minutes. Once all agree, it asks for restart-socket, reason snapd-idle. It stops as usual and exits with status forty two: success, so systemd doesn't restart it. Systemd still holds the sockets, so the next snap command starts snapd again.

08 Where to go from here

ROADS Restart andre-executionThe snapd REST APIwhile the daemon is away The overlorda change outlives the process Refreshwhen a snapd refresh arrives Reboots and snap rebootthe system restart types State format patchesthe new snapd's first step Images and preseedingstop-daemon ends a preseed run
Roads from How snapd restarts and re-executes: the snapd REST API, the overlord, refresh, reboots and snap reboot, state format patches, and preparing images and preseeding

Roads: the REST API during a restart, the overlord and refreshes, reboots, state patches, and image preseeding.

So snapd can replace itself, back out, and step aside. Each of those touches something another deck explains. The REST API deck shows what a client sees while the daemon is away. The overlord deck shows why a change can outlive the process, and the refresh deck shows when a refresh of snapd arrives. The system restart types end in a reboot, which has its own deck. The state patches deck covers the first thing a new snapd does with the old state. And the image deck shows the one place where the daemon is asked to stop for good.

Read more

Restart types

TypeRequested whenHow the daemon ends
restart-daemonsnapd was refreshed, reverted or undone; a feature that needs a restart changedexits; systemd starts snapd.service again
restart-systema change needs a rebootreboot after 1 minute, with a 10 minute fallback scheduled first
restart-system-nowan immediate reboot was requestedreboot at once
restart-system-halt-now, restart-system-poweroff-nowhalt or power offhalt or power off at once
restart-socketthe daemon is idleexits with status 42; socket activation starts it later
stop-daemonimage preseeding reached mark-preseededstops

Reasons recorded with a daemon restart: snapd-update, snapd-revert, snapd-undo, snapd-feature-change, and snapd-idle for idle exit. System restarts are explained in Reboots and snap reboot.

The stop sequence

  • Record the requested restart and write /var/lib/snapd/maintenance.json.
  • Wait for the code that requested the restart to release the state.
  • Shut down the overlord and close /run/snapd-snap.socket.
  • About 3 seconds after the overlord shutdown began, close /run/snapd.socket and stop the idle monitor.
  • Give in-flight requests up to 25 seconds.
  • Tell systemd STOPPING=1 unless the system is restarting.
  • For restart-socket, check the idle conditions once more; if a request made snapd busy, do an ordinary daemon restart instead.
  • Stop the overlord; then end as the restart type says.

Re-execution, in order

  • Already running from a snap: stay.
  • SNAP_REEXEC is false: stay, and log re-exec disabled by user at debug level.
  • Not a classic system, or not like debian or ubuntu: stay, unless SNAP_REEXEC is set to a true value or to force.
  • Look for the same executable under /snap/snapd/current, then /snap/core/current. A path under /usr/libexec/snapd is looked up under usr/lib/snapd. Not found: stay.
  • Read VERSION= from usr/lib/snapd/info in that snap. Unreadable: stay. Older than the running binary: stay, unless SNAP_REEXEC=force. The same version counts as not older.
  • exec the binary from the snap with the same arguments and environment.

Programs that do this: snapd, snap, snapd-apparmor and snap-gpio-helper. snapd skips it when preseeding. The published distribution-support page describes re-execution as the package installing the snapd snap; in the code it is the packaged binary executing the copy in the snap.

Environment variables

VariableEffect
SNAP_REEXECunset or true: re-execute where supported; 0: stay; a true value on an unsupported distribution: re-execute; force: skip the version comparison
SNAPD_REVERT_TO_REVset by snap-failure; tells snapd it was started to handle a failure
SNAPD_STANDBY_WAITreplaces the 5 s idle wait
SNAPD_REBOOT_DELAYreplaces the 1 minute reboot delay; meant for tests
SNAPD_PRESEEDpreseed mode; no re-execution
WATCHDOG_USECset by systemd; snapd reports to the watchdog at half this interval

for administrators for Ubuntu Core developers for snapd developers

What the units say

UnitSettingMeaning
snapd.serviceRequires=snapd.socketthe sockets belong to systemd
snapd.serviceRestart=alwaysan exit is followed by a start
snapd.serviceSuccessExitStatus=42, RestartPreventExitStatus=42idle exit is a success and is not restarted
snapd.serviceOnFailure=snapd.failure.servicea failure runs snap-failure
snapd.serviceWatchdogSec=5msnapd must report to systemd regularly
snapd.failure.serviceType=oneshot, ExecStart=.../snap-failure snapdone attempt at recovery

To keep the packaged daemon on a classic system, add a drop-in such as /etc/systemd/system/snapd.service.d/reexec.conf and run systemctl daemon-reload:

[Service]
Environment=SNAP_REEXEC=0

snap-failure, step by step

  • On a classic system, do nothing unless re-execution is supported and enabled.
  • Read /var/lib/snapd/sequence/snapd.json and find the revision before the current one. No file: no snapd snap in the system, stop.
  • No previous revision: use the current snapd and leave the current link alone. On Ubuntu Core 16 that is the snapd in the core snap; on a classic system the core snap is tried first, then the snapd snap.
  • systemctl stop snapd.socket.
  • Point /snap/snapd/current at the previous revision.
  • Run that snapd with systemd-run --collect --wait and SNAPD_REVERT_TO_REV=<revision>.
  • If the units are not failed, check up to 5 times, 5 s apart, whether the socket and the service are active and both socket files exist; if so, stop.
  • Otherwise systemctl reset-failed, remove leftover socket files and restart snapd.socket and snapd.service; on failure wait 12.5 s and try once more.

The snapd started this way looks for an unfinished change in which link-snap of the snapd snap is done and its auto-connect is waiting. If it finds one, that change is undone. If it finds none it logs detected failure recovery context, but no recovery needed and exits without changing anything.

Known defect in that second case, found by reading the code and not yet reproduced: snap-failure has already pointed /snap/snapd/current at the previous revision and does not point it back, while snapd's state still names the newer revision as current. After a failure that happened outside a snapd refresh, compare readlink /snap/snapd/current with the revision in snap list snapd.

Idle exit

All of these must hold: 5 s since start; every change ready and clean; no open connection; the ensure loop has run once; no snaps installed; the device is seeded. The monitor checks again after a wait that doubles from 5 s while it is below 5 minutes. The daemon prints daemon stop requested to wait for socket activation and exits with status 42.

If an expected reboot did not happen

snapd schedules the reboot again, at once, and waits 5 minutes; it does this up to 3 times and then carries on. See Reboots and snap reboot.

Further reading

No published page explains snap-failure, the restart types or idle exit. The nearest pages:

Roads from here

RUNNING SNAPS

UBUNTU CORE · boot

INSIDE SNAPD · the daemon itself