How snapd restarts and re-executes
daemon lifecycle for snapd developers for administrators
snapd is refreshed as a snap while it is running. On classic Debian and Ubuntu the packaged binaries re-execute into a snapd snap that is not older. A snapd refresh restarts the daemon in the middle of its own change. snap-failure starts the previous snapd when the new one cannot run. With no snaps installed the daemon exits and socket activation starts it again.
On the map: inside snapd district
01 A program that replaces itself
snapd refreshes itself. It restarts into the new revision during its own change, falls back to the previous revision if the new one fails, re-executes from the package into the snap on classic systems, and exits when it has nothing to manage.
Every snap on your machine is refreshed by snapd. So who refreshes snapd? It does, and it has to do it without sawing off the branch it's sitting on. That takes four things. The running daemon has to stop in the middle of its own change and let the new one carry on. If the new one can't start, something that isn't snapd has to bring the previous one back. On a classic system, a packaged snapd has to hand over to the copy in the snap. And when there's nothing to manage, the daemon can leave and be started again on demand.
02 The packaged binary hands over for snapd developers for administrators
A packaged snapd or snap looks for itself under /snap/snapd/current, then /snap/core/current, reads usr/lib/snapd/info there and, unless that version is older, calls exec on the copy in the snap with the same arguments and environment.
On the laptop there are two snapds: one from the Ubuntu archive and one inside the snapd snap. Only one should do the work. Here's how they settle it at every start. The packaged binary asks: am I running from a snap? No. Is there a copy of me at the same path under /snap/snapd/current, or failing that, in the core snap? Yes. What version does its info file give? 2.78, and I'm 2.76. That copy isn't older, so I replace myself with it, same arguments, same environment. The snap command does this too, and so do two helpers. In proc, the daemon's executable is in the snap.
03 When it stays where it is for snapd developers for administrators for Ubuntu Core developers
SNAP_REEXEC=0 keeps the packaged binary. Unset, re-execution happens only on classic systems like Debian or Ubuntu; a true value enables it elsewhere, and force skips the version comparison. On Ubuntu Core and other unsupported systems it is off unless set to a true value.
You've rebuilt snapd with your fix, started it, and nothing changed. Before you doubt the fix, check which snapd is running. Yours probably stepped aside. The switch is one environment variable, SNAP_REEXEC. Set it to zero and the binary stays put. Unset means on, but only where it's supported: classic systems like Debian or Ubuntu. On Ubuntu Core, like the gateway, and on other distributions, it's off unless you set a true value. With no copy in the snapd or core snap, it stays. A snap older than the package is skipped, unless you say force. For the daemon, set it in a drop-in for the service.
04 A refresh that restarts its own worker for snapd developers for administrators
link-snap for the snapd snap requests restart-daemon. Later tasks log Waiting for automatic snapd restart... and retry until the new process is running; it then continues the same change and, on Ubuntu Core, regenerates snapd's own units.
Run snap refresh snapd on the gateway and watch the change. Somewhere in the middle, the program running the change is replaced. The change doesn't notice. Everything goes as usual up to link-snap, which makes revision 102 current. That task ends by asking for a daemon restart, and says so in its log. The next task, auto-connect, sees the restart pending, logs that it's waiting, and asks to be retried. Revision 101 stops. Systemd starts the service again, from 102 this time. It reads the state, finds the change, and the waiting task goes ahead. On Ubuntu Core, 102 also rewrites snapd's own units with its new code.
05 How the daemon stops for snapd developers
Daemon.Stop writes the maintenance notice, shuts down the overlord, closes the snap socket, closes the main socket about 3 s later and waits up to 25 s for requests. The restart type decides the ending: restart-daemon, a system restart, restart-socket (exit 42) or stop-daemon.
Restart snapd. Inside it's a small ceremony, and it's the same whether the daemon comes back in a second, after a reboot, or not at all. A task, or the idle monitor, asks for a restart. First, a maintenance notice for clients. It waits for the requester to release the state, shuts the overlord down and closes the snap socket. Three seconds later it closes the main socket, and gives requests in flight up to twenty five seconds. Only now does the type matter. restart-daemon exits, and systemd starts snapd again. The system types reboot. restart-socket exits with status forty two. stop-daemon just stops; preseeding uses it.
06 "snap-failure: the way back" for snapd developers for administrators for Ubuntu Core developers
snapd.service has OnFailure=snapd.failure.service, which runs snap-failure snapd. It finds the previous revision in the sequence file, points /snap/snapd/current at it and runs that snapd with SNAPD_REVERT_TO_REV set; the old snapd undoes the refresh.
Now the bad day. Revision 102 of snapd reaches the gateway, and it won't start. The only program that knows how to roll a snap back is the one that's broken. So the snapd service names a unit to run when it fails. That unit runs snap-failure, a small program separate from snapd. In the sequence file, the revision before the current one is 101. It stops the socket, points the current link at 101, and starts it with SNAPD_REVERT_TO_REV set. The old snapd finds the half-done refresh, the waiting task fails, and the change is undone. Then the socket and service are restarted.
07 "Nothing to do: exit and wait" for snapd developers for administrators
When no snaps are installed, the device is seeded, no change or connection is open and 5 s have passed, snapd requests restart-socket and exits with status 42. snapd.service treats 42 as success and does not restart; a connection to either socket starts snapd again.
Look at a fresh container: snapd is installed, the sockets are there, and no snapd is running. It left on purpose. A daemon with nothing to manage only uses memory. A monitor checks: five seconds since start, every change finished, no open connection, the main loop has run once, zero snaps installed, the device seeded. If anything's busy, it tries again later, doubling the wait up to about five minutes. Once all agree, it asks for restart-socket, reason snapd-idle. It stops as usual and exits with status forty two: success, so systemd doesn't restart it. Systemd still holds the sockets, so the next snap command starts snapd again.
08 Where to go from here
Roads: the REST API during a restart, the overlord and refreshes, reboots, state patches, and image preseeding.
So snapd can replace itself, back out, and step aside. Each of those touches something another deck explains. The REST API deck shows what a client sees while the daemon is away. The overlord deck shows why a change can outlive the process, and the refresh deck shows when a refresh of snapd arrives. The system restart types end in a reboot, which has its own deck. The state patches deck covers the first thing a new snapd does with the old state. And the image deck shows the one place where the daemon is asked to stop for good.
Read more
Restart types
| Type | Requested when | How the daemon ends |
|---|---|---|
restart-daemon | snapd was refreshed, reverted or undone; a feature that needs a restart changed | exits; systemd starts snapd.service again |
restart-system | a change needs a reboot | reboot after 1 minute, with a 10 minute fallback scheduled first |
restart-system-now | an immediate reboot was requested | reboot at once |
restart-system-halt-now, restart-system-poweroff-now | halt or power off | halt or power off at once |
restart-socket | the daemon is idle | exits with status 42; socket activation starts it later |
stop-daemon | image preseeding reached mark-preseeded | stops |
Reasons recorded with a daemon restart: snapd-update, snapd-revert, snapd-undo, snapd-feature-change, and snapd-idle for idle exit. System restarts are explained in Reboots and snap reboot.
The stop sequence
- Record the requested restart and write
/var/lib/snapd/maintenance.json. - Wait for the code that requested the restart to release the state.
- Shut down the overlord and close
/run/snapd-snap.socket. - About 3 seconds after the overlord shutdown began, close
/run/snapd.socketand stop the idle monitor. - Give in-flight requests up to 25 seconds.
- Tell systemd
STOPPING=1unless the system is restarting. - For
restart-socket, check the idle conditions once more; if a request made snapd busy, do an ordinary daemon restart instead. - Stop the overlord; then end as the restart type says.
Re-execution, in order
- Already running from a snap: stay.
SNAP_REEXECis false: stay, and logre-exec disabled by userat debug level.- Not a classic system, or not like
debianorubuntu: stay, unlessSNAP_REEXECis set to a true value or toforce. - Look for the same executable under
/snap/snapd/current, then/snap/core/current. A path under/usr/libexec/snapdis looked up underusr/lib/snapd. Not found: stay. - Read
VERSION=fromusr/lib/snapd/infoin that snap. Unreadable: stay. Older than the running binary: stay, unlessSNAP_REEXEC=force. The same version counts as not older. execthe binary from the snap with the same arguments and environment.
Programs that do this: snapd, snap, snapd-apparmor and snap-gpio-helper. snapd skips it when preseeding. The published distribution-support page describes re-execution as the package installing the snapd snap; in the code it is the packaged binary executing the copy in the snap.
Environment variables
| Variable | Effect |
|---|---|
SNAP_REEXEC | unset or true: re-execute where supported; 0: stay; a true value on an unsupported distribution: re-execute; force: skip the version comparison |
SNAPD_REVERT_TO_REV | set by snap-failure; tells snapd it was started to handle a failure |
SNAPD_STANDBY_WAIT | replaces the 5 s idle wait |
SNAPD_REBOOT_DELAY | replaces the 1 minute reboot delay; meant for tests |
SNAPD_PRESEED | preseed mode; no re-execution |
WATCHDOG_USEC | set by systemd; snapd reports to the watchdog at half this interval |
for administrators for Ubuntu Core developers for snapd developers
What the units say
| Unit | Setting | Meaning |
|---|---|---|
snapd.service | Requires=snapd.socket | the sockets belong to systemd |
snapd.service | Restart=always | an exit is followed by a start |
snapd.service | SuccessExitStatus=42, RestartPreventExitStatus=42 | idle exit is a success and is not restarted |
snapd.service | OnFailure=snapd.failure.service | a failure runs snap-failure |
snapd.service | WatchdogSec=5m | snapd must report to systemd regularly |
snapd.failure.service | Type=oneshot, ExecStart=.../snap-failure snapd | one attempt at recovery |
To keep the packaged daemon on a classic system, add a drop-in such as /etc/systemd/system/snapd.service.d/reexec.conf and run systemctl daemon-reload:
[Service] Environment=SNAP_REEXEC=0
snap-failure, step by step
- On a classic system, do nothing unless re-execution is supported and enabled.
- Read
/var/lib/snapd/sequence/snapd.jsonand find the revision before the current one. No file:no snapd snap in the system, stop. - No previous revision: use the current snapd and leave the
currentlink alone. On Ubuntu Core 16 that is the snapd in the core snap; on a classic system the core snap is tried first, then the snapd snap. systemctl stop snapd.socket.- Point
/snap/snapd/currentat the previous revision. - Run that snapd with
systemd-run --collect --waitandSNAPD_REVERT_TO_REV=<revision>. - If the units are not failed, check up to 5 times, 5 s apart, whether the socket and the service are active and both socket files exist; if so, stop.
- Otherwise
systemctl reset-failed, remove leftover socket files and restartsnapd.socketandsnapd.service; on failure wait 12.5 s and try once more.
The snapd started this way looks for an unfinished change in which link-snap of the snapd snap is done and its auto-connect is waiting. If it finds one, that change is undone. If it finds none it logs detected failure recovery context, but no recovery needed and exits without changing anything.
Known defect in that second case, found by reading the code and not yet reproduced: snap-failure has already pointed /snap/snapd/current at the previous revision and does not point it back, while snapd's state still names the newer revision as current. After a failure that happened outside a snapd refresh, compare readlink /snap/snapd/current with the revision in snap list snapd.
Idle exit
All of these must hold: 5 s since start; every change ready and clean; no open connection; the ensure loop has run once; no snaps installed; the device is seeded. The monitor checks again after a wait that doubles from 5 s while it is below 5 minutes. The daemon prints daemon stop requested to wait for socket activation and exits with status 42.
If an expected reboot did not happen
snapd schedules the reboot again, at once, and waits 5 minutes; it does this up to 3 times and then carries on. See Reboots and snap reboot.
Further reading
No published page explains snap-failure, the restart types or idle exit. The nearest pages:
- Distribution support: which distributions support re-execution.
- Debug snaps: telling from
/proc/<pid>/exewhether snapd re-executed. - Test snapd fixes: installing a snapd snap from a branch and watching the switch.
- Decommissioning: re-execution on classic Ubuntu and removing snapd.
- Snapd release process: how a new snapd reaches devices.
Roads from here
RUNNING SNAPS
- The overlord — how a change survives the restart
- Refresh — when a refresh of snapd arrives
UBUNTU CORE · boot
- Reboots and snap reboot — the restart types that reboot the whole device
INSIDE SNAPD · the daemon itself
- The snapd REST API — what a client sees while the daemon is away
- State format patches — what the new snapd does to its state first
- Preparing images and preseeding — the stop-daemon type that ends a preseed run