Delayed security setup
sandbox internals for snapd developers
Two mechanisms share the word delayed. Connect tasks made by auto-connect carry delayed-setup-profiles and leave the backends to one setup-profiles task after all of them. And when a content provider is refreshed the mount backend writes each consumer's mount profile at once but postpones updating its running mount namespace. A process-delayed-security-backend-effects task waits for the refresh to succeed and then adds one apply task per consumer.
On the map: sandbox internals district
01 Two things snapd puts off
1. connect tasks made by auto-connect skip the security backends; one setup-profiles does the work for all of them. 2. On a slot provider's refresh the consumer's mount profile is written at once and its mount namespace is updated after the provider's refresh succeeds. acme-vpn and its socket slot are examples.
The word delayed turns up twice in snapd's security setup. It means two different things, so it helps to keep them apart. First, when you install acme-agent, it auto-connects its network plug and its VPN socket. Each connect task only records its connection. One setup-profiles task afterwards writes the profiles once, for both. The second is newer. When acme-vpn, which provides the agent's socket, is refreshed, the agent's mount profile is rewritten straight away. But the namespace the running agent lives in is updated only once acme-vpn's refresh has gone through.
02 Many connects, one setup-profiles for snapd developers
batchConnectTasks forces DelayedSetupProfiles on each connect; one setup-profiles (new-connections, sorted) waits on every ConnectTaskEdge task and the AfterConnectHooksEdge hook waits on it. Only doAutoConnect calls it. undoConnect also skips the backends with the flag.
When the agent is installed, auto-connect builds these tasks itself. For each connection it calls the same helper that snap connect uses, with one difference: the delayed-setup-profiles flag. With it, the connect task updates the repository and the state, and skips the security setup. It says so only in snapd's debug log. One setup-profiles task waits for every connect, and the connect hooks wait for it, so the backends run once, not twice per connection. That task is added even when there's nothing to connect, because it's now the only one that writes the new revision's full profiles. A manual snap connect doesn't set the flag.
03 The mount profile is written now, the namespace waits for snapd developers
Only the mount backend implements ApplyDelayedEffects, and only for delayed-consumer-mount-ns-update. The fstab is written at once; only snap-update-ns (or snap-discard-ns) waits. AppArmor and the other backends are not delayed.
Now the newer mechanism, on a refresh of acme-vpn. Its full setup-profiles task redoes the agent too, because the agent's plug is connected to acme-vpn's slot. The mount backend writes the agent's new mount profile at once, pointing into acme-vpn's new revision. If nothing changed, it stops there. If something did, and this task may delay, it doesn't run snap-update-ns. It hands back one effect, a consumer mount namespace update, and returns. That's the only backend that delays anything. AppArmor, seccomp and the rest apply in the same pass. When every backend is done, the effect is stored on another task of the change, under acme-vpn's name.
04 Who may wait for snapd developers
CanDelayEffects = canDelay && !isNewConnected, set only for SnapSetupReasonConnectedSlotProviderUpdate in setupProfilesForAppSet. canDelay means the change has a process-delayed-security-backend-effects task. Undo passes canDefer = false.
When may the mount update wait? Only in the full setup task of an install or refresh change that also has a process-delayed-security-backend-effects task. Call it the process task; snapd takes the first one. Prepare mode, connect and disconnect, startup regeneration and every undo apply at once. The snap being installed or refreshed isn't delayed, only snaps it affects, and of those only the plug side, where content consumers sit. A snap on a connection made in this change doesn't wait either: its connect-plug hook is about to run and must see the mounts. Seeding, remodel, prerequisites and validation set enforcement switch it all off.
05 The process task watches the lanes for snapd developers
process-delayed-security-backend-effects: created per install or refresh request, retries every 0.5 s, then adds apply-delayed-snap-security-backend-effects per affected snap, for triggering snaps with a Done link-snap in a lane with no failed task. Neither task has an undo handler.
The effect is stored; someone has to apply it. snapstate adds one process task to each install and refresh request, with the lanes to watch. With nothing to wait for, it starts at once and retries twice a second until every task in those lanes is finished. In snap change you'll see it sitting in Doing. A snap counts as successful if its lane is all done, link-snap included. For each snap affected by a successful one, it adds an apply task, in a lane of its own, or in the transaction's lane with --transaction=all-snaps. It all lives in the state, so a snapd restart loses nothing.
06 Discard if idle, update if running for snapd developers
Apply: try-lock and discard when no process runs; otherwise snap-update-ns; on failure snap-discard-ns, except with a refresh-mode: endure daemon, which fails the task. Before the apply, processes joining the preserved namespace see the old mounts.
The apply task for the agent tries the cheap way first. It takes the snap's lock, if it can, and looks for running processes. If none is running, it throws the namespace away, and the next start builds a fresh one. But the agent's daemon is running, so instead snap-update-ns brings the live namespace up to date. If that fails, the namespace is discarded, unless the snap has a daemon with refresh-mode: endure; then the task fails. Until this runs, a process that joins the kept namespace still sees acme-vpn's previous content. One that starts without a kept namespace already sees the new content.
07 When the trigger fails, and a re-refresh for snapd developers
A failed triggering snap: effects skipped, the undo applies immediately (canDefer = false). Current behaviour: delayedEffectsTask returns the first process task, so a re-refresh's effects are stored on a Done task and not applied.
What if acme-vpn's refresh fails? Then its effects are simply skipped, and that's deliberate. The undo puts acme-vpn's old revision back and regenerates the agent with delaying turned off, so its namespace is fixed on the spot. There's one case to know about today. When a refresh is followed by a re-refresh in the same change, the new setup task looks for the first process task in the change. That one's already done, and the effects are stored on it. The second process task finds nothing, and the agent's namespace keeps the in-between revision until something else rebuilds it.
08 Where to go from here
Roads: setting up a snap's security, snap-update-ns, the per-snap mount namespace, the content interface, the security backends, the overlord, services.
Two kinds of later, then. Profiles written once, after all the connects, and a consumer's namespace brought up to date after its provider's refresh. The setup task that does the work is in setting up a snap's security. What the apply step runs is snap-update-ns, and the namespace it touches is the per-snap mount namespace. The content interface is where these mounts come from. The mount backend sits among the security backends. Lanes and retrying tasks belong to the overlord. And enduring services are in the services deck.
Read more
Two mechanisms
delayed-setup-profiles | delayed backend side effects | |
|---|---|---|
| What waits | the security backends of each connect task | the update of a consumer's preserved mount namespace |
| Who sets it | auto-connect, for every connection it creates | the full setup-profiles of a change with a process-delayed-security-backend-effects task |
| Who does the work | one setup-profiles task after all the connect tasks | one apply-delayed-snap-security-backend-effects task per affected snap |
| Backends involved | all of them, once | the mount backend only |
The examples use acme-agent on the gateway, with the daemon agent and a content plug vpn-socket connected to acme-vpn:socket. acme-vpn and its slot are invented.
Tasks and task data
| Task | Summary | Task data |
|---|---|---|
connect | Connect <plug> to <slot> | "auto": true, "delayed-setup-profiles": true when made by auto-connect |
setup-profiles (after the connects) | Setup snap "<name>" (<rev>) security profiles | "new-connections": the sorted connection IDs, only when there are connections |
process-delayed-security-backend-effects | Process delayed security backend side effects for affected snaps | "monitored-lanes", "apply-in-lane", then "delayed-effects-for-snaps" |
apply-delayed-snap-security-backend-effects | Apply delayed security backend side effects for snap "<name>" | "effects-data": "affected-snap-instance" and "effects" |
Neither of the last two tasks has an undo handler. The process task has no wait edges and retries every 0.5 s with the reason pending snap work until every task in its monitored lanes is ready. With a pending system restart it waits for the restart instead. snap watch skips it when choosing the task to display, unless it is the only task in Doing.
When snapd may delay the namespace update
| Situation | Delayed |
|---|---|
| the snap being installed or refreshed | no (SnapSetupReasonOwnUpdate) |
| an affected snap on the plug side, slot provider being updated | yes, unless it is on a connection made in the same change |
| an affected snap on the slot side | no (SnapSetupReasonConnectedPlugConsumerUpdate) |
| an affected snap on both sides | no (SnapSetupReasonCyclicallyConnectedUpdate) |
prepare mode, connect, disconnect, regeneration at startup | no |
the undo of setup-profiles | no (const canDefer = false) |
| seeding, remodel, prerequisites, validation-set enforcement | no: NoDelayedSideEffects, and no process task |
| remove, revert, enable, component-only installs, manual connect | no process task |
The mount backend delays only when its files changed. The fstab and user fstab are written at once; only snap-update-ns or snap-discard-ns waits. The effect ID is delayed-consumer-mount-ns-update with the description mount namespace update triggered by slot provider update. AppArmor, seccomp, udev and the other backends apply in the same pass.
What the apply task does
1. Rejects any effect ID other than delayed-consumer-mount-ns-update (unexpected effect) and collapses duplicates. 2. Under a try-lock of the snap, looks for its cgroups. With none, runs snap-discard-ns --snap-already-locked. 3. Otherwise runs snap-update-ns, which does nothing when the snap has no preserved namespace. 4. If that fails, runs snap-discard-ns, unless an app is a daemon with refresh-mode: endure; then the task fails with cannot update mount namespace of snap "<name>", and cannot discard it because it contains an enduring daemon.
Between the delay and the apply, a process that joins the preserved namespace sees the provider's previous revision; a process that starts without one sees the new revision at once.
for snapd developers
For snapd developers
SetupContext(interfaces/backend.go) carriesReason,CanDelayEffectsandDelayEffect. The delay fields are not inConfinementOptions.- Only
mount.BackendimplementsDelayedSideEffectsBackend. ADelayEffectcall from a backend withoutApplyDelayedEffectsfails the task after the backends have run. delayedEffectsTaskreturns the first process task in the change. With--transaction=all-snapsthe process task and the apply tasks join the transaction lane, so a failed apply undoes the whole transaction; otherwise each apply task gets a lane of its own.- The apply task holds no conflict claim on the affected snap;
overlord/dot/dot.gohas a TODO for it. check-rerefreshwith no explicit task list waits for every other task of the change, including the process task.- The spread test
tests/main/layout-content-delayed-effectscovers the discard of an idle consumer, the update of a running one and two apply tasks from one install. - Suspected bugs: the re-refresh case drawn in scene 7, and effects triggered by a prerequisite update being dropped as if the provider failed. An effect can also be lost when
setup-profilesre-runs after snapd was killed; that is a design limitation, not a confirmed bug. See findings/security-setup/SUSPECTED-BUGS.md (S2, S8) and findings/security-setup/delayed-setup.md.
Not covered
Why the namespace update is delayed is stated in the commits only as robustness; the concrete failure avoided was not determined. Whether the provider's previous revision can be removed by its own refresh before the apply task runs, with a small refresh.retain, was not determined.
Further reading
No published page describes either mechanism.
- The content interface: the connections whose mounts are delayed.
- Transactional updates:
--transaction=all-snapsandper-snapfrom the user's side. - Debugging startup performance: the one page that shows snap-update-ns running.
Roads from here
INTERFACES AND CONFINEMENT
- The content interface — the content connections whose mounts are delayed
RUNNING SNAPS
- The overlord — lanes and tasks that retry
- Services — refresh-mode endure and why a namespace cannot always be discarded
SANDBOX INTERNALS · the mount namespace
- Setting up a snap's security — the setup-profiles task that does the delayed work
- snap-update-ns and the mount profiles — what the postponed namespace update runs
- The per-snap mount namespace — the preserved namespace that is updated or discarded
- The security backends — the mount backend among the eleven