Delayed security setup

sandbox internals for snapd developers

Two mechanisms share the word delayed. Connect tasks made by auto-connect carry delayed-setup-profiles and leave the backends to one setup-profiles task after all of them. And when a content provider is refreshed the mount backend writes each consumer's mount profile at once but postpones updating its running mount namespace. A process-delayed-security-backend-effects task waits for the refresh to succeed and then adds one apply task per consumer.

On the map: sandbox internals district

▶ Watch this deck — 8 scenes, ~6 min

01 Two things snapd puts off

TWO KINDS OF LATER 1 · installing acme-agent2 · refreshing acme-vpn auto-connectconnect · networkconnect · vpn-socketeach one only records its connection setup-profilesprofiles writtenonce, for both setup-profilesof acme-vpnacme-agent's mount profile: nowacme-agent's running namespace: later acme-vpn's refresh succeededupdate acme-agent's namespace
Two kinds of later. Top: installing acme-agent, auto-connect creates two connect tasks, network and vpn-socket, which only record their connections; one setup-profiles task after both writes the profiles once. Bottom: refreshing acme-vpn, its setup-profiles writes acme-agent's mount profile now and leaves acme-agent's running namespace for later; once acme-vpn's refresh has succeeded, acme-agent's namespace is updated

1. connect tasks made by auto-connect skip the security backends; one setup-profiles does the work for all of them. 2. On a slot provider's refresh the consumer's mount profile is written at once and its mount namespace is updated after the provider's refresh succeeds. acme-vpn and its socket slot are examples.

The word delayed turns up twice in snapd's security setup. It means two different things, so it helps to keep them apart. First, when you install acme-agent, it auto-connects its network plug and its VPN socket. Each connect task only records its connection. One setup-profiles task afterwards writes the profiles once, for both. The second is newer. When acme-vpn, which provides the agent's socket, is refreshed, the agent's mount profile is rewritten straight away. But the namespace the running agent lives in is updated only once acme-vpn's refresh has gone through.

02 Many connects, one setup-profiles for snapd developers

DELAYED-SETUP-PROFILES auto-connectbatchConnectTasks(…)opts.DelayedSetupProfiles = true prepare-plug-vpn-socketprepare-slot-socket connect · network"auto": true"delayed-setup-profiles": true connect · vpn-socket"auto": true"delayed-setup-profiles": true snapd debug logConnect handler: skippingsetupSnapSecurity for snaps"acme-agent" and "acme-vpn" ConnectTaskEdgesetup-profiles"new-connections": ["acme-agent:network snapd:network","acme-agent:vpn-socket acme-vpn:socket"] connect-slot-socketAfterConnectHooksEdgeconnect-plug-vpn-socket always added, even with no connection:the earlier setup-profiles ran inprepare mode, without full profilessnap connect and hotplug: no flag;their connect sets up both snapsundo: the connect undo skips thebackends too; the prepare-modesetup-profiles undo regenerates
The tasks auto-connect injects for acme-agent's install. batchConnectTasks sets DelayedSetupProfiles. Network: one connect task. vpn-socket: prepare-plug-vpn-socket, prepare-slot-socket, then connect. Both connect tasks carry auto true and delayed-setup-profiles true, and the debug log says the connect handler skipped setupSnapSecurity. One setup-profiles task with the sorted new-connections waits for both connect tasks, and connect-slot-socket, marked AfterConnectHooksEdge, then connect-plug-vpn-socket wait for it. Side notes: the task is always added because the earlier setup-profiles ran in prepare mode; manual snap connect and hotplug set no flag; on undo the connect undo also skips the backends and the prepare-mode undo regenerates

batchConnectTasks forces DelayedSetupProfiles on each connect; one setup-profiles (new-connections, sorted) waits on every ConnectTaskEdge task and the AfterConnectHooksEdge hook waits on it. Only doAutoConnect calls it. undoConnect also skips the backends with the flag.

When the agent is installed, auto-connect builds these tasks itself. For each connection it calls the same helper that snap connect uses, with one difference: the delayed-setup-profiles flag. With it, the connect task updates the repository and the state, and skips the security setup. It says so only in snapd's debug log. One setup-profiles task waits for every connect, and the connect hooks wait for it, so the backends run once, not twice per connection. That task is added even when there's nothing to connect, because it's now the only one that writes the new revision's full profiles. A manual snap connect doesn't set the flag.

03 The mount profile is written now, the namespace waits for snapd developers

DELAYED-CONSUMER-MOUNT-NS-UPDATE setup-profilesacme-vpn's refresh, full mode, affects:acme-vpnacme-agentSnapSetupReasonConnectedSlotProviderUpdateit has the plug; its slot provideris being updated mount.Backend.Setup(acme-agent)repo.SnapSpecificationEnsureDirState/var/lib/snapd/mount/snap.acme-agent.fstabwritten now · source: acme-vpn's new revisionfiles changed?no: return, no effect sctx.CanDelayEffects &&Reason == ConnectedSlotProviderUpdateno: snap-update-nsacme-agent nowyessctx.DelayEffect(b, {ID: "delayed-consumer-mount-ns-update",Description: "mount namespace update triggeredby slot provider update"}) → return acme-agent's other backendsapparmor ✓seccomp ✓udev ✓applied now, in this same pass after all backends:DelayedBackendEffectsFor(process task, "acme-vpn")→ "delayed-effects-for-snaps"
acme-vpn's full setup-profiles sets up acme-vpn and acme-agent; acme-agent's reason is ConnectedSlotProviderUpdate because it has the plug. In the mount backend's Setup for acme-agent: repo.SnapSpecification, then EnsureDirState writes snap.acme-agent.fstab pointing at acme-vpn's new revision. If no file changed it returns with no effect. If CanDelayEffects and the reason is ConnectedSlotProviderUpdate, it calls DelayEffect with delayed-consumer-mount-ns-update and returns; otherwise it would run snap-update-ns now. acme-agent's other backends, apparmor, seccomp and udev, apply now. After all backends the effect is stored with DelayedBackendEffectsFor on the process task as delayed-effects-for-snaps

Only the mount backend implements ApplyDelayedEffects, and only for delayed-consumer-mount-ns-update. The fstab is written at once; only snap-update-ns (or snap-discard-ns) waits. AppArmor and the other backends are not delayed.

Now the newer mechanism, on a refresh of acme-vpn. Its full setup-profiles task redoes the agent too, because the agent's plug is connected to acme-vpn's slot. The mount backend writes the agent's new mount profile at once, pointing into acme-vpn's new revision. If nothing changed, it stops there. If something did, and this task may delay, it doesn't run snap-update-ns. It hands back one effect, a consumer mount namespace update, and returns. That's the only backend that delays anything. AppArmor, seccomp and the rest apply in the same pass. When every backend is done, the effect is stored on another task of the change, under acme-vpn's name.

04 Who may wait for snapd developers

SETUPCONTEXT.CANDELAYEFFECTS change has a process-delayed-…-effects task?canDelay: snapd takes the first such task in the change the full setup-profiles, not prepare mode or undo?undo: const canDefer = falseno: at once; connect, disconnect andregenerate-all pass false as well an affected snap, not the snap being set up?the snap itself: SnapSetupReasonOwnUpdateno: at once on the plug side of the connection?Reason: ConnectedSlotProviderUpdate, where content consumers sitno: slot side and cyclic, at onceTODO: "the safe bet right now…" not on a connection made in this change?!isNewConnected: its connect-plug hook must see the mountsno: at oncemay delay: CanDelayEffects = true no: seeding, remodel, prerequisites,validation sets: NoDelayedSideEffects
Five questions decide whether the mount backend may delay, each with a no exit that applies at once. Does the change have a process-delayed-security-backend-effects task, the first one in the change? Is this the full setup-profiles, not prepare mode or undo? Is it an affected snap, not the snap being set up? Is it on the plug side? Is it not on a connection made in this change? If all are yes, CanDelayEffects is true. Seeding, remodel, prerequisites and validation sets set NoDelayedSideEffects and have no process task

CanDelayEffects = canDelay && !isNewConnected, set only for SnapSetupReasonConnectedSlotProviderUpdate in setupProfilesForAppSet. canDelay means the change has a process-delayed-security-backend-effects task. Undo passes canDefer = false.

When may the mount update wait? Only in the full setup task of an install or refresh change that also has a process-delayed-security-backend-effects task. Call it the process task; snapd takes the first one. Prepare mode, connect and disconnect, startup regeneration and every undo apply at once. The snap being installed or refreshed isn't delayed, only snaps it affects, and of those only the plug side, where content consumers sit. A snap on a connection made in this change doesn't wait either: its connect-plug hook is about to run and must see the mounts. Seeding, remodel, prerequisites and validation set enforcement switch it all off.

05 The process task watches the lanes for snapd developers

PROCESS-DELAYED-SECURITY-BACKEND-EFFECTS Process delayed security backend side effects for affected snaps"monitored-lanes", "apply-in-lane"added by snapstate: setupDelayedSecurityBackendEffects · no wait edges retry every 0.5 sreason: pending snap workacme-vpn's lane, watchedlink-snapauto-connectsetup-profilespost-refresh hookstart-snap-services… $ snap change 41Done Make snap "acme-vpn" (…) available to the systemDoing Process delayed security backend side effects for affected snaps all Done, link-snap among them: acme-vpn succeeded then, per affected snap:Apply delayed security backend side effects for snap "acme-agent""effects-data" · in a new lane of its own--transaction=all-snaps:in the transaction lane a pending reboot: waits for it · data in the state: survives a snapd restart · no undo handler
The process task, Process delayed security backend side effects for affected snaps, is added by snapstate with monitored-lanes and apply-in-lane and retries every half second while snap work is pending. It watches acme-vpn's lane: link-snap, auto-connect, setup-profiles, the post-refresh hook, start-snap-services. snap change shows link-snap Done and the process task Doing. When the lane is all Done with link-snap Done, acme-vpn has succeeded, and the process task adds Apply delayed security backend side effects for snap acme-agent with effects-data in a new lane, or in the transaction lane with --transaction=all-snaps. A pending reboot is waited for, the data survives a snapd restart, and there is no undo handler

process-delayed-security-backend-effects: created per install or refresh request, retries every 0.5 s, then adds apply-delayed-snap-security-backend-effects per affected snap, for triggering snaps with a Done link-snap in a lane with no failed task. Neither task has an undo handler.

The effect is stored; someone has to apply it. snapstate adds one process task to each install and refresh request, with the lanes to watch. With nothing to wait for, it starts at once and retries twice a second until every task in those lanes is finished. In snap change you'll see it sitting in Doing. A snap counts as successful if its lane is all done, link-snap included. For each snap affected by a successful one, it adds an apply task, in a lane of its own, or in the transaction's lane with --transaction=all-snaps. It all lives in the state, so a snapd restart loses nothing.

06 Discard if idle, update if running for snapd developers

APPLY-DELAYED-SNAP-SECURITY-BACKEND-EFFECTS Apply delayed security backend side effects for snap "acme-agent"interfaces.ApplyDelayedEffects(mount, acme-agent, effects)unknown ID: "unexpected effect" opportunisticDiscardtry-lock acme-agent, look in its cgroupsa process of acme-agent running? nosnap-discard-ns --snap-already-lockedthe next start builds a fresh one from the new fstab yes, the agent daemon:cannot discard when 1 instances of snap are runningsnap-update-ns acme-agent failssnap-discard-nsunless an app is a daemon with refresh-mode: endure:cannot update mount namespace of snap "acme-agent",and cannot discard it because it contains anenduring daemon: … applybefore: joining the kept namespace shows acme-vpn's previous mountsafter: the new onesno kept namespace: snap-confine builds one from the new fstab, so the new content shows at once
The apply task for acme-agent calls ApplyDelayedEffects on the mount backend; an unknown effect ID is an error. opportunisticDiscard try-locks acme-agent and looks in its cgroups. If no process runs, snap-discard-ns --snap-already-locked removes the namespace and the next start builds a fresh one from the new fstab. The agent daemon runs, so snap-update-ns acme-agent updates it. If that fails, snap-discard-ns, unless an app is a daemon with refresh-mode endure, which fails the task with cannot update mount namespace of snap acme-agent, and cannot discard it because it contains an enduring daemon. Timeline: before the apply, a process joining the kept namespace sees the old mounts, after it the new ones; with no kept namespace, snap-confine builds one from the new fstab at once

Apply: try-lock and discard when no process runs; otherwise snap-update-ns; on failure snap-discard-ns, except with a refresh-mode: endure daemon, which fails the task. Before the apply, processes joining the preserved namespace see the old mounts.

The apply task for the agent tries the cheap way first. It takes the snap's lock, if it can, and looks for running processes. If none is running, it throws the namespace away, and the next start builds a fresh one. But the agent's daemon is running, so instead snap-update-ns brings the live namespace up to date. If that fails, the namespace is discarded, unless the snap has a daemon with refresh-mode: endure; then the task fails. Until this runs, a process that joins the kept namespace still sees acme-vpn's previous content. One that starts without a kept namespace already sees the new content.

07 When the trigger fails, and a re-refresh for snapd developers

FAILURE PATHS 1 · acme-vpn's refresh failslink-snapsetup-profilespost-refresh hookErrorsnapd log:skipping effects triggered by failed snap "acme-vpn"no apply task undo: prepare-mode setup-profilesfstab: previous revisionnamespace updated nowregenerates acme-agent with const canDefer = false 2 · a re-refresh in the same changefirst refresh: Doneprocess task 1: Donecheck-rerefreshadds the re-refreshand process task 2 setup-profiles · re-refreshdelayedEffectsTask: the first in the changestored here process task 2monitors the new lanes, has no dataacme-agent's namespace keeps the in-between revision
Two failure paths. Top: acme-vpn's refresh fails in its post-refresh hook; the snapd log says skipping effects triggered by failed snap acme-vpn and no apply task is added. The undo of acme-vpn's prepare-mode setup-profiles regenerates acme-agent with canDefer false, its fstab goes back to the previous revision and its namespace is updated at once. Bottom: a refresh with check-rerefresh; process task 1 is Done; check-rerefresh adds a re-refresh and process task 2. The re-refresh's setup-profiles looks up the first process task in the change and stores its effects on the Done task 1. Process task 2 monitors the new lanes and has no data, so acme-agent's namespace keeps the in-between revision

A failed triggering snap: effects skipped, the undo applies immediately (canDefer = false). Current behaviour: delayedEffectsTask returns the first process task, so a re-refresh's effects are stored on a Done task and not applied.

What if acme-vpn's refresh fails? Then its effects are simply skipped, and that's deliberate. The undo puts acme-vpn's old revision back and regenerates the agent with delaying turned off, so its namespace is fixed on the spot. There's one case to know about today. When a refresh is followed by a re-refresh in the same change, the new setup task looks for the first process task in the change. That one's already done, and the effects are stored on it. The second process task finds nothing, and the agent's namespace keeps the in-between revision until something else rebuilds it.

08 Where to go from here

WHERE NEXT Delayed security setup1 · profiles once, after the connects2 · namespace after the provider Setting up a snap's securitythe setup-profiles task snap-update-nswhat the apply step runs The per-snap mount namespaceupdated or discarded The content interfacewhere the mounts come from The security backendsmount among the eleven The overlordlanes and retrying tasks Servicesenduring services
Delayed security setup in the middle, with its two kinds of later, and roads to setting up a snap's security, snap-update-ns, the per-snap mount namespace, the content interface, the security backends, the overlord and services

Roads: setting up a snap's security, snap-update-ns, the per-snap mount namespace, the content interface, the security backends, the overlord, services.

Two kinds of later, then. Profiles written once, after all the connects, and a consumer's namespace brought up to date after its provider's refresh. The setup task that does the work is in setting up a snap's security. What the apply step runs is snap-update-ns, and the namespace it touches is the per-snap mount namespace. The content interface is where these mounts come from. The mount backend sits among the security backends. Lanes and retrying tasks belong to the overlord. And enduring services are in the services deck.

Read more

Two mechanisms

delayed-setup-profilesdelayed backend side effects
What waitsthe security backends of each connect taskthe update of a consumer's preserved mount namespace
Who sets itauto-connect, for every connection it createsthe full setup-profiles of a change with a process-delayed-security-backend-effects task
Who does the workone setup-profiles task after all the connect tasksone apply-delayed-snap-security-backend-effects task per affected snap
Backends involvedall of them, oncethe mount backend only

The examples use acme-agent on the gateway, with the daemon agent and a content plug vpn-socket connected to acme-vpn:socket. acme-vpn and its slot are invented.

Tasks and task data

TaskSummaryTask data
connectConnect <plug> to <slot>"auto": true, "delayed-setup-profiles": true when made by auto-connect
setup-profiles (after the connects)Setup snap "<name>" (<rev>) security profiles"new-connections": the sorted connection IDs, only when there are connections
process-delayed-security-backend-effectsProcess delayed security backend side effects for affected snaps"monitored-lanes", "apply-in-lane", then "delayed-effects-for-snaps"
apply-delayed-snap-security-backend-effectsApply delayed security backend side effects for snap "<name>""effects-data": "affected-snap-instance" and "effects"

Neither of the last two tasks has an undo handler. The process task has no wait edges and retries every 0.5 s with the reason pending snap work until every task in its monitored lanes is ready. With a pending system restart it waits for the restart instead. snap watch skips it when choosing the task to display, unless it is the only task in Doing.

When snapd may delay the namespace update

SituationDelayed
the snap being installed or refreshedno (SnapSetupReasonOwnUpdate)
an affected snap on the plug side, slot provider being updatedyes, unless it is on a connection made in the same change
an affected snap on the slot sideno (SnapSetupReasonConnectedPlugConsumerUpdate)
an affected snap on both sidesno (SnapSetupReasonCyclicallyConnectedUpdate)
prepare mode, connect, disconnect, regeneration at startupno
the undo of setup-profilesno (const canDefer = false)
seeding, remodel, prerequisites, validation-set enforcementno: NoDelayedSideEffects, and no process task
remove, revert, enable, component-only installs, manual connectno process task

The mount backend delays only when its files changed. The fstab and user fstab are written at once; only snap-update-ns or snap-discard-ns waits. The effect ID is delayed-consumer-mount-ns-update with the description mount namespace update triggered by slot provider update. AppArmor, seccomp, udev and the other backends apply in the same pass.

What the apply task does

1. Rejects any effect ID other than delayed-consumer-mount-ns-update (unexpected effect) and collapses duplicates. 2. Under a try-lock of the snap, looks for its cgroups. With none, runs snap-discard-ns --snap-already-locked. 3. Otherwise runs snap-update-ns, which does nothing when the snap has no preserved namespace. 4. If that fails, runs snap-discard-ns, unless an app is a daemon with refresh-mode: endure; then the task fails with cannot update mount namespace of snap "<name>", and cannot discard it because it contains an enduring daemon.

Between the delay and the apply, a process that joins the preserved namespace sees the provider's previous revision; a process that starts without one sees the new revision at once.

for snapd developers

For snapd developers

  • SetupContext (interfaces/backend.go) carries Reason, CanDelayEffects and DelayEffect. The delay fields are not in ConfinementOptions.
  • Only mount.Backend implements DelayedSideEffectsBackend. A DelayEffect call from a backend without ApplyDelayedEffects fails the task after the backends have run.
  • delayedEffectsTask returns the first process task in the change. With --transaction=all-snaps the process task and the apply tasks join the transaction lane, so a failed apply undoes the whole transaction; otherwise each apply task gets a lane of its own.
  • The apply task holds no conflict claim on the affected snap; overlord/dot/dot.go has a TODO for it.
  • check-rerefresh with no explicit task list waits for every other task of the change, including the process task.
  • The spread test tests/main/layout-content-delayed-effects covers the discard of an idle consumer, the update of a running one and two apply tasks from one install.
  • Suspected bugs: the re-refresh case drawn in scene 7, and effects triggered by a prerequisite update being dropped as if the provider failed. An effect can also be lost when setup-profiles re-runs after snapd was killed; that is a design limitation, not a confirmed bug. See findings/security-setup/SUSPECTED-BUGS.md (S2, S8) and findings/security-setup/delayed-setup.md.

Not covered

Why the namespace update is delayed is stated in the commits only as robustness; the concrete failure avoided was not determined. Whether the provider's previous revision can be removed by its own refresh before the apply task runs, with a small refresh.retain, was not determined.

Further reading

No published page describes either mechanism.

Roads from here

INTERFACES AND CONFINEMENT

RUNNING SNAPS

  • The overlord — lanes and tasks that retry
  • Services — refresh-mode endure and why a namespace cannot always be discarded

SANDBOX INTERNALS · the mount namespace