The security backends

sandbox internals for snapd developers for Ubuntu Core developers

snapd turns a snap's plugs, slots and connections into policy through eleven security backends run in a fixed order, from systemd first to AppArmor last. Each asks the interface repository for its specification of the snap, synchronises the snap's generated files and applies only what changed, with apparmor_parser, snap-seccomp, udevadm, snap-update-ns, modprobe or systemctl. Remove deletes the files but leaves AppArmor profiles and kernel modules loaded.

On the map: sandbox internals district

▶ Watch this deck — 8 scenes, ~6 min

Part of journeys

01 One connection, eleven writers

ONE CONNECTION $ snap connect acme-agent:meter acme-gadget:meteracme-agentmeteracme-gadget: meter set up again: acme-gadget, then acme-agent 1 systemd2 seccomp3 dbus4 udev5 mount6 kmod7 polkit8 ldconfig9 configfiles10 symlinks 11 apparmorlast, and only when the kernel has AppArmor each backend, for this snap:ask for its part of the policywrite its filesrun a command only for what changed for acme-agentnew device rule · udev reloadednew tty rule · profile reloadedunchanged · nothing to rununchanged · nothing to rununchanged · nothing to rununchanged · nothing to rununchanged · nothing to rununchanged · nothing to rununchanged · nothing to rununchanged · nothing to rununchanged · nothing to run
snap connect acme-agent:meter acme-gadget:meter sets up both snaps again, the gadget and then the agent. A column of eleven security backends in order, systemd, seccomp, dbus, udev, mount, kmod, polkit, ldconfig, configfiles, symlinks and apparmor last, which exists only when the kernel has AppArmor. For the agent, udev and apparmor have a new rule and reload; the other nine are unchanged and run nothing

Setting up a snap's security runs every backend in a fixed order: systemd, seccomp, dbus, udev, mount, kmod, polkit, ldconfig, configfiles, symlinks, then apparmor when the kernel supports AppArmor. Each writes one kind of policy and applies only what changed.

You've just connected the agent's meter plug to the gadget's meter slot. Which parts of snapd wake up? snapd sets up the security of both snaps again, the gadget first, then the agent. For each, that means running all its security backends, always in the same order. There are eleven, from systemd first to AppArmor last. Or ten, when the kernel has no AppArmor, because then that backend isn't there. Each one asks for its part of the policy, writes its files, and runs a command only for what changed. For the agent, udev and AppArmor have work. The other nine find their files as they were, and do nothing.

02 A backend's life for snapd developers

interfaces.SecurityBackend snapd startsInitialize(opts)Name()addBackends →repo.AddBackendduplicate name refused before link-snapPrepare(appSet)prepare-modesetup-profilesonly apparmor acts security set upSetup(appSet, opts,sctx, repo, tm)setupSecurityByBackendrepo.SnapSpecification→ NewSpecificationSetup builds its own spec snap removedRemove(instanceName)removeSnapSecurityno snap info, no repository $ snap debug sandbox-featuresSandboxFeatures()no transaction:undo is the caller's job optional interfaces, each with its implementersSetupManyapparmor: batch reloadRemoveLateapparmor, the snapd snapReinitializeapparmorApplyDelayedEffectsmount
The life of a security backend in four moments: when snapd starts, Initialize and Name, registered by addBackends with repo.AddBackend, a duplicate name refused; before link-snap, Prepare, used only by apparmor; whenever a snap's security is set up, Setup, which receives the repository and builds its own specification; when a snap is removed, Remove with only the instance name. snap debug sandbox-features prints SandboxFeatures. No transaction: undo is the caller's job. Four optional interfaces: SetupMany, RemoveLate and Reinitialize for apparmor, ApplyDelayedEffects for mount

Initialize, Name, Prepare, Setup, Remove, NewSpecification, SandboxFeatures. Optional: SetupMany, RemoveLate, Reinitialize (apparmor) and ApplyDelayedEffects (mount). Setup receives the repository and builds its own specification.

Say you're adding a backend: here's the contract you sign. When snapd starts, it calls Initialize, and the repository registers the backend by its name. A duplicate name is refused. Prepare runs before link-snap, and only AppArmor does anything there. Setup runs whenever a snap's security is set up. Notice what it gets: the repository, not a specification. It builds its own. Remove gets the instance name, no snap info, no repository. Snap debug sandbox-features prints its feature list. There's no transaction here; undo is the caller's job. Four optional interfaces add the rest: three only AppArmor uses, one the mount backend uses for delayed effects.

03 The shape of every Setup for snapd developers for Ubuntu Core developers

EVERY SETUP 1 · specificationrepo.SnapSpecification("udev", appSet, …)udev's view of acme-agent 2 · content map70-snap.acme-agent.rules →"# This file isautomatically generated…" 3 · sync the files/etc/udev/rules.donly this snap's namesAtomicWrite, mode 0644changed: the rules fileremoved: none 4 · apply changesonly for changedand removed filesudevadm control --reload-rulesudevadm trigger --subsystem-nomatch=inputudevadm trigger --property-match=ID_INPUT_JOYSTICK=1udevadm settle --timeout=10 "changed" means the file changed,not that the last apply workedapparmor alone reloadsunchanged profiles tooa retry repairs a failed load seccomp today: a failed compile deletes the .bin2;the next Setup sees the same .src and compiles nothingsnap-confine waits up to 120 s for the .bin2, then fails
Four stages of every Setup, followed with udev for acme-agent: ask the repository for the udev specification; map the file name 70-snap.acme-agent.rules to its content; synchronise only this snap's files in /etc/udev/rules.d, which reports the rules file as changed and nothing removed; run udevadm control --reload-rules, the triggers and udevadm settle only for the change. Changed means the file changed, not that the last apply worked: apparmor reloads unchanged profiles too, while seccomp after a failed compile has a new .src and no .bin2, compiles nothing next time, and snap-confine waits up to 120 seconds and fails

Specification → content map → files synchronised by per-snap name (EnsureDirState on a glob; udev's single file with EnsureFileState) → a command for changed files only. Only apparmor reloads unchanged profiles. Current behaviour: after a failed seccomp compile the .src is new but the .bin2 is gone, and an unchanged .src is not recompiled.

Squint, and every backend's Setup looks the same. Here's udev, for the agent. First, ask the repository for udev's specification of the snap. Second, map file names to contents. Third, sync the disk, touching only this snap's files; udev has one. That says what changed or went away. Fourth, act only on those: udevadm reloads rules and re-triggers devices. The catch: changed means the file changed, not that the last apply worked. AppArmor reloads unchanged profiles too, so retries repair it. seccomp doesn't. Today a failed compile deletes the binary filter; the next Setup sees the same source, compiles nothing, and snap-confine waits for a filter that doesn't come.

04 Where it lands, and what applies it for Ubuntu Core developers for snapd developers

ON THE GATEWAY /var/lib/snapd/apparmor/profiles/snap.acme-agent.agent snap-update-ns.acme-agentapparmor_parser--replace --write-cacheloaded into the kernel now /var/lib/snapd/seccomp/bpf/ · per app and hooksnap.acme-agent.agent.src → .bin2snap-seccomp compileread by snap-confineat the app's next start /etc/udev/rules.d/70-snap.acme-agent.rules/var/lib/snapd/cgroup/snap.acme-agent.deviceudevadmdevices re-tagged now/var/lib/snapd/mount/snap.acme-agent.fstabsnap-update-nsrunning namespace updatednow, or discarded /etc/modules-load.d/snap.<snap>.conf/etc/modprobe.d/snap.<snap>.confmodprobeerrors ignored/etc/systemd/system/snap.<snap>.interface.<suffix>.servicesystemctl daemon-reload,enable, restarterrors only logged /etc/dbus-1/system.d · polkit actions, rulesld.so.conf.d · configfiles · symlinksno commandldconfig alone runs ldconfig
On the gateway, files for acme-agent and what applies them: AppArmor profiles in /var/lib/snapd/apparmor/profiles, loaded into the kernel now by apparmor_parser; seccomp sources, one per app and hook, in /var/lib/snapd/seccomp/bpf compiled by snap-seccomp to .bin2, read by snap-confine at the next start; udev rules in /etc/udev/rules.d and the device cgroup file, udevadm re-tags devices now; the fstab in /var/lib/snapd/mount, snap-update-ns updates the running namespace or it is discarded; kmod files with modprobe, errors ignored; systemd helper units with systemctl, errors only logged; dbus, polkit, configfiles and symlinks files with no command, and ldconfig runs ldconfig

apparmor: /var/lib/snapd/apparmor/profiles, loaded now. seccomp: /var/lib/snapd/seccomp/bpf/*.src compiled to .bin2, used at the next start. udev: /etc/udev/rules.d/70-snap.*.rules. mount: /var/lib/snapd/mount/*.fstab. kmod: /etc/modules-load.d, /etc/modprobe.d. systemd: /etc/systemd/system. dbus and polkit: files only.

On the gateway, where does each backend's work land, and when? AppArmor writes one profile per app and hook, plus one for snap-update-ns, and loads them into the kernel at once. seccomp compiles a source per app and hook into a binary filter. snap-confine reads it at the next start, so the running agent keeps the filter it started with. udev writes rules and a device-cgroup file, then re-tags devices. Mount writes an fstab, and snap-update-ns changes the running namespace, or throws it away. Modules get modprobe, helper units get systemctl, both best effort. D-Bus and polkit files are only written.

05 Backend by backend, not snap by snap for snapd developers for Ubuntu Core developers

THE OUTER LOOP for backend in repo.Backends(): SetupMany(all affected snaps)acme-gadgetacme-agentsystemdseccompdbusudevmountkmod … symlinksapparmor snap.acme-gadget.interface.gpio-17.serviceExecStart: echo 17 > /sys/class/gpio/exportapplied appliedappliedappliedappliedappliedappliedappliedappliedappliedapplied appliedevalSymlinks(/sys/class/gpio/gpio17)→ <resolved path>/* rwk,needs the export to exist; a missing pin logs "cannot export not existing gpio" and adds no rule See LP: 1802581systemd before apparmor: the one ordering constraint the code documents; the others are likely implicit ✗if udev fails, the loop ends there: systemd, seccomp and dbus stay applied, the rest do not run
A grid with backends as rows and the affected snaps acme-gadget and acme-agent as columns, filled row by row: the systemd row gives acme-gadget the unit snap.acme-gadget.interface.gpio-17.service, which exports pin 17; the middle rows are applied for both snaps; the apparmor row resolves /sys/class/gpio/gpio17 for acme-agent, which needs the export to exist. systemd before apparmor is the one documented constraint. If udev failed, systemd, seccomp and dbus would stay applied and the later backends would not run

setupSecurityByBackend runs each backend over all affected snaps, in list order. The documented constraint: systemd before apparmor, so the gpio unit exports the pin before the apparmor rule resolves it. The first error stops the loop. Pin 17 and the relay plug are examples.

One task, several snaps: in which order? Refresh the agent, and setup-profiles sets up the agent and the gadget it's connected to. It goes backend by backend, each over all the snaps. Here's why. Say the gadget has a gpio slot for pin 17, and the agent a relay plug. The gadget gets a systemd unit that exports the pin. The agent's AppArmor rule must resolve the pin's link in sysfs, which exists only after the export. So systemd runs for everyone before AppArmor runs for anyone. It's the one ordering rule the code documents. If a backend fails, the loop stops there, and the earlier backends stay applied.

06 "Strict, devmode, classic: the same snap three times" for Ubuntu Core developers for snapd developers

ConfinementOptions strictdevmodeclassicapparmorseccompudevDevMode · JailMode · Classic · ExtraLayouts · AppArmorPrompting · KernelSnap flags=(attach_disconnected,mediate_deleted)flags=(…,mediate_deleted,complain) classicTemplatesnippets ignoredflags=(…,complain) template + snippets@complaintemplate + snippets@unrestrictedtemplate + snippets TAG+= rulesdevices tagged for the app# udev tagging/device cgroups disabledwith non-strict mode snapsthen every rule, each commented out with # jailmode: strict output even for devmode or classic dbus, polkit, kmod, systemd, ldconfig, configfiles, symlinks, mount: the same files whatever the flags say
The same snap as strict, devmode and classic, for three backends. AppArmor: strict has flags attach_disconnected and mediate_deleted; devmode adds complain; classic uses classicTemplate, ignores snippets and adds complain. seccomp: strict is the template plus snippets; devmode has @complain and classic @unrestricted right after the version header. udev: strict has active TAG rules; devmode and classic write the header udev tagging/device cgroups disabled with non-strict mode snaps and comment every rule out. The other eight backends write the same output; jailmode gives strict rules even for devmode or classic

(DevMode || Classic) && !JailMode gives AppArmor complain and udev rules commented out; Classic && !JailMode gives classicTemplate (snippets ignored) and seccomp @unrestricted; DevMode && !JailMode gives seccomp @complain.

Picture the agent three times: strict, in devmode, and as a classic snap. What changes in its generated files? Only three backends care. AppArmor adds the complain flag for devmode and classic, so violations are logged, not denied. For classic it also swaps in a separate template that ignores every interface snippet. seccomp puts complain, or unrestricted, right after the version header. udev keeps the rules file but comments every rule out. Jailmode turns all of that back to strict. The other eight backends ignore the confinement flags and write the same files either way.

07 "Remove: the files go, the kernel keeps some" for snapd developers for Ubuntu Core developers

SNAP REMOVE $ snap remove acme-agent… → unlink-snap →remove-profiles→ … discard-snapRemove("acme-agent")for each backend, in list ordersystemdseccompdbusudevmountkmodpolkitldconfigconfigfilessymlinksapparmor files removed, then disable and stopfiles removedfiles removedremoved; udev reloaded if rules existedfstab removed, namespace discardedfiles removed files gone; profiles stay in the kernelservices may still runRemoveLate: snap-confine.snapd.<rev>apparmor, for the snapd snap only files removed; modules stay loaded no-op: system snap onlyno-op: system snap onlyno-op: system snap only not reached: these keep their files✗today the first failing backend ends the loopcannot setup udev for snap "acme-agent": …
snap remove acme-agent runs unlink-snap, then remove-profiles, which calls Remove for each backend in order. systemd removes files then disables and stops units; seccomp, dbus and polkit remove files; udev removes files and reloads if rules existed; mount removes the fstab and discards the namespace; kmod removes files but modules stay loaded; ldconfig, configfiles and symlinks do nothing; apparmor removes files and cache but profiles stay loaded in the kernel, and RemoveLate removes the snapd snap's snap-confine profile. Today a failure at udev stops the loop: mount to apparmor keep their files and the log says cannot setup udev

removeSnapSecurity calls Remove in list order. AppArmor profiles and kernel modules stay loaded; ldconfig, configfiles and symlinks are no-ops. Current behaviour: the first failing backend stops the loop and is logged as "cannot setup".

Now remove the agent. The remove-profiles task calls Remove on every backend, in the same order. Files go by name, and the mount backend throws the namespace away. Two things stay in the kernel on purpose. AppArmor profiles stay loaded, because snapd can't be sure every process of the snap has stopped. And kernel modules stay loaded, by design. ldconfig, configfiles and symlinks do nothing here, since their files belong to the system snap. One catch, today: the loop stops at the first backend that fails. AppArmor comes last, so it keeps its files, and the log says cannot setup, though it was a removal.

08 Where to go from here

WHERE NEXT The security backendseleven, in one order Specifications and snippetshow each spec is filled Setting up a snap's securitythe tasks that call the backends Delayed security setupthe mount update that can waitHow interfaces are implementedthe repository they read Confinementwhere the profiles take effectsnap-update-ns and the mount profileswhat becomes of the fstab files The system keywhen they all run againInterfaces: buses and boardsthe gadget's serial and gpio slots
Roads from The security backends: Specifications and snippets, Setting up a snap's security, Delayed security setup, How interfaces are implemented, Confinement, snap-update-ns and the mount profiles, The system key, and Interfaces: buses and boards

Roads: specifications and snippets, setting up a snap's security, delayed security setup, how interfaces are implemented, confinement, snap-update-ns, the system key, buses and boards.

Eleven backends, one order, and each applies only what changed, except AppArmor, which reloads all of a snap's profiles. How each one's specification gets filled is next door, in specifications and snippets. Which tasks call the backends, and for which snaps, is in setting up a snap's security. The mount update that can wait is in delayed security setup. The repository they all read is in how interfaces are implemented. Where the profiles take effect is confinement, and what becomes of the fstab files is in snap-update-ns. When they all run again is the system key. And the gadget's slots live in buses and boards.

Read more

The backends in order

snapd runs the backends in this order when it sets up a snap, and in the same order when it removes one. The apparmor backend exists only when the kernel's AppArmor support is partial or full; without it there are ten backends.

#BackendFiles it writesWhat applies themOn remove
1systemd/etc/systemd/system/snap.<snap>.interface.<suffix>.service, drop-ins <unit>.d/snap.<name>.confsystemctl daemon-reload, enable, then stop and start; errors are only loggedfiles removed, units disabled and stopped, daemon-reload
2seccomp/var/lib/snapd/seccomp/bpf/<tag>.src, compiled to <tag>.bin2snap-seccomp compile for changed sources; snap-confine loads the .bin2 at the next startfiles removed
3dbus/etc/dbus-1/system.d/<tag>.confnonefiles removed
4udev/etc/udev/rules.d/70-snap.<snap>.rules, /var/lib/snapd/cgroup/snap.<snap>.deviceudevadm control --reload-rules, udevadm trigger, udevadm settle --timeout=10 when the rules changedfiles removed, udev reloaded if the rules file existed
5mount/var/lib/snapd/mount/snap.<snap>.fstab, .user-fstabsnap-update-ns, or the namespace is discarded; the update can be delayedfiles removed, namespace discarded
6kmod/etc/modules-load.d/snap.<snap>.conf, /etc/modprobe.d/snap.<snap>.confmodprobe for each module when the modules file changed; errors are ignoredfiles removed; no module is unloaded
7polkit/usr/share/polkit-1/actions/snap.<snap>.interface.<suffix>.policy, /etc/polkit-1/rules.d/70-snap.<snap>.<suffix>.rulesnonefiles removed
8ldconfig/etc/ld.so.conf.d/snap.system.conf, for the system snap onlyldconfig when the file changednothing
9configfileshost files that interfaces register, for the system snap onlynonenothing
10symlinkssymlinks in host directories that interfaces register, for the system snap onlynonenothing
11apparmor/var/lib/snapd/apparmor/profiles/<tag> and snap-update-ns.<snap>; cache in /var/cache/apparmorapparmor_parser --replace --write-cache for changed and unchanged profilesfiles and cache removed; profiles stay loaded in the kernel

<tag> is a security tag such as snap.acme-agent.agent. Generated files are written atomically with mode 0644, and directories are created with mode 0755.

The code documents one reason for the order: the systemd backend must run before the apparmor backend, because the gpio interface's systemd unit exports the pin that the AppArmor rule then resolves. The code says the other constraints are "implicit and not well-known".

Effects of confinement

Backendstrictdevmodeclassic
apparmorflags=(attach_disconnected,mediate_deleted)adds complainclassicTemplate, interface snippets ignored, adds complain
seccomptemplate and snippets@complain first@unrestricted first
udevrules activethe file starts # udev tagging/device cgroups disabled with non-strict mode snaps and every rule is commented outas devmode

Jailmode gives strict output for devmode and classic snaps. A classic snap in jailmode gets an extra AppArmor snippet that gives access to the core snap, for the dynamic linker and shared libraries. The other eight backends ignore the confinement flags; the mount backend reads only the extra layouts.

for snapd developers for Ubuntu Core developers for administrators

Sandbox features

snap debug sandbox-features prints the features that each backend reports.

BackendFeatures
apparmorkernel:<feature>, parser:<feature>, support-level:full or support-level:partial, policy:default
seccompkernel:<action> for each action the kernel supports, bpf-argument-filtering, bpf-actlog when the compiler supports it
udevtagging, device-filtering, device-cgroup-v2 or device-cgroup-v1
mountlayouts, mount-namespace, per-snap-persistency, per-snap-profiles, per-snap-updates, per-snap-user-profiles, stale-base-invalidation, and freezer-cgroup-v1 when cgroups are not unified
kmodmediated-modprobe
dbusmediated-bus-access
ldconfigmediated-ldconfig
configfilesmediated-configfiles
symlinksmediated-symlinks
systemd, polkitnone

for snapd developers

The interface

interfaces.SecurityBackend has seven methods:

MethodWhen snapd calls it
Initialize(opts)at startup, before repo.AddBackend; an error stops the interface manager from starting
Name()when the backend is added; a second backend with the same name is refused with cannot add backend %q, security system name is in use
Prepare(appSet)before link-snap, from the setup-profiles task in prepare mode; only apparmor does any work
Setup(appSet, opts, sctx, repo, tm)on install, refresh, connect, disconnect and regeneration
Remove(instanceName)from remove-profiles
NewSpecification(appSet, opts)from repo.SnapSpecification, which Setup calls
SandboxFeatures()from snap debug sandbox-features

Optional interfaces:

InterfaceImplemented byPurpose
SecurityBackendSetupManyapparmorloads the changed and the unchanged profiles of many snaps in two batch calls of apparmor_parser; on failure it falls back to Setup for each snap
SecurityBackendDiscardingLateapparmorRemoveLate removes the snap-confine.snapd.<rev> profile of a removed snapd snap revision
ReinitializableSecurityBackendapparmorReinitialize rewrites the snap-confine policy snippets and reloads the snap-confine profile if they changed
DelayedSideEffectsBackendmountApplyDelayedEffects updates a consumer's mount namespace later in the change

A backend without SetupMany is called through interfaces.SetupMany, which calls Setup for each snap and collects the errors.

for snapd developers

Errors and retries

  • setupSecurityByBackend runs each backend over every affected snap and returns the first error. Backends that ran before keep their changes; undo is the job of the task's undo handler.
  • removeSnapSecurity calls Remove in list order and returns at the first error, logged as cannot setup <backend> for snap "<snap>": <error>. The later backends, apparmor among them, keep their files.
  • Only apparmor reloads unchanged profiles on every Setup. The other backends run their command only when a file changed, so a retry with the same content does not repeat a command that failed.
  • seccomp removes the old .bin2 before compiling, and removes every .bin2 of the batch when one compilation fails. The next Setup with the same .src compiles nothing. snap-confine waits for a missing .bin2 for 120 seconds by default (SNAP_CONFINE_MAX_PROFILE_WAIT, at most 3600), then fails.
  • The systemd backend only logs systemctl errors and returns success. The kmod backend ignores modprobe errors on purpose.
  • A failed mount namespace update is followed by a discard, unless the snap has a daemon with refresh-mode: endure; then Setup fails with cannot update mount namespace of snap %q, and cannot discard it because it contains an enduring daemon.

for snapd developers for Ubuntu Core developers

Preseeding

When an image is preseeded, apparmor adds --skip-kernel-load to every apparmor_parser call, udev does not run udevadm, kmod does not run modprobe, and systemd works in emulation mode: no daemon-reload and no stop or start, with enable and disable run as systemctl --root <dir>. In a container the udev backend still writes its files but does not run udevadm. See Preparing images and preseeding.

Not covered

The examples use invented snaps: acme-agent with the daemon agent, and the gadget acme-gadget. The gpio slot with number: 17 and the plug relay are examples. How the bus daemon and polkitd pick up new files was not determined from the snapd code; snapd runs no command for them. The reasons for the positions of the backends other than systemd and apparmor were not determined.

Further reading

No published page describes the security backends. These are the nearest.

Roads from here

INTERFACES AND CONFINEMENT

INTERFACE FAMILIES · every interface, grouped by what it touches

SANDBOX INTERNALS · the mount namespace

AppArmor map ↗