The system key

sandbox internals for snapd developers for administrators

The system key is a small file that records what the security profiles on disk were generated for including the snapd build and the kernel's AppArmor and seccomp features. snapd compares it at startup and regenerates every profile on a mismatch. snap run compares it before each start and asks snapd whether to proceed or wait.

On the map: sandbox internals district

▶ Watch this deck — 8 scenes, ~6 min

01 Profiles written for a system that has changed

AFTER THE REBOOT snapd refreshed, gateway rebootstime since boot systemdacme-agent.agentearly in bootnew snapdlater profiles on diskwritten by the old snapd ?old profiles, new snapd: may not start system key: what the profiles were generated forsnapd: compares at startupsnap run: compares before every start
A boot time line: snapd was refreshed and the gateway reboots; early in boot systemd starts acme-agent.agent, and the new snapd starts later; the agent meets the profiles on disk, written by the old snapd, so it may not start; the system key records what the profiles were generated for, and snapd compares it at startup while snap run compares it before every start

The system key records what the security profiles on disk were generated for. snapd compares it when it starts, and snap run compares it before each app starts.

The gateway has just taken a new snapd and rebooted. systemd is already starting the agent, as early in boot as it can. Has anyone rewritten the security profiles yet? Maybe not. The profiles on disk were written by the old snapd, and the new snapd's own helper programs may need a permission those profiles don't grant. Then the agent would fail to start. So snapd keeps a small note next to the profiles, saying what they were generated for. That note is the system key. snapd checks it every time it starts, and snap run checks it before every app it starts.

02 What the key records for Ubuntu Core developers for snapd developers for administrators

ELEVEN VALUES /var/lib/snapd/system-key{"version": 11,} snapd binary"build-id": "…", AppArmor: kernel and parser"apparmor-features": […],"apparmor-parser-mtime": …,"apparmor-parser-features": […],"apparmor-prompting": …, seccomp"seccomp-features": […],"seccomp-compiler-version": "…", home and root filesystems"nfs-home": …,"overlay-root": "…",cgroups"cgroup-version": "…" a new snapd, a kernel with other AppArmor or seccomp features, or a home directory moved to NFS changes a value
The file /var/lib/snapd/system-key with its eleven keys and where each comes from: version 11; build-id from the snapd binary; apparmor-features, apparmor-parser-mtime, apparmor-parser-features and apparmor-prompting from the kernel and the AppArmor parser; seccomp-features and seccomp-compiler-version from seccomp; nfs-home and overlay-root from the home directory and the root filesystem; cgroup-version from cgroups

/var/lib/snapd/system-key holds version (11), build-id, four AppArmor values, two seccomp values, nfs-home, overlay-root and cgroup-version.

So what goes on that note? Only things that would change what a profile says, and cheap to read, because snap run works the key out on every start. The file holds eleven values. First the format version, which happens to be eleven too. Then the build ID of the snapd binary, so any new snapd counts as a change. Four values about AppArmor: what the kernel supports, when the parser last changed, its features, and whether prompting is on. Two about seccomp: its actions and the compiler version. Then whether home is on a network filesystem, whether the root is an overlay, and the cgroup version.

03 "snapd at startup: remove, regenerate, write" for Ubuntu Core developers for snapd developers for administrators

SNAPD STARTS snapdkey on disk = key of this system?yes: nothing to do no · missing · cannot compare1 · remove the keysystem-key 2 · regenerate all profileseach security backend, all snapsapparmorseccomp… error: logged, carry on 3 · write the new keyonly if no backend failed snapd, next starta backend failed: no key on diskno key is a mismatch, so the next start removes, regenerates and tries to write again
snapd starts and asks whether the key on disk equals the key of this system; if yes there is nothing to do. If not, or the file is missing or unreadable: 1, remove the key; 2, regenerate all profiles, each security backend over all snaps, with an error logged and the loop carrying on; 3, write the new key, only if no backend failed. If a backend failed, no key is written and the next start of snapd regenerates again

Any mismatch or comparison error triggers regeneration: the key is removed, SetupMany runs per backend over all snaps, and the key is written only if no backend reported an error.

The new snapd is up. Early on, its interface manager works out the key for this system and compares it with the one on disk. Any difference counts, and so does a missing file or one it can't compare. The order matters. First it removes the old key, so that if this snapd is reverted halfway, the older one can't mistake a stale key for a valid one. Next, every security backend rewrites the profiles of every snap. An error is logged and the loop carries on. Only if nothing failed is the new key written. A failed run leaves no key, so the next start does it all again.

04 A start that waits general version

WHEN A START PAUSES $ acme-agent.ctl status snap runsnapdare the profiles current? no answer yet? ask again a second laterusually: go ahead, the profiles are current sometimes:snapd: regenerating security profiles $ snap changesID Status Spawn Ready Summary12 Doing today at 07:02 UTC - Regenerate security profiles the command startsthe command waits for that change to finish, then starts as usual
A terminal runs acme-agent.ctl status and waits. snap run asks snapd whether the profiles are current, and asks again a second later if snapd is still starting. snapd is regenerating security profiles; snap changes shows change 12, Doing, Regenerate security profiles. When the change is done the command starts

After an update or a reboot, starting an app can wait for snapd. If snapd regenerates the profiles, the start waits for the change "Regenerate security profiles", visible in snap changes, and then proceeds.

You run the agent's control command right after the gateway updated, and for a second or two nothing happens. Nothing's stuck. Before starting anything, snap run noticed that the profiles on disk might not fit this system, and asked snapd. If snapd is still starting, it asks again a second later. Usually snapd has already rewritten the profiles and says go ahead. Sometimes it starts rewriting them and tells the command to wait for that change. You can see it in snap changes, as Regenerate security profiles. When it's done, the command starts as usual.

05 snap run asks snapd for Ubuntu Core developers for snapd developers for administrators in place of the general version

BEFORE EVERY START snap run acme-agent.agentsnapd my key = key on disk/tmp/snap-private-tmp existsboth true: start at once system stopping: skip the checkPOST /v2/system-info{"action": "advise-system-key-mismatch", "system-key": "{…}"} proceedstart the appor 405 from an older snapdwait-for-change: change 12wait for it, then starteven if the change failed no answer: 12 tries by default, about a minuteSNAPD_DEBUG_SYSTEM_KEY_RETRY error: timeout waiting for snap system profiles to get updated
snap run acme-agent.agent checks that its key equals the key on disk and that /tmp/snap-private-tmp exists; if both are true it starts at once. Otherwise, unless the system is stopping, it posts advise-system-key-mismatch with its key to /v2/system-info. snapd answers proceed, and the app starts, or wait-for-change with a change id, and the app starts after the change even if it failed; an older snapd that answers 405 counts as proceed. Without an answer it tries twelve times, about a minute in all, overridden by SNAPD_DEBUG_SYSTEM_KEY_RETRY, and then fails with timeout waiting for snap system profiles to get updated

snap run compares keys. On a mismatch it posts advise-system-key-mismatch to /v2/system-info up to 12 times, a second apart plus up to 5 s per try while snapd is starting, about a minute in all, and acts on proceed or wait-for-change. After that: timeout waiting for snap system profiles to get updated.

Back to the agent that systemd started early. This check sits at the top of every snap run. If the keys match and the private tmp directory exists, the app starts at once. If not, and the machine isn't shutting down, snap run posts its own key to /v2/system-info and asks for advice. snapd says proceed, or names a change to wait for. The app starts after that change, even if it failed. An older snapd that doesn't know the question counts as proceed. No answer? Try again a second later, twelve times by default. Then snap run gives up with a timeout.

06 What snapd advises for snapd developers

THE ADVICE seeded?no: error "system not yet seeded", snap run proceeds regeneration change still running?yes: return that change compare with the key on disk same versionparser features skippedif the caller left them outequalnot equalproceedregenerateregenerate: a new change and task of kind regenerate-security-profiles caller's version loweronly nfs-home is comparedequaldiffersproceedregenerate caller's version highersnapd is behind: refusesystem-key version higherthan supported
snapd's advice as a decision tree. Not seeded: the error system not yet seeded, and snap run proceeds. An unfinished regeneration change: it is returned. Otherwise the caller's key is compared with the key on disk. Same version: equal keys proceed, parser features skipped if the caller left them out, anything else regenerates. Caller's version lower: only nfs-home is compared; equal proceeds, different regenerates. Caller's version higher: system-key version higher than supported. Regenerate means a new change and task of kind regenerate-security-profiles

SystemKeyMismatchAdvice returns none for equal keys, regenerate for an older caller whose nfs-home differs, and ErrSystemKeyMismatchVersionTooHigh for a newer caller. Regeneration is a change and task of kind regenerate-security-profiles.

The question arrives at snapd as a key and a claim: this doesn't match. Not seeded yet? It says so, and snap run carries on. A regeneration change already running? It hands that one back. Otherwise it compares the caller's key with the key on disk, which is current, because snapd checked it at startup. Same version: equal keys mean proceed, anything else a new change. Parser features are skipped when the caller left them empty; comparing them once caused an endless regeneration loop. An older caller: only nfs-home is compared, since home can move after snapd started. A newer caller means snapd is behind, so it refuses.

07 Checking it on the gateway for administrators for Ubuntu Core developers for snapd developers

WHAT YOU CAN SEE $ cat /var/lib/snapd/system-key{"version":11,"build-id":"…",…}missing? a regeneration is running or the last one failed $ snap changesID Status Spawn Ready Summary12 Done today at 07:02 UTC today at 07:02 UTC Regenerate security profiles only a regeneration that snap run asked for; the one at snapd's startup is not a change $ snap logs acme-agent.agent… acme-agent.agent[812]: error: timeout waiting for snap system profiles to get updatedsnap run got no answer from snapd in twelve tries a profile edited by hand is overwritten by the next regeneration
Three places to look on the gateway. cat /var/lib/snapd/system-key prints the key, which is missing while a regeneration runs or after a failed one. snap changes lists a done change Regenerate security profiles, only for a regeneration that snap run asked for. snap logs acme-agent.agent shows error: timeout waiting for snap system profiles to get updated. A profile edited by hand is overwritten by the next regeneration

/var/lib/snapd/system-key is absent during and after a failed regeneration. A regeneration requested by snap run appears in snap changes as "Regenerate security profiles"; a start that gave up logs the timeout.

Say the agent didn't come up after last night's update. Where do you look? First, the key itself. Anyone can read the file. If it's missing, a regeneration is running or the last one failed. Second, snap changes. A regeneration that snap run asked for shows up there as Regenerate security profiles. The one snapd does by itself at startup doesn't. Third, the agent's log, where a start that gave up says so. And one thing to remember: a mismatch rewrites every profile, so a profile you edited by hand while debugging won't survive the next new snapd.

08 Where to go from here

ROADS The system key Confinementthe profiles the key stands forHow interfaces are implementedthe backends that rewrite them The overlordthe regeneration changeServiceswhat starts at boot How snapd restarts and re-executeswhen snapd itself is replacedPreparing images and preseedingprofiles made at image build The per-snap mount namespacethis district's hub
Roads from The system key: Confinement, How interfaces are implemented, The overlord, Services, How snapd restarts and re-executes, Preparing images and preseeding, and The per-snap mount namespace

Roads: confinement, how interfaces are implemented, the overlord, services, snapd restart and re-exec, images and preseeding, and the per-snap mount namespace.

One small file, compared in two places, so that no app starts on profiles written for a different system. That's the system key. The profiles it stands for are in the confinement deck. The backends that rewrite them are in how interfaces are implemented. The regeneration change is a change like any other, so the overlord explains it, and the services deck covers what starts at boot. Why snapd restarts at all is in the daemon's own district, next to profiles prepared when an image is built. And the mount namespace an app starts in is this district's hub.

Read more

The fields

The key is the JSON file /var/lib/snapd/system-key, written with mode 0644. Format version 11.

JSON keySource
versionthe format version, 11
build-idthe build ID of the snapd binary
apparmor-featuresthe AppArmor features of the kernel
apparmor-parser-mtimethe modification time of the AppArmor parser
apparmor-parser-featuresthe features of the AppArmor parser; filled only when snapd writes the key
apparmor-promptingthe prompting feature flag, and support for prompting by the kernel and the parser
nfs-homewhether home directories are on a remote filesystem such as NFS
overlay-rootwhether the root filesystem is a writable overlay
seccomp-featuresthe seccomp actions the kernel supports
seccomp-compiler-versionthe version of the seccomp compiler shipped with snapd
cgroup-versionthe cgroup version, as a string

Every value is cheap to compute, because snap run computes the whole key before each start. snap run does not run the AppArmor parser; it reads only its modification time, and a changed parser changes that time.

Comparison rules

CaseResult
no filesystem-key missing on disk, treated as a mismatch
different versionsystem-key versions not comparable
the binary runs from outside /usr and the snap mount directorylogged as running from non-installed location <path>: ignoring system-key
otherwiseapparmor-parser-features is cleared on both sides and the rest is compared field by field

snapd at startup treats any comparison error as a mismatch.

snapd at startup

When the interface manager starts and the keys do not match:

1. The key file is removed, so that a snapd reverted after a failed start cannot take the old key as valid. 2. Every security backend regenerates the profiles of every snap. An error is logged as cannot regenerate <backend> profiles and the loop continues. 3. The new key is written only if no backend reported an error. A failed regeneration leaves no key, and the next start regenerates again.

This regeneration is not a change and does not appear in snap changes.

Two other places remove the key. When snapd uses its own AppArmor parser, refreshing snapd to an older version removes all snap AppArmor profiles and the key, so the downgraded snapd regenerates them. Resetting a preseeded image removes the key file with the other files it resets.

snap run: outcomes

snap run checks the key before it starts any app, hook or service.

SituationWhat snap run does
keys match and /tmp/snap-private-tmp existsstarts at once
systemctl is-system-running prints stoppingstarts; the mismatch is ignored during shutdown
snapd answers proceedstarts
snapd answers wait-for-change with a change IDwaits for that change, then starts, also when the change failed
HTTP 405: an older snapd without the advice APIstarts
another API error, such as system not yet seededstarts; the sandbox may not be set up completely
system-key version higher than supported while snapd is restartingtries again
no answer, for example while snapd is not runningtries again
no decision after the last tryfails with timeout waiting for snap system profiles to get updated

The request is POST /v2/system-info with {"action": "advise-system-key-mismatch", "system-key": "<the key as a JSON string>"}. There are 12 tries by default, with a one-second pause between them; the environment variable SNAPD_DEBUG_SYSTEM_KEY_RETRY sets another count.

snapd's advice

CheckAdvice
the device is not seedederror system not yet seeded
a regenerate-security-profiles change is not finishedthat change
same key version, keys equalproceed
same key version, keys differa new change
caller's version lower, nfs-home equalproceed
caller's version lower, nfs-home differsa new change
caller's version highererror system-key version higher than supported

snapd compares the caller's key with the key on disk, which it rewrote at startup. When the caller's key has no apparmor-parser-features, the field is left out of the comparison. Comparing it made every request look like a mismatch and caused an endless regeneration loop (LP: #2161845).

A new change has kind regenerate-security-profiles and summary Regenerate security profiles, with one task of the same kind. The task re-initialises the security backends and then regenerates every profile as at startup, including removing and rewriting the key.

for snapd developers

For snapd developers

  • Bump systemKeyVersion in interfaces/system_key.go whenever an input of the key is added, removed or changed.
  • The regeneration task unlocks the state while the backends work; the startup path keeps it locked.
  • The spread test tests/main/snap-system-key checks that the key has a build-id, that a restart with an unchanged key does not rewrite the file, that a file holding {"version":1} is rewritten at restart, that with snapd stopped and SNAPD_DEBUG_SYSTEM_KEY_RETRY=1 an app fails to start, and that with snapd running the same bad file does not stop the app.

Not covered

The examples use an invented app acme-agent.ctl.

The system keys kept for preseeding, preseed-system-key and seed-restart-system-key, were not researched; see Preparing images and preseeding. Whether anything besides snap run consults the key, such as snap-confine, was not determined.

Further reading

No published page names the system key.

  • Security policies: where profiles live and when snapd generates them.
  • Optimise boot speed: the one published consequence, differing AppArmor features that "nullify the pre-created security profiles".
  • Debug snaps: editing a profile in place, which a regeneration overwrites.

Roads from here

INTERFACES AND CONFINEMENT

RUNNING SNAPS

  • The overlord — changes and tasks such as the regeneration change
  • Services — services that start at boot before snapd is ready

SANDBOX INTERNALS · the mount namespace

INSIDE SNAPD · the daemon itself