The system key
sandbox internals for snapd developers for administrators
The system key is a small file that records what the security profiles on disk were generated for including the snapd build and the kernel's AppArmor and seccomp features. snapd compares it at startup and regenerates every profile on a mismatch. snap run compares it before each start and asks snapd whether to proceed or wait.
On the map: sandbox internals district
01 Profiles written for a system that has changed
The system key records what the security profiles on disk were generated for. snapd compares it when it starts, and snap run compares it before each app starts.
The gateway has just taken a new snapd and rebooted. systemd is already starting the agent, as early in boot as it can. Has anyone rewritten the security profiles yet? Maybe not. The profiles on disk were written by the old snapd, and the new snapd's own helper programs may need a permission those profiles don't grant. Then the agent would fail to start. So snapd keeps a small note next to the profiles, saying what they were generated for. That note is the system key. snapd checks it every time it starts, and snap run checks it before every app it starts.
02 What the key records for Ubuntu Core developers for snapd developers for administrators
/var/lib/snapd/system-key holds version (11), build-id, four AppArmor values, two seccomp values, nfs-home, overlay-root and cgroup-version.
So what goes on that note? Only things that would change what a profile says, and cheap to read, because snap run works the key out on every start. The file holds eleven values. First the format version, which happens to be eleven too. Then the build ID of the snapd binary, so any new snapd counts as a change. Four values about AppArmor: what the kernel supports, when the parser last changed, its features, and whether prompting is on. Two about seccomp: its actions and the compiler version. Then whether home is on a network filesystem, whether the root is an overlay, and the cgroup version.
03 "snapd at startup: remove, regenerate, write" for Ubuntu Core developers for snapd developers for administrators
Any mismatch or comparison error triggers regeneration: the key is removed, SetupMany runs per backend over all snaps, and the key is written only if no backend reported an error.
The new snapd is up. Early on, its interface manager works out the key for this system and compares it with the one on disk. Any difference counts, and so does a missing file or one it can't compare. The order matters. First it removes the old key, so that if this snapd is reverted halfway, the older one can't mistake a stale key for a valid one. Next, every security backend rewrites the profiles of every snap. An error is logged and the loop carries on. Only if nothing failed is the new key written. A failed run leaves no key, so the next start does it all again.
04 A start that waits general version
After an update or a reboot, starting an app can wait for snapd. If snapd regenerates the profiles, the start waits for the change "Regenerate security profiles", visible in snap changes, and then proceeds.
You run the agent's control command right after the gateway updated, and for a second or two nothing happens. Nothing's stuck. Before starting anything, snap run noticed that the profiles on disk might not fit this system, and asked snapd. If snapd is still starting, it asks again a second later. Usually snapd has already rewritten the profiles and says go ahead. Sometimes it starts rewriting them and tells the command to wait for that change. You can see it in snap changes, as Regenerate security profiles. When it's done, the command starts as usual.
05 snap run asks snapd for Ubuntu Core developers for snapd developers for administrators in place of the general version
snap run compares keys. On a mismatch it posts advise-system-key-mismatch to /v2/system-info up to 12 times, a second apart plus up to 5 s per try while snapd is starting, about a minute in all, and acts on proceed or wait-for-change. After that: timeout waiting for snap system profiles to get updated.
Back to the agent that systemd started early. This check sits at the top of every snap run. If the keys match and the private tmp directory exists, the app starts at once. If not, and the machine isn't shutting down, snap run posts its own key to /v2/system-info and asks for advice. snapd says proceed, or names a change to wait for. The app starts after that change, even if it failed. An older snapd that doesn't know the question counts as proceed. No answer? Try again a second later, twelve times by default. Then snap run gives up with a timeout.
06 What snapd advises for snapd developers
SystemKeyMismatchAdvice returns none for equal keys, regenerate for an older caller whose nfs-home differs, and ErrSystemKeyMismatchVersionTooHigh for a newer caller. Regeneration is a change and task of kind regenerate-security-profiles.
The question arrives at snapd as a key and a claim: this doesn't match. Not seeded yet? It says so, and snap run carries on. A regeneration change already running? It hands that one back. Otherwise it compares the caller's key with the key on disk, which is current, because snapd checked it at startup. Same version: equal keys mean proceed, anything else a new change. Parser features are skipped when the caller left them empty; comparing them once caused an endless regeneration loop. An older caller: only nfs-home is compared, since home can move after snapd started. A newer caller means snapd is behind, so it refuses.
07 Checking it on the gateway for administrators for Ubuntu Core developers for snapd developers
/var/lib/snapd/system-key is absent during and after a failed regeneration. A regeneration requested by snap run appears in snap changes as "Regenerate security profiles"; a start that gave up logs the timeout.
Say the agent didn't come up after last night's update. Where do you look? First, the key itself. Anyone can read the file. If it's missing, a regeneration is running or the last one failed. Second, snap changes. A regeneration that snap run asked for shows up there as Regenerate security profiles. The one snapd does by itself at startup doesn't. Third, the agent's log, where a start that gave up says so. And one thing to remember: a mismatch rewrites every profile, so a profile you edited by hand while debugging won't survive the next new snapd.
08 Where to go from here
Roads: confinement, how interfaces are implemented, the overlord, services, snapd restart and re-exec, images and preseeding, and the per-snap mount namespace.
One small file, compared in two places, so that no app starts on profiles written for a different system. That's the system key. The profiles it stands for are in the confinement deck. The backends that rewrite them are in how interfaces are implemented. The regeneration change is a change like any other, so the overlord explains it, and the services deck covers what starts at boot. Why snapd restarts at all is in the daemon's own district, next to profiles prepared when an image is built. And the mount namespace an app starts in is this district's hub.
Read more
The fields
The key is the JSON file /var/lib/snapd/system-key, written with mode 0644. Format version 11.
| JSON key | Source |
|---|---|
version | the format version, 11 |
build-id | the build ID of the snapd binary |
apparmor-features | the AppArmor features of the kernel |
apparmor-parser-mtime | the modification time of the AppArmor parser |
apparmor-parser-features | the features of the AppArmor parser; filled only when snapd writes the key |
apparmor-prompting | the prompting feature flag, and support for prompting by the kernel and the parser |
nfs-home | whether home directories are on a remote filesystem such as NFS |
overlay-root | whether the root filesystem is a writable overlay |
seccomp-features | the seccomp actions the kernel supports |
seccomp-compiler-version | the version of the seccomp compiler shipped with snapd |
cgroup-version | the cgroup version, as a string |
Every value is cheap to compute, because snap run computes the whole key before each start. snap run does not run the AppArmor parser; it reads only its modification time, and a changed parser changes that time.
Comparison rules
| Case | Result |
|---|---|
| no file | system-key missing on disk, treated as a mismatch |
different version | system-key versions not comparable |
the binary runs from outside /usr and the snap mount directory | logged as running from non-installed location <path>: ignoring system-key |
| otherwise | apparmor-parser-features is cleared on both sides and the rest is compared field by field |
snapd at startup treats any comparison error as a mismatch.
snapd at startup
When the interface manager starts and the keys do not match:
1. The key file is removed, so that a snapd reverted after a failed start cannot take the old key as valid. 2. Every security backend regenerates the profiles of every snap. An error is logged as cannot regenerate <backend> profiles and the loop continues. 3. The new key is written only if no backend reported an error. A failed regeneration leaves no key, and the next start regenerates again.
This regeneration is not a change and does not appear in snap changes.
Two other places remove the key. When snapd uses its own AppArmor parser, refreshing snapd to an older version removes all snap AppArmor profiles and the key, so the downgraded snapd regenerates them. Resetting a preseeded image removes the key file with the other files it resets.
snap run: outcomes
snap run checks the key before it starts any app, hook or service.
| Situation | What snap run does |
|---|---|
keys match and /tmp/snap-private-tmp exists | starts at once |
systemctl is-system-running prints stopping | starts; the mismatch is ignored during shutdown |
snapd answers proceed | starts |
snapd answers wait-for-change with a change ID | waits for that change, then starts, also when the change failed |
| HTTP 405: an older snapd without the advice API | starts |
another API error, such as system not yet seeded | starts; the sandbox may not be set up completely |
system-key version higher than supported while snapd is restarting | tries again |
| no answer, for example while snapd is not running | tries again |
| no decision after the last try | fails with timeout waiting for snap system profiles to get updated |
The request is POST /v2/system-info with {"action": "advise-system-key-mismatch", "system-key": "<the key as a JSON string>"}. There are 12 tries by default, with a one-second pause between them; the environment variable SNAPD_DEBUG_SYSTEM_KEY_RETRY sets another count.
snapd's advice
| Check | Advice |
|---|---|
| the device is not seeded | error system not yet seeded |
a regenerate-security-profiles change is not finished | that change |
| same key version, keys equal | proceed |
| same key version, keys differ | a new change |
caller's version lower, nfs-home equal | proceed |
caller's version lower, nfs-home differs | a new change |
| caller's version higher | error system-key version higher than supported |
snapd compares the caller's key with the key on disk, which it rewrote at startup. When the caller's key has no apparmor-parser-features, the field is left out of the comparison. Comparing it made every request look like a mismatch and caused an endless regeneration loop (LP: #2161845).
A new change has kind regenerate-security-profiles and summary Regenerate security profiles, with one task of the same kind. The task re-initialises the security backends and then regenerates every profile as at startup, including removing and rewriting the key.
for snapd developers
For snapd developers
- Bump
systemKeyVersionininterfaces/system_key.gowhenever an input of the key is added, removed or changed. - The regeneration task unlocks the state while the backends work; the startup path keeps it locked.
- The spread test
tests/main/snap-system-keychecks that the key has abuild-id, that a restart with an unchanged key does not rewrite the file, that a file holding{"version":1}is rewritten at restart, that with snapd stopped andSNAPD_DEBUG_SYSTEM_KEY_RETRY=1an app fails to start, and that with snapd running the same bad file does not stop the app.
Not covered
The examples use an invented app acme-agent.ctl.
The system keys kept for preseeding, preseed-system-key and seed-restart-system-key, were not researched; see Preparing images and preseeding. Whether anything besides snap run consults the key, such as snap-confine, was not determined.
Further reading
No published page names the system key.
- Security policies: where profiles live and when snapd generates them.
- Optimise boot speed: the one published consequence, differing AppArmor features that "nullify the pre-created security profiles".
- Debug snaps: editing a profile in place, which a regeneration overwrites.
Roads from here
INTERFACES AND CONFINEMENT
- Confinement — the profiles that the key stands for
- How interfaces are implemented — the security backends that regenerate them
RUNNING SNAPS
- The overlord — changes and tasks such as the regeneration change
- Services — services that start at boot before snapd is ready
SANDBOX INTERNALS · the mount namespace
- The per-snap mount namespace — the district hub and the namespace an app starts in
INSIDE SNAPD · the daemon itself
- How snapd restarts and re-executes — what happens when snapd itself is replaced
- Preparing images and preseeding — profiles generated when an image is built