snap-update-ns and the mount profiles
sandbox internals for snapd developers
snap-update-ns makes a snap's mount namespace match the desired mount profile that snapd wrote. It compares it with the current profile and unmounts and mounts the difference with the snap's processes frozen. Where a mount point is missing on a read-only filesystem it builds a writable mimic and a trespassing check keeps it from creating files on the host.
On the map: sandbox internals district
01 A connection made while the app is running
snapd's mount backend writes the desired mount profile; snap-update-ns compares it with the current profile and applies the difference to the preserved namespace, while the app keeps running. snap-confine runs the same tool when it first builds the namespace.
The photo editor is open, and you connect a pack of filters to it. The app's view of the filesystem was built before that connection existed. So who adds the directory? Nobody is going to restart a running program for this. Instead, snapd writes down what the mounts should be: the desired mount profile. A small tool, snap-update-ns, compares that with what's mounted now, works out the difference, and applies only that, inside the app's own mount namespace. The filters just appear. The same tool also finishes the job when a namespace is first built, so there's one way to make the mounts match.
02 Two profiles, one line per mount for app developers for Ubuntu Core developers for snapd developers for administrators
Desired: /var/lib/snapd/mount/snap.<instance>.fstab. Current: /run/snapd/ns/snap.<instance>.fstab. Both use the fstab format; x-snapd.origin, x-snapd.kind, x-snapd.synthetic and x-snapd.needed-by are among the options snapd adds.
Both sides of the comparison are plain text files, and you can read them with cat. The desired profile sits with snapd's other security profiles, and snapd rewrites it whenever a connection or the snap changes. The current profile sits next to the kept namespace and records what was actually done; snap-confine starts it with one line, the root tmpfs. Every line is ordinary fstab: source, mount point, type, options. The x-snapd options say what fstab can't: who asked for the entry, what kind of thing to create, and whether snap-update-ns made it itself. A line with no origin, like the filters, is an interface mount.
03 Inside, locked and frozen for snapd developers
The C bootstrap opens /run/snapd/ns/<instance>.mnt and calls setns; a missing namespace means success with nothing done. The update holds the snap lock and freezes the snap's processes so they cannot redirect a mount; With --from-snap-confine the join is skipped and the lock is checked, not taken; the processes are still frozen.
Before any mount, snap-update-ns has to get three things right: where it is, who else is working, and who might be watching. First, it must be inside the snap's namespace. setns won't switch mount namespaces for a process with several threads, and Go starts threads early, so a few lines of C do the join first. No namespace yet? Then there's nothing to update. Next, it takes the snap's lock, the one snap-confine takes, so no app starts halfway. Then it freezes the snap's processes, so none can plant a symlink to redirect a mount, and thaws them after. Called from snap-confine, it's already inside.
04 "The difference: keep, unmount, mount" for snapd developers
NeededChanges(current, desired) emits keep, unmount and mount. The rootfs entry is always kept; a synthetic entry lives as long as the entry named in x-snapd.needed-by is desired; a changed parent invalidates its children.
Here's the comparison for our connection. Three things are mounted, two are wanted, and one of the wanted ones is new. The tool walks the current entries and asks of each: is this still wanted, exactly as it is? The root tmpfs from snap-confine is always kept. The photo editor's layout matches its desired line, so it stays. The mimic under it isn't desired at all, but it's synthetic, and the layout that needed it is still wanted, so it stays too. Anything else is unmounted, children first, and nothing below a dropped directory is kept. Last, the filters line becomes a mount.
05 Four passes, two attitudes to failure for app developers for snapd developers
Pass 1 keeps and unmounts; 2.1 overname; 2.2 entries without an origin; 3 layouts; then the current profile is saved. A failed layout or overname mount aborts, and snapd discards the namespace unless the snap has a daemon with refresh-mode: endure. An aborted update returns before the current profile is saved.
Mounts stack, so the order they're made in isn't a matter of taste. First, whatever should go is unmounted; a failure there is logged and skipped. Next, for a parallel instance, the overname mounts, since every other path depends on them. Then interface mounts such as content, before any layout, as a layout may point into them. Layouts come last, then the new current profile is saved. A failed content mount is logged, and the rest carries on. A failed overname or layout mount stops the update. Snapd throws the namespace away, and the next start builds it cleanly. With an enduring service it can't, so the change fails.
06 The writable mimic for app developers for snapd developers
On ReadOnlyFsError or TrespassingError the directory is rbind-mounted to /tmp/.snap/<dir>, covered with a tmpfs, each entry is bound back and the copy is detached; creation is then retried once. The entries carry x-snapd.synthetic and x-snapd.needed-by. The mimic exists only in this snap's mount namespace.
The filters need a directory inside the photo editor's own snap. That snap is a read-only squashfs image. You can't mkdir on that. So snap-update-ns sets the directory aside with a recursive bind mount, and covers the original with an empty tmpfs of the same mode and owner. Then it binds every entry back in from the copy, and re-creates the symlinks. The place looks the same, and its top level is writable. It lets go of the copy, makes the filters directory, and mounts the filters there. Every piece is recorded as synthetic, so a later run can take the mimic down.
07 The trespassing check for snapd developers
A path is created only in directories that are read-only, are a tmpfs mounted by a recorded change, or are under an unrestricted path. Otherwise TrespassingError, and a mimic is built over the violated directory. For / the error is "cannot recover from trespassing over /".
Now the opposite problem: a directory that's writable, and that's the trouble. The photo editor has a layout that puts a config file in /etc. Inside the snap, that's the laptop's own /etc, bind mounted in, and writable. An empty file made there as a mount point would stay on the host. So each directory on the way is checked. Read-only is harmless. A tmpfs that snap-confine or this tool mounted exactly there is ours. A few paths are open. Anything else is the host's. The root passes; /etc doesn't. So the mimic goes over /etc. Over the root, there's no mimic to build.
08 Where to go from here
Roads: the per-snap mount namespace, the content interface, layouts, parallel installs, the security backends and confinement.
That's the whole tool: read two files, work out the difference, and apply it without the snap noticing, frozen and in order, with a mimic wherever a mount point can't be made. The namespace it edits is next door, in the per-snap mount namespace deck. Content and layouts fill most of the desired profile, and parallel installs add the overname mounts. How the profiles get written is with the security backends, and where it all sits is in confinement.
Read more
Invocation
snap-update-ns [--from-snap-confine] [--user-mounts] [-u <uid>] <snap>
| Caller | When | Options |
|---|---|---|
| snap-confine | while it builds a new mount namespace, for the system profile and then the user profile | --from-snap-confine; --from-snap-confine --user-mounts for the user profile |
| snapd's mount backend | after the desired system profile of a snap changed and the snap has a kept namespace | the instance name only |
- Without
--from-snap-confineor--user-mounts, C code that runs before the Go runtime opens/run/snapd/ns/<instance>.mntand callssetns(fd, CLONE_NEWNS). This is done in C becausesetnswithCLONE_NEWNSfails once a process has more than one thread, and Go starts threads at will. ENOENTfrom that open orEINVALfromsetnsbecomes "cannot update mount namespace that was not created yet", which is treated as success: there is nothing to update.- Any other failure prints
cannot update snap namespace: <error>and exits with status 1. -uis parsed by the C code and by the Go options, but the Go code only branches on--user-mounts. No caller that passes-uwas found; not determined.- The umask is set to 0.
Note: the "no namespace" case is recognised from the error number of the bootstrap step alone. Other bootstrap steps store their error number in the same place, so an EINVAL from a capability call would also end in a silent success (suspected snapd bug, plausible but not found on a supported system). Do not read exit status 0 as proof that an update happened; the opt-in log below shows what was done.
Files
| File | What |
|---|---|
/var/lib/snapd/mount/snap.<instance>.fstab | desired system profile, written by snapd's mount backend |
/run/snapd/ns/snap.<instance>.fstab | current system profile; snap-confine writes the first line, tmpfs / tmpfs x-snapd.origin=rootfs 0 0, and snap-update-ns rewrites it after each update |
/var/lib/snapd/mount/snap.<instance>.user-fstab | desired user profile; the current user profile is neither loaded nor saved |
/run/snapd/ns/<instance>.mnt | the kept mount namespace that snap-update-ns joins |
/run/snapd/lock/<instance>.lock | the per-snap lock, shared with snap-confine |
/run/snapd/ns/snap.<instance>.log | opt-in debug log of system profile updates; it is used only if it exists |
/run/snapd/ns/snap.<instance>.user.<uid>.log | opt-in debug log of user profile updates |
Nothing in snapd creates or removes the two log files. Create an empty one with touch and every later run for that snap appends its debug log to it.
The desired system profile holds, in this order: the overname entries of a parallel instance, the snap's layouts (and extra layouts), then the interface mount entries such as content.
Mount entry options
Each line is an fstab line: source, mount point, type, options. Fields are split on spaces or tabs and # starts a comment. snapd adds these options:
| Option | Meaning |
|---|---|
x-snapd.kind=file | the mount point is a file |
x-snapd.kind=symlink | create a symlink instead of mounting |
x-snapd.kind=ensure-dir | only create missing directories; nothing is mounted |
x-snapd.symlink=<target> | the target of a symlink entry |
x-snapd.origin=layout | the entry comes from a layout |
x-snapd.origin=overname | the instance-name mapping of a parallel instance |
x-snapd.origin=rootfs | the root tmpfs recorded by snap-confine; always kept |
x-snapd.synthetic | made by snap-update-ns itself, for example a part of a writable mimic |
x-snapd.needed-by=<id> | the entry that a synthetic entry exists for |
x-snapd.id=<id> | a stable identifier; without it the mount point is the identifier |
x-snapd.detach | unmount with MNT_DETACH after making the subtree private |
x-snapd.ignore-missing | skip the entry if its source or mount point is missing |
x-snapd.must-exist-dir=<dir> | for ensure-dir: the directory that must already exist |
x-snapd.mode | mode of a created directory or file, default 0755 |
x-snapd.uid, x-snapd.gid | owner of a created directory or file, default root; these are the names snap-update-ns reads |
x-snapd.user, x-snapd.group | written by the layout code only for a non-root owner, which layouts do not accept yet |
The user profile
The user profile is applied only when snap-confine builds a namespace (see the per-snap mount namespace). It takes no lock and freezes nothing. $XDG_RUNTIME_DIR in it becomes /run/user/<uid>, and $HOME (only in ensure-dir entries) becomes SNAP_REAL_HOME. The unrestricted path for the trespassing check is the real home directory, if it is valid.
Rules of the comparison
NeededChanges(current, desired) returns a list of keep, unmount and mount changes.
- Every mount point is cleaned. The desired entries are sorted by origin (overname, then no origin, then layout) and by mount point.
- A current entry is reused (kept) if it is the
x-snapd.origin=rootfsentry; or it is synthetic and itsx-snapd.needed-bynames a desired entry, unless a different desired entry now targets the same directory; or it equals the desired entry for the same directory. - Once an entry is not reused, the entries below its directory are not reused either.
- The current profile is walked backwards: reused entries become
keep, the restunmount. A tmpfs,bindorrbindentry is unmounted withx-snapd.detach. - Desired entries that were not reused become
mount: first those whose mount point exists, then those that will need a mimic, grouped by their first existing parent, parents before children.
Passes and failures
| Pass | What | On failure |
|---|---|---|
| 1 | all keep and unmount changes | logged ("cannot change mount namespace according to change …"), skipped |
| 2.1 | overname mounts, prepared and performed | the update stops |
| 2.2 | entries without an origin, such as content: all prepared, then all performed | logged and skipped; ErrIgnoredMissingMount is silent |
| 3 | layout mounts: all prepared, then all performed | the update stops |
| last | the current profile is saved: kept entries in their order, then the mounts made, synthetic ones included |
When snap-update-ns fails, snapd's mount backend discards the namespace with snap-discard-ns, so the next start builds it again. The code names the case that motivated this: a layout that changes from a bind mount to a symlink between revisions. If the snap has a daemon with refresh-mode: endure, the namespace is not discarded and the error is cannot update mount namespace of snap "<snap>", and cannot discard it because it contains an enduring daemon: <error>.
When a content slot provider is updated, the update of each consumer's namespace can be delayed: the backend records delayed-consumer-mount-ns-update and the task process-delayed-security-backend-effects applies it later, first trying to discard the namespace if no app of the consumer runs.
Note: when a layout or overname mount fails, the update returns before the current profile is saved, although pass 1 has already unmounted entries and passes 2 and 3 may have mounted some. For most snaps the namespace is then discarded. For a snap with an enduring daemon it is kept, and the current profile may no longer describe it (suspected snapd bug, plausible, not reproduced).
Mimic and trespassing details
A missing mount point is created with MkdirAll, MkfileAll or MksymlinkAll. A missing bind-mount source is created too, but a mimic for the source is allowed only for layout entries. If creation fails with ReadOnlyFsError or TrespassingError, a writable mimic is built over the reported directory and creation is retried once.
The mimic over a directory <dir>:
- rbind
<dir>to/tmp/.snap/<dir>; - mount a tmpfs on
<dir>with the mode, uid and gid of the original; - for each entry of the original: rbind a directory, bind a file (
x-snapd.kind=file), re-create a symlink; other file types are skipped with "skipping unsupported file …"; - detach
/tmp/.snap/<dir>.
Every entry carries x-snapd.synthetic and x-snapd.needed-by=<id>. If building the mimic fails, what was done is unmounted; if that fails too the error is fatal: cannot undo change "…" while recovering from earlier error …: …. The mimic exists only in the snap's own namespace.
Mount point type errors: cannot use "<path>" as mount point: not a directory, … not a regular file, cannot create symlink in "<path>": existing file in the way.
The trespassing check applies to paths outside the unrestricted ones. Each directory on the way is checked before something is created in it; writing is allowed on a read-only filesystem or squashfs, or in a tmpfs that a recorded change mounted exactly at that directory. The current profile is replayed as past changes first, so the root tmpfs and earlier mimics count. Otherwise the error is cannot write to "<path>" because it would affect the host in "<dir>", and the mimic goes over <dir>. For / the error is cannot recover from trespassing over /.
| Profile | Unrestricted paths |
|---|---|
| system | /tmp, /var/snap, /snap/<instance>, /dev/shm, /run/systemd, /var/lib/snapd/hostfs/tmp, and /snap/<snap> for a parallel instance |
| user | the real home directory, if valid |
Created directories get mode 0755 unless a mode hint matches; for example /tmp/.X11-unix and /dev/shm/snap.* get 1777.
Examples
The snap photo-filters and its filters content slot are invented for the deck.
for snapd developers
Tests to read
tests/main/snap-update-ns: a content plug and slot; checks the namespace after connect and disconnect.tests/main/interfaces-content-mimic: a mimic over$SNAPkeeps files, directories and symlinks.tests/main/layout-symlink-bind-revert: a layout that is a symlink in one revision and a bind mount in the next. It looks inside the namespace withsnap debug mount-namespace --shell app -- /bin/ls -l /opt.- Also present:
layout,layout-change,layout-remove,layout-content-delayed-effects,interfaces-content-mimic-missing-target, and the Go tests incmd/snap-update-ns/.
Further reading
No published page explains snap-update-ns. These are the nearest.
- Debugging startup performance: the one page that shows snap-update-ns running, as a line of
snap run --trace-execoutput. - Experimental features:
robust-mount-namespace-updatesandper-user-mount-namespacein the table of former flags. - Test snaps with snap try: layout and interface changes while trying a snap.
Roads from here
INTERFACES AND CONFINEMENT
- The content interface — the content interface whose mounts are applied here
- How interfaces are implemented — the security backends that write the desired profile
- Confinement — where the mount namespace sits in the sandbox
ADMINISTRATION · managing a system of snaps
- Parallel installs — the overname mounts of a parallel instance
DEVELOPING SNAPS · try, pack, run, debug
- Layouts — layouts from the snap developer's side
SANDBOX INTERNALS · the mount namespace
- The per-snap mount namespace — how the namespace is built kept joined and discarded