snap-update-ns and the mount profiles

sandbox internals for snapd developers

snap-update-ns makes a snap's mount namespace match the desired mount profile that snapd wrote. It compares it with the current profile and unmounts and mounts the difference with the snap's processes frozen. Where a mount point is missing on a read-only filesystem it builds a writable mimic and a trespassing check keeps it from creating files on the host.

On the map: sandbox internals district

▶ Watch this deck — 8 scenes, ~6 min

01 A connection made while the app is running

CHANGE IT IN PLACE photo-editor's mount namespacephoto-editor (running)built an hour ago, before the connection $ snap connect photo-editor:filters photo-filters:filters desired mount profilewritten by snapd+ filters current mount profilewhat is mounted nowsnap-update-nsdifference filtersmounted in place; the app keeps running snap-confine runs the same tool when it first builds a namespace: one way to make the mounts match
photo-editor runs inside its mount namespace; snap connect adds a filters connection; snapd writes the desired mount profile, snap-update-ns compares it with the current profile and mounts the difference, filters, into the running namespace; snap-confine runs the same tool when it first builds the namespace

snapd's mount backend writes the desired mount profile; snap-update-ns compares it with the current profile and applies the difference to the preserved namespace, while the app keeps running. snap-confine runs the same tool when it first builds the namespace.

The photo editor is open, and you connect a pack of filters to it. The app's view of the filesystem was built before that connection existed. So who adds the directory? Nobody is going to restart a running program for this. Instead, snapd writes down what the mounts should be: the desired mount profile. A small tool, snap-update-ns, compares that with what's mounted now, works out the difference, and applies only that, inside the app's own mount namespace. The filters just appear. The same tool also finishes the job when a namespace is first built, so there's one way to make the mounts match.

02 Two profiles, one line per mount for app developers for Ubuntu Core developers for snapd developers for administrators

DESIRED AND CURRENT desired · /var/lib/snapd/mount/snap.photo-editor.fstab/snap/photo-editor/42/usr/share/photo-editor/usr/share/photo-editor none rbind,rw,x-snapd.origin=layout 0 0/snap/photo-filters/7/filters/snap/photo-editor/42/filters none bind,ro 0 0 current · /run/snapd/ns/snap.photo-editor.fstabtmpfs / tmpfs x-snapd.origin=rootfs 0 0# then one line per mount made, synthetic ones toowritten first by snap-confine /snap/photo-filters/7/filters/snap/photo-editor/42/filtersnonebind,rosourcemount pointtypeoptions x-snapd.origin=layoutwho asked: layout, overname, rootfsx-snapd.kind=filea file, a symlink or ensure-dirx-snapd.syntheticmade by snap-update-ns itself no origin: an interface mount, such as content · desired = interface mounts + parallel-instance overname + layouts
The desired profile in /var/lib/snapd/mount with a layout line and a content line, the current profile in /run/snapd/ns starting with the rootfs line written by snap-confine, the four fields of a line, and three x-snapd options: origin, kind and synthetic

Desired: /var/lib/snapd/mount/snap.<instance>.fstab. Current: /run/snapd/ns/snap.<instance>.fstab. Both use the fstab format; x-snapd.origin, x-snapd.kind, x-snapd.synthetic and x-snapd.needed-by are among the options snapd adds.

Both sides of the comparison are plain text files, and you can read them with cat. The desired profile sits with snapd's other security profiles, and snapd rewrites it whenever a connection or the snap changes. The current profile sits next to the kept namespace and records what was actually done; snap-confine starts it with one line, the root tmpfs. Every line is ordinary fstab: source, mount point, type, options. The x-snapd options say what fstab can't: who asked for the entry, what kind of thing to create, and whether snap-update-ns made it itself. A line with no origin, like the filters, is an interface mount.

03 Inside, locked and frozen for snapd developers

BEFORE ANY MOUNT snapdsnap-update-ns photo-editorphoto-editor's mount namespacephoto-editor 1 · join, in C, before the Go runtime startsopen /run/snapd/ns/photo-editor.mntsetns(fd, CLONE_NEWNS)snap-update-ns no namespace yet: nothing to do, exit 0 2 · lock/run/snapd/lock/photo-editor.lockthe same lock snap-confine takes 3 · freeze, update, thawno snap process can plant a symlink mid-updatephoto-editor · frozen --from-snap-confine: already inside,no setns; checks the lock is held, still freezes
snapd runs snap-update-ns photo-editor; C code opens /run/snapd/ns/photo-editor.mnt and calls setns before Go starts; a missing namespace means nothing to do; it takes /run/snapd/lock/photo-editor.lock, freezes photo-editor's processes, updates and thaws them; from snap-confine there is no setns, the lock is already held and the processes are still frozen

The C bootstrap opens /run/snapd/ns/<instance>.mnt and calls setns; a missing namespace means success with nothing done. The update holds the snap lock and freezes the snap's processes so they cannot redirect a mount; With --from-snap-confine the join is skipped and the lock is checked, not taken; the processes are still frozen.

Before any mount, snap-update-ns has to get three things right: where it is, who else is working, and who might be watching. First, it must be inside the snap's namespace. setns won't switch mount namespaces for a process with several threads, and Go starts threads early, so a few lines of C do the join first. No namespace yet? Then there's nothing to update. Next, it takes the snap's lock, the one snap-confine takes, so no app starts halfway. Then it freezes the snap's processes, so none can plant a symlink to redirect a mount, and thaws them after. Called from snap-confine, it's already inside.

04 "The difference: keep, unmount, mount" for snapd developers

NEEDED CHANGES currentchangedesired / (rootfs tmpfs)/usr/share (mimic, synthetic)/usr/share/photo-editor (layout) /usr/share/photo-editor (layout)…/42/filters (content) keepalwayskeepsame entry keepneeded by layout anything else: unmount, in reverse order, children before their parentsa parent entry that is not kept takes everything below it: unmounted, then mounted again mountnot there yet
Current entries: the rootfs tmpfs, a mimic over /usr/share and the photo-editor layout; desired entries: the layout and the new filters content mount; rootfs is always kept, the layout is kept as the same entry, the mimic is kept because the layout still needs it, and filters becomes a mount; a changed parent takes its children with it

NeededChanges(current, desired) emits keep, unmount and mount. The rootfs entry is always kept; a synthetic entry lives as long as the entry named in x-snapd.needed-by is desired; a changed parent invalidates its children.

Here's the comparison for our connection. Three things are mounted, two are wanted, and one of the wanted ones is new. The tool walks the current entries and asks of each: is this still wanted, exactly as it is? The root tmpfs from snap-confine is always kept. The photo editor's layout matches its desired line, so it stays. The mimic under it isn't desired at all, but it's synthetic, and the layout that needed it is still wanted, so it stays too. Anything else is unmounted, children first, and nothing below a dropped directory is kept. Last, the filters line becomes a mount.

05 Four passes, two attitudes to failure for app developers for snapd developers

ORDER OF WORK 1 · keep, unmountwhat should gofailure: logged, skipped 2.1 · overnameparallel instancesfailure: stop 2.2 · interface mountscontent and othersfailure: logged, skipped 3 · layoutslastfailure: stopsave the current profilereached only if no pass stopped when the update stopssnapdsnap-discard-ns photo-editorthe next start builds the namespace again a daemon with refresh-mode: endureno discard: the change fails
Four passes in order: keep and unmount, failures logged; overname for parallel instances, a failure stops; interface mounts such as content, failures logged; layouts, a failure stops; then the current profile is saved. When the update stops, snapd runs snap-discard-ns and the next start rebuilds the namespace, unless a daemon has refresh-mode: endure, and then the change fails

Pass 1 keeps and unmounts; 2.1 overname; 2.2 entries without an origin; 3 layouts; then the current profile is saved. A failed layout or overname mount aborts, and snapd discards the namespace unless the snap has a daemon with refresh-mode: endure. An aborted update returns before the current profile is saved.

Mounts stack, so the order they're made in isn't a matter of taste. First, whatever should go is unmounted; a failure there is logged and skipped. Next, for a parallel instance, the overname mounts, since every other path depends on them. Then interface mounts such as content, before any layout, as a layout may point into them. Layouts come last, then the new current profile is saved. A failed content mount is logged, and the rest carries on. A failed overname or layout mount stops the update. Snapd throws the namespace away, and the next start builds it cleanly. With an enduring service it can't, so the change fails.

06 The writable mimic for app developers for snapd developers

A DIRECTORY THAT LOOKS THE SAME /snap/photo-editor/42$SNAP · squashfs, read-onlybinlibmetamkdir filters: read-only file system rbind/tmp/.snap/snap/photo-editor/42set asidebin lib meta tmpfsempty, same mode and owner as before binlibmetabound back: directories rbind, files bind, symlinks re-created detachedfiltersphoto-filters every piece is recorded in the current profilex-snapd.synthetic,x-snapd.needed-by=/snap/photo-editor/42/filtersonly in this snap's namespace
mkdir filters fails in the read-only squashfs /snap/photo-editor/42; the directory is rbind-mounted aside to /tmp/.snap/snap/photo-editor/42, covered with a tmpfs of the same mode and owner, its entries are bound back, the copy is detached, filters is created and photo-filters is mounted on it; every piece is recorded as synthetic

On ReadOnlyFsError or TrespassingError the directory is rbind-mounted to /tmp/.snap/<dir>, covered with a tmpfs, each entry is bound back and the copy is detached; creation is then retried once. The entries carry x-snapd.synthetic and x-snapd.needed-by. The mimic exists only in this snap's mount namespace.

The filters need a directory inside the photo editor's own snap. That snap is a read-only squashfs image. You can't mkdir on that. So snap-update-ns sets the directory aside with a recursive bind mount, and covers the original with an empty tmpfs of the same mode and owner. Then it binds every entry back in from the copy, and re-creates the symlinks. The place looks the same, and its top level is writable. It lets go of the copy, makes the filters directory, and mounts the filters there. Every piece is recorded as synthetic, so a later run can take the mimic down.

07 The trespassing check for snapd developers

WOULD THIS WRITE REACH THE HOST? /etc/photo-editor.conflayout: a file bound from $SNAP_COMMON may create in a directory that isread-only, or a squashfsa tmpfs that a recorded changemounted exactly thereunder an unrestricted path:/tmp /var/snap /dev/shm/snap/photo-editor … /tmpfs mounted by snap-confine: fine/etcthe host's /etc, writable: stop cannot write to "/etc/photo-editor.conf"because it would affect the host in "/etc" writable mimic over /etccreate the empty file on the tmpfsbind the real file onto it the root has no mimic: "cannot recover from trespassing over /"
Creating /etc/photo-editor.conf for a layout: the root, a tmpfs mounted by snap-confine, passes; /etc, the host's writable directory, fails with a trespassing error; a writable mimic goes over /etc, the empty file is created on the tmpfs and the real file is bound onto it. Writing is allowed in read-only directories, in a tmpfs a recorded change mounted there, and under a few unrestricted paths; there is no mimic over the root

A path is created only in directories that are read-only, are a tmpfs mounted by a recorded change, or are under an unrestricted path. Otherwise TrespassingError, and a mimic is built over the violated directory. For / the error is "cannot recover from trespassing over /".

Now the opposite problem: a directory that's writable, and that's the trouble. The photo editor has a layout that puts a config file in /etc. Inside the snap, that's the laptop's own /etc, bind mounted in, and writable. An empty file made there as a mount point would stay on the host. So each directory on the way is checked. Read-only is harmless. A tmpfs that snap-confine or this tool mounted exactly there is ours. A few paths are open. Anything else is the host's. The root passes; /etc doesn't. So the mimic goes over /etc. Over the root, there's no mimic to build.

08 Where to go from here

ROADS snap-update-ns The per-snap mount namespacebuilt, kept, joined, discarded The content interfacemounts applied hereLayoutsfrom the snap developer's side Parallel installsthe overname mounts How interfaces are implementedbackends write the profileConfinementwhere the namespace sits
Roads from snap-update-ns: the per-snap mount namespace, the content interface, layouts, parallel installs, the security backends and confinement

Roads: the per-snap mount namespace, the content interface, layouts, parallel installs, the security backends and confinement.

That's the whole tool: read two files, work out the difference, and apply it without the snap noticing, frozen and in order, with a mimic wherever a mount point can't be made. The namespace it edits is next door, in the per-snap mount namespace deck. Content and layouts fill most of the desired profile, and parallel installs add the overname mounts. How the profiles get written is with the security backends, and where it all sits is in confinement.

Read more

Invocation

snap-update-ns [--from-snap-confine] [--user-mounts] [-u <uid>] <snap>
CallerWhenOptions
snap-confinewhile it builds a new mount namespace, for the system profile and then the user profile--from-snap-confine; --from-snap-confine --user-mounts for the user profile
snapd's mount backendafter the desired system profile of a snap changed and the snap has a kept namespacethe instance name only
  • Without --from-snap-confine or --user-mounts, C code that runs before the Go runtime opens /run/snapd/ns/<instance>.mnt and calls setns(fd, CLONE_NEWNS). This is done in C because setns with CLONE_NEWNS fails once a process has more than one thread, and Go starts threads at will.
  • ENOENT from that open or EINVAL from setns becomes "cannot update mount namespace that was not created yet", which is treated as success: there is nothing to update.
  • Any other failure prints cannot update snap namespace: <error> and exits with status 1.
  • -u is parsed by the C code and by the Go options, but the Go code only branches on --user-mounts. No caller that passes -u was found; not determined.
  • The umask is set to 0.

Note: the "no namespace" case is recognised from the error number of the bootstrap step alone. Other bootstrap steps store their error number in the same place, so an EINVAL from a capability call would also end in a silent success (suspected snapd bug, plausible but not found on a supported system). Do not read exit status 0 as proof that an update happened; the opt-in log below shows what was done.

Files

FileWhat
/var/lib/snapd/mount/snap.<instance>.fstabdesired system profile, written by snapd's mount backend
/run/snapd/ns/snap.<instance>.fstabcurrent system profile; snap-confine writes the first line, tmpfs / tmpfs x-snapd.origin=rootfs 0 0, and snap-update-ns rewrites it after each update
/var/lib/snapd/mount/snap.<instance>.user-fstabdesired user profile; the current user profile is neither loaded nor saved
/run/snapd/ns/<instance>.mntthe kept mount namespace that snap-update-ns joins
/run/snapd/lock/<instance>.lockthe per-snap lock, shared with snap-confine
/run/snapd/ns/snap.<instance>.logopt-in debug log of system profile updates; it is used only if it exists
/run/snapd/ns/snap.<instance>.user.<uid>.logopt-in debug log of user profile updates

Nothing in snapd creates or removes the two log files. Create an empty one with touch and every later run for that snap appends its debug log to it.

The desired system profile holds, in this order: the overname entries of a parallel instance, the snap's layouts (and extra layouts), then the interface mount entries such as content.

Mount entry options

Each line is an fstab line: source, mount point, type, options. Fields are split on spaces or tabs and # starts a comment. snapd adds these options:

OptionMeaning
x-snapd.kind=filethe mount point is a file
x-snapd.kind=symlinkcreate a symlink instead of mounting
x-snapd.kind=ensure-dironly create missing directories; nothing is mounted
x-snapd.symlink=<target>the target of a symlink entry
x-snapd.origin=layoutthe entry comes from a layout
x-snapd.origin=overnamethe instance-name mapping of a parallel instance
x-snapd.origin=rootfsthe root tmpfs recorded by snap-confine; always kept
x-snapd.syntheticmade by snap-update-ns itself, for example a part of a writable mimic
x-snapd.needed-by=<id>the entry that a synthetic entry exists for
x-snapd.id=<id>a stable identifier; without it the mount point is the identifier
x-snapd.detachunmount with MNT_DETACH after making the subtree private
x-snapd.ignore-missingskip the entry if its source or mount point is missing
x-snapd.must-exist-dir=<dir>for ensure-dir: the directory that must already exist
x-snapd.modemode of a created directory or file, default 0755
x-snapd.uid, x-snapd.gidowner of a created directory or file, default root; these are the names snap-update-ns reads
x-snapd.user, x-snapd.groupwritten by the layout code only for a non-root owner, which layouts do not accept yet

The user profile

The user profile is applied only when snap-confine builds a namespace (see the per-snap mount namespace). It takes no lock and freezes nothing. $XDG_RUNTIME_DIR in it becomes /run/user/<uid>, and $HOME (only in ensure-dir entries) becomes SNAP_REAL_HOME. The unrestricted path for the trespassing check is the real home directory, if it is valid.

Rules of the comparison

NeededChanges(current, desired) returns a list of keep, unmount and mount changes.

  • Every mount point is cleaned. The desired entries are sorted by origin (overname, then no origin, then layout) and by mount point.
  • A current entry is reused (kept) if it is the x-snapd.origin=rootfs entry; or it is synthetic and its x-snapd.needed-by names a desired entry, unless a different desired entry now targets the same directory; or it equals the desired entry for the same directory.
  • Once an entry is not reused, the entries below its directory are not reused either.
  • The current profile is walked backwards: reused entries become keep, the rest unmount. A tmpfs, bind or rbind entry is unmounted with x-snapd.detach.
  • Desired entries that were not reused become mount: first those whose mount point exists, then those that will need a mimic, grouped by their first existing parent, parents before children.

Passes and failures

PassWhatOn failure
1all keep and unmount changeslogged ("cannot change mount namespace according to change …"), skipped
2.1overname mounts, prepared and performedthe update stops
2.2entries without an origin, such as content: all prepared, then all performedlogged and skipped; ErrIgnoredMissingMount is silent
3layout mounts: all prepared, then all performedthe update stops
lastthe current profile is saved: kept entries in their order, then the mounts made, synthetic ones included

When snap-update-ns fails, snapd's mount backend discards the namespace with snap-discard-ns, so the next start builds it again. The code names the case that motivated this: a layout that changes from a bind mount to a symlink between revisions. If the snap has a daemon with refresh-mode: endure, the namespace is not discarded and the error is cannot update mount namespace of snap "<snap>", and cannot discard it because it contains an enduring daemon: <error>.

When a content slot provider is updated, the update of each consumer's namespace can be delayed: the backend records delayed-consumer-mount-ns-update and the task process-delayed-security-backend-effects applies it later, first trying to discard the namespace if no app of the consumer runs.

Note: when a layout or overname mount fails, the update returns before the current profile is saved, although pass 1 has already unmounted entries and passes 2 and 3 may have mounted some. For most snaps the namespace is then discarded. For a snap with an enduring daemon it is kept, and the current profile may no longer describe it (suspected snapd bug, plausible, not reproduced).

Mimic and trespassing details

A missing mount point is created with MkdirAll, MkfileAll or MksymlinkAll. A missing bind-mount source is created too, but a mimic for the source is allowed only for layout entries. If creation fails with ReadOnlyFsError or TrespassingError, a writable mimic is built over the reported directory and creation is retried once.

The mimic over a directory <dir>:

  • rbind <dir> to /tmp/.snap/<dir>;
  • mount a tmpfs on <dir> with the mode, uid and gid of the original;
  • for each entry of the original: rbind a directory, bind a file (x-snapd.kind=file), re-create a symlink; other file types are skipped with "skipping unsupported file …";
  • detach /tmp/.snap/<dir>.

Every entry carries x-snapd.synthetic and x-snapd.needed-by=<id>. If building the mimic fails, what was done is unmounted; if that fails too the error is fatal: cannot undo change "…" while recovering from earlier error …: …. The mimic exists only in the snap's own namespace.

Mount point type errors: cannot use "<path>" as mount point: not a directory, … not a regular file, cannot create symlink in "<path>": existing file in the way.

The trespassing check applies to paths outside the unrestricted ones. Each directory on the way is checked before something is created in it; writing is allowed on a read-only filesystem or squashfs, or in a tmpfs that a recorded change mounted exactly at that directory. The current profile is replayed as past changes first, so the root tmpfs and earlier mimics count. Otherwise the error is cannot write to "<path>" because it would affect the host in "<dir>", and the mimic goes over <dir>. For / the error is cannot recover from trespassing over /.

ProfileUnrestricted paths
system/tmp, /var/snap, /snap/<instance>, /dev/shm, /run/systemd, /var/lib/snapd/hostfs/tmp, and /snap/<snap> for a parallel instance
userthe real home directory, if valid

Created directories get mode 0755 unless a mode hint matches; for example /tmp/.X11-unix and /dev/shm/snap.* get 1777.

Examples

The snap photo-filters and its filters content slot are invented for the deck.

for snapd developers

Tests to read

  • tests/main/snap-update-ns: a content plug and slot; checks the namespace after connect and disconnect.
  • tests/main/interfaces-content-mimic: a mimic over $SNAP keeps files, directories and symlinks.
  • tests/main/layout-symlink-bind-revert: a layout that is a symlink in one revision and a bind mount in the next. It looks inside the namespace with snap debug mount-namespace --shell app -- /bin/ls -l /opt.
  • Also present: layout, layout-change, layout-remove, layout-content-delayed-effects, interfaces-content-mimic-missing-target, and the Go tests in cmd/snap-update-ns/.

Further reading

No published page explains snap-update-ns. These are the nearest.

Roads from here

INTERFACES AND CONFINEMENT

ADMINISTRATION · managing a system of snaps

DEVELOPING SNAPS · try, pack, run, debug

  • Layouts — layouts from the snap developer's side

SANDBOX INTERNALS · the mount namespace