The per-snap mount namespace

sandbox internals for snapd developers for administrators

snap-confine builds one mount namespace per snap on the first run with the base snap as the root and keeps it as a file under /run/snapd/ns. Later runs join it. A namespace that no longer matches the base is rebuilt once no process of the snap is left in it, and a refresh or removal discards it.

On the map: sandbox internals district

▶ Watch this deck — 8 scenes, ~6 min

01 One view of the filesystem per snap

ONE VIEW PER SNAP $ photo-editorthe laptopthe system's own rootyour homeremovable mediatemporary files photo-editorroot: the base snapits own private temporary files your homehost mounts show up inside mounts made inside stay inside removable mediashared both ways built on the first run · kept · shared by every app and hook · classic snaps: none
The laptop and photo-editor side by side: photo-editor's view has the base snap as its root, sees your home, receives mounts from the host but keeps its own mounts inside, and shares removable media both ways; the view is built on the first run and shared by every app and hook

Each strictly confined snap gets one mount namespace, with its base snap as the root. Mounts on the host propagate into it, mounts made inside do not propagate out, and removable media is shared both ways. It is built on the first run and shared by every app and hook of the snap. Classic snaps do not get one.

You list a system directory in a terminal, then open the same directory in photo editor's file dialog, and the two don't agree. Nothing is broken. Photo editor has its own view of the filesystem. Its root is the base snap, so it finds the same libraries on every distribution. Your home is there as well. When something is mounted on the laptop, it shows up inside, but a mount made inside stays inside. removable-media is the exception: it's shared both ways. The view is built the first time photo editor runs, then kept, and every app and hook of the snap shares it. Classic snaps don't get one.

02 The first run builds it for app developers for Ubuntu Core developers for snapd developers for administrators

FIRST RUN 1 · lock/run/snapd/lock/photo-editor.lockhelper processstays outside, waits 2 · unshareunshare(CLONE_NEWNS)/tmp/snap-private-tmp/snap.rootfs_XXXXXXa tmpfs, to become the new root/usr/var/opt/srvfrom core24 3 · bind host directories/dev /etc /home /proc /sys /run …/dev/etc/home/proc/sys/run/var/snap/var/lib/snapd/mediaboth ways 4 · pivot_rootthe scratch directory becomes /now //var/lib/snapd/hostfsthe old root 5 · finishprivate /tmp, devpts, the mount profileprivate /tmpnew devptssnap-update-ns --from-snap-confine photo-editorapplies the mount profile: interface mounts and layouts
First run of photo-editor: snap-confine takes the snap lock and forks a helper, unshares a mount namespace, mounts a tmpfs root and binds the top-level directories of core24 onto it, binds host directories with /media shared both ways, pivots into the new root leaving the old one at /var/lib/snapd/hostfs, then sets up a private /tmp, a new devpts and runs snap-update-ns

Lock, helper, unshare(CLONE_NEWNS), a tmpfs root carrying the base's top-level entries, bind mounts from the host, pivot_root with the old root at /var/lib/snapd/hostfs, a private /tmp, a new devpts, then snap-update-ns --from-snap-confine.

It's the first time photo editor runs since boot, so there's nothing to join yet. snap-confine builds the view. It takes photo editor's lock, so two starts can't build at once, and forks a helper that stays outside. Then it unshares. An empty in-memory filesystem becomes the new root, and core24's top-level directories are bound onto it. The host's devices, configuration, homes, proc, sys and run follow, with media shared both ways. pivot_root swaps the roots and leaves the old one at /var/lib/snapd/hostfs. Last come a private tmp, a fresh devpts, and one run of snap-update-ns for interface mounts and layouts.

03 Kept as a file for Ubuntu Core developers for snapd developers for administrators

PRESERVED snap-confineinside the new namespace/proc/<pid>/ns/mnt capture, over a pipehelperoutside, where /run/snapd/ns is visiblebind mount/run/snapd/ns/photo-editor.mnt $ ls /run/snapd/nsphoto-editor.mntsnap.photo-editor.fstabsnap.photo-editor.info what is mounted now$ cat /run/snapd/ns/snap.photo-editor.infobase-snap-name=core24 $ snap debug mount-namespace photo-editormount namespace of snap "photo-editor" bound to/run/snapd/ns/photo-editor.mnt, use --shell to enter $ sudo snap debug mount-namespace --shell photo-editora shell that sees what photo-editor sees
snap-confine, inside the new namespace, asks the helper outside to bind-mount the namespace file onto /run/snapd/ns/photo-editor.mnt; the directory also holds the .info file with base-snap-name=core24 and the .fstab file; snap debug mount-namespace reports the file and --shell enters it

<instance>.mnt is a bind mount of the namespace's nsfs file, made by the helper process. snap.<instance>.info records base-snap-name and snap.<instance>.fstab is the current mount profile. snap debug mount-namespace <snap> reports it and --shell enters it.

A namespace normally vanishes when its last process exits. So how does photo editor's survive you closing the window? snap-confine pins it. Every namespace has a file under proc, and bind-mounting that file somewhere keeps the namespace alive. The mount has to be made from outside, where the /run/snapd/ns directory is visible, and that's the helper's job. The result is photo-editor.mnt. Next to it sit an info file with the base's name, and the current mount profile. Snap debug mount-namespace tells you whether one exists, and with the shell option it gives you a shell that sees what photo editor sees.

04 Later runs join for Ubuntu Core developers for snapd developers

JOIN /run/snapd/ns/photo-editor.mntsnap-confineopen, check it is nsfs forkchild processsetns, look inside votesSC_DISCARD_NOfits: reuseSC_DISCARD_SHOULDstale: discard if nobody is insideSC_DISCARD_MUSTbase name changed: discard setns(fd, CLONE_NEWNS)joined preserved mount namespace photo-editor no mounts are redone here; snapd keeps the namespace current with snap-update-ns
snap-confine opens /run/snapd/ns/photo-editor.mnt and checks it is nsfs; a forked child joins it and votes SC_DISCARD_NO, SC_DISCARD_SHOULD or SC_DISCARD_MUST; on reuse snap-confine joins with setns and mounts nothing

snap-confine opens <instance>.mnt and checks it is nsfs. A child joins and votes SC_DISCARD_NO, SC_DISCARD_SHOULD or SC_DISCARD_MUST, then the parent calls setns. Later changes are applied by snapd through snap-update-ns.

You open a second photo editor window. snap-confine finds the file from last time, and the question becomes: is what's in there still good? It opens photo-editor.mnt and checks it's really a namespace. Then it forks a child to go in and look, because the kernel misbehaves if one process calls setns twice. The child votes. It fits, so reuse it. It's stale, so discard it if nobody's inside. Or the snap moved to another base: discard it regardless. On reuse, snap-confine joins with a single setns. Nothing is mounted on this path, which is why a later connection needs snap-update-ns.

05 Stale, but someone is still inside

AFTER A BASE REFRESH core24 revision 1100view: core24 revision 1100photo-editor window1 · while it runs $ snap refresh core24core24 revision 1150 new photo-editor window2 · after the refresh stale but occupied: reused all windows closed view: core24 revision 1150next startrebuilt3 · after the last exit
core24 refreshes from revision 1100 to 1150 while photo-editor is open; a new photo-editor window joins the old view, which is stale but occupied; after every window is closed the next start gets a view rebuilt on revision 1150

A view whose root is not the current revision of the base is stale. It is reused while any process of the snap runs, and rebuilt on the first start after the last one exits. The revision numbers are invented for this deck.

core24 just refreshed, and photo editor is still open on your screen. Which core24 does a new photo editor window see? The old one. Photo editor's view was built on revision 1100, and the window that's still open lives in it. snap-confine notices the view is stale, sees that photo editor is still running, and lets the new window join. Two halves of one app looking at two different bases would be worse. Once every photo editor process exits, the next start throws the old view away and builds a fresh one on 1150. So if a base refresh seems to change nothing, check what's still running.

06 A second layer for each user for app developers for Ubuntu Core developers for snapd developers for administrators

PER USER per-snap namespace (kept)joined or built, as before unshare(CLONE_NEWNS) againonly when the real uid is not 0per-user namespace, uid 1000not kept /var/lib/snapd/mount/snap.photo-editor.user-fstab$XDG_RUNTIME_DIR/doc/by-app/snap.photo-editor→ $XDG_RUNTIME_DIR/docadded by the desktop interface snap-update-ns --from-snap-confine --user-mounts photo-editor$XDG_RUNTIME_DIR=/run/user/1000 NOT preserving per-user mount namespacebuilt again on every start
On top of the kept per-snap namespace, snap-confine unshares a per-user namespace for uid 1000 when the real uid is not 0; snap-update-ns applies the user mount profile from the desktop interface with $XDG_RUNTIME_DIR set to /run/user/1000; the per-user namespace is built again on every start

For a non-root user snap-confine unshares a second time and runs snap-update-ns --user-mounts if snap.<instance>.user-fstab exists. The per-user namespace is not preserved. The entry shown is the one the desktop interface adds.

Everything so far is shared by every user of the laptop. But some mounts belong to one person. The desktop interface, for example, wants the document portal's folder under your own runtime directory. So when anyone but root starts photo editor, snap-confine unshares once more, on top of the shared view. If the snap has a user mount profile, snap-update-ns applies it, with $XDG_RUNTIME_DIR filled in as /run/user/1000. If there's no profile, the step returns at once. This layer isn't kept. Every start by a user builds it again on top of the shared one.

07 Who throws it away for app developers for Ubuntu Core developers for snapd developers for administrators

DISCARD $ snap refresh photo-editorno app running, snap lock held snap-discard-ns photo-editorphoto-editor.mntsnap.photo-editor.fstabsnap.photo-editor.info $ snap remove photo-editor snap-confinestale and nobody inside $ sudo snap debug mount-namespace --discard photo-editorby hand not at refresh if a service of the snap has refresh-mode: enduredo not count on a mount surviving a refresh
snap-discard-ns photo-editor removes photo-editor.mnt, snap.photo-editor.fstab and snap.photo-editor.info; it is run by snap refresh, snap remove, snap-confine for a stale unused namespace and by hand with snap debug mount-namespace --discard; not at refresh if a service uses refresh-mode endure

snap-discard-ns <instance> unmounts and removes <instance>.mnt and the .fstab and .info files. snapd runs it on a refresh that waited for the apps to close (not with --ignore-running, and not when a service uses refresh-mode: endure), on removal and when an update of the namespace fails; snap-confine runs it for a stale, unoccupied namespace.

You refresh photo editor to 2.5. Its view was built for revision 42, so what happens to it? Snapd discards it while the snap is locked and no app is running. The tool is snap-discard-ns. It unmounts the namespace file and deletes it, with the info file and the current profile, and the next start builds a fresh view. Removing the snap does the same. snap-confine calls it for a stale view nobody's using, and you can run it by hand. One exception: a service with refresh-mode: endure keeps running across a refresh, so its view stays. Don't count on a mount surviving a refresh.

08 Where to go from here

ROADS The per-snapmount namespacesnap-update-nschanges the kept view in place The system keydo the profiles still fit? Hotplug slotsslots for devices plugged in Confinementthe rest of the sandboxsnap debugmore tools for looking inside Serviceskept running across a refreshParallel installsthe instance mapping insideHow interfaces are implementedthe mount backend
Roads from the per-snap mount namespace: snap-update-ns, the system key, hotplug slots, how interfaces are implemented, confinement, snap debug, services and parallel installs

Roads: snap-update-ns, the system key, hotplug slots, confinement, snap debug, services, parallel installs and the security backends.

So that's the view: built once, kept as a file, joined by everything that follows. When a connection or a layout changes, snap-update-ns edits the kept view in place; that's next door. The system key answers a different question: do the profiles on disk still fit this system? Hotplug covers slots that appear when you plug a device in. Confinement shows the rest of what snap-confine sets up, and snap debug lists the other tools for looking inside. Services explains services that keep running across a refresh.

Read more

for administrators for snapd developers for Ubuntu Core developers

Files

<instance> is the snap instance name, such as photo-editor or photo-editor_beta.

PathWritten byContent
/run/snapd/ns/snap-confinea directory, bind-mounted over itself and made private (MS_PRIVATE)
/run/snapd/ns/<instance>.mntthe snap-confine helper processa bind mount of the nsfs file of the snap's mount namespace; mode 0600
/run/snapd/ns/<instance>.<uid>.mntnothing at presentthe per-user namespace file; the code to capture it exists but is not called
/run/snapd/ns/snap.<instance>.infosnap-confinebase-snap-name=<name>, and managed-ca-certs-generation=<id> when managed CA certificates are mounted
/run/snapd/ns/snap.<instance>.fstabsnap-confine, then snap-update-nsthe current mount profile; its first line, written by snap-confine, is tmpfs / tmpfs x-snapd.origin=rootfs 0 0
/run/snapd/ns/snap.<instance>.log and snap.<instance>.user.<uid>.logyou, by handif present, snap-update-ns appends debug logs there. Nothing in snapd creates or removes them
/run/snapd/lock/<instance>.lock and /run/snapd/lock/.locksnap-confinethe per-snap and the global flock files
/var/lib/snapd/mount/snap.<instance>.fstabsnapd's mount backendthe desired mount profile
/var/lib/snapd/mount/snap.<instance>.user-fstabsnapd's mount backendthe desired per-user mount profile
/tmp/snap-private-tmp/snap.<instance>/tmpsnap-confinebecomes /tmp inside the namespace
/tmp/snap-private-tmp/snap.rootfs_XXXXXXsnap-confinethe scratch directory for the new root; removed after pivot_root

What snap-confine bind-mounts from the host

  • Always: /dev, /etc, /home, /root, /proc, /sys, /tmp, /var/snap, /var/lib/snapd, /var/tmp, /run, /usr/src, /var/log.
  • When present: /lib/modules, /lib/firmware, /mnt, /var/lib/extrausers.
  • Shared both ways: /media (on systems built with a merged /usr, /run/media with /media as the alternative) and /run/netns.
  • Home directories configured outside /home, added on every build of the namespace.
  • The old root stays reachable at /var/lib/snapd/hostfs. Its sys, dev and proc are detached, and on Ubuntu Core so is /var/lib/snapd/hostfs/writable.

The other top-level directories come from the base snap: directories are bind-mounted onto the new tmpfs root, symbolic links are recreated and regular files are bind-mounted. /snap and the directories in the lists above are skipped.

When a namespace is stale

snap-confine treats a kept namespace as stale when any of these holds:

  • the current revision of the base snap is not what is mounted at /, /usr or under /usr/ in the namespace. This is what a base refresh causes;
  • a configured home directory is not a mount point in the namespace;
  • the managed CA certificate generation differs from the one in the .info file, or that mount is missing. The mount is only made on Ubuntu Core-like distributions whose base has /etc/ssl/certs.

A stale namespace that a process of the snap still uses is reused, with the debug message "preserved mount namespace is stale but occupied, reusing". Use is read from the freezer cgroup on cgroup v1 and from the tracking cgroups on cgroup v2.

A change of base, for example from core22 to core24, is read from base-snap-name in the .info file. It discards the namespace even when it is in use. When the file or the key is absent no change is detected.

Stale detection is off in legacy mode (Ubuntu Core 16 with the base core); the base change check is not.

Who discards

CallerWhenMessage
snap-confinestale and not in use, or the base name changed
the unlink-current-snap taskon refresh, while the snap lock is held and no app runs; skipped when a service has refresh-mode: endure; also skipped with snap refresh --ignore-running and for snap types that refresh without waiting for apps to closenot discarding snap mount namespace: service "<name>" uses endure refresh-mode in the task log
the discard-snap taskwhen the last revision is removedcannot discard snap namespace "<snap>", will retry in 3 mins: ...
undo of link-snapwhen a first install fails
a snapd downgrade below 2.57for every snapDowngrading snapd to version "<version>", discarding preserved namespaces
the mount backend, on removal of a snap's profilesafter removal
the mount backend, after a failed updatewhen snap-update-ns fails, unless a daemon uses refresh-mode: endurecannot update mount namespace of snap "<snap>", and cannot discard it because it contains an enduring daemon: ...
the mount backend, delayedfor a snap that consumes an updated slot provider, if none of its apps runs
snap debug mount-namespace --discard <snap>by handcannot discard mount namespace of snap "<snap>": ...

snapd runs the tool only if /run/snapd/ns/<instance>.mnt exists. Its own error is cannot discard preserved mount namespace of snap "<snap>": ....

Classic snaps and parallel instances

Classic snaps do not get a per-snap mount namespace. The snap documentation says the same. There is one exception in the code: with the parallel-instances feature on and an instance key, snap-confine unshares a mount namespace for a classic snap and bind-mounts the <snap>_<key> directories over <snap>. That namespace is not kept. snapd's code calls these parallel instances; the documentation calls them parallel installs.

Commands

CommandEffect
snap debug mount-namespace <snap>prints mount namespace of snap "<snap>" bound to /run/snapd/ns/<snap>.mnt, use --shell to enter or no mount namespace of snap "<snap>" found
snap debug mount-namespace --shell <snap> [-- <command>]runs nsenter -m on the file with /bin/bash or the given command. The help says the command may need root
snap debug mount-namespace --discard <snap>runs snap-discard-ns. --shell and --discard cannot be used together
snap-discard-ns [--snap-already-locked or --from-snap-confine] <instance>an internal tool; needs CAP_SYS_ADMIN, CAP_DAC_OVERRIDE and CAP_CHOWN. With either flag it checks that the snap lock is already held; without, it takes the lock. It unmounts and removes <instance>.mnt, <instance>.*.mnt, snap.<instance>.fstab, snap.<instance>.*.user-fstab and snap.<instance>.info, and exits 0 if /run/snapd/ns does not exist

for snapd developers for administrators

Messages

From snap-confine, each ending the run:

  • cannot create directory %s, cannot self-bind mount %s
  • cannot open preserved mount namespace %s, cannot join preserved mount namespace %s
  • support process for mount namespace inspection exited abnormally
  • cannot preserve mount namespace of process %d as %s, helper process exited abnormally
  • cannot unshare the mount namespace
  • cannot perform operation: pivot_root %s %s
  • %s failed with code %i, when snap-update-ns or snap-discard-ns fails
  • cannot find mount entry for snap %s revision %s
  • sanity timeout expired, when a lock is not acquired within 30 seconds
  • snap is currently being removed

From snap-discard-ns: missing capability %s.

From snapd: cannot discard preserved mount namespace of snap %q: %s.

for snapd developers

Details for snapd developers

  • snap-confine first moves into the mount namespace of PID 1 if it is not there already, so that /run/snapd/ns shows the kept namespaces even when it is started from inside a snap.
  • Legacy mode is SC_DISTRO_CORE16 with the base core. It does not pivot into a base, bind-mounts / recursively instead of building a tmpfs root, and skips stale detection.
  • On kernel 6.18 only, after unshare(CLONE_NEWNS) snap-confine checks with the NS_GET_ID ioctl that the new namespace id is greater than the helper's, and unshares again on other CPUs until it is.
  • pivot_root is used rather than chroot so that AppArmor sees / as the root in every case.
  • The child that inspects a kept namespace reports through an eventfd: 1 for SC_DISCARD_NO, 2 for SC_DISCARD_SHOULD, 3 for SC_DISCARD_MUST.
  • sc_preserve_populated_per_user_mount_ns() exists but snap-confine does not call it, so <instance>.<uid>.mnt is not created.
  • The mount backend advertises the per-snap-user-profiles sandbox feature.
  • The README next to the code (cmd/snap-confine/README.mount_namespace) describes older paths for the scratch directory and the private /tmp; the code is as described here.
  • Spread tests: tests/main/mount-ns (mountinfo tables for core16 and core18 snaps, the per-user namespace and a classic snap), tests/main/snap-discard-ns, tests/main/stale-base-snap (a running process keeps the old base visible to new ones), tests/main/user-mounts (the desktop user-fstab entry).

Further reading

  • Security policies: the launch steps as the snap documentation lists them, including the private mount namespace.
  • Security and sandboxing: the same steps for Ubuntu Core.
  • Classic confinement: what is skipped for classic snaps. It says classic snaps have no mount namespace; see the section on parallel instances above for the one exception in the code.
  • Parallel installs: the path mapping done inside the namespace for an instance.

No published page explains how the namespace is kept between runs or what snap-discard-ns does.

Roads from here

INTERFACES AND CONFINEMENT

RUNNING SNAPS

  • Services — services and what refresh-mode endure means for them

TOOLS

ADMINISTRATION · managing a system of snaps

SANDBOX INTERNALS · the mount namespace