Living with strict confinement

scratch space, paths and other limits for users for CI operators

What strict confinement changes for the snap. QEMU's -snapshot scratch cannot go to /var/tmp, so the snap keeps it on disk in ~/snap/image-garden/common/tmp, falling back to its private /tmp, which may be RAM, and allocate counts neither; the runtime directory moves; launchers and the virtiofsd app still run confined; docker comes through a content interface; and a sideloaded snap cannot fetch components.

On the map: snap

▶ Watch this deck — 8 scenes, ~6 min

Worth seeing first

01 Where do the guest's writes go

WHERE DO THE WRITES GO a VM with -m 2048 the test installs packages, builds something big 10 GB of guest writes snapshot mode: not in the prepared image somewhere on the host, and confinement has a say QEMU's usual place /var/tmp the snap's place, on disk ~/snap/image-garden/common/tmp the fallback /tmp: maybe memory
A VM allocated with 2 GB of memory runs a test that installs packages and builds; it writes 10 GB, which snapshot mode keeps out of the prepared image; the writes still need a place on the host, and confinement has a say: QEMU's usual /var/tmp, the snap's ~/snap/image-garden/common/tmp on disk, or the fallback /tmp, which may be memory

In snapshot mode the guest's writes go to a scratch file on the host. With the snap, it lives in ~/snap/image-garden/common/tmp, on disk, unless a fallback sends it to /tmp.

Say you allocate a machine with two gigabytes of memory, and your test installs a lot of packages or builds something big. The guest may write ten gigabytes. allocate runs it in snapshot mode, so none of that lands in your prepared image. It still has to go somewhere on the host, and with the snap, strict confinement has a say in where. This deck follows those writes: where QEMU normally keeps them, where the snap keeps them instead, on the disk that holds your home, and the one case where they can end up in memory. Then it looks at the other, smaller things confinement changes.

Details and sources

Details

  • allocate runs the .run script with ${QEMU_SNAPSHOT_OPTION=-snapshot}, so every allocated machine runs in snapshot mode unless that variable is set (image-garden:1153-1163).
  • The snap's wrapper says where the overlay goes: QEMU keeps it "in a file under $TMPDIR, or /var/tmp when TMPDIR is unset or /tmp"; the snap uses $SNAP_USER_COMMON/tmp, and /tmp/./ only when that cannot be made (qemu-var-tmp:6-20).
  • The README: "Scratch storage on disk, not in RAM", in ~/snap/image-garden/common/tmp (README.md:155-161).
  • The 2 GB and 10 GB on the slide are an example, not a measurement.

Read more

02 Snapshot mode writes somewhere else

SNAPSHOT MODE $ image-garden allocate … → ./SYSTEM.run -snapshot … prepared image: unchanged guest writes: packages, builds an overlay file grows with every write created in QEMU's temporary directory $TMPDIR, else /var/tmp TMPDIR unset or exactly /tmp → /var/tmp: disk
allocate runs the .run script with -snapshot; the prepared disk image stays unchanged; the guest's writes go into an overlay file that grows with every write, which QEMU creates in its temporary directory: the one TMPDIR names, or /var/tmp when TMPDIR is unset or /tmp; on a normal system that is disk

-snapshot keeps the image unchanged by writing the guest's changes to an overlay file in QEMU's temporary directory: $TMPDIR, or /var/tmp when that is unset or /tmp.

image-garden allocate starts every machine with -snapshot, unless you say otherwise. In that mode QEMU never writes to your prepared disk image, so the next run starts from it unchanged. But the guest still writes: every package it installs and every file it builds has to go somewhere. QEMU puts those writes in an overlay file of its own, which grows with every write. It creates that file in its temporary directory: the one TMPDIR names, or /var/tmp when TMPDIR is unset or is exactly /tmp. On a normal system, that's disk.

Details and sources

Details

  • allocate passes -snapshot unless QEMU_SNAPSHOT_OPTION is set, for example to the empty string (image-garden:1153-1163).
  • The snap's wrapper states the rule: "QEMU keeps the overlay of -snapshot, where everything a machine writes to its disk goes, in a file under $TMPDIR, or /var/tmp when TMPDIR is unset or /tmp. That file grows with the machine's writes, so it belongs on disk" (qemu-var-tmp:6-8).
  • Derived, from QEMU's behaviour rather than image-garden's tree: with -snapshot, QEMU opens each disk with a temporary qcow2 overlay of its own and deletes it when it exits. Writes go to that overlay; the image file is not modified.

03 The snap keeps it in your home

THE WRAPPER /var/tmp blocked by strict confinement qemu-var-tmp, in every QEMU app's chain if ! touch /var/tmp/image-garden-canary if mkdir -p "$SNAP_USER_COMMON"/tmp export TMPDIR="$SNAP_USER_COMMON"/tmp else export TMPDIR=/tmp/./ fi fi exec "$@" ~/snap/image-garden/common/tmp on the disk that holds your home every -snapshot overlay grows here gone when its machine stops
/var/tmp is blocked for the snap; the qemu-var-tmp wrapper in every QEMU app's command chain tries to touch a canary file there and, when that fails, makes $SNAP_USER_COMMON/tmp, that is ~/snap/image-garden/common/tmp, and points TMPDIR at it; that directory is on the disk that holds your home, and every -snapshot overlay grows there until its machine stops

qemu-var-tmp sets TMPDIR=$SNAP_USER_COMMON/tmp, that is ~/snap/image-garden/common/tmp, when /var/tmp is not writable. Every -snapshot overlay grows there, on disk.

Strict confinement doesn't let the snap write to /var/tmp. So every QEMU the snap runs, and image-garden itself, starts through a small wrapper in its command chain. The wrapper tries to create a canary file in /var/tmp. When that fails, it makes a directory called tmp in the snap's own directory in your home, tilde slash snap slash image-garden slash common, and points TMPDIR at it. That's on the same disk as your home. From then on, the snapshot overlay of every machine grows there, and it goes away when the machine stops.

Details and sources

Details

  • qemu-var-tmp touches /var/tmp/image-garden-canary. If that fails and SNAP_USER_COMMON is set and mkdir -p "$SNAP_USER_COMMON"/tmp succeeds, it exports TMPDIR="$SNAP_USER_COMMON"/tmp; otherwise it exports TMPDIR=/tmp/./. Then it execs the rest of the command chain (qemu-var-tmp:9-23). The slide's panel leaves out the then keywords and the error redirections.
  • The image-garden app and every qemu-system-* app list usr/bin/qemu-var-tmp in their command-chain (snapcraft.yaml:122-131, snapcraft.yaml:149-160); so does the spread app (snapcraft.yaml:211-226).
  • The README: the snap keeps the overlay "in ~/snap/image-garden/common/tmp instead of its private /tmp, which is a tmpfs on many hosts" (README.md:155-161).
  • Derived, from snapd: $SNAP_USER_COMMON is ~/snap/image-garden/common for the user who runs the app, so /root/snap/image-garden/common/tmp for root, as under sudo or spread. /var/tmp is not writable under strict confinement (confinement: strict, snapcraft.yaml:43), which is why the canary fails there.
  • Derived, from QEMU: the overlay is removed when QEMU exits, so the space is freed when the machine stops or is discarded.

04 The fallback, and why it matters

THE FALLBACK no ~/snap/…/tmp → TMPDIR=/tmp/./ host /tmp the snap's private /tmp tmpfs = memory -snapshot overlay files one VM 2 GB + 10 GB of writes = 12 GB four VMs with spread × 4: 48 GB of host memory $ ls -ld ~/snap/image-garden/common/tmp there and writable?
When $SNAP_USER_COMMON/tmp cannot be made, the wrapper sets TMPDIR to /tmp/./, the snap's private directory inside the host's /tmp; where that is a tmpfs it is memory, so the -snapshot overlays take RAM: a 2 GB VM that writes 10 GB holds 12 GB, four with spread hold 48 GB; that is why the snap uses your home; check that ~/snap/image-garden/common/tmp exists and is writable

If $SNAP_USER_COMMON/tmp cannot be made, TMPDIR=/tmp/./: the snap's private /tmp. On a tmpfs host the guest's writes then take RAM on top of -m.

There's a fallback. If the wrapper can't make that directory, it sets TMPDIR to /tmp/./ instead. That's the snap's private /tmp, spelled so that QEMU doesn't swap it for /var/tmp. It works, but on many hosts /tmp is a tmpfs, which lives in memory. Then every byte a guest writes lands in host memory. A machine with two gigabytes that writes ten holds twelve on the host, and four of them with spread hold forty-eight. That's exactly why the snap keeps the overlay in your home. If host memory runs out during write-heavy tests, check whether that directory exists and can be written.

Details and sources

Details

  • The fallback: when SNAP_USER_COMMON is empty or mkdir -p "$SNAP_USER_COMMON"/tmp fails, the wrapper exports TMPDIR=/tmp/./; its comment says the path "is still /tmp but does not compare to /tmp exactly, which QEMU would replace with /var/tmp" (qemu-var-tmp:13-19). Nothing is printed when it falls back.
  • The README says the snap avoids its private /tmp because it "is a tmpfs on many hosts", and that a scratch directory on tmpfs "would use RAM" allocate's memory check "does not see" (README.md:155-161).
  • Derived, from snapd rather than image-garden's tree: a strictly confined snap gets a private /tmp, a directory inside the host's /tmp. Where that is a tmpfs, files in it take RAM (and swap) until they are deleted. Derived as well: snapd always sets SNAP_USER_COMMON for an app, so the fallback needs a home directory where the directory cannot be made, for example one that is not writable.
  • findmnt /tmp (util-linux) shows whether /tmp is a tmpfs.
  • The 12 GB and 48 GB are arithmetic on the example, not measurements.

05 What allocate does not count

WHAT ALLOCATE COUNTS memory check: -m only the guest's writes: not counted scratch on disk: four VMs × 10 GB of writes 40 GB free under your home while they run $ df -h ~/snap/image-garden/common under sudo, or spread as root: /root/snap/image-garden/common/tmp QEMU_SNAPSHOT_OPTION="": no scratch, the guest writes into the image; use a copy
allocate's memory check compares only the machine's -m with free memory, not what it writes; with the scratch on disk, four VMs writing 10 GB each need 40 GB free under your home while they run; df shows the free space; under sudo or spread as root the scratch is in /root/snap/image-garden/common/tmp; QEMU_SNAPSHOT_OPTION set empty writes into the image instead, so use a throwaway copy

The memory check counts only -m. Four VMs that write 10 GB each need 40 GB free in ~/snap/image-garden/common/tmp (root's home under sudo). QEMU_SNAPSHOT_OPTION="" writes into the image: use a copy.

One thing stays the same wherever the overlay lives. allocate's memory check counts only the machine's own memory, its dash m, and nothing it writes. With the scratch on disk, a write-heavy test costs disk space instead: four machines that each write ten gigabytes need forty gigabytes free under your home, for as long as they run. So check with df before a big run. Under sudo, or when spread runs as root, that's root's home, not yours. And to keep the writes out of scratch altogether, set QEMU_SNAPSHOT_OPTION to empty, but then the guest writes into the prepared image, so do that on a copy you throw away.

Details and sources

Details

  • allocate's memory check reads vm.memory_mb from .run --query and compares it with MemAvailable less a reserve (GARDEN_MEMORY_RESERVE), divided by 1.5; it knows nothing of disk writes (image-garden:711-782). The README: "allocate's memory check counts only the machine's memory (-m)" (README.md:155-161).
  • Nothing in image-garden checks free disk space for the overlays either. Derived: the overlays are files in $SNAP_USER_COMMON/tmp, so they take space on that file system until QEMU exits; df shows it even when the files have no name in the directory.
  • $SNAP_USER_COMMON belongs to the user who runs the app: /root/snap/image-garden/common for root (derived, snapd's layout).
  • Snapshot mode is chosen by QEMU_SNAPSHOT_OPTION; set to the empty string, allocate runs the machine without -snapshot and the guest writes into the system's .qcow2 (image-garden:1153-1163). Copy the .qcow2 (and, for UEFI systems, the .efi-vars.qcow2) first, or rebuild with make afterwards.
  • The container and a native install run QEMU without the wrapper, so QEMU's own default, /var/tmp, holds the overlays there unless TMPDIR says otherwise (README.md:96-106 compares the installs).
  • The 40 GB is arithmetic on the example, not a measurement.

06 Paths that move

PATHS THAT MOVE the home interface ~/project/.image-garden/ reachable /srv/project/ not reachable the runtime directory: pid files and locks /run/user/0/snap.image-garden root under sudo or spread: /run/user/0 does not exist /run/snap.image-garden/image-garden/ where image-garden ps looks then
Through the home interface the snap reaches projects under the home directory, not elsewhere; when /run/user/UID does not exist, as for root under sudo or spread, pid files and locks go to /run/snap.image-garden/image-garden instead, where image-garden ps looks

Plugs include home, not removable media. Without /run/user/UID, pid files and locks go to /run/snap.image-garden/image-garden/.

Two smaller things move too. The snap reaches your files through the home interface, so keep projects and their .image-garden directories under your home directory. And the runtime directory, where allocate keeps pid files and locks, normally sits under /run/user. snapd points the snap at it even when it doesn't exist, as for root under sudo or spread, and the snap can't create it. Then image-garden uses /run/snap.image-garden, the snap's own place in /run. Look there when image-garden ps finds nothing.

Details and sources

Details

  • The snap's apps plug home, kvm, network, network-bind and pkgproxy-sockets (snapcraft.yaml:122-131); there is no removable-media plug, so paths outside the home directory, such as /srv, are not reachable. Derived from the plug list.
  • snapd sets XDG_RUNTIME_DIR to /run/user/UID/snap.NAME even where /run/user/UID does not exist, as for root under sudo or spread. The snap cannot create it, so image-garden uses /run/snap.$SNAP_INSTANCE_NAME (image-garden:64-73). allocate's pid files, locks and TPM run directories go under $XDG_RUNTIME_DIR/image-garden/.
  • The cache directory is $SNAP_USER_COMMON/cache in the snap (image-garden:20-29).

07 Reaching out of the sandbox

THROUGH THE SANDBOX $ ./SYSTEM.run (host shell) image-garden.qemu-system-x86-64 image-garden.swtpm (+tpm systems) still confined: scratch in ~/snap/image-garden/common/tmp confined, so its own sandbox is off image-garden.virtiofsd --sandbox none spread, docker backend docker snap, via docker-executables snap install ./image-garden.snap no components from the store install them from files, or the snap from the store
A .run script made by the snap and run from a host shell calls image-garden.qemu-system-x86-64, and a TPM system's calls image-garden.swtpm, so QEMU still runs confined with its scratch in ~/snap/image-garden/common/tmp; the image-garden.virtiofsd app runs confined too, with --sandbox none; spread's docker backend reaches docker through the docker-executables content interface; a snap installed from a local file cannot fetch components

Launchers call image-garden.qemu-system-* and image-garden.swtpm; image-garden.virtiofsd runs with --sandbox none. docker comes from the docker snap through docker-executables. A sideloaded snap installs no components.

Some things cross the boundary on purpose. A .run script made by the snap names the snap's own QEMU app, and a TPM system's script names the snap's swtpm app. So when you run it from an ordinary shell, it still runs confined, and its scratch still goes to the snap's directory in your home. The virtiofsd app runs confined too, which is why it turns off virtiofsd's own sandbox. spread's docker backend reaches docker through the docker snap's executables, connected with a content interface. And a snap you install from a local file can't fetch components: install them from files too, or install the snap from the store.

Details and sources

Details

  • A launcher made inside the snap names image-garden.qemu-system-x86-64 when run from outside the snap, and qemu-system-x86_64 inside it (010-qemu-x86_64.mk:4); the same holds for swtpm and image-garden.swtpm (010-qemu-x86_64.mk:115-119). The QEMU app starts through qemu-var-tmp too, so its overlay goes to $SNAP_USER_COMMON/tmp as well (snapcraft.yaml:149-160).
  • The virtiofsd app plugs home and network-bind (snapcraft.yaml:140-148). Its stub adds --sandbox none unless the caller passes a --sandbox option: virtiofsd's default sandbox "enters new namespaces, which strict confinement does not allow", and "the snap's confinement still limits what virtiofsd can reach" (virtiofsd:6-15). See sharing host directories.
  • Only the spread app plugs docker and docker-executables (snapcraft.yaml:211-226). The content plug mounts the docker snap's executables at $SNAP/docker-snap (snapcraft.yaml:267-270); the docker stub prints how to connect them when they are missing (docker:6-35).
  • The install hook installs components only when the revision is not a local one (x1, x2…) (install:6); the configure hook does the same (configure:47-53).

08 Where to go from here

ROADS Confinement The snap what it carries and installs allocate every option, -snapshot too Three ways in container and native: /var/tmp virtiofs and 9p share host data, don't copy
Four roads from Living with strict confinement: the snap overview, allocate and snapshot mode, the three ways to install, and sharing host directories with virtiofs

Next: the snap overview, allocate and its snapshot mode, the three ways to install, and sharing host directories with virtiofs.

So that's strict confinement: snapshot overlays on disk in the snap's directory in your home, a fallback to /tmp, which may be memory, a runtime directory that moves, and a few deliberate holes in the sandbox. To see the rest of the snap, go back to its overview. allocate's own deck shows every option it passes to QEMU, including snapshot mode. To compare the container and native installs, where QEMU uses /var/tmp, see the three ways in. And if your guests need large host data sets, the virtiofs deck shows how to share them without copying.

Details and sources

Details

  • The deck in one line: confinement blocks /var/tmp; the wrapper sends QEMU's -snapshot overlays to ~/snap/image-garden/common/tmp, on disk, or to the snap's /tmp, which may be memory, when that directory cannot be made; allocate counts neither.

Read more

Basis of the statements on this page

Read in the snap's snapcraft.yaml, hooks and stubs, the README and the image-garden script at the commit this map describes ("code"). Statements about QEMU's -snapshot overlay, snapd's private /tmp, $SNAP_USER_COMMON and tmpfs are general behaviour, not image-garden's code ("derived").

Confinement limits, their symptoms and what to do

LimitWhat you noticeWhat image-garden doesWhat you can do
/var/tmp is not writableQEMU's -snapshot overlays cannot go to /var/tmpqemu-var-tmp sets TMPDIR=$SNAP_USER_COMMON/tmp, that is ~/snap/image-garden/common/tmp, on disk (qemu-var-tmp:9-23, README.md:155-161)keep enough free space under your home (root's home under sudo or spread)
$SNAP_USER_COMMON/tmp cannot be madethe overlays go to the snap's private /tmp; on a tmpfs host, write-heavy tests use host RAM far beyond -mTMPDIR=/tmp/./, spelled so QEMU does not replace it (qemu-var-tmp:13-19)make sure that directory exists and is writable; findmnt /tmp shows a tmpfs
allocate's memory checkthe overlays are not counted, wherever they arecounts only -m (image-garden:711-782)budget disk (or RAM) for what tests write; fewer parallel VMs; QEMU_SNAPSHOT_OPTION="" on a copy; or the container or a native install
only homeprojects outside your home directory are not reachableplugs home, kvm, network, network-bind, pkgproxy-sockets (snapcraft.yaml:122-131)keep projects under $HOME
no /run/user/UID for root without a sessionpid files would have nowhere to goXDG_RUNTIME_DIR=/run/snap.image-garden (image-garden:64-73)look in /run/snap.image-garden/image-garden/
launchers run from a host shellQEMU is still the snap'slaunchers name image-garden.qemu-system-* and image-garden.swtpm (010-qemu-x86_64.mk:4, 010-qemu-x86_64.mk:115-119)expect the same scratch directory
no docker inside the sandboxspread's docker backend fails with a noticedocker stub, docker-executables content plug (docker:6-35)snap connect image-garden:docker-executables docker:docker-executables and snap connect image-garden:docker docker:docker
a sideloaded snap has no storecomponents are not installedhooks skip x revisions (install:6)install the .comp files too, or the snap from the store
virtiofsd's own sandboxits namespace sandbox is not allowedthe virtiofsd app's stub adds --sandbox none (virtiofsd:6-15); plugs home, network-bind (snapcraft.yaml:140-148)share directories under your home
pkgproxy snap's socketspkgproxy=snap needs themcontent plug pkgproxy-sockets (snapcraft.yaml:255-266)snap connect image-garden:pkgproxy-sockets pkgproxy:sockets

Roads from here

START HERE

RUNTIME · make rules and the CLI

SNAP · snap-only features