Living with strict confinement
scratch space, paths and other limits for users for CI operators
What strict confinement changes for the snap. QEMU's -snapshot scratch cannot go to /var/tmp, so the snap keeps it on disk in ~/snap/image-garden/common/tmp, falling back to its private /tmp, which may be RAM, and allocate counts neither; the runtime directory moves; launchers and the virtiofsd app still run confined; docker comes through a content interface; and a sideloaded snap cannot fetch components.
On the map: snap
▶ Watch this deck — 8 scenes, ~6 min
Worth seeing first
- The image-garden snap — the snap, its components and the short version of confinement
01 Where do the guest's writes go
In snapshot mode the guest's writes go to a scratch file on the host. With the snap, it lives in ~/snap/image-garden/common/tmp, on disk, unless a fallback sends it to /tmp.
Say you allocate a machine with two gigabytes of memory, and your test installs a lot of packages or builds something big. The guest may write ten gigabytes. allocate runs it in snapshot mode, so none of that lands in your prepared image. It still has to go somewhere on the host, and with the snap, strict confinement has a say in where. This deck follows those writes: where QEMU normally keeps them, where the snap keeps them instead, on the disk that holds your home, and the one case where they can end up in memory. Then it looks at the other, smaller things confinement changes.
Details and sources
Details
- allocate runs the
.runscript with${QEMU_SNAPSHOT_OPTION=-snapshot}, so every allocated machine runs in snapshot mode unless that variable is set (image-garden:1153-1163). - The snap's wrapper says where the overlay goes: QEMU keeps it "in a file under $TMPDIR, or /var/tmp when TMPDIR is unset or /tmp"; the snap uses
$SNAP_USER_COMMON/tmp, and/tmp/./only when that cannot be made (qemu-var-tmp:6-20). - The README: "Scratch storage on disk, not in RAM", in
~/snap/image-garden/common/tmp(README.md:155-161). - The 2 GB and 10 GB on the slide are an example, not a measurement.
02 Snapshot mode writes somewhere else
-snapshot keeps the image unchanged by writing the guest's changes to an overlay file in QEMU's temporary directory: $TMPDIR, or /var/tmp when that is unset or /tmp.
image-garden allocate starts every machine with -snapshot, unless you say otherwise. In that mode QEMU never writes to your prepared disk image, so the next run starts from it unchanged. But the guest still writes: every package it installs and every file it builds has to go somewhere. QEMU puts those writes in an overlay file of its own, which grows with every write. It creates that file in its temporary directory: the one TMPDIR names, or /var/tmp when TMPDIR is unset or is exactly /tmp. On a normal system, that's disk.
Details and sources
Details
- allocate passes
-snapshotunlessQEMU_SNAPSHOT_OPTIONis set, for example to the empty string (image-garden:1153-1163). - The snap's wrapper states the rule: "QEMU keeps the overlay of -snapshot, where everything a machine writes to its disk goes, in a file under $TMPDIR, or /var/tmp when TMPDIR is unset or /tmp. That file grows with the machine's writes, so it belongs on disk" (qemu-var-tmp:6-8).
- Derived, from QEMU's behaviour rather than image-garden's tree: with
-snapshot, QEMU opens each disk with a temporary qcow2 overlay of its own and deletes it when it exits. Writes go to that overlay; the image file is not modified.
Read more
03 The snap keeps it in your home
qemu-var-tmp sets TMPDIR=$SNAP_USER_COMMON/tmp, that is ~/snap/image-garden/common/tmp, when /var/tmp is not writable. Every -snapshot overlay grows there, on disk.
Strict confinement doesn't let the snap write to /var/tmp. So every QEMU the snap runs, and image-garden itself, starts through a small wrapper in its command chain. The wrapper tries to create a canary file in /var/tmp. When that fails, it makes a directory called tmp in the snap's own directory in your home, tilde slash snap slash image-garden slash common, and points TMPDIR at it. That's on the same disk as your home. From then on, the snapshot overlay of every machine grows there, and it goes away when the machine stops.
Details and sources
Details
qemu-var-tmptouches/var/tmp/image-garden-canary. If that fails andSNAP_USER_COMMONis set andmkdir -p "$SNAP_USER_COMMON"/tmpsucceeds, it exportsTMPDIR="$SNAP_USER_COMMON"/tmp; otherwise it exportsTMPDIR=/tmp/./. Then it execs the rest of the command chain (qemu-var-tmp:9-23). The slide's panel leaves out thethenkeywords and the error redirections.- The
image-gardenapp and everyqemu-system-*app listusr/bin/qemu-var-tmpin theircommand-chain(snapcraft.yaml:122-131, snapcraft.yaml:149-160); so does thespreadapp (snapcraft.yaml:211-226). - The README: the snap keeps the overlay "in
~/snap/image-garden/common/tmpinstead of its private/tmp, which is a tmpfs on many hosts" (README.md:155-161). - Derived, from snapd:
$SNAP_USER_COMMONis~/snap/image-garden/commonfor the user who runs the app, so/root/snap/image-garden/common/tmpfor root, as under sudo or spread./var/tmpis not writable under strict confinement (confinement: strict, snapcraft.yaml:43), which is why the canary fails there. - Derived, from QEMU: the overlay is removed when QEMU exits, so the space is freed when the machine stops or is discarded.
Read more
04 The fallback, and why it matters
If $SNAP_USER_COMMON/tmp cannot be made, TMPDIR=/tmp/./: the snap's private /tmp. On a tmpfs host the guest's writes then take RAM on top of -m.
There's a fallback. If the wrapper can't make that directory, it sets TMPDIR to /tmp/./ instead. That's the snap's private /tmp, spelled so that QEMU doesn't swap it for /var/tmp. It works, but on many hosts /tmp is a tmpfs, which lives in memory. Then every byte a guest writes lands in host memory. A machine with two gigabytes that writes ten holds twelve on the host, and four of them with spread hold forty-eight. That's exactly why the snap keeps the overlay in your home. If host memory runs out during write-heavy tests, check whether that directory exists and can be written.
Details and sources
Details
- The fallback: when
SNAP_USER_COMMONis empty ormkdir -p "$SNAP_USER_COMMON"/tmpfails, the wrapper exportsTMPDIR=/tmp/./; its comment says the path "is still /tmp but does not compare to /tmp exactly, which QEMU would replace with /var/tmp" (qemu-var-tmp:13-19). Nothing is printed when it falls back. - The README says the snap avoids its private
/tmpbecause it "is a tmpfs on many hosts", and that a scratch directory on tmpfs "would use RAM" allocate's memory check "does not see" (README.md:155-161). - Derived, from snapd rather than image-garden's tree: a strictly confined snap gets a private
/tmp, a directory inside the host's/tmp. Where that is a tmpfs, files in it take RAM (and swap) until they are deleted. Derived as well: snapd always setsSNAP_USER_COMMONfor an app, so the fallback needs a home directory where the directory cannot be made, for example one that is not writable. findmnt /tmp(util-linux) shows whether/tmpis a tmpfs.- The 12 GB and 48 GB are arithmetic on the example, not measurements.
Read more
05 What allocate does not count
The memory check counts only -m. Four VMs that write 10 GB each need 40 GB free in ~/snap/image-garden/common/tmp (root's home under sudo). QEMU_SNAPSHOT_OPTION="" writes into the image: use a copy.
One thing stays the same wherever the overlay lives. allocate's memory check counts only the machine's own memory, its dash m, and nothing it writes. With the scratch on disk, a write-heavy test costs disk space instead: four machines that each write ten gigabytes need forty gigabytes free under your home, for as long as they run. So check with df before a big run. Under sudo, or when spread runs as root, that's root's home, not yours. And to keep the writes out of scratch altogether, set QEMU_SNAPSHOT_OPTION to empty, but then the guest writes into the prepared image, so do that on a copy you throw away.
Details and sources
Details
- allocate's memory check reads
vm.memory_mbfrom.run --queryand compares it withMemAvailableless a reserve (GARDEN_MEMORY_RESERVE), divided by 1.5; it knows nothing of disk writes (image-garden:711-782). The README: "allocate's memory check counts only the machine's memory (-m)" (README.md:155-161). - Nothing in image-garden checks free disk space for the overlays either. Derived: the overlays are files in
$SNAP_USER_COMMON/tmp, so they take space on that file system until QEMU exits;dfshows it even when the files have no name in the directory. $SNAP_USER_COMMONbelongs to the user who runs the app:/root/snap/image-garden/commonfor root (derived, snapd's layout).- Snapshot mode is chosen by
QEMU_SNAPSHOT_OPTION; set to the empty string, allocate runs the machine without-snapshotand the guest writes into the system's.qcow2(image-garden:1153-1163). Copy the.qcow2(and, for UEFI systems, the.efi-vars.qcow2) first, or rebuild with make afterwards. - The container and a native install run QEMU without the wrapper, so QEMU's own default,
/var/tmp, holds the overlays there unlessTMPDIRsays otherwise (README.md:96-106 compares the installs). - The 40 GB is arithmetic on the example, not a measurement.
Read more
06 Paths that move
Plugs include home, not removable media. Without /run/user/UID, pid files and locks go to /run/snap.image-garden/image-garden/.
Two smaller things move too. The snap reaches your files through the home interface, so keep projects and their .image-garden directories under your home directory. And the runtime directory, where allocate keeps pid files and locks, normally sits under /run/user. snapd points the snap at it even when it doesn't exist, as for root under sudo or spread, and the snap can't create it. Then image-garden uses /run/snap.image-garden, the snap's own place in /run. Look there when image-garden ps finds nothing.
Details and sources
Details
- The snap's apps plug
home,kvm,network,network-bindandpkgproxy-sockets(snapcraft.yaml:122-131); there is noremovable-mediaplug, so paths outside the home directory, such as/srv, are not reachable. Derived from the plug list. - snapd sets
XDG_RUNTIME_DIRto/run/user/UID/snap.NAMEeven where/run/user/UIDdoes not exist, as for root under sudo or spread. The snap cannot create it, so image-garden uses/run/snap.$SNAP_INSTANCE_NAME(image-garden:64-73). allocate's pid files, locks and TPM run directories go under$XDG_RUNTIME_DIR/image-garden/. - The cache directory is
$SNAP_USER_COMMON/cachein the snap (image-garden:20-29).
Read more
07 Reaching out of the sandbox
Launchers call image-garden.qemu-system-* and image-garden.swtpm; image-garden.virtiofsd runs with --sandbox none. docker comes from the docker snap through docker-executables. A sideloaded snap installs no components.
Some things cross the boundary on purpose. A .run script made by the snap names the snap's own QEMU app, and a TPM system's script names the snap's swtpm app. So when you run it from an ordinary shell, it still runs confined, and its scratch still goes to the snap's directory in your home. The virtiofsd app runs confined too, which is why it turns off virtiofsd's own sandbox. spread's docker backend reaches docker through the docker snap's executables, connected with a content interface. And a snap you install from a local file can't fetch components: install them from files too, or install the snap from the store.
Details and sources
Details
- A launcher made inside the snap names
image-garden.qemu-system-x86-64when run from outside the snap, andqemu-system-x86_64inside it (010-qemu-x86_64.mk:4); the same holds for swtpm andimage-garden.swtpm(010-qemu-x86_64.mk:115-119). The QEMU app starts throughqemu-var-tmptoo, so its overlay goes to$SNAP_USER_COMMON/tmpas well (snapcraft.yaml:149-160). - The
virtiofsdapp plugshomeandnetwork-bind(snapcraft.yaml:140-148). Its stub adds--sandbox noneunless the caller passes a--sandboxoption: virtiofsd's default sandbox "enters new namespaces, which strict confinement does not allow", and "the snap's confinement still limits what virtiofsd can reach" (virtiofsd:6-15). See sharing host directories. - Only the
spreadapp plugsdockeranddocker-executables(snapcraft.yaml:211-226). The content plug mounts the docker snap's executables at$SNAP/docker-snap(snapcraft.yaml:267-270); thedockerstub prints how to connect them when they are missing (docker:6-35). - The install hook installs components only when the revision is not a local one (
x1,x2…) (install:6); the configure hook does the same (configure:47-53).
08 Where to go from here
Next: the snap overview, allocate and its snapshot mode, the three ways to install, and sharing host directories with virtiofs.
So that's strict confinement: snapshot overlays on disk in the snap's directory in your home, a fallback to /tmp, which may be memory, a runtime directory that moves, and a few deliberate holes in the sandbox. To see the rest of the snap, go back to its overview. allocate's own deck shows every option it passes to QEMU, including snapshot mode. To compare the container and native installs, where QEMU uses /var/tmp, see the three ways in. And if your guests need large host data sets, the virtiofs deck shows how to share them without copying.
Details and sources
Details
- The deck in one line: confinement blocks
/var/tmp; the wrapper sends QEMU's-snapshotoverlays to~/snap/image-garden/common/tmp, on disk, or to the snap's/tmp, which may be memory, when that directory cannot be made; allocate counts neither.
Read more
Read more
Basis of the statements on this page
Read in the snap's snapcraft.yaml, hooks and stubs, the README and the image-garden script at the commit this map describes ("code"). Statements about QEMU's -snapshot overlay, snapd's private /tmp, $SNAP_USER_COMMON and tmpfs are general behaviour, not image-garden's code ("derived").
Confinement limits, their symptoms and what to do
| Limit | What you notice | What image-garden does | What you can do |
|---|---|---|---|
/var/tmp is not writable | QEMU's -snapshot overlays cannot go to /var/tmp | qemu-var-tmp sets TMPDIR=$SNAP_USER_COMMON/tmp, that is ~/snap/image-garden/common/tmp, on disk (qemu-var-tmp:9-23, README.md:155-161) | keep enough free space under your home (root's home under sudo or spread) |
$SNAP_USER_COMMON/tmp cannot be made | the overlays go to the snap's private /tmp; on a tmpfs host, write-heavy tests use host RAM far beyond -m | TMPDIR=/tmp/./, spelled so QEMU does not replace it (qemu-var-tmp:13-19) | make sure that directory exists and is writable; findmnt /tmp shows a tmpfs |
| allocate's memory check | the overlays are not counted, wherever they are | counts only -m (image-garden:711-782) | budget disk (or RAM) for what tests write; fewer parallel VMs; QEMU_SNAPSHOT_OPTION="" on a copy; or the container or a native install |
only home | projects outside your home directory are not reachable | plugs home, kvm, network, network-bind, pkgproxy-sockets (snapcraft.yaml:122-131) | keep projects under $HOME |
no /run/user/UID for root without a session | pid files would have nowhere to go | XDG_RUNTIME_DIR=/run/snap.image-garden (image-garden:64-73) | look in /run/snap.image-garden/image-garden/ |
| launchers run from a host shell | QEMU is still the snap's | launchers name image-garden.qemu-system-* and image-garden.swtpm (010-qemu-x86_64.mk:4, 010-qemu-x86_64.mk:115-119) | expect the same scratch directory |
| no docker inside the sandbox | spread's docker backend fails with a notice | docker stub, docker-executables content plug (docker:6-35) | snap connect image-garden:docker-executables docker:docker-executables and snap connect image-garden:docker docker:docker |
| a sideloaded snap has no store | components are not installed | hooks skip x revisions (install:6) | install the .comp files too, or the snap from the store |
| virtiofsd's own sandbox | its namespace sandbox is not allowed | the virtiofsd app's stub adds --sandbox none (virtiofsd:6-15); plugs home, network-bind (snapcraft.yaml:140-148) | share directories under your home |
| pkgproxy snap's sockets | pkgproxy=snap needs them | content plug pkgproxy-sockets (snapcraft.yaml:255-266) | snap connect image-garden:pkgproxy-sockets pkgproxy:sockets |
Roads from here
START HERE
- Snap, container or native package — the container and native installs, where /var/tmp works
RUNTIME · make rules and the CLI
- image-garden allocate — allocate, which runs every machine with -snapshot
- Where image-garden keeps its files — where image-garden keeps its other files
SNAP · snap-only features
- The image-garden snap — what the snap carries and installs
- Sharing host directories: virtiofs and 9p — sharing host directories with a guest
- Snap components — the components a sideloaded snap cannot fetch