Sharing host directories: virtiofs and 9p
share host files, advanced for CI operators for contributors
Advanced. When a test reads a small part of a large data set, share the host directory instead of copying it in. The snap runs its bundled virtiofsd as image-garden.virtiofsd, without virtiofsd's own sandbox, for near-native sharing, and the snap's and the container's QEMU also have 9p, which is easier to set up but slower. image-garden passes the QEMU options through the .run script and allocate.
On the map: snap
01 Advanced: a big data set, a few files
Advanced. When a test touches a small part of a large data set, share the host directory instead of copying it in: the guest reads only what it opens.
This one is advanced, so here's the situation it's for. Your test needs data from the host: a CI cache, a package mirror, a large source tree. You could copy it into the VM over SSH, or bake it into the image with cloud-init. But if the data set is gigabytes and a test opens only a handful of files, most of that copying is wasted, and it happens again for every run. A shared directory turns it around. The guest mounts the host directory and reads only the files it opens, when it opens them. Nothing is copied up front.
Details and sources
Details
- The README describes the bundled virtiofsd, the
image-garden.virtiofsdapp, as "very efficient host-to-guest file system sharing", "useful for CI systems with native cache directories (e.g., GitHub Actions cache)", letting guests "directly access host caches without copying data" (README.md:280-290). - The sizes on the slide are invented for the example.
- Derived: that a shared directory moves only the files a guest reads is how both virtiofs and 9p work, as file systems served on demand; image-garden's tree does not say it in these words.
Read more
03 virtiofs: a daemon and a socket
virtiofsd serves a host directory over a unix socket; QEMU adds vhost-user-fs-pci with a tag; the guest runs mount -t virtiofs data /mnt.
With virtiofs, a separate program does the file serving. virtiofsd runs on the host, points at the directory you want to share, and listens on a unix socket. QEMU connects to that socket through a character device, and adds a virtiofs device with a tag, a short name you choose, such as data. Inside the guest, you mount by that tag, with the file system type virtiofs. File operations go straight from the guest kernel to virtiofsd and back, which is why it runs close to native speed. The price is one more process to start before QEMU, and to stop after it.
Details and sources
Details
- The snap stages
usr/libexec/virtiofsdfrom the distribution'svirtiofsdpackage (snapcraft.yaml:602-607); QEMU is built with--enable-vhost-user(snapcraft.yaml:356), whichvhost-user-fs-pcineeds. - Derived from QEMU's and virtiofsd's documented options, not from image-garden's tree: virtiofsd is started with
--socket-path=SOCKET --shared-dir=DIR; QEMU takes-chardev socket,id=fs0,path=SOCKET -device vhost-user-fs-pci,chardev=fs0,tag=data; the guest runsmount -t virtiofs data /mnt. The namesvfsd.sock,fs0,dataand~/dataare examples. In the snap, virtiofsd runs as theimage-garden.virtiofsdapp (snapcraft.yaml:140-148); see running virtiofsd from the snap. - The device name on the slide is for PCI machines (x86_64, aarch64); other machines use other device names.
Read more
05 9p: built into QEMU
One option, -virtfs local,path=DIR,mount_tag=data,security_model=none,readonly=on; in the guest, mount -t 9p -o trans=virtio,version=9p2000.L data /mnt. No daemon, but slower with many small files.
If that sounds like a lot, there's an easier way, and it's already inside QEMU. 9p, also called virtfs, needs no daemon and no shared memory. One option names the host directory, a mount tag and a security model, and can make the share read-only. In the guest you mount it with the file system type 9p over the virtio transport. The snap and the container both build QEMU with it. The catch is speed. Every file operation is a round trip through QEMU, so lots of small files, or anything that walks directories, is noticeably slower than virtiofs.
Details and sources
Details
- The snap's QEMU is built with
--enable-virtfs(snapcraft.yaml:357), and so is the container's (Dockerfile:51). The README lists virtfs among the snap's and the container's QEMU features (README.md:153, README.md:358). A distribution QEMU may or may not have it. - Derived from QEMU's documented options:
-virtfs local,path=DIR,mount_tag=TAG,security_model=none|mapped-xattr|passthrough[,readonly=on](or the longer-fsdevwith-device virtio-9p-pci); in the guestmount -t 9p -o trans=virtio,version=9p2000.L TAG /mnt, with the guest's 9p modules. - Derived: 9p performance is lower than virtiofs's, most visibly for many small files and metadata-heavy work; nothing was measured for this map.
Read more
06 Snapshot mode does not cover it
-snapshot protects the disk image, not a shared directory. A guest's writes reach the host directory: share it read-only when the test only reads.
Here's a trap worth knowing about. allocate starts machines in snapshot mode, so whatever a test writes to the disk is thrown away when the machine stops. That only covers the disk image. A shared directory is not part of it: it's the real directory on the host, and a write from the guest lands there for good. If a test can damage your cache, it will, sooner or later. So share read-only whenever the test only needs to read. Both 9p and virtiofs can do that, and the guest then gets an error instead of changing your files.
Details and sources
Details
- allocate runs the
.runscript with${QEMU_SNAPSHOT_OPTION=-snapshot}(image-garden:1154).-snapshotapplies to the disk drives, here the system's.qcow2(010-qemu-x86_64.mk:101). - Derived from QEMU's documented options:
-snapshotdoes not cover a 9p or virtiofs share;readonly=on(9p) or a read-only mount makes the guest's writes fail. virtiofsd can also be started on a read-only bind mount of the directory.
Read more
07 Running virtiofsd from the snap
image-garden.virtiofsd --socket-path=$HOME/snap/image-garden/common/cache.sock --shared-dir=$HOME/cache &. The app adds --sandbox none unless you pass --sandbox; it reaches your home.
In the snap, virtiofsd is an app: image-garden dot virtiofsd. You start it from your shell, with the socket to listen on and the directory to share, and leave it running in the background. One thing differs from a plain virtiofsd. Its default sandbox enters new namespaces, which strict confinement doesn't allow, so the snap's small wrapper adds dash dash sandbox none, unless you choose a sandbox yourself. Confinement is the fence instead: the app reaches your home and not much else, so share a directory under it. And put the socket in the snap's own directory, tilde slash snap slash image-garden slash common, where the snap's QEMU can reach it too.
Details and sources
Details
- The
virtiofsdapp runsusr/bin/virtiofsd, a stub, and plugshome, "for the shared directory", andnetwork-bind, for "the vhost-user UNIX socket" (snapcraft.yaml:140-148). - The stub,
qemu-stubs/virtiofsd: "Its default sandbox enters new namespaces, which strict confinement does not allow, so the app runs without one unless the caller chooses another with --sandbox. The snap's confinement still limits what virtiofsd can reach: the home interface, the user's own files." It prepends--sandbox nonewhen no argument starts with--sandbox, then execs$SNAP/usr/libexec/virtiofsd(virtiofsd:6-15). - virtiofsd itself is staged from Ubuntu's
virtiofsdpackage into the snap, not into a component (snapcraft.yaml:602-607). - The README: the app "can reach what the snap can, the user's home. Put the socket in the snap's directory, where the snap's QEMU can reach it too", with the command on the slide (README.md:292-301).
- The snap's own test runs the app as a user who is not root, checks
--version, serves~/sharedon~/snap/image-garden/common/virtiofsd.sockand waits for "Waiting for vhost-user socket connection"; no guest connects (tests/snap/virtiofsd/task.yaml). So a guest mounting a share through the snap was not tested there, nor for this map. --socket-pathand--shared-dirare virtiofsd's own options (derived, from its documentation).
Read more
08 Passing the options in
Extra arguments to X.run, or after the system name in image-garden allocate, go to QEMU: the shared memory backend, -chardev socket,… and -device vhost-user-fs-pci,…,tag=cache. In the guest, mount -t virtiofs cache /mnt.
Now QEMU needs the other end. image-garden has no rule that sets this up for you, but it leaves the door open. The run script adds whatever arguments you give it to the end of the QEMU command. So you pass the shared memory backend, sized like the machine, two thousand and forty-eight megabytes unless you changed it, then a character device for the socket, and the virtiofs device with its tag. allocate does the same with anything after the system name, so a background machine can have its share too. In the guest, you mount the tag. And 9p needs nothing extra: its one option is all there is.
Details and sources
Details
- The
.runscript appends"$@"to the QEMU command (010-qemu-x86_64.mk:105). - allocate's usage is
image-garden allocate [--spread] [-p PORT] SYSTEM.ARCH [QEMU-OPTION...](image-garden:1037); the options after the system name reach the.runscript (image-garden:1160). - The README's example, for
debian-cloud-13.x86_64.run: startimage-garden.virtiofsdwith the socket in$HOME/snap/image-garden/common, then pass-object memory-backend-memfd,id=mem,size=2048M,share=on -numa node,memdev=mem,-chardev socket,id=cache,path=…/cache.sockand-device vhost-user-fs-pci,chardev=cache,tag=cacheto the.runscript, and in the guestmount -t virtiofs cache /mnt. The backend's size is the machine's memory, "2048M unlessQEMU_MEM_OPTIONsays otherwise" (README.md:292-307, 010-qemu.mk:9). The slide uses the running example instead. - No make rule in image-garden sets up a share, and the example was not run for this map. Start virtiofsd before QEMU, so the socket exists when QEMU connects (derived).
Read more
09 Where to go from here
Next: the .run script and allocate for passing options, caching on GitHub, and the snap and its confinement.
So: when a test needs a little of a lot, share instead of copying. virtiofs is fast but needs a daemon and shared memory; 9p is one option and slower. Either way, share read-only when you can. To see where the extra options and the memory variable go, open the deck on the run script, and the one on allocate. For the CI caches that make this worth doing, see how the GitHub action caches. And for what else the snap carries, and what confinement changes, there are the snap decks.
Details and sources
Details
- Every QEMU option in this deck that is not quoted from image-garden's tree is derived from QEMU's and virtiofsd's documented options and was not run for this map.
Read more
Read more
Basis of the statements on this page
What image-garden provides (virtiofsd in the snap and its image-garden.virtiofsd app, virtfs in the snap's and the container's QEMU, the .run script's QEMU_MEM_OPTION and "$@", allocate's extra options) is read in image-garden at the commit this map describes ("code"); the virtiofs example follows the README's. The QEMU and virtiofsd options and the performance comparison are derived from their documented behaviour; the example command lines are illustrative and were not run for this map.
virtiofs or 9p
| virtiofs | 9p (virtfs) | |
|---|---|---|
| Where it comes from | virtiofsd, bundled in the snap and run as image-garden.virtiofsd, with --sandbox none (snapcraft.yaml:140-148, virtiofsd:6-15) | built into the snap's and the container's QEMU |
| Setup steps | start virtiofsd; add a chardev and a vhost-user-fs-pci device; shared memory; mount | one -virtfs option; mount |
| Daemon | yes: start before QEMU, stop after | no |
| Memory | guest RAM from a shared memory-backend-memfd, same size as -m | nothing special |
| Performance | close to native | slower, most for many small files and directory walks |
| Guest kernel | virtiofs file system | 9p and 9pnet_virtio |
| Read-only | a read-only mount, or serve a read-only directory | readonly=on |
| Typical use | large caches and trees read by many tests | a quick share, a few files, little metadata work |
A virtiofs example with the snap (illustrative, not run for this map)
After the README's example (README.md:292-307). On the host, start the daemon in the background:
image-garden.virtiofsd --socket-path="$HOME"/snap/image-garden/common/cache.sock \ --shared-dir="$HOME"/cache &
Then start the system with shared memory, as big as the machine's (-m 2048 unless QEMU_MEM_OPTION says otherwise), and the device:
./ubuntu-cloud-24.04.x86_64.run \ -object memory-backend-memfd,id=mem,size=2048M,share=on -numa node,memdev=mem \ -chardev socket,id=cache,path="$HOME"/snap/image-garden/common/cache.sock \ -device vhost-user-fs-pci,chardev=cache,tag=cache
In the guest:
mount -t virtiofs cache /mnt
A 9p example (illustrative, not run for this map)
./ubuntu-cloud-24.04.x86_64.run \ -virtfs local,path=$HOME/cache,mount_tag=data,security_model=none,readonly=on
In the guest:
mount -t 9p -o trans=virtio,version=9p2000.L data /mnt
With allocate
Options after the system name go to the .run script:
image-garden allocate ubuntu-cloud-24.04.x86_64 \ -virtfs local,path=$HOME/cache,mount_tag=data,security_model=none,readonly=on
allocate runs the machine with -snapshot, which does not cover the shared directory: share read-only when the tests only read.
Things to remember
- In the snap, QEMU and virtiofsd run confined: the shared directory must be one the snap can reach (the
homeinterface covers your home directory). - Put the socket in
~/snap/image-garden/common, where both the virtiofsd app and the snap's QEMU reach it, as the README does. - The snap's virtiofsd runs without its own sandbox; pass
--sandboxyourself to choose another, which confinement may refuse. - The snap's test of the app checks that it starts as a user and listens on its socket; no guest mounts a share in that test (tests/snap/virtiofsd/task.yaml).
Roads from here
RUNTIME · make rules and the CLI
- The .run script — where the extra QEMU options and QEMU_MEM_OPTION go
- image-garden allocate — extra QEMU options after the system name
BUILT ON IT · CI integrations
- How the action caches — a CI cache that a shared directory can serve
SNAP · snap-only features
- The image-garden snap — what else the snap bundles
- Living with strict confinement — what strict confinement changes for QEMU
CONTAINER · OCI image
- The OCI container — the container's QEMU, which also has 9p