Sharing host directories: virtiofs and 9p

share host files, advanced for CI operators for contributors

Advanced. When a test reads a small part of a large data set, share the host directory instead of copying it in. The snap runs its bundled virtiofsd as image-garden.virtiofsd, without virtiofsd's own sandbox, for near-native sharing, and the snap's and the container's QEMU also have 9p, which is easier to set up but slower. image-garden passes the QEMU options through the .run script and allocate.

On the map: snap

▶ Watch this deck — 9 scenes, ~7 min

01 Advanced: a big data set, a few files

ADVANCED · SHARING HOST FILES the host ci-cache/ 12 GB mirror/ 40 GB source-tree/ 8 GB the test VM copy it all in opens 3 files most of it wasted, on every run share the directory mount: reads 3 files nothing is copied up front: the guest reads what it opens, when it opens it
Advanced: a host holds a large data set such as a CI cache, a package mirror or a source tree; copying all of it into the VM on every run is wasted when a test opens only a few files; a shared directory lets the guest read only the files it opens

Advanced. When a test touches a small part of a large data set, share the host directory instead of copying it in: the guest reads only what it opens.

This one is advanced, so here's the situation it's for. Your test needs data from the host: a CI cache, a package mirror, a large source tree. You could copy it into the VM over SSH, or bake it into the image with cloud-init. But if the data set is gigabytes and a test opens only a handful of files, most of that copying is wasted, and it happens again for every run. A shared directory turns it around. The guest mounts the host directory and reads only the files it opens, when it opens them. Nothing is copied up front.

Details and sources

Details

  • The README describes the bundled virtiofsd, the image-garden.virtiofsd app, as "very efficient host-to-guest file system sharing", "useful for CI systems with native cache directories (e.g., GitHub Actions cache)", letting guests "directly access host caches without copying data" (README.md:280-290).
  • The sizes on the slide are invented for the example.
  • Derived: that a shared directory moves only the files a guest reads is how both virtiofs and 9p work, as file systems served on demand; image-garden's tree does not say it in these words.

02 Copy everything, or share it

COPY OR SHARE copy: eager every file crosses into the guest test grows with the size of the data share: lazy test a file moves when it is read grows with what the test touches virtiofs: fast, the snap has virtiofsd 9p: built into QEMU, simpler, slower
Two timelines: copying moves every file before the test starts, so its cost grows with the whole data set; sharing lets the test start at once and moves only the files it reads; virtiofs is the fast way to share and 9p the simple one

Copying costs time in proportion to the whole data set; sharing, in proportion to what the test reads. virtiofs is fast; 9p is simpler.

Picture the two ways side by side. Copying is eager. Before the test starts, every file crosses into the guest, whether the test needs it or not, and the time grows with the size of the data. Sharing is lazy. The guest sees the whole directory at once, but a file's bytes move only when something reads them, so the cost grows with what the test actually touches. There are two ways to share a directory with a QEMU guest. virtiofs is the fast one, and the snap bundles its daemon. 9p is built into QEMU itself: slower, but simpler to set up.

Details and sources

Details

  • The snap stages virtiofsd (snapcraft.yaml:602-607), runs it as the image-garden.virtiofsd app (snapcraft.yaml:140-148), and lists it in its description (snapcraft.yaml:38). QEMU in the snap is built with --enable-virtfs, which gives 9p (snapcraft.yaml:357).
  • Derived: the cost model (eager copy grows with the data set, lazy sharing with what is read) follows from what each approach transfers; nothing was measured for this map.

03 virtiofs: a daemon and a socket

VIRTIOFS virtiofsd on the host shares ~/data vfsd.sock QEMU -chardev socket,id=fs0,path=vfsd.sock -device vhost-user-fs-pci,chardev=fs0,tag=data # mount -t virtiofs data /mnt in the guest, by tag guest kernel ↔ virtiofsd close to native speed the price: start virtiofsd before QEMU, stop it after
virtiofsd runs on the host, serves a directory and listens on a unix socket; QEMU connects to the socket with a chardev and adds a vhost-user-fs-pci device tagged data; the guest mounts the tag with mount -t virtiofs; file operations go from the guest kernel to virtiofsd directly; virtiofsd must start before QEMU and stop after it

virtiofsd serves a host directory over a unix socket; QEMU adds vhost-user-fs-pci with a tag; the guest runs mount -t virtiofs data /mnt.

With virtiofs, a separate program does the file serving. virtiofsd runs on the host, points at the directory you want to share, and listens on a unix socket. QEMU connects to that socket through a character device, and adds a virtiofs device with a tag, a short name you choose, such as data. Inside the guest, you mount by that tag, with the file system type virtiofs. File operations go straight from the guest kernel to virtiofsd and back, which is why it runs close to native speed. The price is one more process to start before QEMU, and to stop after it.

Details and sources

Details

  • The snap stages usr/libexec/virtiofsd from the distribution's virtiofsd package (snapcraft.yaml:602-607); QEMU is built with --enable-vhost-user (snapcraft.yaml:356), which vhost-user-fs-pci needs.
  • Derived from QEMU's and virtiofsd's documented options, not from image-garden's tree: virtiofsd is started with --socket-path=SOCKET --shared-dir=DIR; QEMU takes -chardev socket,id=fs0,path=SOCKET -device vhost-user-fs-pci,chardev=fs0,tag=data; the guest runs mount -t virtiofs data /mnt. The names vfsd.sock, fs0, data and ~/data are examples. In the snap, virtiofsd runs as the image-garden.virtiofsd app (snapcraft.yaml:140-148); see running virtiofsd from the snap.
  • The device name on the slide is for PCI machines (x86_64, aarch64); other machines use other device names.

04 The memory has to be shared

SHARED MEMORY virtiofsd the guest's memory read and written directly -object memory-backend-memfd,id=mem,size=2G,share=on -numa node,memdev=mem the size must match the guest's memory, -m QEMU_MEM_OPTION="-m 2G -object memory-backend-memfd,id=mem,size=2G,share=on -numa node,memdev=mem" the .run script splits the variable into words both sizes in one place
virtiofsd reads and writes guest memory directly, so the memory must be a shared memfd backend used by a NUMA node and as big as -m; the .run script splits QEMU_MEM_OPTION into words, so the size, the backend and the NUMA node can all go into that one variable

virtiofs needs guest memory shared with virtiofsd: -object memory-backend-memfd,id=mem,size=2G,share=on -numa node,memdev=mem, with the same size as -m. All of it fits in QEMU_MEM_OPTION.

There's one more condition, and it's the one people miss. virtiofsd reads and writes the guest's memory directly, so that memory has to be shared with it. You give QEMU a memory backend of the memfd kind, marked as shared, and a NUMA node that uses it, and its size must match the guest's memory size. The run script makes that easy. Its memory setting is one variable, QEMU_MEM_OPTION, which it splits into words before it starts QEMU. So you can put the size and the shared backend into that one variable, and keep both numbers next to each other.

Details and sources

Details

  • The default memory is -m 2048 (MiB) (010-qemu.mk:9, 010-qemu.mk:12). The .run script puts it in QEMU_MEM_OPTION unless already set (010-qemu-x86_64.mk:64, 010-qemu-x86_64.mk:73), splits it with read -ra (010-qemu-x86_64.mk:84) and passes the words to QEMU (010-qemu-x86_64.mk:97). So several options can share the one variable.
  • Derived from QEMU's documented options: virtiofs needs guest RAM from a shared backend: -object memory-backend-memfd,id=mem,size=2G,share=on -numa node,memdev=mem, with size equal to -m.
  • Illustrative, not run for this map: QEMU_MEM_OPTION="-m 2G -object memory-backend-memfd,id=mem,size=2G,share=on -numa node,memdev=mem" ./ubuntu-cloud-24.04.x86_64.run …

Read more

05 9p: built into QEMU

9P · VIRTFS QEMU, nothing else no daemon, no shared memory -virtfs local,path=$HOME/data,mount_tag=data,security_model=none,readonly=on # mount -t 9p -o trans=virtio,version=9p2000.L data /mnt snap and container: built with virtfs every operation is a round trip through QEMU: many small files and directory walks are slower than virtiofs
9p, also called virtfs, is built into QEMU: one -virtfs option names the directory, a mount tag, a security model and read-only; the guest mounts it with mount -t 9p over virtio; the snap and the container build QEMU with it; it is slower than virtiofs for many small files and directory walks

One option, -virtfs local,path=DIR,mount_tag=data,security_model=none,readonly=on; in the guest, mount -t 9p -o trans=virtio,version=9p2000.L data /mnt. No daemon, but slower with many small files.

If that sounds like a lot, there's an easier way, and it's already inside QEMU. 9p, also called virtfs, needs no daemon and no shared memory. One option names the host directory, a mount tag and a security model, and can make the share read-only. In the guest you mount it with the file system type 9p over the virtio transport. The snap and the container both build QEMU with it. The catch is speed. Every file operation is a round trip through QEMU, so lots of small files, or anything that walks directories, is noticeably slower than virtiofs.

Details and sources

Details

  • The snap's QEMU is built with --enable-virtfs (snapcraft.yaml:357), and so is the container's (Dockerfile:51). The README lists virtfs among the snap's and the container's QEMU features (README.md:153, README.md:358). A distribution QEMU may or may not have it.
  • Derived from QEMU's documented options: -virtfs local,path=DIR,mount_tag=TAG,security_model=none|mapped-xattr|passthrough[,readonly=on] (or the longer -fsdev with -device virtio-9p-pci); in the guest mount -t 9p -o trans=virtio,version=9p2000.L TAG /mnt, with the guest's 9p modules.
  • Derived: 9p performance is lower than virtiofs's, most visibly for many small files and metadata-heavy work; nothing was measured for this map.

Read more

06 Snapshot mode does not cover it

SNAPSHOT DOES NOT COVER IT a test writes the disk image -snapshot: thrown away the shared host directory kept for good readonly=on the guest gets an error instead of changing your files
allocate runs the VM with -snapshot, so disk writes are thrown away, but a shared directory is the real host directory and guest writes land there for good; sharing read-only makes the guest get an error instead

-snapshot protects the disk image, not a shared directory. A guest's writes reach the host directory: share it read-only when the test only reads.

Here's a trap worth knowing about. allocate starts machines in snapshot mode, so whatever a test writes to the disk is thrown away when the machine stops. That only covers the disk image. A shared directory is not part of it: it's the real directory on the host, and a write from the guest lands there for good. If a test can damage your cache, it will, sooner or later. So share read-only whenever the test only needs to read. Both 9p and virtiofs can do that, and the guest then gets an error instead of changing your files.

Details and sources

Details

  • allocate runs the .run script with ${QEMU_SNAPSHOT_OPTION=-snapshot} (image-garden:1154). -snapshot applies to the disk drives, here the system's .qcow2 (010-qemu-x86_64.mk:101).
  • Derived from QEMU's documented options: -snapshot does not cover a 9p or virtiofs share; readonly=on (9p) or a read-only mount makes the guest's writes fail. virtiofsd can also be started on a read-only bind mount of the directory.

07 Running virtiofsd from the snap

THE VIRTIOFSD APP image-garden.virtiofsd $ image-garden.virtiofsd \ --socket-path=$HOME/snap/image-garden/common/cache.sock \ --shared-dir=$HOME/cache & its own sandbox namespaces: not allowed --sandbox none $SNAP/usr/libexec/virtiofsd added by the stub, unless you pass --sandbox confinement: your home, little else socket where the snap's QEMU reaches it
The snap's image-garden.virtiofsd app is started from a shell with a socket under ~/snap/image-garden/common and a shared directory under the home directory, in the background; its stub adds --sandbox none, because virtiofsd's own namespace sandbox is not allowed under strict confinement, then runs $SNAP/usr/libexec/virtiofsd; confinement limits it to the home directory; the socket sits where the snap's QEMU can reach it

image-garden.virtiofsd --socket-path=$HOME/snap/image-garden/common/cache.sock --shared-dir=$HOME/cache &. The app adds --sandbox none unless you pass --sandbox; it reaches your home.

In the snap, virtiofsd is an app: image-garden dot virtiofsd. You start it from your shell, with the socket to listen on and the directory to share, and leave it running in the background. One thing differs from a plain virtiofsd. Its default sandbox enters new namespaces, which strict confinement doesn't allow, so the snap's small wrapper adds dash dash sandbox none, unless you choose a sandbox yourself. Confinement is the fence instead: the app reaches your home and not much else, so share a directory under it. And put the socket in the snap's own directory, tilde slash snap slash image-garden slash common, where the snap's QEMU can reach it too.

Details and sources

Details

  • The virtiofsd app runs usr/bin/virtiofsd, a stub, and plugs home, "for the shared directory", and network-bind, for "the vhost-user UNIX socket" (snapcraft.yaml:140-148).
  • The stub, qemu-stubs/virtiofsd: "Its default sandbox enters new namespaces, which strict confinement does not allow, so the app runs without one unless the caller chooses another with --sandbox. The snap's confinement still limits what virtiofsd can reach: the home interface, the user's own files." It prepends --sandbox none when no argument starts with --sandbox, then execs $SNAP/usr/libexec/virtiofsd (virtiofsd:6-15).
  • virtiofsd itself is staged from Ubuntu's virtiofsd package into the snap, not into a component (snapcraft.yaml:602-607).
  • The README: the app "can reach what the snap can, the user's home. Put the socket in the snap's directory, where the snap's QEMU can reach it too", with the command on the slide (README.md:292-301).
  • The snap's own test runs the app as a user who is not root, checks --version, serves ~/shared on ~/snap/image-garden/common/virtiofsd.sock and waits for "Waiting for vhost-user socket connection"; no guest connects (tests/snap/virtiofsd/task.yaml). So a guest mounting a share through the snap was not tested there, nor for this map.
  • --socket-path and --shared-dir are virtiofsd's own options (derived, from its documentation).

08 Passing the options in

PASSING THE OPTIONS no make rule sets this up $ ./ubuntu-cloud-24.04.x86_64.run \ the .run script adds "$@" to the end of the QEMU command -object memory-backend-memfd,id=mem,size=2048M,share=on -numa node,memdev=mem \ -chardev socket,id=cache,path=$HOME/snap/image-garden/common/cache.sock \ -device vhost-user-fs-pci,chardev=cache,tag=cache $ image-garden allocate ubuntu-cloud-24.04.x86_64 -object … -chardev … -device … allocate passes what follows the system name # mount -t virtiofs cache /mnt 9p: one -virtfs option is all there is
image-garden has no rule for sharing, but the .run script appends its arguments to the QEMU command: a shared memory backend of 2048M with a NUMA node, a socket chardev for ~/snap/image-garden/common/cache.sock and a vhost-user-fs-pci device tagged cache; allocate passes anything after the system name the same way; in the guest, mount -t virtiofs cache /mnt; 9p needs only its one option

Extra arguments to X.run, or after the system name in image-garden allocate, go to QEMU: the shared memory backend, -chardev socket,… and -device vhost-user-fs-pci,…,tag=cache. In the guest, mount -t virtiofs cache /mnt.

Now QEMU needs the other end. image-garden has no rule that sets this up for you, but it leaves the door open. The run script adds whatever arguments you give it to the end of the QEMU command. So you pass the shared memory backend, sized like the machine, two thousand and forty-eight megabytes unless you changed it, then a character device for the socket, and the virtiofs device with its tag. allocate does the same with anything after the system name, so a background machine can have its share too. In the guest, you mount the tag. And 9p needs nothing extra: its one option is all there is.

Details and sources

Details

  • The .run script appends "$@" to the QEMU command (010-qemu-x86_64.mk:105).
  • allocate's usage is image-garden allocate [--spread] [-p PORT] SYSTEM.ARCH [QEMU-OPTION...] (image-garden:1037); the options after the system name reach the .run script (image-garden:1160).
  • The README's example, for debian-cloud-13.x86_64.run: start image-garden.virtiofsd with the socket in $HOME/snap/image-garden/common, then pass -object memory-backend-memfd,id=mem,size=2048M,share=on -numa node,memdev=mem, -chardev socket,id=cache,path=…/cache.sock and -device vhost-user-fs-pci,chardev=cache,tag=cache to the .run script, and in the guest mount -t virtiofs cache /mnt. The backend's size is the machine's memory, "2048M unless QEMU_MEM_OPTION says otherwise" (README.md:292-307, 010-qemu.mk:9). The slide uses the running example instead.
  • No make rule in image-garden sets up a share, and the example was not run for this map. Start virtiofsd before QEMU, so the socket exists when QEMU connects (derived).

09 Where to go from here

ROADS virtiofs and 9p The .run script extra options, QEMU_MEM_OPTION allocate options after the system name Caching on GitHub a cache worth sharing The snap what it bundles Confinement what strict changes The container its QEMU has 9p too
Six roads from virtiofs and 9p: the .run script, allocate, caching on GitHub, the snap, strict confinement and the container

Next: the .run script and allocate for passing options, caching on GitHub, and the snap and its confinement.

So: when a test needs a little of a lot, share instead of copying. virtiofs is fast but needs a daemon and shared memory; 9p is one option and slower. Either way, share read-only when you can. To see where the extra options and the memory variable go, open the deck on the run script, and the one on allocate. For the CI caches that make this worth doing, see how the GitHub action caches. And for what else the snap carries, and what confinement changes, there are the snap decks.

Details and sources

Details

  • Every QEMU option in this deck that is not quoted from image-garden's tree is derived from QEMU's and virtiofsd's documented options and was not run for this map.

Read more

Basis of the statements on this page

What image-garden provides (virtiofsd in the snap and its image-garden.virtiofsd app, virtfs in the snap's and the container's QEMU, the .run script's QEMU_MEM_OPTION and "$@", allocate's extra options) is read in image-garden at the commit this map describes ("code"); the virtiofs example follows the README's. The QEMU and virtiofsd options and the performance comparison are derived from their documented behaviour; the example command lines are illustrative and were not run for this map.

virtiofs or 9p

virtiofs9p (virtfs)
Where it comes fromvirtiofsd, bundled in the snap and run as image-garden.virtiofsd, with --sandbox none (snapcraft.yaml:140-148, virtiofsd:6-15)built into the snap's and the container's QEMU
Setup stepsstart virtiofsd; add a chardev and a vhost-user-fs-pci device; shared memory; mountone -virtfs option; mount
Daemonyes: start before QEMU, stop afterno
Memoryguest RAM from a shared memory-backend-memfd, same size as -mnothing special
Performanceclose to nativeslower, most for many small files and directory walks
Guest kernelvirtiofs file system9p and 9pnet_virtio
Read-onlya read-only mount, or serve a read-only directoryreadonly=on
Typical uselarge caches and trees read by many testsa quick share, a few files, little metadata work

A virtiofs example with the snap (illustrative, not run for this map)

After the README's example (README.md:292-307). On the host, start the daemon in the background:

image-garden.virtiofsd --socket-path="$HOME"/snap/image-garden/common/cache.sock \ --shared-dir="$HOME"/cache &

Then start the system with shared memory, as big as the machine's (-m 2048 unless QEMU_MEM_OPTION says otherwise), and the device:

./ubuntu-cloud-24.04.x86_64.run \ -object memory-backend-memfd,id=mem,size=2048M,share=on -numa node,memdev=mem \ -chardev socket,id=cache,path="$HOME"/snap/image-garden/common/cache.sock \ -device vhost-user-fs-pci,chardev=cache,tag=cache

In the guest:

mount -t virtiofs cache /mnt

A 9p example (illustrative, not run for this map)

./ubuntu-cloud-24.04.x86_64.run \ -virtfs local,path=$HOME/cache,mount_tag=data,security_model=none,readonly=on

In the guest:

mount -t 9p -o trans=virtio,version=9p2000.L data /mnt

With allocate

Options after the system name go to the .run script:

image-garden allocate ubuntu-cloud-24.04.x86_64 \ -virtfs local,path=$HOME/cache,mount_tag=data,security_model=none,readonly=on

allocate runs the machine with -snapshot, which does not cover the shared directory: share read-only when the tests only read.

Things to remember

  • In the snap, QEMU and virtiofsd run confined: the shared directory must be one the snap can reach (the home interface covers your home directory).
  • Put the socket in ~/snap/image-garden/common, where both the virtiofsd app and the snap's QEMU reach it, as the README does.
  • The snap's virtiofsd runs without its own sandbox; pass --sandbox yourself to choose another, which confinement may refuse.
  • The snap's test of the app checks that it starts as a user and listens on its socket; no guest mounts a share in that test (tests/snap/virtiofsd/task.yaml).

Roads from here

RUNTIME · make rules and the CLI

BUILT ON IT · CI integrations

SNAP · snap-only features

CONTAINER · OCI image