Living with the distribution QEMU

no guestfwd patch for users for CI operators

A native install runs the qemu-system emulator make finds on PATH, written into each .run script, with its release's features and without the guestfwd unix socket patch; PKGPROXY_URL gives guests and downloads a pkgproxy anyway, and QEMU_SYSTEM_ARCH, QEMU_IMG and GARDEN_ARCH_PREFIX point at another QEMU.

On the map: native

▶ Watch this deck — 6 scenes, ~4 min

Worth seeing first

01 Whichever QEMU make finds

WHICHEVER QEMU MAKE FINDS $ image-garden make ubuntu-cloud-24.04.x86_64.run command -v qemu-system-x86_64 /usr/bin/qemu-system-x86_64 from PATH the provisioning boot qemu_cmd=("/usr/bin/qemu-system-x86_64") the .run script keeps this path none found: the script holds the bare name, looked up on every run qemu_cmd=("qemu-system-x86_64") the snap and the container: their own QEMU, one release, one patch
image-garden make looks up qemu-system-x86_64 on PATH and writes its full path into the provisioning boot and the .run script, which keeps it; with none found the script holds the bare name, looked up on every run; the snap and the container bring their own patched QEMU

make runs command -v qemu-system-x86_64 and writes the result into the provisioning command and the .run script. The snap and the container bring their own patched QEMU.

On a native install, which QEMU boots your guests? Whichever one make finds. When it builds ubuntu-cloud-24.04.x86_64, it looks up qemu-system-x86_64 on your PATH, takes its full path, and writes it into two places: the provisioning boot, and the .run script. So the script keeps that QEMU, even if you install another one later. If make found none, the script holds the bare name, and the shell looks it up each time you run it. The snap and the container don't ask your PATH at all. Each brings its own QEMU, built from one release, with one extra patch.

Details and sources

Details

  • Outside the snap, QEMU_SYSTEM_X86_64 ?= $(or $(shell command -v qemu-system-x86_64),$(warning program qemu-system-x86_64 is required)qemu-system-x86_64) (mk/010-qemu-x86_64.mk:3-4); in the snap it names the snap's qemu-system-x86-64 app. The other architectures do the same in their own mk/010-qemu-ARCH.mk.
  • The launcher gets qemu_cmd=("$(QEMU_SYSTEM_X86_64)") (mk/010-qemu-x86_64.mk:94), and the provisioning boot runs exec $(QEMU_SYSTEM_X86_64) through QEMU_SYSTEM_X86_64_EFI_CMDLINE (rulegen.py:593-606, mk/010-qemu-x86_64.mk:13-15). With no emulator on PATH, make -n showed qemu_cmd=("qemu-system-x86_64") (run with v0.8.0).
  • The path /usr/bin/qemu-system-x86_64 on the slide is an example: it is wherever command -v finds the emulator.
  • The README: packages and source use the "System-provided" QEMU; the snap and the container build one release from source (README.md:96-100), both with qemu-patches/guestfwd-unix-socket.patch (Dockerfile:29-32, snap/snapcraft.yaml:327-341).

02 pkgproxy without the patch

PKGPROXY WITHOUT THE PATCH a distribution QEMU no guestfwd unix: patch in the snap guest guestfwd 10.0.2.100:3142 pkgproxy.sock outside the snap, make does not look for the socket $ PKGPROXY_URL=http://pkgproxy.example.lan:3142 image-garden make ubuntu-cloud-24.04.x86_64.run guest QEMU's user network pkgproxy at its address any QEMU image-garden downloads
A distribution QEMU has no guestfwd unix socket patch; the socket path to the snap's pkgproxy is the snap's only and is not set up outside it; with PKGPROXY_URL the guest reaches pkgproxy at its address through QEMU's user network, and image-garden's downloads use the same pkgproxy, with any QEMU

Without the guestfwd unix socket patch there is no socket bridge, and outside the snap none is set up. PKGPROXY_URL=http://host:port works with any QEMU, for guests and for image downloads.

That patch is the first thing a distribution QEMU lacks. It lets guestfwd hand each guest connection to a unix socket, and that's how guests in the snap reach its bundled pkgproxy. Outside the snap, image-garden doesn't look for that socket, so you lose nothing you'd otherwise get. You reach pkgproxy by its address instead. Set PKGPROXY_URL to a pkgproxy you run, on your network or on this machine. make writes its address into each system it prepares, and the guest connects out through QEMU's ordinary user network. image-garden's own downloads go through the same pkgproxy. Any QEMU can do that.

Details and sources

Details

  • The patch adds the unix: form of guestfwd, one unix socket connection per guest connection (qemu-patches/guestfwd-unix-socket.patch:6-20); the guestfwd patch has the rest.
  • make looks for pkgproxy.sock in $SNAP_COMMON, or under it for pkgproxy=snap; a PKGPROXY_URL wins over any socket (mk/010-qemu.mk:20-44). Outside the snap SNAP_COMMON is unset unless you set it (derived).
  • With PKGPROXY_URL=http://host[:port], make resolves the host to IPv4 with getent ahostsv4 and the guest gets that address as pkgproxy; no guestfwd is added (mk/010-qemu.mk:24-30). "Outside the snap, as in image-garden's container, make PKGPROXY_URL=http://host:port ... does the same" (proxy/README.md:122-127).
  • Image downloads use PKGPROXY_URL too, unless GARDEN_DL_PKGPROXY is set (image-garden:31-38).
  • For a pkgproxy on the machine that runs the guests, localhost and 127.0.0.1 mean the guest itself; use the host's network address or 10.0.2.2, see an address that points back.
  • proxy/README.md's early-access note says the same: "The snap bundles pkgproxy and sets it up; outside the snap, as in image-garden's container, guests use a pkgproxy that PKGPROXY_URL names" (proxy/README.md:7-12).

03 Your release decides the rest

YOUR RELEASE DECIDES your distribution's release QEMU: older or newer than the bundled one which emulators, in which packages the EDK2 firmware what image-garden asks of QEMU -machine q35 -cpu max virtio-blk disks, virtio-net-pci network, virtio-rng-pci max: whatever this QEMU release offers $ qemu-system-x86_64 -accel help kvm listed: accel=kvm:tcg otherwise: accel=tcg asked on every start of a .run script
Your distribution's release decides the QEMU version, the emulators and their packages, and the firmware; image-garden asks for a machine such as q35, the max CPU and virtio devices; max means what that release offers; every start of a .run script asks -accel help and uses kvm:tcg or tcg

A distribution QEMU has its release's features, emulators and firmware. The .run script asks -accel help on every start and uses kvm:tcg or tcg.

The rest follows your distribution's release. Its QEMU may be older than the one the snap and the container build, or newer. Which emulators you can install, and in which packages, is your distribution's choice, and so is the firmware. What image-garden asks of QEMU is plain: a machine type such as q35, the CPU model called max, virtio disks and network. And max means whatever your QEMU's release knows how to offer. One thing is checked every time a script starts: it asks QEMU which accelerators it has, and picks kvm, with tcg to fall back on, or tcg alone.

Details and sources

Details

  • The README's comparison: QEMU version "System-provided" and architectures "System-dependent" for packages and source, one QEMU release built from source for the snap and the container (README.md:96-100); the container's comes from one pinned tarball (Dockerfile:5-6).
  • x86_64 asks for -machine q35 and -cpu max (mk/010-qemu-x86_64.mk:5-6), with virtio-blk, virtio-net-pci and virtio-rng-pci devices (mk/010-qemu-x86_64.mk:8-11, :23-25, mk/010-qemu.mk:45). The other machines are in architectures.
  • That -cpu max enables every CPU feature the QEMU release can offer is QEMU's documented meaning of the model (derived).
  • Every .run script computes default_qemu_accel_machine_prop by running the emulator with -accel help and choosing kvm:tcg when kvm is listed, tcg otherwise; QEMU_ACCEL_MACHINE_PROP overrides it (mk/010-qemu-x86_64.mk:63, :72).
  • Which emulators and firmware a distribution packages, and in which packages, is not in image-garden's tree (derived).

04 Pointing at another QEMU

ANOTHER QEMU the distribution's QEMU stays a newer one of your own, in /opt/qemu $ QEMU_SYSTEM_X86_64=/opt/qemu/bin/qemu-system-x86_64 \ QEMU_IMG=/opt/qemu/bin/qemu-img \ image-garden make ubuntu-cloud-24.04.x86_64.run GARDEN_X86_64_PREFIX=/opt/qemu \ + share/qemu/edk2-x86_64-code.fd, last in the list the container sets its prefixes to /usr/local, where its own QEMU is ubuntu-cloud-24.04.x86_64.run exists: keeps the old QEMU path a new variable does not make it out of date $ rm ubuntu-cloud-24.04.x86_64.run $ QEMU_SYSTEM_X86_64=… image-garden make ubuntu-cloud-24.04.x86_64.run
Your distribution's QEMU stays; QEMU_SYSTEM_X86_64, QEMU_IMG and GARDEN_X86_64_PREFIX point image-garden make at another one and its firmware, as the container does with /usr/local; an existing .run script keeps the old path until it is deleted and made again

QEMU_SYSTEM_X86_64=… QEMU_IMG=… GARDEN_X86_64_PREFIX=… image-garden make SYSTEM.run. An existing .run keeps its QEMU path until you delete it.

Maybe your distribution's QEMU is too old for a guest, or you've built a newer one yourself. You don't have to replace the system's. Set QEMU_SYSTEM_X86_64 to the emulator you want, and QEMU_IMG to its qemu-img, when you run image-garden make. Its firmware comes along with GARDEN_X86_64_PREFIX, which adds that prefix's EDK2 files to the end of the search list. The container does exactly this, with /usr/local. One catch: a .run script that already exists keeps the old path, because a new variable doesn't make it out of date. Delete the script, and the next make writes it again.

Details and sources

Details

  • The README's "Tool Paths": QEMU_SYSTEM_X86_64, QEMU_SYSTEM_AARCH64, QEMU_SYSTEM_RISCV64, QEMU_SYSTEM_S390X, QEMU_SYSTEM_PPC64, QEMU_SYSTEM_LOONGARCH64, QEMU_IMG, XORRISO, CURL, SWTPM (README.md:913-921). They are set with ?= (mk/010-qemu-x86_64.mk:3-4, mk/010-qemu.mk:3), so the environment or image-garden make VAR=… wins.
  • GARDEN_X86_64_PREFIX appends PREFIX/share/qemu/edk2-x86_64-code.fd and edk2-i386-vars.fd to the end of the x86_64 lists (mk/010-qemu-x86_64.mk:277-280); a distribution's OVMF earlier in the list still wins. aarch64, riscv64 and loongarch64 have their own prefix variables (UEFI firmware per system).
  • The container copies its QEMU to /usr/local and sets the prefixes there (Dockerfile:98-102); its emulators are found through PATH (derived: /usr/local/bin comes first in the image's default PATH).
  • The .run rule depends on $(MAKEFILE_LIST) and, order-only, on the image and firmware (rulegen.py:593-599); a changed variable does not make it out of date (derived from GNU make). Deleting the .run makes only the script again; the prepared .qcow2 stays.
  • The firmware copies use $(QEMU_IMG) as well (mk/010-qemu-x86_64.mk:325-329), and so does image-garden rebase when QEMU_IMG is in the environment (image-garden:970-980).
  • /opt/qemu on the slide is an example prefix.

05 What you are running

WHAT YOU ARE RUNNING $ grep qemu_cmd= ubuntu-cloud-24.04.x86_64.run qemu_cmd=("/usr/bin/qemu-system-x86_64") $ /usr/bin/qemu-system-x86_64 --version QEMU emulator version … $ grep -q slirp_add_unix /usr/bin/qemu-system-x86_64 && echo patched the smoke test's own check: only the patch calls it a distribution QEMU: no output the snap's QEMU: patched with PKGPROXY_URL it does not matter; it does for the snap's socket modes outside the snap
grep qemu_cmd= in the .run script shows the QEMU path; --version shows its release; grep -q slirp_add_unix on the binary finds nothing in a distribution QEMU and a match in the snap's; with PKGPROXY_URL it does not matter

grep qemu_cmd= SYSTEM.run shows the path, --version the release, and grep -q slirp_add_unix on the binary whether it has the patch.

To see which QEMU a system uses, open its .run script. The line that starts the QEMU command names the program, with its full path. Ask that program for its version with --version. And to know whether it carries the guestfwd patch, do what image-garden's own smoke test does: search the binary for the name slirp_add_unix. Only the patch calls it, so a distribution QEMU won't mention it, and the snap's will. With PKGPROXY_URL the answer doesn't matter. It matters only when you test the snap's socket modes outside the snap.

Details and sources

Details

  • Every launcher has one line qemu_cmd=("PATH-TO-EMULATOR") (mk/010-qemu-x86_64.mk:94).
  • image-garden's smoke test: "The patch is the only caller of slirp_add_unix in QEMU", grep -q slirp_add_unix "$qemu", and without it SKIP: $mode needs QEMU patched with qemu-patches/guestfwd-unix-socket.patch (tests/smoke-pkgproxy.sh:84-90).
  • That a QEMU linked to a shared libslirp carries the name of each function it imports, and so contains slirp_add_unix only when the patch calls it, is derived. The snap's QEMU is patched (snap/snapcraft.yaml:327-341).
  • The --version output line is QEMU's own (derived, not run).
  • The socket modes are made only by the snap; see what you lose without the patch.

06 Where to go from here

ROADS The distribution QEMU The guestfwd patch what it adds, who builds it Reaching pkgproxy the URL path, the right address Host tools what else your system supplies The container its own patched QEMU The snap QEMU components
Five roads from The distribution QEMU: the guestfwd patch, how a guest reaches pkgproxy, host tools, the container and the snap

Next: the guestfwd patch, how a guest reaches pkgproxy, host tools, the container, the snap and the architectures.

So a distribution QEMU is whichever one make finds, with your release's features and without the socket patch, and PKGPROXY_URL covers what the patch would give you. To see the patch itself, take the patch deck. The reaching deck shows the URL path, and the address to use for a pkgproxy on your own machine. The host tools deck lists the rest of what your system supplies. And the container and the snap decks show the two installs that bring a QEMU of their own.

Details and sources

Read more

Basis of the statements on this page

Read in the image-garden source at the pinned commit ("code"): mk/010-qemu*.mk, rulegen.py, Dockerfile, snap/snapcraft.yaml, tests/smoke-pkgproxy.sh and the README. What make -n writes into a .run script with and without an emulator on PATH was checked by running v0.8.0 from a source tree ("run"). What a distribution packages, and QEMU's own behaviour, are "derived".

The QEMU each install runs

InstallEmulatorPatchedFirmware
Distribution package or sourcecommand -v qemu-system-ARCH when make writes the rulesnothe distribution's packages
Container/usr/local/bin, built in the imageyesbuilt with QEMU, plus Ubuntu's ovmf
Snapthe image-garden.qemu-system-* apps, from the QEMU componentsyesin the QEMU components

Variables that choose a QEMU

VariableDefault outside the snapWhat it changes
QEMU_SYSTEM_X86_64 (and _AARCH64, _RISCV64, _S390X, _PPC64, _LOONGARCH64)the emulator on PATHthe provisioning boot and the .run script
QEMU_IMGqemu-img on PATHoverlays, unpacked images and firmware copies; rebase reads it from the environment
GARDEN_X86_64_PREFIX (and _AARCH64, _RISCV64, _LOONGARCH64)unsetappends PREFIX/share/qemu/edk2-* to the firmware lists
QEMU_ACCEL_MACHINE_PROP,accel=kvm:tcg or ,accel=tcg, asked on every startthe accelerator, per run

Quick checks

grep qemu_cmd= ubuntu-cloud-24.04.x86_64.run
/usr/bin/qemu-system-x86_64 --version
grep -q slirp_add_unix /usr/bin/qemu-system-x86_64 && echo patched

The last line is the check image-garden's smoke test makes before it runs the socket modes (tests/smoke-pkgproxy.sh:84-90).

Roads from here

ARCHITECTURES · machines and firmware

PKGPROXY · caching for guests

SNAP · snap-only features

CONTAINER · OCI image

NATIVE · packages