Living with the distribution QEMU
no guestfwd patch for users for CI operators
A native install runs the qemu-system emulator make finds on PATH, written into each .run script, with its release's features and without the guestfwd unix socket patch; PKGPROXY_URL gives guests and downloads a pkgproxy anyway, and QEMU_SYSTEM_ARCH, QEMU_IMG and GARDEN_ARCH_PREFIX point at another QEMU.
On the map: native
▶ Watch this deck — 6 scenes, ~4 min
Worth seeing first
- What a native install needs from the host — the emulators and firmware a native host supplies
01 Whichever QEMU make finds
make runs command -v qemu-system-x86_64 and writes the result into the provisioning command and the .run script. The snap and the container bring their own patched QEMU.
On a native install, which QEMU boots your guests? Whichever one make finds. When it builds ubuntu-cloud-24.04.x86_64, it looks up qemu-system-x86_64 on your PATH, takes its full path, and writes it into two places: the provisioning boot, and the .run script. So the script keeps that QEMU, even if you install another one later. If make found none, the script holds the bare name, and the shell looks it up each time you run it. The snap and the container don't ask your PATH at all. Each brings its own QEMU, built from one release, with one extra patch.
Details and sources
Details
- Outside the snap,
QEMU_SYSTEM_X86_64 ?= $(or $(shell command -v qemu-system-x86_64),$(warning program qemu-system-x86_64 is required)qemu-system-x86_64)(mk/010-qemu-x86_64.mk:3-4); in the snap it names the snap'sqemu-system-x86-64app. The other architectures do the same in their ownmk/010-qemu-ARCH.mk. - The launcher gets
qemu_cmd=("$(QEMU_SYSTEM_X86_64)")(mk/010-qemu-x86_64.mk:94), and the provisioning boot runsexec $(QEMU_SYSTEM_X86_64)throughQEMU_SYSTEM_X86_64_EFI_CMDLINE(rulegen.py:593-606, mk/010-qemu-x86_64.mk:13-15). With no emulator onPATH,make -nshowedqemu_cmd=("qemu-system-x86_64")(run with v0.8.0). - The path
/usr/bin/qemu-system-x86_64on the slide is an example: it is wherevercommand -vfinds the emulator. - The README: packages and source use the "System-provided" QEMU; the snap and the container build one release from source (README.md:96-100), both with
qemu-patches/guestfwd-unix-socket.patch(Dockerfile:29-32, snap/snapcraft.yaml:327-341).
Read more
02 pkgproxy without the patch
Without the guestfwd unix socket patch there is no socket bridge, and outside the snap none is set up. PKGPROXY_URL=http://host:port works with any QEMU, for guests and for image downloads.
That patch is the first thing a distribution QEMU lacks. It lets guestfwd hand each guest connection to a unix socket, and that's how guests in the snap reach its bundled pkgproxy. Outside the snap, image-garden doesn't look for that socket, so you lose nothing you'd otherwise get. You reach pkgproxy by its address instead. Set PKGPROXY_URL to a pkgproxy you run, on your network or on this machine. make writes its address into each system it prepares, and the guest connects out through QEMU's ordinary user network. image-garden's own downloads go through the same pkgproxy. Any QEMU can do that.
Details and sources
Details
- The patch adds the
unix:form ofguestfwd, one unix socket connection per guest connection (qemu-patches/guestfwd-unix-socket.patch:6-20); the guestfwd patch has the rest. - make looks for
pkgproxy.sockin$SNAP_COMMON, or under it forpkgproxy=snap; aPKGPROXY_URLwins over any socket (mk/010-qemu.mk:20-44). Outside the snapSNAP_COMMONis unset unless you set it (derived). - With
PKGPROXY_URL=http://host[:port], make resolves the host to IPv4 withgetent ahostsv4and the guest gets that address aspkgproxy; noguestfwdis added (mk/010-qemu.mk:24-30). "Outside the snap, as in image-garden's container,make PKGPROXY_URL=http://host:port ...does the same" (proxy/README.md:122-127). - Image downloads use
PKGPROXY_URLtoo, unlessGARDEN_DL_PKGPROXYis set (image-garden:31-38). - For a pkgproxy on the machine that runs the guests,
localhostand127.0.0.1mean the guest itself; use the host's network address or10.0.2.2, see an address that points back. - proxy/README.md's early-access note says the same: "The snap bundles pkgproxy and sets it up; outside the snap, as in image-garden's container, guests use a pkgproxy that
PKGPROXY_URLnames" (proxy/README.md:7-12).
03 Your release decides the rest
A distribution QEMU has its release's features, emulators and firmware. The .run script asks -accel help on every start and uses kvm:tcg or tcg.
The rest follows your distribution's release. Its QEMU may be older than the one the snap and the container build, or newer. Which emulators you can install, and in which packages, is your distribution's choice, and so is the firmware. What image-garden asks of QEMU is plain: a machine type such as q35, the CPU model called max, virtio disks and network. And max means whatever your QEMU's release knows how to offer. One thing is checked every time a script starts: it asks QEMU which accelerators it has, and picks kvm, with tcg to fall back on, or tcg alone.
Details and sources
Details
- The README's comparison: QEMU version "System-provided" and architectures "System-dependent" for packages and source, one QEMU release built from source for the snap and the container (README.md:96-100); the container's comes from one pinned tarball (Dockerfile:5-6).
- x86_64 asks for
-machine q35and-cpu max(mk/010-qemu-x86_64.mk:5-6), withvirtio-blk,virtio-net-pciandvirtio-rng-pcidevices (mk/010-qemu-x86_64.mk:8-11, :23-25, mk/010-qemu.mk:45). The other machines are in architectures. - That
-cpu maxenables every CPU feature the QEMU release can offer is QEMU's documented meaning of the model (derived). - Every
.runscript computesdefault_qemu_accel_machine_propby running the emulator with-accel helpand choosingkvm:tcgwhenkvmis listed,tcgotherwise;QEMU_ACCEL_MACHINE_PROPoverrides it (mk/010-qemu-x86_64.mk:63, :72). - Which emulators and firmware a distribution packages, and in which packages, is not in image-garden's tree (derived).
Read more
04 Pointing at another QEMU
QEMU_SYSTEM_X86_64=… QEMU_IMG=… GARDEN_X86_64_PREFIX=… image-garden make SYSTEM.run. An existing .run keeps its QEMU path until you delete it.
Maybe your distribution's QEMU is too old for a guest, or you've built a newer one yourself. You don't have to replace the system's. Set QEMU_SYSTEM_X86_64 to the emulator you want, and QEMU_IMG to its qemu-img, when you run image-garden make. Its firmware comes along with GARDEN_X86_64_PREFIX, which adds that prefix's EDK2 files to the end of the search list. The container does exactly this, with /usr/local. One catch: a .run script that already exists keeps the old path, because a new variable doesn't make it out of date. Delete the script, and the next make writes it again.
Details and sources
Details
- The README's "Tool Paths":
QEMU_SYSTEM_X86_64,QEMU_SYSTEM_AARCH64,QEMU_SYSTEM_RISCV64,QEMU_SYSTEM_S390X,QEMU_SYSTEM_PPC64,QEMU_SYSTEM_LOONGARCH64,QEMU_IMG,XORRISO,CURL,SWTPM(README.md:913-921). They are set with?=(mk/010-qemu-x86_64.mk:3-4, mk/010-qemu.mk:3), so the environment orimage-garden make VAR=…wins. GARDEN_X86_64_PREFIXappendsPREFIX/share/qemu/edk2-x86_64-code.fdandedk2-i386-vars.fdto the end of the x86_64 lists (mk/010-qemu-x86_64.mk:277-280); a distribution's OVMF earlier in the list still wins. aarch64, riscv64 and loongarch64 have their own prefix variables (UEFI firmware per system).- The container copies its QEMU to
/usr/localand sets the prefixes there (Dockerfile:98-102); its emulators are found throughPATH(derived:/usr/local/bincomes first in the image's defaultPATH). - The
.runrule depends on$(MAKEFILE_LIST)and, order-only, on the image and firmware (rulegen.py:593-599); a changed variable does not make it out of date (derived from GNU make). Deleting the.runmakes only the script again; the prepared.qcow2stays. - The firmware copies use
$(QEMU_IMG)as well (mk/010-qemu-x86_64.mk:325-329), and so doesimage-garden rebasewhenQEMU_IMGis in the environment (image-garden:970-980). /opt/qemuon the slide is an example prefix.
05 What you are running
grep qemu_cmd= SYSTEM.run shows the path, --version the release, and grep -q slirp_add_unix on the binary whether it has the patch.
To see which QEMU a system uses, open its .run script. The line that starts the QEMU command names the program, with its full path. Ask that program for its version with --version. And to know whether it carries the guestfwd patch, do what image-garden's own smoke test does: search the binary for the name slirp_add_unix. Only the patch calls it, so a distribution QEMU won't mention it, and the snap's will. With PKGPROXY_URL the answer doesn't matter. It matters only when you test the snap's socket modes outside the snap.
Details and sources
Details
- Every launcher has one line
qemu_cmd=("PATH-TO-EMULATOR")(mk/010-qemu-x86_64.mk:94). - image-garden's smoke test: "The patch is the only caller of slirp_add_unix in QEMU",
grep -q slirp_add_unix "$qemu", and without itSKIP: $mode needs QEMU patched with qemu-patches/guestfwd-unix-socket.patch(tests/smoke-pkgproxy.sh:84-90). - That a QEMU linked to a shared libslirp carries the name of each function it imports, and so contains
slirp_add_unixonly when the patch calls it, is derived. The snap's QEMU is patched (snap/snapcraft.yaml:327-341). - The
--versionoutput line is QEMU's own (derived, not run). - The socket modes are made only by the snap; see what you lose without the patch.
Read more
06 Where to go from here
Next: the guestfwd patch, how a guest reaches pkgproxy, host tools, the container, the snap and the architectures.
So a distribution QEMU is whichever one make finds, with your release's features and without the socket patch, and PKGPROXY_URL covers what the patch would give you. To see the patch itself, take the patch deck. The reaching deck shows the URL path, and the address to use for a pkgproxy on your own machine. The host tools deck lists the rest of what your system supplies. And the container and the snap decks show the two installs that bring a QEMU of their own.
Details and sources
Read more
Basis of the statements on this page
Read in the image-garden source at the pinned commit ("code"): mk/010-qemu*.mk, rulegen.py, Dockerfile, snap/snapcraft.yaml, tests/smoke-pkgproxy.sh and the README. What make -n writes into a .run script with and without an emulator on PATH was checked by running v0.8.0 from a source tree ("run"). What a distribution packages, and QEMU's own behaviour, are "derived".
The QEMU each install runs
| Install | Emulator | Patched | Firmware |
|---|---|---|---|
| Distribution package or source | command -v qemu-system-ARCH when make writes the rules | no | the distribution's packages |
| Container | /usr/local/bin, built in the image | yes | built with QEMU, plus Ubuntu's ovmf |
| Snap | the image-garden.qemu-system-* apps, from the QEMU components | yes | in the QEMU components |
Variables that choose a QEMU
| Variable | Default outside the snap | What it changes |
|---|---|---|
QEMU_SYSTEM_X86_64 (and _AARCH64, _RISCV64, _S390X, _PPC64, _LOONGARCH64) | the emulator on PATH | the provisioning boot and the .run script |
QEMU_IMG | qemu-img on PATH | overlays, unpacked images and firmware copies; rebase reads it from the environment |
GARDEN_X86_64_PREFIX (and _AARCH64, _RISCV64, _LOONGARCH64) | unset | appends PREFIX/share/qemu/edk2-* to the firmware lists |
QEMU_ACCEL_MACHINE_PROP | ,accel=kvm:tcg or ,accel=tcg, asked on every start | the accelerator, per run |
Quick checks
grep qemu_cmd= ubuntu-cloud-24.04.x86_64.run /usr/bin/qemu-system-x86_64 --version grep -q slirp_add_unix /usr/bin/qemu-system-x86_64 && echo patched
The last line is the check image-garden's smoke test makes before it runs the socket modes (tests/smoke-pkgproxy.sh:84-90).
Roads from here
ARCHITECTURES · machines and firmware
- Six architectures, six QEMU machines — the machine and accelerator each emulator runs with
PKGPROXY · caching for guests
- The guestfwd unix socket patch — the patch a distribution QEMU lacks, and who builds with it
- How a guest reaches pkgproxy — the URL path, and the address to give a pkgproxy on your own machine
SNAP · snap-only features
- The image-garden snap — the snap, whose QEMU components carry the patch
CONTAINER · OCI image
- Running image-garden in a container — the container, which brings a patched QEMU into any runner
NATIVE · packages
- What a native install needs from the host — everything else your host supplies