The guestfwd unix socket patch

TCP in, unix socket out for contributors

Stock QEMU's guestfwd can start a command or open one shared character device, but not a fresh unix socket connection per guest connection. image-garden's qemu-patches/guestfwd-unix-socket.patch adds that unix: form with libslirp's slirp_add_unix; the snap and the container build their QEMU with it, a distribution QEMU lacks it, and a URL works without it.

On the map: pkgproxy

▶ Watch this deck — 7 scenes, ~5 min

Worth seeing first

01 TCP in, unix socket out

TCP IN, UNIX SOCKET OUT pkgproxy, in the snap unix: pkgproxy.sock the guest TCP 10.0.2.100:3142 QEMU user network ? guestfwd-unix-socket.patch what it adds who builds with it what you lose without it
The guest speaks TCP to 10.0.2.100:3142; the snap's pkgproxy listens on the unix socket pkgproxy.sock. QEMU's user network sits between them, and with guestfwd-unix-socket.patch it joins each guest connection to the socket. The deck covers what the patch adds, who builds with it and what is lost without it.

The guest speaks TCP to 10.0.2.100:3142; the snap's pkgproxy listens on a unix socket. qemu-patches/guestfwd-unix-socket.patch lets QEMU join the two.

In the snap, pkgproxy doesn't listen on a TCP port. It listens on a unix socket, a file in the snap's data directory. The guest, though, only speaks TCP: it connects to 10.0.2.100, port 3142, and expects someone to answer. Something has to join the two, once for every connection the guest opens. QEMU's user network almost can, and that gap is where image-garden steps in. It carries one small patch for QEMU, so that its guestfwd option can send each guest connection straight to a unix socket. This deck is about that patch: what it adds, who builds with it, and what you lose without it.

Details and sources

Details

Read more

02 What stock QEMU offers

STOCK GUESTFWD TARGETS …-cmd:nc -U SOCKET nc -U nc -U nc -U pkgproxy.sock the old bridge: a process per connection, one more program …-unix:SOCKET (stock) chardev pkgproxy.sock one every guest connection shares one host connection: wrong for HTTP wanted pkgproxy.sock a fresh socket connection for each guest connection
Stock guestfwd targets. cmd: starts one process per guest connection, as the old nc -U bridge did: a process per connection and one more program. A bare unix path becomes a character device: one host connection shared by every guest connection, wrong for HTTP. Wanted: one fresh socket connection per guest connection.

cmd: starts a process per connection (the old nc -U bridge); a chardev shares one host connection between all guest connections. Neither gives one socket connection per guest connection.

Stock QEMU's guestfwd has two kinds of target. The first is a command: QEMU starts a process for every guest connection and passes the traffic through it. image-garden used to do exactly that, starting nc for each connection to reach the socket. It works, but it's one more program to ship, and a new process per connection. The second is a character device, and a bare unix path ends up there. But a character device is a single connection, opened once and shared by every guest connection. For HTTP, where requests run side by side, that's wrong. What's missing is in between: a fresh socket connection for each guest connection.

Details and sources

Details

  • From QEMU's command line, "only cmd:command (spawn a process per connection) and a generic chardev form (one connection shared across *all* guest connections — wrong for HTTP) were reachable" (proxy/README.md:25-31).
  • "Before this patch existed, the bridge instead spawned nc -U per connection via guestfwd=...-cmd:...; the patch removes that external-process dependency entirely" (proxy/README.md:34-36).
  • The patch's own header: a bare unix:/path already falls through to the chardev path (qemu_chr_new_mux_mon), which "opens one chardev shared across the whole config-line's lifetime, multiplexing every guest TCP connection onto a single host-side connection — wrong for a protocol like HTTP where concurrent requests need independent connections" (guestfwd-unix-socket.patch:15-20).
  • SOCKET on the slide stands for the socket's path.

03 Six lines in QEMU's network code for contributors

THE PATCH libslirp slirp_add_unix() QEMU's guestfwd parser net/slirp.c never called @@ -1153,6 +1153,12 @@ + } else if (g_str_has_prefix(p, "unix:")) { + if (slirp_add_unix(s->slirp, &p[5], &server, port) < 0) { + error_setg(errp, "Conflicting/invalid host:port in guest " + "forwarding rule '%s'", config_str); + return -1; + } } else { net/slirp.c the path after "unix:" on failure: the same error as the cmd: form gives

A unix: branch in net/slirp.c passes the path, guest address and port to libslirp's slirp_add_unix(), which connects afresh for every guest connection.

The fix is small because the hard part already exists. libslirp, the library behind QEMU's user network, has a function for exactly this: slirp_add_unix, which opens a new connection to a unix socket for each guest connection. QEMU's option parser never called it. The patch adds one branch to the file net/slirp.c, after the command form. When the target starts with unix and a colon, the rest is the socket path, and the branch hands it to slirp_add_unix with the guest address and port. If that fails, you get the same error the command form gives: conflicting or invalid host and port in the guest forwarding rule.

Details and sources

Details

  • The patch's header: "Add a "unix:" form to -netdev user's guestfwd=, forwarding each new guest TCP connection to a fresh AF_UNIX connection to the given host socket path (one connection per guest connection, same semantics as the existing "cmd:" exec form). libslirp already implements exactly this via slirp_add_unix()/open_unix() … it's just never wired up from QEMU's own guestfwd= option parser" (guestfwd-unix-socket.patch:6-13).
  • The hunk adds six lines to net/slirp.c at line 1153 of the QEMU source the snap and the container build, between the cmd: branch and the chardev fallback (guestfwd-unix-socket.patch:22-36). &p[5] is the text after unix:, the socket path; server and port are, by libslirp's slirp_add_unix() signature, the guest address and port from tcp:10.0.2.100:3142 (derived: QEMU's net/slirp.c and libslirp are not in the trees this map reads).
  • The error text is the same as the cmd: branch's, shown as the context line above the hunk (guestfwd-unix-socket.patch:24-27).
  • The patch carries an SPDX header (Apache-2.0) and an author line; it is applied with patch -p1 from the QEMU source directory.

04 Who builds QEMU with it

WHO BUILDS WITH IT the snap, snap/snapcraft.yaml qemu: source: https://download.qemu.org/… override-pull: | craftctl default patch -p1 < …/guestfwd-unix-socket.patch the container, Dockerfile FROM ubuntu:latest AS build-qemu ADD --checksum=sha256:5a82… … COPY qemu-patches/…patch /src/ RUN patch -d /src/qemu-… -p1 <… RUN /src/qemu-*/configure … qemu-patches/guestfwd-unix-socket.patch a distribution's QEMU, or one you build yourself: no unix: form
The snap's qemu part runs craftctl default and then patch -p1 with guestfwd-unix-socket.patch in override-pull; the container's build-qemu stage downloads the same tarball by checksum, copies the patch in and runs patch before configure. Both take qemu-patches/guestfwd-unix-socket.patch from the image-garden tree. A distribution's QEMU, or one built without the patch, lacks it.

The snap's qemu part and the container's build-qemu stage both run patch -p1 with qemu-patches/guestfwd-unix-socket.patch. A distribution's QEMU lacks it.

Two of image-garden's installs build their own QEMU, and both apply the patch. The snap builds QEMU once, from the release tarball, and its QEMU part runs the patch right after pulling the source, so every QEMU component of the snap has it. The container's build stage downloads the same tarball, checked against the same hash, copies the patch in and applies it before configuring. Both take it from the same place, the qemu-patches directory of the image-garden tree. A QEMU from your distribution doesn't have it, and neither does one you build yourself, unless you apply the patch too.

Details and sources

Details

  • The snap builds QEMU once, in the qemu part, from the release tarball with a pinned source-checksum; override-pull runs craftctl default and then patch -p1 < "${CRAFT_PROJECT_DIR}"/qemu-patches/guestfwd-unix-socket.patch (snapcraft.yaml:327-341). The patch is "applied to all bundled QEMU system-emulator parts" (proxy/README.md:32-33).
  • The container's build-qemu stage adds the same tarball with ADD --checksum=sha256:5a8289c7… (Dockerfile:5-6), then "The same patch as the snap's QEMU", COPY qemu-patches/guestfwd-unix-socket.patch /src/ and RUN patch -d /src/qemu-11.1.3 -p1 </src/guestfwd-unix-socket.patch, before configure (Dockerfile:28-36). The container's NEWS records it under 0.7.0 (NEWS.docker:18-21).
  • The snap and the container build the same QEMU release from the same checksum (5a8289c7… in both files).
  • A distribution's QEMU is built from QEMU's own sources without this patch (derived: the patch is image-garden's and not part of QEMU).

05 What you lose without it

WITHOUT THE PATCH the snap socket modes, guestfwd needs the patch: has it native install, container PKGPROXY_URL any QEMU will do where it matters: the socket modes, tested outside the snap # The patch is the only caller of slirp_add_unix in QEMU. if [ -z "$qemu" ] || ! grep -q slirp_add_unix "$qemu"; then echo "SKIP: $mode needs QEMU patched with …" >&2 exit 77 fi tests/smoke-pkgproxy.sh exit 77: the socket modes are skipped, not failed
The snap uses the socket modes, which need its patched QEMU. A native install or the container names pkgproxy with PKGPROXY_URL, which works with any QEMU. Only a test of the socket modes outside the snap needs the patch: tests/smoke-pkgproxy.sh greps the QEMU binary for slirp_add_unix and exits 77, skipped, when it is missing.

Outside the snap, pkgproxy is reached by URL, which needs no patch. The smoke test skips the socket modes when the QEMU binary does not mention slirp_add_unix.

So what does a missing patch cost you? Less than you might think. The socket modes belong to the snap: make looks for a socket only where SNAP_COMMON points, and outside the snap nothing sets that for you. A native install, or the container, names a pkgproxy with PKGPROXY_URL instead, and a URL works with any QEMU, because the guest simply connects out. Where it matters is a test that runs the socket modes outside the snap. Only the patch calls slirp_add_unix, so image-garden's smoke test looks for that name in the QEMU binary, and skips the socket modes when it isn't there.

Details and sources

Details

  • make looks for pkgproxy.sock in $SNAP_COMMON or $SNAP_COMMON/pkgproxy-host-snap/sockets (010-qemu.mk:20-23, :31). snapd sets SNAP_COMMON for the snap's apps; outside the snap it is unset unless you set it, as image-garden's own tests do with a fixture directory (derived from check-pkgproxy-config.sh:103-120).
  • "Outside the snap, as in image-garden's container, make PKGPROXY_URL=http://host:port ... does the same, and so does PKGPROXY_URL in the environment" (proxy/README.md:122-127). With a URL make adds no guestfwd (010-qemu.mk:24-30).
  • The smoke test, for component and snap: "The patch is the only caller of slirp_add_unix in QEMU", grep -q slirp_add_unix "$qemu", else SKIP: $mode needs QEMU patched with qemu-patches/guestfwd-unix-socket.patch; … is not and exit 77 (smoke-pkgproxy.sh:84-90). A QEMU linked to a shared libslirp names the function it imports, so the name is in the binary only when something calls it (derived).
  • Given a socket path on an unpatched QEMU, unix:PATH falls through to the chardev path: one host connection shared by every guest connection, per the patch's description (guestfwd-unix-socket.patch:15-20). image-garden itself sets up the socket modes only in the snap, whose QEMU has the patch.
  • proxy/README.md's "Early-Access" section says the same: "The snap bundles pkgproxy and sets it up; outside the snap, as in image-garden's container, guests use a pkgproxy that PKGPROXY_URL names" (proxy/README.md:7-12).

06 How the bridge is tested for contributors

CHEAP TO THOROUGH cheap thorough check-pkgproxy-config.sh make per mode, files as sockets component: …3142-unix:… + lxd: …3143-unix:… snap: …/sockets/… url, none: (nothing) boots nothing, no patch needed smoke-pkgproxy.sh pkgproxy and one guest downloads land in the cache socket modes: patched QEMU tests/snap/ (spread) the smoke test, every mode nested in a VM
Three tests from cheap to thorough. tests/check-pkgproxy-config.sh runs make per mode with fixture sockets and compares the guestfwd options, booting nothing and needing no patched QEMU. tests/smoke-pkgproxy.sh starts pkgproxy and boots one guest in a mode and checks its downloads land in the cache. The snap's spread suite, tests/snap/, runs the smoke test in every mode nested in a VM.

tests/check-pkgproxy-config.sh checks the guestfwd options without booting; tests/smoke-pkgproxy.sh MODE SYSTEM.ARCH boots a guest per mode; the snap's spread suite runs the smoke test nested.

Three tests keep the bridge honest, from cheap to thorough. The config check runs make for every mode, with plain files standing in for pkgproxy's sockets, and compares the guestfwd options it gets: one rule for the bundled socket, a second for the LXD one when it's there, the pkgproxy snap's sockets in snap mode, and nothing for a URL. It boots nothing, and needs no patched QEMU. The smoke test does boot: it starts pkgproxy, brings up one guest in the chosen mode, and checks that what the guest downloads lands in the cache. And the snap's spread suite runs that smoke test in every mode, nested in a VM.

Details and sources

Details

  • tests/check-pkgproxy-config.sh "Check[s] what image-garden generates for each way of choosing a pkgproxy, without booting anything", with $SNAP_COMMON a fixture directory and "plain files standing in for pkgproxy's sockets" (check-pkgproxy-config.sh:5-9). Its cases: component with and without the LXD socket, component down, snap, snap disconnected, external with and without a port, an unresolvable URL, PKGPROXY_URL from the environment, none and nothing (check-pkgproxy-config.sh:115-170). It also checks the user-data of six families with tests/check-pkgproxy-user-data.py.
  • tests/smoke-pkgproxy.sh MODE SYSTEM.ARCH, MODE one of component, snap, url, none, runs pkgproxy on the host and boots one VM (smoke-pkgproxy.sh:5-23); "It checks the guest's setup, and that what the guest downloads ends up in pkgproxy's cache, or, for none, does not" (proxy/README.md:247-253).
  • The snap's spread suite in tests/snap/ runs tests/smoke-pkgproxy.sh with SMOKE_SNAP=1 for each mode, "with the guest nested in the VM"; it runs only when named (proxy/README.md:261-274).
  • "CI runs them before the slow tests of every system" (proxy/README.md:235-238).

Read more

07 Where to go from here

ROADS The guestfwd patch Reaching pkgproxy a socket or a URL A distribution QEMU no patch, PKGPROXY_URL The container patched QEMU, a URL The snap QEMU components
Four roads from The guestfwd patch: how a guest reaches pkgproxy, a distribution QEMU, the container's patched QEMU with PKGPROXY_URL, and the snap's QEMU components

Next: reaching pkgproxy, a distribution QEMU, the container, the snap and why pkgproxy at all.

That's the whole patch: six lines that let QEMU hand each guest connection to a unix socket, carried by the snap and the container. To see where the bridge sits among the ways a guest finds pkgproxy, take the reaching deck. If you run a distribution's QEMU, the native QEMU deck says what else differs from the bundled one. The container deck shows its patched QEMU and PKGPROXY_URL at work, and the snap overview shows the QEMU components that carry the patch.

Details and sources

Read more

Basis of the statements on this page

Read in qemu-patches/guestfwd-unix-socket.patch, snap/snapcraft.yaml, Dockerfile, mk/010-qemu.mk, proxy/README.md and tests/ of image-garden at the v0.8.0 tag ("code"). QEMU's net/slirp.c and libslirp are not in the trees this map reads: what the slides say about them comes from the patch's description and proxy/README.md, and the rest is labelled "derived".

guestfwd targets

Target after tcp:ADDR:PORT-In stock QEMUWith the patchHost side
cmd:COMMANDyesyesa new process per guest connection
unix:PATHread as a chardevthe patch's brancha new socket connection per guest connection
a chardevyesyesone connection shared by every guest connection

The snap's .run scripts use guestfwd=tcp:10.0.2.100:3142-unix:$SNAP_COMMON/pkgproxy.sock, and …:3143-unix:…/pkgproxy-lxd-tls.sock when that socket exists (010-qemu.mk:31-39).

The patch

--- a/net/slirp.c
+++ b/net/slirp.c
@@ -1153,6 +1153,12 @@
                        "forwarding rule '%s'", config_str);
             return -1;
         }
+    } else if (g_str_has_prefix(p, "unix:")) {
+        if (slirp_add_unix(s->slirp, &p[5], &server, port) < 0) {
+            error_setg(errp, "Conflicting/invalid host:port in guest "
+                       "forwarding rule '%s'", config_str);
+            return -1;
+        }
     } else {

Source: guestfwd-unix-socket.patch:22-36.

Which QEMU has it

InstallQEMUPatchpkgproxy reached by
the snapbuilt in the qemu part, one build for every QEMU componentapplied in override-pull (snapcraft.yaml:327-341)a socket (the default) or a URL
the containerbuilt in the build-qemu stage, installed in /usr/localapplied before configure (Dockerfile:28-32)PKGPROXY_URL
native packages, a source installthe distribution'snot appliedPKGPROXY_URL

Checking a QEMU binary

The smoke test's check, usable by hand: grep -q slirp_add_unix "$(command -v qemu-system-x86_64)" succeeds only for a patched QEMU, since "The patch is the only caller of slirp_add_unix in QEMU" (smoke-pkgproxy.sh:84-90).

Roads from here

PKGPROXY · caching for guests

SNAP · snap-only features

CONTAINER · OCI image

NATIVE · packages