How a guest reaches pkgproxy

10.0.2.100 or a URL for CI operators

Guests ask for the name pkgproxy, and make fixes what it means when it prepares a system. A PKGPROXY_URL wins and becomes the host's IPv4 address in /etc/hosts; otherwise an existing pkgproxy socket is bridged by QEMU's guestfwd from 10.0.2.100 port 3142; otherwise guests get none.

On the map: pkgproxy

▶ Watch this deck — 7 scenes, ~5 min

Worth seeing first

01 One name, two paths

ONE NAME, TWO PATHS the guest's private network ubuntu-cloud-24.04.x86_64 http://pkgproxy:3142 every guest asks for this name; make decides what it means QEMU guestfwd pkgproxy.sock the snap's own pkgproxy, on the host user network pkgproxy at a URL like any other connection
The guest ubuntu-cloud-24.04.x86_64 sits on QEMU's private user network and asks for http://pkgproxy:3142. One path goes through QEMU's guestfwd to the snap pkgproxy's unix socket on the host; the other goes out through the user network to a pkgproxy at a URL.

Every guest asks for pkgproxy. make decides what the name means: a QEMU guestfwd to a unix socket on the host, or the address of a pkgproxy at a URL.

Your guest lives on a small private network that QEMU makes up for it. It can reach out to the internet, but nothing on your host listens on that network. So how does it find pkgproxy? With a name. Every guest asks for pkgproxy, and make decides what that name means when it prepares ubuntu-cloud-24.04.x86_64. There are two answers. With the snap's own pkgproxy, the name leads through a forward inside QEMU to a unix socket on the host. With a pkgproxy at a URL, it leads to that machine's address, like any other connection the guest makes.

Details and sources

Details

  • Guests reach pkgproxy as the host name pkgproxy on port 3142 for the sockets, or the URL's port; make chooses the address when it makes a system's files (010-qemu.mk:15-44).
  • The socket path: "QEMU's guestfwd forwards guest TCP connections to 10.0.2.100 on port 3142 straight to that UNIX socket" (proxy/README.md:16-23).
  • The URL path: the host is resolved to an IPv4 address when an instance is built, and guests get it as pkgproxy in /etc/hosts (proxy/README.md:117-120). "Whichever pkgproxy it is, guests know it as pkgproxy" (proxy/README.md:129-130).
  • QEMU's user network (-netdev user, libslirp) gives the guest a private address range and translates its outgoing connections into connections made by the QEMU process on the host (derived: general QEMU behaviour).

02 make picks the path, in a fixed order

IN A FIXED ORDER 1. A URL? PKGPROXY_URL=http://host[:port] snap set image-garden pkgproxy=http://… yes the guest connects to the URL's host no 2. A pkgproxy socket that exists? component: $SNAP_COMMON/pkgproxy.sock snap: …/pkgproxy-host-snap/sockets/ yes guestfwd from 10.0.2.100:3142 no 3. Neither no pkgproxy: straight to the mirrors
make asks three questions in order. A URL, from PKGPROXY_URL or from snap set image-garden pkgproxy=http://…, wins and the guest connects to its address. Otherwise a pkgproxy socket that exists, the bundled one or the pkgproxy snap's, is bridged with guestfwd from 10.0.2.100:3142. Otherwise the guest gets no pkgproxy and downloads straight from the mirrors.

A URL, from PKGPROXY_URL or snap set image-garden pkgproxy=http://…, wins. Else an existing pkgproxy socket is bridged. Else the guest gets no pkgproxy.

Which path you get is settled by make, in a fixed order. First, a URL. Outside the snap, that's the PKGPROXY_URL variable, from the environment or the make command line. In the snap, it's the URL you gave to snap set, which the configure hook saves for make to read. A URL always wins. Without one, make looks for a socket: the bundled pkgproxy's, or the pkgproxy snap's when you chose that mode. The socket has to exist right then, while make writes the files. And when there's neither, the guest simply gets no pkgproxy, and downloads everything straight from the mirrors.

Details and sources

Details

  • In the snap, the configure hook turns snap set image-garden pkgproxy=… into a mode (component, snap, external or none) and, for a URL, checks it against ^http://[A-Za-z0-9.-]+(:[0-9]+)?/?$ (configure:17-39). It writes the mode to $SNAP_COMMON/pkgproxy-mode and the URL to $SNAP_COMMON/pkgproxy-url (configure:76-82).
  • make reads the mode, and the URL only when the mode is external; PKGPROXY_URL ?= … means a PKGPROXY_URL from the environment or the command line takes precedence over the saved URL (010-qemu.mk:20-21). That a PKGPROXY_URL in the snap's environment also overrides the component, snap and none modes follows from the same line (derived, not run).
  • The socket directory is $SNAP_COMMON, or $SNAP_COMMON/pkgproxy-host-snap/sockets in snap mode, where snapd mounts the pkgproxy snap's sockets slot (010-qemu.mk:22-23). The bridge is set up only when pkgproxy.sock exists there (010-qemu.mk:31-39).
  • Without either: "No pkgproxy: guests get none." (010-qemu.mk:40-44).
  • Outside the snap $SNAP_COMMON is empty, so make looks for /pkgproxy-mode and /pkgproxy.sock at the file system's root, which normally do not exist: PKGPROXY_URL is the only setting there (derived from the same lines).

Read more

03 A socket, through guestfwd

THE SOCKET PATH the guest 10.0.2.100 pkgproxy -netdev user,id=netdev0 ,guestfwd=tcp:10.0.2.100:3142-unix:$SNAP_COMMON/pkgproxy.sock QEMU user network :3142 one each pkgproxy.sock on the host ,guestfwd=tcp:10.0.2.100:3143-unix:$SNAP_COMMON/pkgproxy-lxd-tls.sock pkgproxy-lxd-tls.sock :3143, if it exists written into ubuntu-cloud-24.04.x86_64.run: every boot has the bridge
The .run script's network option, -netdev user with guestfwd=tcp:10.0.2.100:3142-unix:$SNAP_COMMON/pkgproxy.sock. The guest's connections to 10.0.2.100 go into QEMU, and each becomes its own connection to pkgproxy.sock on the host; a second rule sends port 3143 to pkgproxy-lxd-tls.sock when that socket exists. The option is written into ubuntu-cloud-24.04.x86_64.run.

-netdev user,guestfwd=tcp:10.0.2.100:3142-unix:$SNAP_COMMON/pkgproxy.sock: one host connection per guest connection, written into the .run script.

Here's the socket path, the snap's default. Inside the guest, pkgproxy is 10.0.2.100. Nothing really has that address: it's a rule in QEMU's user network. make adds a guestfwd option to the network device, and it says: connections to 10.0.2.100, port 3142, go to pkgproxy.sock on the host. Each connection the guest opens gets a fresh connection to the socket, so parallel downloads stay out of each other's way. When pkgproxy also has its LXD image socket, a second rule does the same on port 3143. All of this is written into the .run script, so every boot of the system brings the bridge along.

Details and sources

Details

  • With pkgproxy.sock in the socket directory, make sets PKGPROXY_GUEST_IP := 10.0.2.100, PKGPROXY_PORT := 3142 and QEMU_DEFAULT_NETDEV_SYS_EXTRA = ,guestfwd=tcp:10.0.2.100:3142-unix:$(PKGPROXY_SOCKET_DIR)/pkgproxy.sock, and adds ,guestfwd=tcp:10.0.2.100:3143-unix:…/pkgproxy-lxd-tls.sock when that socket exists too (010-qemu.mk:31-39).
  • Every architecture's network option is -netdev user,id=netdev0$(QEMU_DEFAULT_NETDEV_SYS_EXTRA)${QEMU_NETDEV_USER_EXTRA:-} (010-qemu-x86_64.mk:8-11). The first part is expanded when the .run script is written; QEMU_NETDEV_USER_EXTRA is read when it runs.
  • Run for this deck: image-garden make -n ubuntu-cloud-24.04.x86_64.run with a fixture $SNAP_COMMON holding a pkgproxy.sock printed -netdev user,id=netdev0,guestfwd=tcp:10.0.2.100:3142-unix:<fixture>/pkgproxy.sock${QEMU_NETDEV_USER_EXTRA:-}.
  • "One fresh UNIX connection per guest TCP connection, handled natively inside QEMU." Port 3142 is apt-cacher-ng's default, kept for continuity (proxy/README.md:16-23).
  • The LXD image server listens on a second socket rather than TCP port 3143, "where it would collide with a standalone pkgproxy snap on the same host" (proxy/README.md:69-74). Nothing sets LXD up at boot; curl -fsSL http://pkgproxy:3142/setup/lxd-remote | sh in a guest does (proxy/README.md:76-88).
  • 10.0.2.100 is inside the user network's default 10.0.2.0/24 range but is no machine: libslirp answers it with the forwarding rule (derived: general QEMU behaviour). The snap's $SNAP_COMMON is /var/snap/image-garden/common (derived: snapd's layout).
  • $SNAP_COMMON on the slide stands for the directory: the .run script holds the expanded path.

Read more

04 A URL, through the user network

THE URL PATH PKGPROXY_URL=http://pkgproxy.example.lan:3142 plain http; port 80 when none is given getent ahostsv4 192.0.2.9 the first IPv4 address user-data, bootcmd and runcmd: … echo '192.0.2.9 pkgproxy'; } >>/etc/hosts the guest QEMU user network the pkgproxy machine connects from your host The name resolves, and the address routes, from the machine that runs make
PKGPROXY_URL=http://pkgproxy.example.lan:3142 is resolved by getent ahostsv4 to 192.0.2.9 when make prepares the image; the user-data has the guest add 192.0.2.9 pkgproxy to /etc/hosts; the guest then connects out through QEMU's user network to the pkgproxy machine. The name must resolve, and the address route, from the machine that runs make.

http://host[:port], port 80 by default. make resolves the host with getent ahostsv4 and the guest gets that address as pkgproxy in /etc/hosts. No guestfwd.

With a URL there's no forward at all. The URL is plain http, a host and an optional port, with port 80 when you leave it out. When make prepares the image, it looks the host up on your machine with getent and takes the first IPv4 address. That address goes into the user-data, and the guest writes it to /etc/hosts as pkgproxy. From then on the guest connects to it like to any other server: out through QEMU's user network, which makes the connection from your host. So the name has to resolve on the machine that runs make, and the address has to be reachable from there.

Details and sources

Details

  • make takes the host with ^http://([A-Za-z0-9.-]+)(:[0-9]+)?/?$ and the port from the same pattern, or 80 (010-qemu.mk:24-26), then "/etc/hosts maps names to addresses only, so resolve the host now": getent ahostsv4 HOST, the first address (010-qemu.mk:27-28). No guestfwd is added (010-qemu.mk:30).
  • The hosts line is grep -q '^IP pkgproxy$' /etc/hosts || { echo '# image-garden: pkgproxy, see http://pkgproxy:PORT/'; echo 'IP pkgproxy'; } >>/etc/hosts, in bootcmd and in runcmd (010-qemu.mk:132-134). Run for this deck: with PKGPROXY_URL=http://192.0.2.9:3142, ubuntu-cloud-24.04.x86_64.user-data differs from the socket mode's only in 192.0.2.9 replacing 10.0.2.100 on those two lines.
  • pkgproxy.example.lan is the README's example name (proxy/README.md:103-104); 192.0.2.9 is a documentation address.
  • "Outside the snap, as in image-garden's container, make PKGPROXY_URL=http://host:port ... does the same, and so does PKGPROXY_URL in the environment" (proxy/README.md:117-127). The same variable sends image downloads through that pkgproxy unless GARDEN_DL_PKGPROXY says otherwise: see image downloads.
  • With a URL, pkgproxy's LXD image server is used directly, on the port its -lxd-tls-addr names (proxy/README.md:120-122).
  • The user network makes the guest's outgoing connection from the QEMU process on the host (derived: general QEMU behaviour).

Read more

05 An address that points back at the guest

ADDRESSES THAT WORK your machine pkgproxy on port 3142 PKGPROXY_URL=http://127.0.0.1:3142 the guest 127.0.0.1 pkgproxy back to the guest itself: nothing answers, no setup, no cache snap set image-garden pkgproxy=http://$host_ip:3142 the GitHub action: the host's address, from hostname -I PKGPROXY_URL=http://10.0.2.2:3142 10.0.2.2: your host's loopback, as the guest sees it
PKGPROXY_URL=http://127.0.0.1:3142 gives the guest 127.0.0.1 pkgproxy, which is the guest itself: nothing answers, so no setup and no cache. The GitHub action's snap set image-garden pkgproxy=http://$host_ip:3142, with the host's network address, and a URL with 10.0.2.2, the host's loopback seen from the guest, both reach pkgproxy on your machine.

127.0.0.1 and localhost mean the guest itself. Use the host's network address, or 10.0.2.2, the host's loopback as the guest sees it.

The most common slip is an address that means something else inside the guest. Say pkgproxy runs on your own machine, on port 3142, and you give it as localhost, or 127.0.0.1. make dutifully writes 127.0.0.1 as pkgproxy, and in the guest, that's the guest itself. Nothing answers there, so the guest skips its setup and downloads without a cache. Give your host's address on the network instead, as the GitHub action does. Or use 10.0.2.2: that's how QEMU's user network shows your host's loopback to the guest, and image-garden's own tests use it.

Details and sources

Details

  • make resolves the URL's host on the machine that runs it and writes that address into the guest's /etc/hosts (010-qemu.mk:24-28); localhost resolves to 127.0.0.1, which in the guest is the guest's own loopback (derived).
  • image-garden-action, when a pkgproxy already listens on the runner: "Guests reach the pkgproxy at the address of the host, as 127.0.0.1 would be the guest itself", host_ip="$(hostname -I | awk '{ print $1 }')" and snap set image-garden pkgproxy=http://"$host_ip":3142 (action.yaml:386-389). Its own image downloads, made on the host, use GARDEN_DL_PKGPROXY=http://127.0.0.1:3142 (action.yaml:391-393).
  • image-garden's smoke test runs pkgproxy on 127.0.0.1:PORT and gives guests http://10.0.2.2:PORT: "QEMU's user networking makes the host's loopback 10.0.2.2" (smoke-pkgproxy.sh:229-233).
  • A guest that gets no answer skips its setup scripts in that boot: see an unreachable pkgproxy. On the host, image-garden's own image downloads go to the same URL from the host, where 127.0.0.1 does work, so the downloads may be cached while the guests are not (derived).
  • Inside a CI job container the address must route from the container, not be Docker's host-gateway: see an address the VM can reach.

06 When make finds no pkgproxy

NO PKGPROXY, WITH AND WITHOUT A WORD $ PKGPROXY_URL=https://192.0.2.9:3142 image-garden make ubuntu-cloud-24.04.x86_64.run …/image-garden.mk:…: cannot use pkgproxy at "https://192.0.2.9:3142": want http://host[:port] with a host that resolves to IPv4; guests get no pkgproxy the same for a host name that does not resolve a socket that isn't there yet daemon not started, plug not connected no message at all ubuntu-cloud-24.04.x86_64.user-data ubuntu-cloud-24.04.x86_64.run no pkgproxy, baked in: fix, then make them again
An https PKGPROXY_URL makes make warn: cannot use pkgproxy at the URL, want http://host[:port] with a host that resolves to IPv4; guests get no pkgproxy. A socket that is not there yet, because the daemon has not started or the plug is not connected, gives no message at all. Either way ubuntu-cloud-24.04.x86_64.user-data and .run are written without pkgproxy and must be made again.

An https URL or a host that does not resolve: cannot use pkgproxy at "…": … guests get no pkgproxy. A missing socket: no message. Both are baked into the files make wrote.

Two other mistakes leave the guest with no pkgproxy at all. If the URL isn't plain http with a host and maybe a port, or the host doesn't resolve to an IPv4 address, make warns you: cannot use pkgproxy at that URL, guests get no pkgproxy. In a long build that's easy to miss. The quiet one is a socket that isn't there yet: the bundled pkgproxy hasn't started, or the pkgproxy snap's plug isn't connected. Then there's no message at all. Either way, the choice is now written into the user-data and the .run script, so fix the cause, then make the system's files again.

Details and sources

Details

  • When PKGPROXY_URL does not match http://host[:port] or its host has no IPv4 address, make prints cannot use pkgproxy at "URL": want http://host[:port] with a host that resolves to IPv4; guests get no pkgproxy with $(warning …) (010-qemu.mk:24-29). Run for this deck with https://192.0.2.9:3142 and with http://pkgproxy.invalid:3142: both print it, prefixed with the make file and line, and leave PKGPROXY_GUEST_IP empty.
  • In the snap, the configure hook rejects such a URL before make sees it: pkgproxy must be component, snap, none or a URL such as http://192.0.2.1:3142 (configure:28-38). A name that stops resolving later still gets the warning.
  • With no socket in the socket directory, make falls through to "No pkgproxy: guests get none." without a message (010-qemu.mk:31, :40-44). The test cases "component-down" and "snap-disconnected" check exactly that (check-pkgproxy-config.sh:128-145).
  • With pkgproxy=snap set while the plug is disconnected, the configure hook says pkgproxy=snap, but the pkgproxy snap's sockets are not connected: guests get no pkgproxy until snap connect … once, when the setting is applied (configure:56-68); make itself says nothing.
  • The .user-data and .run rules do not depend on the pkgproxy setting, so they are not made again by themselves: see changing the proxy does not rebuild.

Read more

07 Where to go from here

ROADS Reaching pkgproxy In the user-data what the guest does with it The guestfwd patch TCP in, unix socket out From a CI container an address the VM can reach When it is down seconds, not minutes No rebuild the setting is baked in
Five roads from Reaching pkgproxy: what the user-data does with the name, the guestfwd patch, an address from a CI job container, an unreachable pkgproxy, and why a new setting needs the files made again

Next: the user-data, the guestfwd patch, addresses from a CI container, the container's PKGPROXY_URL, an unreachable pkgproxy and rebuilding.

So the guest only ever knows one name, and make decides, once, what it points to: a socket bridged by QEMU, or an address you gave it. Next, the user-data deck shows what the guest does with that name in its first boot. The patch deck explains how QEMU learned to bridge a unix socket at all. If your VM runs inside a CI job container, the craft components' deck is about picking an address from there. And two decks cover what happens when nothing answers, and why a new setting needs the files made again.

Details and sources

Read more

Basis of the statements on this page

Read in mk/010-qemu.mk, snap/hooks/configure, proxy/README.md and tests/ of image-garden at the v0.8.0 tag ("code"). The QEMU option and the user-data lines were produced by running image-garden make from an export of that tag with a fixture $SNAP_COMMON, as tests/check-pkgproxy-config.sh does, and with PKGPROXY_URL ("run"). How QEMU's user network treats addresses is general QEMU behaviour ("derived").

How make decides, mode by mode

Setting when the files are madeGuest's pkgproxyPortQEMU -netdev user gets
PKGPROXY_URL=http://host[:port], or pkgproxy=http://… in the snapthe host's first IPv4 address from getent ahostsv4the URL's, else 80nothing
a URL that is not http://host[:port], or a host without IPv4none, with a warning-nothing
pkgproxy=component (the snap's default), $SNAP_COMMON/pkgproxy.sock exists10.0.2.1003142,guestfwd=tcp:10.0.2.100:3142-unix:$SNAP_COMMON/pkgproxy.sock
pkgproxy=snap, $SNAP_COMMON/pkgproxy-host-snap/sockets/pkgproxy.sock exists10.0.2.1003142the same, with the pkgproxy snap's socket
either socket mode, with pkgproxy-lxd-tls.sock beside itas aboveas abovealso ,guestfwd=tcp:10.0.2.100:3143-unix:…/pkgproxy-lxd-tls.sock
a socket mode whose socket does not exist yetnone, silently-nothing
pkgproxy=none, or nothing set outside the snapnone-nothing

Source: 010-qemu.mk:20-44. PKGPROXY_URL from the environment or the make command line wins over the snap's saved URL and mode (?=, line 21).

The make variables

VariableSet fromUsed by
PKGPROXY_MODE$SNAP_COMMON/pkgproxy-modethe socket directory and whether the saved URL is read
PKGPROXY_URLthe environment, the command line, or $SNAP_COMMON/pkgproxy-url in external modethe URL path, and image downloads
PKGPROXY_SOCKET_DIR$SNAP_COMMON, or …/pkgproxy-host-snap/sockets in snap modethe socket path
PKGPROXY_HOST, PKGPROXY_PORTparsed from the URLthe guest's port
PKGPROXY_GUEST_IPgetent ahostsv4, or 10.0.2.100the /etc/hosts line; empty means no pkgproxy
QEMU_DEFAULT_NETDEV_SYS_EXTRAthe guestfwd rulesevery architecture's -netdev user

QEMU_NETDEV_USER_EXTRA, read by the .run script when it runs, is appended after them (010-qemu-x86_64.mk:8-11).

Addresses, seen from the guest

AddressIn the guest
127.0.0.1, localhostthe guest itself
10.0.2.2the host's loopback, through the user network (smoke-pkgproxy.sh:230-233)
10.0.2.100pkgproxy's socket, only with a guestfwd rule
the host's LAN addressthe host, through the user network
Docker's host-gatewaynot defined in the guest (snapcraft-gitlab-ci README.md:218-224)

Roads from here

BUILT ON IT · CI integrations

PKGPROXY · caching for guests

CONTAINER · OCI image