How a guest reaches pkgproxy
10.0.2.100 or a URL for CI operators
Guests ask for the name pkgproxy, and make fixes what it means when it prepares a system. A PKGPROXY_URL wins and becomes the host's IPv4 address in /etc/hosts; otherwise an existing pkgproxy socket is bridged by QEMU's guestfwd from 10.0.2.100 port 3142; otherwise guests get none.
On the map: pkgproxy
▶ Watch this deck — 7 scenes, ~5 min
Worth seeing first
- What pkgproxy gives a garden — what pkgproxy caches and why a garden uses it
01 One name, two paths
Every guest asks for pkgproxy. make decides what the name means: a QEMU guestfwd to a unix socket on the host, or the address of a pkgproxy at a URL.
Your guest lives on a small private network that QEMU makes up for it. It can reach out to the internet, but nothing on your host listens on that network. So how does it find pkgproxy? With a name. Every guest asks for pkgproxy, and make decides what that name means when it prepares ubuntu-cloud-24.04.x86_64. There are two answers. With the snap's own pkgproxy, the name leads through a forward inside QEMU to a unix socket on the host. With a pkgproxy at a URL, it leads to that machine's address, like any other connection the guest makes.
Details and sources
Details
- Guests reach pkgproxy as the host name
pkgproxyon port 3142 for the sockets, or the URL's port; make chooses the address when it makes a system's files (010-qemu.mk:15-44). - The socket path: "QEMU's
guestfwdforwards guest TCP connections to 10.0.2.100 on port 3142 straight to that UNIX socket" (proxy/README.md:16-23). - The URL path: the host is resolved to an IPv4 address when an instance is built, and guests get it as
pkgproxyin/etc/hosts(proxy/README.md:117-120). "Whichever pkgproxy it is, guests know it aspkgproxy" (proxy/README.md:129-130). - QEMU's user network (
-netdev user, libslirp) gives the guest a private address range and translates its outgoing connections into connections made by the QEMU process on the host (derived: general QEMU behaviour).
02 make picks the path, in a fixed order
A URL, from PKGPROXY_URL or snap set image-garden pkgproxy=http://…, wins. Else an existing pkgproxy socket is bridged. Else the guest gets no pkgproxy.
Which path you get is settled by make, in a fixed order. First, a URL. Outside the snap, that's the PKGPROXY_URL variable, from the environment or the make command line. In the snap, it's the URL you gave to snap set, which the configure hook saves for make to read. A URL always wins. Without one, make looks for a socket: the bundled pkgproxy's, or the pkgproxy snap's when you chose that mode. The socket has to exist right then, while make writes the files. And when there's neither, the guest simply gets no pkgproxy, and downloads everything straight from the mirrors.
Details and sources
Details
- In the snap, the configure hook turns
snap set image-garden pkgproxy=…into a mode (component,snap,externalornone) and, for a URL, checks it against^http://[A-Za-z0-9.-]+(:[0-9]+)?/?$(configure:17-39). It writes the mode to$SNAP_COMMON/pkgproxy-modeand the URL to$SNAP_COMMON/pkgproxy-url(configure:76-82). - make reads the mode, and the URL only when the mode is
external;PKGPROXY_URL ?= …means aPKGPROXY_URLfrom the environment or the command line takes precedence over the saved URL (010-qemu.mk:20-21). That aPKGPROXY_URLin the snap's environment also overrides thecomponent,snapandnonemodes follows from the same line (derived, not run). - The socket directory is
$SNAP_COMMON, or$SNAP_COMMON/pkgproxy-host-snap/socketsinsnapmode, where snapd mounts the pkgproxy snap'ssocketsslot (010-qemu.mk:22-23). The bridge is set up only whenpkgproxy.sockexists there (010-qemu.mk:31-39). - Without either: "No pkgproxy: guests get none." (010-qemu.mk:40-44).
- Outside the snap
$SNAP_COMMONis empty, so make looks for/pkgproxy-modeand/pkgproxy.sockat the file system's root, which normally do not exist:PKGPROXY_URLis the only setting there (derived from the same lines).
Read more
- The snap's pkgproxy setting, with the table of values.
03 A socket, through guestfwd
-netdev user,guestfwd=tcp:10.0.2.100:3142-unix:$SNAP_COMMON/pkgproxy.sock: one host connection per guest connection, written into the .run script.
Here's the socket path, the snap's default. Inside the guest, pkgproxy is 10.0.2.100. Nothing really has that address: it's a rule in QEMU's user network. make adds a guestfwd option to the network device, and it says: connections to 10.0.2.100, port 3142, go to pkgproxy.sock on the host. Each connection the guest opens gets a fresh connection to the socket, so parallel downloads stay out of each other's way. When pkgproxy also has its LXD image socket, a second rule does the same on port 3143. All of this is written into the .run script, so every boot of the system brings the bridge along.
Details and sources
Details
- With
pkgproxy.sockin the socket directory, make setsPKGPROXY_GUEST_IP := 10.0.2.100,PKGPROXY_PORT := 3142andQEMU_DEFAULT_NETDEV_SYS_EXTRA = ,guestfwd=tcp:10.0.2.100:3142-unix:$(PKGPROXY_SOCKET_DIR)/pkgproxy.sock, and adds,guestfwd=tcp:10.0.2.100:3143-unix:…/pkgproxy-lxd-tls.sockwhen that socket exists too (010-qemu.mk:31-39). - Every architecture's network option is
-netdev user,id=netdev0$(QEMU_DEFAULT_NETDEV_SYS_EXTRA)${QEMU_NETDEV_USER_EXTRA:-}(010-qemu-x86_64.mk:8-11). The first part is expanded when the.runscript is written;QEMU_NETDEV_USER_EXTRAis read when it runs. - Run for this deck:
image-garden make -n ubuntu-cloud-24.04.x86_64.runwith a fixture$SNAP_COMMONholding apkgproxy.sockprinted-netdev user,id=netdev0,guestfwd=tcp:10.0.2.100:3142-unix:<fixture>/pkgproxy.sock${QEMU_NETDEV_USER_EXTRA:-}. - "One fresh UNIX connection per guest TCP connection, handled natively inside QEMU." Port 3142 is apt-cacher-ng's default, kept for continuity (proxy/README.md:16-23).
- The LXD image server listens on a second socket rather than TCP port 3143, "where it would collide with a standalone pkgproxy snap on the same host" (proxy/README.md:69-74). Nothing sets LXD up at boot;
curl -fsSL http://pkgproxy:3142/setup/lxd-remote | shin a guest does (proxy/README.md:76-88). 10.0.2.100is inside the user network's default10.0.2.0/24range but is no machine: libslirp answers it with the forwarding rule (derived: general QEMU behaviour). The snap's$SNAP_COMMONis/var/snap/image-garden/common(derived: snapd's layout).$SNAP_COMMONon the slide stands for the directory: the.runscript holds the expanded path.
Read more
- The guestfwd unix socket patch: why stock QEMU cannot do this.
- Changing the proxy does not rebuild: the option stays in the
.runscript.
04 A URL, through the user network
http://host[:port], port 80 by default. make resolves the host with getent ahostsv4 and the guest gets that address as pkgproxy in /etc/hosts. No guestfwd.
With a URL there's no forward at all. The URL is plain http, a host and an optional port, with port 80 when you leave it out. When make prepares the image, it looks the host up on your machine with getent and takes the first IPv4 address. That address goes into the user-data, and the guest writes it to /etc/hosts as pkgproxy. From then on the guest connects to it like to any other server: out through QEMU's user network, which makes the connection from your host. So the name has to resolve on the machine that runs make, and the address has to be reachable from there.
Details and sources
Details
- make takes the host with
^http://([A-Za-z0-9.-]+)(:[0-9]+)?/?$and the port from the same pattern, or 80 (010-qemu.mk:24-26), then "/etc/hosts maps names to addresses only, so resolve the host now":getent ahostsv4 HOST, the first address (010-qemu.mk:27-28). Noguestfwdis added (010-qemu.mk:30). - The hosts line is
grep -q '^IP pkgproxy$' /etc/hosts || { echo '# image-garden: pkgproxy, see http://pkgproxy:PORT/'; echo 'IP pkgproxy'; } >>/etc/hosts, in bootcmd and in runcmd (010-qemu.mk:132-134). Run for this deck: withPKGPROXY_URL=http://192.0.2.9:3142,ubuntu-cloud-24.04.x86_64.user-datadiffers from the socket mode's only in192.0.2.9replacing10.0.2.100on those two lines. pkgproxy.example.lanis the README's example name (proxy/README.md:103-104);192.0.2.9is a documentation address.- "Outside the snap, as in image-garden's container,
make PKGPROXY_URL=http://host:port ...does the same, and so doesPKGPROXY_URLin the environment" (proxy/README.md:117-127). The same variable sends image downloads through that pkgproxy unlessGARDEN_DL_PKGPROXYsays otherwise: see image downloads. - With a URL, pkgproxy's LXD image server is used directly, on the port its
-lxd-tls-addrnames (proxy/README.md:120-122). - The user network makes the guest's outgoing connection from the QEMU process on the host (derived: general QEMU behaviour).
Read more
- An address the VM can reach, for a VM inside a CI job container; pkgproxy by URL in the container.
05 An address that points back at the guest
127.0.0.1 and localhost mean the guest itself. Use the host's network address, or 10.0.2.2, the host's loopback as the guest sees it.
The most common slip is an address that means something else inside the guest. Say pkgproxy runs on your own machine, on port 3142, and you give it as localhost, or 127.0.0.1. make dutifully writes 127.0.0.1 as pkgproxy, and in the guest, that's the guest itself. Nothing answers there, so the guest skips its setup and downloads without a cache. Give your host's address on the network instead, as the GitHub action does. Or use 10.0.2.2: that's how QEMU's user network shows your host's loopback to the guest, and image-garden's own tests use it.
Details and sources
Details
- make resolves the URL's host on the machine that runs it and writes that address into the guest's
/etc/hosts(010-qemu.mk:24-28);localhostresolves to127.0.0.1, which in the guest is the guest's own loopback (derived). - image-garden-action, when a pkgproxy already listens on the runner: "Guests reach the pkgproxy at the address of the host, as 127.0.0.1 would be the guest itself",
host_ip="$(hostname -I | awk '{ print $1 }')"andsnap set image-garden pkgproxy=http://"$host_ip":3142(action.yaml:386-389). Its own image downloads, made on the host, useGARDEN_DL_PKGPROXY=http://127.0.0.1:3142(action.yaml:391-393). - image-garden's smoke test runs pkgproxy on
127.0.0.1:PORTand gives guestshttp://10.0.2.2:PORT: "QEMU's user networking makes the host's loopback 10.0.2.2" (smoke-pkgproxy.sh:229-233). - A guest that gets no answer skips its setup scripts in that boot: see an unreachable pkgproxy. On the host, image-garden's own image downloads go to the same URL from the host, where
127.0.0.1does work, so the downloads may be cached while the guests are not (derived). - Inside a CI job container the address must route from the container, not be Docker's
host-gateway: see an address the VM can reach.
Read more
06 When make finds no pkgproxy
An https URL or a host that does not resolve: cannot use pkgproxy at "…": … guests get no pkgproxy. A missing socket: no message. Both are baked into the files make wrote.
Two other mistakes leave the guest with no pkgproxy at all. If the URL isn't plain http with a host and maybe a port, or the host doesn't resolve to an IPv4 address, make warns you: cannot use pkgproxy at that URL, guests get no pkgproxy. In a long build that's easy to miss. The quiet one is a socket that isn't there yet: the bundled pkgproxy hasn't started, or the pkgproxy snap's plug isn't connected. Then there's no message at all. Either way, the choice is now written into the user-data and the .run script, so fix the cause, then make the system's files again.
Details and sources
Details
- When
PKGPROXY_URLdoes not matchhttp://host[:port]or its host has no IPv4 address, make printscannot use pkgproxy at "URL": want http://host[:port] with a host that resolves to IPv4; guests get no pkgproxywith$(warning …)(010-qemu.mk:24-29). Run for this deck withhttps://192.0.2.9:3142and withhttp://pkgproxy.invalid:3142: both print it, prefixed with the make file and line, and leavePKGPROXY_GUEST_IPempty. - In the snap, the configure hook rejects such a URL before make sees it:
pkgproxy must be component, snap, none or a URL such as http://192.0.2.1:3142(configure:28-38). A name that stops resolving later still gets the warning. - With no socket in the socket directory, make falls through to "No pkgproxy: guests get none." without a message (010-qemu.mk:31, :40-44). The test cases "component-down" and "snap-disconnected" check exactly that (check-pkgproxy-config.sh:128-145).
- With
pkgproxy=snapset while the plug is disconnected, the configure hook sayspkgproxy=snap, but the pkgproxy snap's sockets are not connected: guests get no pkgproxy until snap connect …once, when the setting is applied (configure:56-68); make itself says nothing. - The
.user-dataand.runrules do not depend on the pkgproxy setting, so they are not made again by themselves: see changing the proxy does not rebuild.
Read more
- An unreachable pkgproxy: a pkgproxy that is configured but does not answer.
07 Where to go from here
Next: the user-data, the guestfwd patch, addresses from a CI container, the container's PKGPROXY_URL, an unreachable pkgproxy and rebuilding.
So the guest only ever knows one name, and make decides, once, what it points to: a socket bridged by QEMU, or an address you gave it. Next, the user-data deck shows what the guest does with that name in its first boot. The patch deck explains how QEMU learned to bridge a unix socket at all. If your VM runs inside a CI job container, the craft components' deck is about picking an address from there. And two decks cover what happens when nothing answers, and why a new setting needs the files made again.
Details and sources
Read more
Basis of the statements on this page
Read in mk/010-qemu.mk, snap/hooks/configure, proxy/README.md and tests/ of image-garden at the v0.8.0 tag ("code"). The QEMU option and the user-data lines were produced by running image-garden make from an export of that tag with a fixture $SNAP_COMMON, as tests/check-pkgproxy-config.sh does, and with PKGPROXY_URL ("run"). How QEMU's user network treats addresses is general QEMU behaviour ("derived").
How make decides, mode by mode
| Setting when the files are made | Guest's pkgproxy | Port | QEMU -netdev user gets |
|---|---|---|---|
PKGPROXY_URL=http://host[:port], or pkgproxy=http://… in the snap | the host's first IPv4 address from getent ahostsv4 | the URL's, else 80 | nothing |
a URL that is not http://host[:port], or a host without IPv4 | none, with a warning | - | nothing |
pkgproxy=component (the snap's default), $SNAP_COMMON/pkgproxy.sock exists | 10.0.2.100 | 3142 | ,guestfwd=tcp:10.0.2.100:3142-unix:$SNAP_COMMON/pkgproxy.sock |
pkgproxy=snap, $SNAP_COMMON/pkgproxy-host-snap/sockets/pkgproxy.sock exists | 10.0.2.100 | 3142 | the same, with the pkgproxy snap's socket |
either socket mode, with pkgproxy-lxd-tls.sock beside it | as above | as above | also ,guestfwd=tcp:10.0.2.100:3143-unix:…/pkgproxy-lxd-tls.sock |
| a socket mode whose socket does not exist yet | none, silently | - | nothing |
pkgproxy=none, or nothing set outside the snap | none | - | nothing |
Source: 010-qemu.mk:20-44. PKGPROXY_URL from the environment or the make command line wins over the snap's saved URL and mode (?=, line 21).
The make variables
| Variable | Set from | Used by |
|---|---|---|
PKGPROXY_MODE | $SNAP_COMMON/pkgproxy-mode | the socket directory and whether the saved URL is read |
PKGPROXY_URL | the environment, the command line, or $SNAP_COMMON/pkgproxy-url in external mode | the URL path, and image downloads |
PKGPROXY_SOCKET_DIR | $SNAP_COMMON, or …/pkgproxy-host-snap/sockets in snap mode | the socket path |
PKGPROXY_HOST, PKGPROXY_PORT | parsed from the URL | the guest's port |
PKGPROXY_GUEST_IP | getent ahostsv4, or 10.0.2.100 | the /etc/hosts line; empty means no pkgproxy |
QEMU_DEFAULT_NETDEV_SYS_EXTRA | the guestfwd rules | every architecture's -netdev user |
QEMU_NETDEV_USER_EXTRA, read by the .run script when it runs, is appended after them (010-qemu-x86_64.mk:8-11).
Addresses, seen from the guest
| Address | In the guest |
|---|---|
127.0.0.1, localhost | the guest itself |
10.0.2.2 | the host's loopback, through the user network (smoke-pkgproxy.sh:230-233) |
10.0.2.100 | pkgproxy's socket, only with a guestfwd rule |
| the host's LAN address | the host, through the user network |
Docker's host-gateway | not defined in the guest (snapcraft-gitlab-ci README.md:218-224) |
Roads from here
BUILT ON IT · CI integrations
- Why the craft components keep no cache — choosing an address for a VM inside a CI job container
PKGPROXY · caching for guests
- What pkgproxy adds to user-data — what the guest does with the name once it has it
- The guestfwd unix socket patch — the QEMU patch that makes the socket bridge possible
- An unreachable pkgproxy — what a guest and the host do when nothing answers
- Changing the proxy does not rebuild — why a changed setting needs the system's files made again
CONTAINER · OCI image
- Running image-garden in a container — PKGPROXY_URL with the container