What pkgproxy gives a garden
download once for everyone
Test VMs download the same packages, snaps, Go modules and cloud images on every run. pkgproxy keeps them in one cache on the host, so only the first run pays for the downloads. In CI, where every job starts empty, keeping pkgproxy's cache between jobs is what makes runs fast.
On the map: pkgproxy
▶ Watch this deck — 6 scenes, ~4 min
Part of journeys
- From your laptop to a git forge, step 6 of 6 · previous: image-garden-action for GitHub
01 The same downloads, every time
Every run fetches the same cloud image, packages, snaps and Go modules again. pkgproxy keeps one copy.
Think about what a test run downloads. The cloud image itself, hundreds of megabytes. Then, inside the VM, the packages your tests install, maybe a few snaps, maybe Go modules for a build. Your next run does exactly the same, and so does the VM next to it, and the one after. Each of them fetches the same files from the same mirrors, again and again. That's slow, it's hard on the mirrors, and in CI it happens on every single job. pkgproxy is image-garden's answer.
Details and sources
Details
- Every system's image is downloaded by
image-garden download-with-…(rulegen.py:480-518); inside the guest, packages, snaps and Go modules come from the distribution's mirrors and the snap store unless pkgproxy is set up (010-qemu.mk:127-185). - "Hundreds of megabytes" is typical for a cloud image; the exact size depends on the distribution and release.
Read more
02 One cache on the host
The first request fetches and keeps a copy; later ones are served from the host's cache. The snap runs a bundled pkgproxy service by default.
pkgproxy is a caching proxy that runs on the host, next to your VMs. The first time a guest asks for a file, pkgproxy fetches it from the real mirror, keeps a copy in its cache directory, and hands it over. The next time anyone asks for the same file, the next VM or the next run, it comes straight from that copy, at the speed of your disk. So only the very first run pays for the downloads. In the snap, pkgproxy is bundled and runs as a service, and it's on by default.
Details and sources
Details
- In the snap, pkgproxy runs as the
image-garden.pkgproxyservice:pkgproxy -addr pkgproxy:3142 -socket-path $SNAP_COMMON/pkgproxy.sock -cache-dir $SNAP_COMMON/cache/pkgproxy …(snapcraft.yaml:241). - It listens on a UNIX socket that QEMU forwards guest connections to (proxy/README.md:16-23).
pkgproxy=component(or unset) is the default (configure:11-39).- How pkgproxy keys, keeps and refreshes its cache is pkgproxy's own design: see the pkgproxy map.
Read more
03 What goes through it
apt, snapd and Go in the guest, and image-garden's own image and checksum downloads on the host, all go through one pkgproxy.
Three kinds of guest traffic go through pkgproxy. apt uses it as an ordinary forward proxy. snapd talks to it as if it were the snap store, and pkgproxy relays and caches. The Go command uses it as its module proxy. And image-garden itself uses it on the host: it asks pkgproxy for each checksum list and each cloud image, by the image's checksum, and falls back to the origin when pkgproxy can't help. So even an image whose mirror is down can come from the cache.
Details and sources
Details
- pkgproxy shares one listener between apt (a forward proxy via
Acquire::http::Proxy), snapd (a reverse relay to the store viaSNAPPY_FORCE_API_URL) and Go (a module proxy viaGOPROXY) (README.md:38-48). - image-garden asks pkgproxy for checksum lists at
/files/v1/volatile(image-garden:211-269) and for images at/files/v1/fetch?url=…&sha256=…, falling back to the origin (image-garden:281-327).
Read more
04 How a guest finds it
cloud-init adds a pkgproxy hosts entry, runs the apt and snap setup scripts and sets GOPROXY. QEMU's guestfwd or a URL carries the traffic to the host.
A guest doesn't need to know where pkgproxy lives. When make prepares the image, cloud-init gets a few extra lines: a hosts entry that names pkgproxy, a run of pkgproxy's own setup script for apt on Debian and Ubuntu, one for snapd on most systems, and a Go proxy variable. With the snap's pkgproxy, QEMU forwards that name's address to pkgproxy's socket on the host. With a proxy at a URL, the guest just connects there. You set none of this up by hand.
Details and sources
Details
PKGPROXY_BOOTCMD_SETUPisaptfor Debian and Ubuntu;PKGPROXY_RUNCMD_SETUPissnapfor every group but Alpine (010-qemu.mk:129-130); the hosts entry isPKGPROXY_HOSTS_SHELL(010-qemu.mk:134);GOPROXY="…/go,direct"goes into/etc/environment(010-qemu.mk:179).- With the snap's own pkgproxy or the pkgproxy snap, QEMU's
guestfwdsends the guest's10.0.2.100:3142to the socket; with a URL, the guest connects to the URL's address (010-qemu.mk:20-44). - The lines are added only when a pkgproxy is configured when the user-data is made.
Read more
05 Where it matters most: CI
CI jobs start empty. Keep pkgproxy's cache between jobs, as image-garden-action does, or use a shared pkgproxy, as the GitLab components do.
On your laptop, pkgproxy's cache simply stays between runs. In CI it doesn't: every job starts on an empty machine. That's why the projects built on image-garden treat pkgproxy as their cache. The GitHub action saves pkgproxy's cache in the GitHub cache and restores it in the next job. The GitLab components point the VM at a pkgproxy your runners can reach. Either way the first job downloads everything, and every job after it gets the image, the packages and the snaps from the cache.
Details and sources
Details
- image-garden-action uses pkgproxy as its only download cache and keeps it as two GitHub caches between jobs (image-garden-action README.md:152-176).
- The GitLab craft components keep no cache of their own: "pkgproxy is what makes the second build fast" (snapcraft-gitlab-ci README.md:205-210). Their
pkgproxy-urlinput names a pkgproxy the VM can reach.
06 Where to go from here
Next: what pkgproxy adds to user-data, how a guest reaches the host, image downloads, an unreachable pkgproxy, the snap setting and CI caching.
That's the idea: download once, on the host, and let every guest and every run share it. To see exactly what cloud-init adds to a guest, take the user-data deck. How a guest reaches the host, through QEMU's forward or a URL, has its own deck. So do the image downloads, and what happens when pkgproxy is down. In the snap, one setting picks which pkgproxy to use. And for CI, the GitHub action deck shows the cache kept between jobs.
Details and sources
Read more
- In the user-data, reaching the host, image downloads, when it is down.
- The pkgproxy setting, image-garden-action.
- The pkgproxy map, the design of pkgproxy itself.
Roads from here
BUILT ON IT · CI integrations
- image-garden-action for GitHub — keeping the cache between GitHub jobs
PKGPROXY · caching for guests
- What pkgproxy adds to user-data — what cloud-init adds to each guest
- How a guest reaches pkgproxy — how a guest reaches the host
- Cloud images through pkgproxy — cloud images and checksum lists through pkgproxy
- An unreachable pkgproxy — what happens when it is down
SNAP · snap-only features
- The pkgproxy snap setting — choosing a pkgproxy in the snap
pkgproxy map ↗
- Why pkgproxy — on the pkgproxy map, why one cache for every client
- A request, end to end — on the pkgproxy map, one request end to end