What pkgproxy gives a garden

download once for everyone

Test VMs download the same packages, snaps, Go modules and cloud images on every run. pkgproxy keeps them in one cache on the host, so only the first run pays for the downloads. In CI, where every job starts empty, keeping pkgproxy's cache between jobs is what makes runs fast.

On the map: pkgproxy

▶ Watch this deck — 6 scenes, ~4 min

Part of journeys

01 The same downloads, every time

WHY PKGPROXY run 1 cloud image packages snaps Go modules run 2 cloud image packages snaps Go modules the VM next to it cloud image packages snaps Go modules the same files, from the same mirrors, again and again slow · hard on the mirrors · on every CI job pkgproxy
Three runs, each with its own VMs, each downloading the cloud image, packages, snaps and Go modules from the same mirrors again; slow, hard on the mirrors, and on every CI job

Every run fetches the same cloud image, packages, snaps and Go modules again. pkgproxy keeps one copy.

Think about what a test run downloads. The cloud image itself, hundreds of megabytes. Then, inside the VM, the packages your tests install, maybe a few snaps, maybe Go modules for a build. Your next run does exactly the same, and so does the VM next to it, and the one after. Each of them fetches the same files from the same mirrors, again and again. That's slow, it's hard on the mirrors, and in CI it happens on every single job. pkgproxy is image-garden's answer.

Details and sources

Details

  • Every system's image is downloaded by image-garden download-with-… (rulegen.py:480-518); inside the guest, packages, snaps and Go modules come from the distribution's mirrors and the snap store unless pkgproxy is set up (010-qemu.mk:127-185).
  • "Hundreds of megabytes" is typical for a cloud image; the exact size depends on the distribution and release.

02 One cache on the host

ONE CACHE ON THE HOST the host pkgproxy VM 1: first request the real mirror a copy in its cache VM 2, run 2 … served from the copy, at disk speed only the first run pays for the downloads image-garden.pkgproxy · bundled in the snap · on by default
pkgproxy runs on the host beside the VMs; the first request for a file is fetched from the mirror and a copy kept in pkgproxy's cache; later requests from other VMs and runs are served from that copy at disk speed; in the snap pkgproxy is a bundled service, on by default

The first request fetches and keeps a copy; later ones are served from the host's cache. The snap runs a bundled pkgproxy service by default.

pkgproxy is a caching proxy that runs on the host, next to your VMs. The first time a guest asks for a file, pkgproxy fetches it from the real mirror, keeps a copy in its cache directory, and hands it over. The next time anyone asks for the same file, the next VM or the next run, it comes straight from that copy, at the speed of your disk. So only the very first run pays for the downloads. In the snap, pkgproxy is bundled and runs as a service, and it's on by default.

Details and sources

Details

  • In the snap, pkgproxy runs as the image-garden.pkgproxy service: pkgproxy -addr pkgproxy:3142 -socket-path $SNAP_COMMON/pkgproxy.sock -cache-dir $SNAP_COMMON/cache/pkgproxy … (snapcraft.yaml:241).
  • It listens on a UNIX socket that QEMU forwards guest connections to (proxy/README.md:16-23).
  • pkgproxy=component (or unset) is the default (configure:11-39).
  • How pkgproxy keys, keeps and refreshes its cache is pkgproxy's own design: see the pkgproxy map.

03 What goes through it

WHAT GOES THROUGH pkgproxy in the guest apt: a forward proxy snapd: "the snap store" Go: the module proxy on the host image-garden checksum lists · cloud images by checksum falls back to the origin a mirror that is down: the image can still come from the cache
Guest traffic through pkgproxy: apt as a forward proxy, snapd as if pkgproxy were the snap store, Go as its module proxy; on the host image-garden asks pkgproxy for checksum lists and cloud images by checksum and falls back to the origin, so an image can come from the cache while its mirror is down

apt, snapd and Go in the guest, and image-garden's own image and checksum downloads on the host, all go through one pkgproxy.

Three kinds of guest traffic go through pkgproxy. apt uses it as an ordinary forward proxy. snapd talks to it as if it were the snap store, and pkgproxy relays and caches. The Go command uses it as its module proxy. And image-garden itself uses it on the host: it asks pkgproxy for each checksum list and each cloud image, by the image's checksum, and falls back to the origin when pkgproxy can't help. So even an image whose mirror is down can come from the cache.

Details and sources

Details

  • pkgproxy shares one listener between apt (a forward proxy via Acquire::http::Proxy), snapd (a reverse relay to the store via SNAPPY_FORCE_API_URL) and Go (a module proxy via GOPROXY) (README.md:38-48).
  • image-garden asks pkgproxy for checksum lists at /files/v1/volatile (image-garden:211-269) and for images at /files/v1/fetch?url=…&sha256=…, falling back to the origin (image-garden:281-327).

04 How a guest finds it

HOW A GUEST FINDS IT #cloud-config (made by make) /etc/hosts: … pkgproxy setup/apt Debian and Ubuntu setup/snap all but Alpine GOPROXY=http://pkgproxy:3142/go,direct the guest QEMU guestfwd pkgproxy's socket a pkgproxy at a URL: the guest connects there instead nothing to set up by hand
When make prepares an image, cloud-init gets a pkgproxy hosts entry, pkgproxy's apt setup on Debian and Ubuntu, its snap setup on all but Alpine, and GOPROXY; QEMU forwards the pkgproxy address to the snap's socket, or the guest connects to a URL; nothing to set up by hand

cloud-init adds a pkgproxy hosts entry, runs the apt and snap setup scripts and sets GOPROXY. QEMU's guestfwd or a URL carries the traffic to the host.

A guest doesn't need to know where pkgproxy lives. When make prepares the image, cloud-init gets a few extra lines: a hosts entry that names pkgproxy, a run of pkgproxy's own setup script for apt on Debian and Ubuntu, one for snapd on most systems, and a Go proxy variable. With the snap's pkgproxy, QEMU forwards that name's address to pkgproxy's socket on the host. With a proxy at a URL, the guest just connects there. You set none of this up by hand.

Details and sources

Details

  • PKGPROXY_BOOTCMD_SETUP is apt for Debian and Ubuntu; PKGPROXY_RUNCMD_SETUP is snap for every group but Alpine (010-qemu.mk:129-130); the hosts entry is PKGPROXY_HOSTS_SHELL (010-qemu.mk:134); GOPROXY="…/go,direct" goes into /etc/environment (010-qemu.mk:179).
  • With the snap's own pkgproxy or the pkgproxy snap, QEMU's guestfwd sends the guest's 10.0.2.100:3142 to the socket; with a URL, the guest connects to the URL's address (010-qemu.mk:20-44).
  • The lines are added only when a pkgproxy is configured when the user-data is made.

05 Where it matters most: CI

IN CI your laptop the cache stays a CI job starts on an empty machine GitHub: image-garden-action saves pkgproxy's cache in the GitHub cache, restores it next job GitLab: the components point the VM at a pkgproxy your runners can reach job 1: downloads job 2 and after: image, packages, snaps from the cache
On a laptop pkgproxy's cache stays; in CI every job starts on an empty machine; image-garden-action saves pkgproxy's cache in the GitHub cache and restores it next job; the GitLab components point the VM at a pkgproxy the runners reach; the first job downloads everything, later jobs get it from the cache

CI jobs start empty. Keep pkgproxy's cache between jobs, as image-garden-action does, or use a shared pkgproxy, as the GitLab components do.

On your laptop, pkgproxy's cache simply stays between runs. In CI it doesn't: every job starts on an empty machine. That's why the projects built on image-garden treat pkgproxy as their cache. The GitHub action saves pkgproxy's cache in the GitHub cache and restores it in the next job. The GitLab components point the VM at a pkgproxy your runners can reach. Either way the first job downloads everything, and every job after it gets the image, the packages and the snaps from the cache.

Details and sources

Details

06 Where to go from here

ROADS Why pkgproxy In the user-data hosts, apt, snap, GOPROXY Reaching the host guestfwd or a URL Image downloads images and checksum lists When it is down seconds, not minutes pkgproxy= which pkgproxy, in the snap GitHub action the cache between jobs
Six roads from Why pkgproxy: the user-data, reaching the host, image downloads, an unreachable pkgproxy, the snap's pkgproxy setting, and caching in the GitHub action

Next: what pkgproxy adds to user-data, how a guest reaches the host, image downloads, an unreachable pkgproxy, the snap setting and CI caching.

That's the idea: download once, on the host, and let every guest and every run share it. To see exactly what cloud-init adds to a guest, take the user-data deck. How a guest reaches the host, through QEMU's forward or a URL, has its own deck. So do the image downloads, and what happens when pkgproxy is down. In the snap, one setting picks which pkgproxy to use. And for CI, the GitHub action deck shows the cache kept between jobs.

Details and sources

Roads from here

BUILT ON IT · CI integrations

PKGPROXY · caching for guests

SNAP · snap-only features

pkgproxy map ↗