Why pkgproxy
start here for everyone
pkgproxy caches what apt, snapd, Go, Docker and LXD download, so every machine on your network gets each file from the internet once; the problem it solves, how it is set up, and what it leaves to others.
On the map: start here
01 What this topic covers
pkgproxy caches what apt, snapd, Go, Docker and LXD download, for every machine on your network. This topic covers the problem, the solution, the setup and the limits.
pkgproxy is a cache for the software your machines download: apt packages, snaps, Go modules, Docker images and LXD images. Start here to see the problem it solves, the same files downloaded again and again. Then see how one shared cache solves it, how you install pkgproxy and point your tools at it, and what it deliberately leaves to other tools.
02 The same download, again and again
Every machine, VM, container and CI job downloads the same packages and images from the internet on its own, and an empty build environment does it again on every run.
Imagine a laptop, two virtual machines, a few build containers and a CI job, all installing the same software. Each of them downloads it from the internet on its own: the same Debian package from deb.debian.org, the same snap, the same Go modules, Docker layers and LXD images. Every fresh container and every CI run starts empty and fetches it all again. Downloading the same files again and again only makes everyone wait.
03 One shared cache
One pkgproxy for the whole network: the first request for a file downloads it once, and every later one is served from its disk. Without the internet, it serves what is cached.
pkgproxy solves this with one shared cache. You run it on one machine on your network and point the others at it. Apt, snapd, the Go command, Docker and LXD use it in the way each of them already expects. The first machine to ask for a file causes one download from the internet. pkgproxy keeps that copy on disk, and every other machine, container and CI job gets it from there, at the speed of your network. When the internet is gone, it keeps serving what it already has.
04 Install, then point each tool at it
sudo snap install pkgproxy starts the service on
3142 (and 3143 for LXD).
pkgproxyctl setup <tool> prints a script; other machines
fetch it from http://<host>:3142/setup/<tool>.
-undo prints the way back.
Installing the snap starts the service right away, with no settings to change, on port 3142, and 3143 for LXD. Your tools do not use it until you set each one up: pkgproxyctl setup prints a short script for one tool, which you pipe to a shell. Other machines on the network open the setup page in a browser, or fetch the same script with curl. Adding dash undo to a setup command gives the commands that remove it again.
05 What it leaves to others
No TLS termination: https apt sources go through an uncached
CONNECT tunnel to port 443. Files by checksum are the
exception. No signature checks (apt does them) and no garbage collection.
Some things pkgproxy deliberately does not do. It does not decrypt HTTPS: an apt source on https passes through as a blind tunnel to port 443, and is not cached. The one exception is files by checksum, where pkgproxy is itself the HTTPS client and keeps only bytes that match the checksum. It does not check signatures; apt does, so indexes are served exactly as the mirror sent them. And it deletes nothing on its own: the cache only grows until you free space.
06 Where to go next for CI operators in place of the general version
Next: pkgproxy for CI, for many runners behind one cache; A request, for one request end to end; The store, for where the cache lives.
Instead of every runner downloading the same files, each file now comes from the internet once, and every runner shares it. To put many runners behind one cache, watch pkgproxy for CI. To follow one request through pkgproxy, watch A request. To see where the cache lives and how it grows, watch The store.
07 Where to go next general version
Next: A request, for one request end to end; pkgproxyctl
setup, for pointing your tools at it; Offline mode, for working
without the internet.
Instead of every machine downloading the same files, each file now comes from the internet once, and your whole network shares it. To follow one request through pkgproxy, watch A request. To point your tools at it, watch pkgproxyctl setup. To keep working when the internet is gone, watch Offline mode.
Read more
In the documentation
- pkgproxy: what it caches, and how to install it
- Check that it works
Roads from here
START HERE
- A request, end to end — what happens to one request
CORE · the path every side shares
- Offline mode — working without the internet
- The store — where the cache lives and how it grows
APT
- HTTPS apt sources — what happens to https apt sources
FILES
- Files by checksum — the one way it fetches over https
SETUP AND THE COMMAND LINE
- Setting up clients — pointing your tools at it
OPERATIONS
- pkgproxy for CI — many runners behind one cache