Why pkgproxy

start here for everyone

pkgproxy caches what apt, snapd, Go, Docker and LXD download, so every machine on your network gets each file from the internet once; the problem it solves, how it is set up, and what it leaves to others.

On the map: start here

▶ Watch this deck — 7 scenes, ~4 min

01 What this topic covers

ABOUT THIS TOPIC Why pkgproxy a cache for the software your machines download 1 The same download, again and again 2 One shared cache 3 Install, then point each tool at it 4 What it leaves to others
The topic title, Why pkgproxy, and its 4 scenes in order: The same download, again and again, One shared cache, Install, then point each tool at it, What it leaves to others

pkgproxy caches what apt, snapd, Go, Docker and LXD download, for every machine on your network. This topic covers the problem, the solution, the setup and the limits.

pkgproxy is a cache for the software your machines download: apt packages, snaps, Go modules, Docker images and LXD images. Start here to see the problem it solves, the same files downloaded again and again. Then see how one shared cache solves it, how you install pkgproxy and point your tools at it, and what it deliberately leaves to other tools.

02 The same download, again and again

THE PROBLEM laptop VM 1 VM 2 build container build container CI job the internet deb.debian.org api.snapcraft.io proxy.golang.org registry-1.docker.io images.lxd.canonical.com hello_2.10-3_amd64.deb hello_2.10-3_amd64.deb six times build container build container CI job these start empty on every run the same files, downloaded again and again: waiting for nothing
Six machines on the left, a laptop, two VMs, two containers and a CI job, each with its own line to the internet on the right, all downloading the same package

Every machine, VM, container and CI job downloads the same packages and images from the internet on its own, and an empty build environment does it again on every run.

Imagine a laptop, two virtual machines, a few build containers and a CI job, all installing the same software. Each of them downloads it from the internet on its own: the same Debian package from deb.debian.org, the same snap, the same Go modules, Docker layers and LXD images. Every fresh container and every CI run starts empty and fetches it all again. Downloading the same files again and again only makes everyone wait.

03 One shared cache

THE SOLUTION laptop VM 1 VM 2 build container build container CI job pkgproxy on your network 1 download the internet cache on disk everyone else: from the cache, at the speed of your network internet gone: what is cached is still served
The same six machines now point at one pkgproxy on the network, which makes a single download from the internet and serves everyone from its disk

One pkgproxy for the whole network: the first request for a file downloads it once, and every later one is served from its disk. Without the internet, it serves what is cached.

pkgproxy solves this with one shared cache. You run it on one machine on your network and point the others at it. Apt, snapd, the Go command, Docker and LXD use it in the way each of them already expects. The first machine to ask for a file causes one download from the internet. pkgproxy keeps that copy on disk, and every other machine, container and CI job gets it from there, at the speed of your network. When the internet is gone, it keeps serving what it already has.

04 Install, then point each tool at it

INSTALL, THEN SET UP EACH TOOL $ sudo snap install pkgproxy the service starts at once: port 3142, and 3143 for LXD $ pkgproxyctl setup apt | sudo sh $ pkgproxyctl setup snap | sudo sh $ pkgproxyctl setup go | sh $ pkgproxyctl setup docker | sudo sh one script per tool on another machine $ curl -fsSL 'http://<host>:3142/setup/apt' | sudo sh and back $ pkgproxyctl setup apt -undo | sudo sh
A terminal: install the snap, then one setup command per tool piped to a shell; below, another machine fetches the same script from the setup page with curl, and an undo command

sudo snap install pkgproxy starts the service on 3142 (and 3143 for LXD). pkgproxyctl setup <tool> prints a script; other machines fetch it from http://<host>:3142/setup/<tool>. -undo prints the way back.

Installing the snap starts the service right away, with no settings to change, on port 3142, and 3143 for LXD. Your tools do not use it until you set each one up: pkgproxyctl setup prints a short script for one tool, which you pipe to a shell. Other machines on the network open the setup page in a browser, or fetch the same script with curl. Adding dash undo to a setup command gives the commands that remove it again.

05 What it leaves to others

NON-GOALS pkgproxy apt · https:// mirror :443 blind tunnel CONNECT not cached files by checksum https origin HTTPS client kept if it matches apt InRelease served as sent apt checks the signature cache nothing deleted on its own · it only grows
Three rows through pkgproxy: an https apt source tunnels straight through and is not cached; a file by checksum is fetched over https by pkgproxy and kept only if it matches; apt checks signatures itself on indexes served unchanged. Below, the cache only grows

No TLS termination: https apt sources go through an uncached CONNECT tunnel to port 443. Files by checksum are the exception. No signature checks (apt does them) and no garbage collection.

Some things pkgproxy deliberately does not do. It does not decrypt HTTPS: an apt source on https passes through as a blind tunnel to port 443, and is not cached. The one exception is files by checksum, where pkgproxy is itself the HTTPS client and keeps only bytes that match the checksum. It does not check signatures; apt does, so indexes are served exactly as the mirror sent them. And it deletes nothing on its own: the cache only grows until you free space.

06 Where to go next for CI operators in place of the general version

WHERE NEXT each file comes from the internet once, and every runner shares it you are here Why pkgproxy pkgproxy for CI many runners behind one cache A request one request, end to end The store where the cache lives and how it grows
You are here, Why pkgproxy; next: pkgproxy for CI, A request, The store

Next: pkgproxy for CI, for many runners behind one cache; A request, for one request end to end; The store, for where the cache lives.

Instead of every runner downloading the same files, each file now comes from the internet once, and every runner shares it. To put many runners behind one cache, watch pkgproxy for CI. To follow one request through pkgproxy, watch A request. To see where the cache lives and how it grows, watch The store.

07 Where to go next general version

WHERE NEXT each file comes from the internet once, and your whole network shares it you are here Why pkgproxy A request one request, end to end pkgproxyctl setup point your tools at it Offline mode keep working when the internet is gone
You are here, Why pkgproxy; next: A request, pkgproxyctl setup, Offline mode

Next: A request, for one request end to end; pkgproxyctl setup, for pointing your tools at it; Offline mode, for working without the internet.

Instead of every machine downloading the same files, each file now comes from the internet once, and your whole network shares it. To follow one request through pkgproxy, watch A request. To point your tools at it, watch pkgproxyctl setup. To keep working when the internet is gone, watch Offline mode.

Read more

In the documentation

Roads from here

START HERE

CORE · the path every side shares

APT

FILES

SETUP AND THE COMMAND LINE

OPERATIONS