A request, end to end
overview for everyone
What happens between a client asking for a package and getting it, from the one listener through the cache key, the store and one shared fetch upstream.
On the map: start here
01 What this topic covers
One request through pkgproxy, from the client to the cache on disk and back.
Every download your machines make through pkgproxy takes the same path, from the client to the cache on disk and back. Follow one request along it. Every client uses one address, and pkgproxy first works out which client is asking. The request then becomes a cache key with a place on disk. The first time, it is a miss, fetched from the internet once. After that come the hits, and the files that get checked again.
02 One address for every client
Five kinds of client share one daemon and one port, 3142; LXD
image servers also need the HTTPS port 3143. Each download is
fetched upstream once and served from disk after that.
pkgproxy runs on one machine, and your other machines download their software through it. Apt, snapd, the Go command, Docker and LXD are all pointed at the same address, port 3142 by default, and LXD also uses an HTTPS port, 3143. Each client still asks for the same files from the same places. pkgproxy fetches each thing from the real upstream once, keeps a copy on disk, and answers every later request for it from that copy.
03 Telling the clients apart
apt is recognized by its absolute request line. The others are told apart
by path: /go, /lxd, /files, Docker's
/v2/…/manifests|blobs|referrers|tags/list,
/v2/_catalog and /v2/, and the rest of
/v2 for snapd. Anything else is answered with 404.
Everything arrives on one listener, so pkgproxy first works out who is asking. Apt uses it as a forward proxy: the request line names the real host, such as deb.debian.org. Every other client talks to pkgproxy as a server, and the path tells them apart. Go modules live under /go, LXD images under /lxd, files under /files. Docker's paths start with /v2, like the Snap Store's, but name manifests, blobs, referrers or tag lists, or are the catalog or the bare version check. What is left under /v2 goes to the snap relay, and a path no side claims gets a 404.
04 A key, and a place on disk
The key names the archive and the path inside it. The store has one directory per kind of download, and a write becomes visible only on commit: a temporary file renamed into place.
Each side turns the request into a cache key. For apt the key is the archive's host and path prefix, such as deb.debian.org /debian, followed by the path from pool or dists onward. The store keeps one directory for each kind of download: apt, snap, go, docker, lxd and files. A download is written to a temporary file and renamed into place only when it is complete, so a half written download is never served as a hit.
05 A miss, fetched once
Concurrent requests for one key share one upstream fetch, a fill. It runs detached from the clients, streams to them while it writes, and is committed only after the content checks out.
When nothing is cached, one fetch goes upstream and every request for the same thing joins it, so fifty machines running apt update at once cost one download. Clients get the response headers as soon as the upstream sends them, then read the bytes as they are written to disk. The fetch does not belong to any one client. If the first one gives up, the download carries on, is checked, and is cached for the next request.
06 Hits, and what gets rechecked
Immutable content is cached forever without revalidation. Mutable indexes are served for a short TTL and then revalidated with a conditional GET. Nothing is ever evicted.
The next request is a hit, answered from disk without asking the upstream. Files that never change, such as a package in apt's pool or a Go module's zip, are kept forever and never checked again. Files that do change, such as apt's Release and Packages indexes, are served from the cache for a short time, five minutes by default, and then checked with a conditional request. pkgproxy deletes nothing on its own, so the cache only grows.
07 Where to go next for administrators in place of the general version
Next: One listener, for the ports and sockets; The store, for the cache on disk; Settings, for the flags and snap options.
You have followed one request: the first request for a file downloads it from the internet, and every later one gets the copy on disk. To see the port every client uses and the other sockets the daemon opens, watch One listener. To see where the cache lives on disk and how to free space, watch The store. To change the port, the cache directory and the rest, watch Settings.
08 Where to go next for CI operators in place of the general version
Next: pkgproxy for CI, for many machines behind one cache; Fetch once, for one download shared by many jobs; Offline mode, for builds that see only what was cached.
You have followed one request: the first job to ask for a file downloads it from the internet, and every later job gets the copy on disk. To put runners, virtual machines and build containers behind one cache, watch pkgproxy for CI. To see many jobs share one download, watch Fetch once. To make builds use only what was cached, watch Offline mode.
09 Where to go next general version
Next: pkgproxyctl setup, for pointing your machines at it;
Fetch once, for one download shared by every client; Immutable or not,
for what is kept and what is rechecked.
You have followed one request: the first request for a file downloads it from the internet, and every later one gets the copy on disk. To point your own machines at pkgproxy, watch pkgproxyctl setup. To see one download shared by every client that asks for it at the same time, watch Fetch once. To see which files are kept forever and which are checked again, watch Immutable or not.
Read more
In the documentation
- pkgproxy: installing it and pointing a client at it
- Where the cache lives
Roads from here
START HERE
- Why pkgproxy — the problem this solves
CORE · the path every side shares
- One listener — how one port serves every client
- Fetch once — one fetch shared by every request
- The store — where the bytes end up
- Immutable or revalidated — what is kept forever and what is rechecked
- Offline mode — serving what is cached without the internet
APT
- apt through a forward proxy — the apt side
SNAP
- Snap Store relay — the snap side
GO MODULES
- Go module proxy — the Go modules side
DOCKER
- Docker Hub registry mirror — the Docker side
LXD
- LXD image servers — the LXD side
FILES
- Files by checksum — files by checksum
SETUP AND THE COMMAND LINE
- Setting up clients — pointing a client at it
OPERATIONS
- Settings — the port and the cache directory
- pkgproxy for CI — many machines behind one cache