A request, end to end

overview for everyone

What happens between a client asking for a package and getting it, from the one listener through the cache key, the store and one shared fetch upstream.

On the map: start here

▶ Watch this deck — 9 scenes, ~5 min

01 What this topic covers

ABOUT THIS TOPIC A request the path every download takes, from the client to the cache and back 1 One address for every client 2 Telling the clients apart 3 A key, and a place on disk 4 A miss, fetched once 5 Hits, and what gets rechecked
The topic title, A request, and its 5 scenes in order: One address for every client, Telling the clients apart, A key, and a place on disk, A miss, fetched once, Hits, and what gets rechecked

One request through pkgproxy, from the client to the cache on disk and back.

Every download your machines make through pkgproxy takes the same path, from the client to the cache on disk and back. Follow one request along it. Every client uses one address, and pkgproxy first works out which client is asking. The request then becomes a cache key with a place on disk. The first time, it is a miss, fetched from the internet once. After that come the hits, and the files that get checked again.

02 One address for every client

ONE ADDRESS apt snapd go docker lxc pkgproxy :3142 :3143 for lxc deb.debian.org api.snapcraft.io proxy.golang.org registry-1.docker.io cloud-images.ubuntu.com each download fetched once cache on disk fetched upstream once · every later request is answered from disk
apt, snapd, go, docker and lxc on the left all point at one pkgproxy in the middle, which fetches from five upstreams on the right and keeps a cache on disk

Five kinds of client share one daemon and one port, 3142; LXD image servers also need the HTTPS port 3143. Each download is fetched upstream once and served from disk after that.

pkgproxy runs on one machine, and your other machines download their software through it. Apt, snapd, the Go command, Docker and LXD are all pointed at the same address, port 3142 by default, and LXD also uses an HTTPS port, 3143. Each client still asks for the same files from the same places. pkgproxy fetches each thing from the real upstream once, keeps a copy on disk, and answers every later request for it from that copy.

03 Telling the clients apart

DISPATCH GET http://deb.debian.org/debian/pool/… apt a forward-proxy request names the real host GET /go/golang.org/x/mod/@v/list Go modules GET /lxd/images/streams/v1/index.json LXD images GET /files/v1/fetch?url=…&sha256=… files GET /v2/library/debian/manifests/12 Docker also blobs, referrers, tags/list, /v2/_catalog and /v2/ GET /v2/snaps/info/hello snap relay GET /anything/else 404
Request lines on the left, each routed to the side that serves it: an absolute URL to apt, /go to Go modules, /lxd to LXD, /files to files by checksum, /v2 manifests, blobs, referrers, tag lists and the catalog to Docker, the rest of /v2 to snap, anything else to 404

apt is recognized by its absolute request line. The others are told apart by path: /go, /lxd, /files, Docker's /v2/…/manifests|blobs|referrers|tags/list, /v2/_catalog and /v2/, and the rest of /v2 for snapd. Anything else is answered with 404.

Everything arrives on one listener, so pkgproxy first works out who is asking. Apt uses it as a forward proxy: the request line names the real host, such as deb.debian.org. Every other client talks to pkgproxy as a server, and the path tells them apart. Go modules live under /go, LXD images under /lxd, files under /files. Docker's paths start with /v2, like the Snap Store's, but name manifests, blobs, referrers or tag lists, or are the catalog or the bare version check. What is left under /v2 goes to the snap relay, and a path no side claims gets a 404.

04 A key, and a place on disk

CACHE KEY GET http://deb.debian.org/debian/pool/main/h/hello/hello_2.10-3_amd64.deb deb.debian.org/debian/pool/main/h/hello/hello_2.10-3_amd64.deb archive: host and prefix then the path from pool/ or dists/ onward the store · one directory per kind of download apt/ snap/ go/ docker/ lxd/ files/ temporary file, writing… rename committed: now a hit a half written download is never served as a hit
An apt request line becomes a cache key made of the archive and the path; the store holds one directory per kind of download; a temporary file is renamed into place when complete

The key names the archive and the path inside it. The store has one directory per kind of download, and a write becomes visible only on commit: a temporary file renamed into place.

Each side turns the request into a cache key. For apt the key is the archive's host and path prefix, such as deb.debian.org /debian, followed by the path from pool or dists onward. The store keeps one directory for each kind of download: apt, snap, go, docker, lxd and files. A download is written to a temporary file and renamed into place only when it is complete, so a half written download is never served as a hit.

05 A miss, fetched once

ONE FETCH client 1 client 2 client 3 one fill every request for the key joins the same fetch 1 request upstream writing… headers at once, then the bytes as they arrive gives up checked, committed cached for the next request fifty machines running apt update at once cost one download
Three clients join one fetch inside pkgproxy, which makes a single request upstream; bytes stream back to all of them while being written to disk; one client leaves and the fetch carries on

Concurrent requests for one key share one upstream fetch, a fill. It runs detached from the clients, streams to them while it writes, and is committed only after the content checks out.

When nothing is cached, one fetch goes upstream and every request for the same thing joins it, so fifty machines running apt update at once cost one download. Clients get the response headers as soon as the upstream sends them, then read the bytes as they are written to disk. The fetch does not belong to any one client. If the first one gives up, the download carries on, is checked, and is cached for the next request.

06 Hits, and what gets rechecked

HITS a hit is answered from disk; the upstream is not asked hit kept forever pool/main/h/hello/hello_2.10-3_amd64.deb golang.org/x/mod/@v/v0.20.0.zip never change · never checked again rechecked dists/trixie/InRelease dists/trixie/main/binary-amd64/Packages.xz served from the cache for 5 minutes, then: If-Modified-Since: … 304 Not Modified nothing is deleted on its own · the cache only grows
Two columns: files that never change are kept forever; indexes that change are served for five minutes and then rechecked with a conditional request answered 304

Immutable content is cached forever without revalidation. Mutable indexes are served for a short TTL and then revalidated with a conditional GET. Nothing is ever evicted.

The next request is a hit, answered from disk without asking the upstream. Files that never change, such as a package in apt's pool or a Go module's zip, are kept forever and never checked again. Files that do change, such as apt's Release and Packages indexes, are served from the cache for a short time, five minutes by default, and then checked with a conditional request. pkgproxy deletes nothing on its own, so the cache only grows.

07 Where to go next for administrators in place of the general version

WHERE NEXT the first request for a file downloads it from the internet, and every later one gets the copy on disk you are here A request One listener one port for every client, and the other sockets The store where the bytes end up and how to free space Settings the port, the cache directory and the rest
You are here, A request; next: One listener, The store, Settings

Next: One listener, for the ports and sockets; The store, for the cache on disk; Settings, for the flags and snap options.

You have followed one request: the first request for a file downloads it from the internet, and every later one gets the copy on disk. To see the port every client uses and the other sockets the daemon opens, watch One listener. To see where the cache lives on disk and how to free space, watch The store. To change the port, the cache directory and the rest, watch Settings.

08 Where to go next for CI operators in place of the general version

WHERE NEXT the first job to ask for a file downloads it from the internet, and every later job gets the copy on disk you are here A request pkgproxy for CI runners, VMs and build containers behind one cache Fetch once many jobs, one download Offline mode builds that see only what was cached
You are here, A request; next: pkgproxy for CI, Fetch once, Offline mode

Next: pkgproxy for CI, for many machines behind one cache; Fetch once, for one download shared by many jobs; Offline mode, for builds that see only what was cached.

You have followed one request: the first job to ask for a file downloads it from the internet, and every later job gets the copy on disk. To put runners, virtual machines and build containers behind one cache, watch pkgproxy for CI. To see many jobs share one download, watch Fetch once. To make builds use only what was cached, watch Offline mode.

09 Where to go next general version

WHERE NEXT the first request for a file downloads it from the internet, and every later one gets the copy on disk you are here A request pkgproxyctl setup point your own machines at pkgproxy Fetch once one download shared by every client that asks Immutable or not what is kept forever and what is rechecked
You are here, A request; next: pkgproxyctl setup, Fetch once, Immutable or not

Next: pkgproxyctl setup, for pointing your machines at it; Fetch once, for one download shared by every client; Immutable or not, for what is kept and what is rechecked.

You have followed one request: the first request for a file downloads it from the internet, and every later one gets the copy on disk. To point your own machines at pkgproxy, watch pkgproxyctl setup. To see one download shared by every client that asks for it at the same time, watch Fetch once. To see which files are kept forever and which are checked again, watch Immutable or not.

Read more

In the documentation

Roads from here

START HERE

CORE · the path every side shares

APT

SNAP

GO MODULES

DOCKER

LXD

FILES

SETUP AND THE COMMAND LINE

OPERATIONS