The proxy engines' packages
contributors for contributors
How each proxy engine (proxy, snapproxy, gomodproxy, registryproxy, lxdproxy, filesproxy) is built from a store, the upstream transport and shared helpers, what the Go and Docker proxy engines do inside, and how the snap proxy engine copes with a store in trouble.
On the map: inside pkgproxy
01 What this topic covers
How each proxy engine is built from a store, a transport and shared helpers, what the Go and Docker proxy engines do inside, and the snap proxy engine's code for a store in trouble.
A proxy engine is the part of pkgproxy that serves one kind of client: apt, snap, Go, Docker, LXD or files. If you have not seen A request in code, start there. You will see how the daemon command builds every proxy engine from a store, a transport and shared helpers, and the revalidate package that several of them share. Then you look inside the Go and Docker proxy engines, and at how the snap proxy engine copes with a store in trouble.
02 How a proxy engine is built
newAptHandler … newFilesHandler in
cmd/pkgproxy, each with fslocal.New(cacheDir/<engine>),
after setUpChain. A disabled proxy engine's constructor returns nil, and the gateway field stays a
nil interface. Each handler names the object a request is for with
reqlog.SetObject, for the cache page's live views.
Each proxy engine is a Handler type in its own package, and the daemon command builds them one after another before it listens. Every proxy engine gets a local store of its own, rooted at a subdirectory of the cache directory named after it, with the logger and the statistics recorder. Four fetch through the shared upstream transport; apt and the files proxy engine wrap the same layers around a transport of their own. Docker and LXD can be turned off, and are then left out of the gateway. Inside, the proxy engines share helpers: fills for large blobs, flight for one shared fetch, revalidate for documents that change.
03 The revalidate package
revalidate.Group.Do shares one detached fetch per key, bounded
by Policy.Wait (10 s) and Policy.RetryInterval
(1 min). Doc.ServedStale tells the upstream status of a stale
copy served. The snap and Docker proxy engines serve stale entries with
their own code.
In the code, the apt, Go and LXD handlers share the revalidate package. A Group runs one revalidation per key, detached from the request and joined by every request for that key. Do waits for it up to the policy's Wait, ten seconds by default, then returns ErrSlow and records the host as failing. Within the RetryInterval, one minute, Do returns ErrFailing at once, without fetching. Each failure carries the offline switch's epoch, so a change of mode forgets it. The apt handler names the two fields RevalidateWait and OriginRetryInterval.
04 Inside the Go proxy engine
gomod.Request names the endpoint; Endpoint.Immutable
picks the path. fetchAndStoreImmutable runs under
flight, hashes with dirhash and commits after
verifyAgainstSumdb; serveTTL uses
revalidate.ServeDoc. See design §16.
The gateway hands every path under the go base path to the Go handler. The gomod package parses the request into a module, a version and one of the five endpoints. A go.mod or zip miss is one shared fetch, detached from the client that started it: the body goes to the store and, for a zip, to a temporary file, and is hashed with dirhash. The entry is committed only after the checksum database check, then served from the store. Unlike apt's pool files, it is not streamed while it downloads. The three changing endpoints go through the shared revalidation code.
05 Inside the Docker proxy engine
tags/<name>/_/<tag>/<id> holds a digest;
manifests/sha256/… and blobs/sha256/… hold
immutable content. revalidateTag uses headTag;
a stale pointer is served when the registry cannot be asked.
The registry proxy answers the /v2 version check itself, so Docker sends no credentials, and fetches with an anonymous pull token of its own. Everything in its store is addressed by digest. Manifests and blobs are stored once per digest, whatever repository they came through, and are checked against that digest on the way in; a blob streams through a fill. Only a tag can change. It is stored as a small pointer to a digest, one per set of accepted media types, and after five minutes it is checked with a HEAD request, which Docker Hub does not count as a pull.
06 When the snap store is in trouble
Placeholders are pkgproxy-<16 hex>-n-… or
-s-…, the prefix kept in
<cache-dir>/snap-session-prefix.
revalidationBackoff: Retry-After, or the TTL
doubled per failure, up to maxRevalidationBackoff;
markRepeatable; StoreChecks.
Four files of the snap proxy engine deal with a store in trouble. session.go answers nonce and session requests with placeholders while the store's sign-in is refused, and later answers a request that carries one with a 401, so snapd gets a real session. backoff.go remembers, by cache key, until when an expired answer whose revalidation failed is served without asking the store again, up to an hour. repeatable.go lets net/http send a nonce or a refresh again on a new connection. check.go holds the store's idle checks.
07 Where to go next general version
Next: internal/fill, for the fills blob proxy engines stream through; Store and keys, for the store every proxy engine writes; Go module proxy, for the Go proxy engine from outside.
You now know what every proxy engine is built from, and what the Go, Docker and snap proxy engines add for their own clients. To see the fills that the proxy engines with large blobs stream through, watch internal/fill. To see the store they all write to, watch Store and keys. To see the Go proxy engine as a developer uses it, watch Go module proxy.
Read more
Each proxy engine as the daemon builds it
RunFlag calls setUpChain (§27), then the constructors in this order, after opening the stats recorder, and before it listens. Each opens fslocal.New(-cache-dir/<dir>) and passes the logger, stats.Recorder and the offline switch.
| Proxy engine | Package and type | Constructor | Store dir | Gateway field |
|---|---|---|---|---|
| apt | proxy.Handler | newAptHandler | apt/ | Apt |
| snap | snapproxy.Handler | newSnapHandler | snap/ | Snap, inside offline.MiddlewareAnswer |
| Go | gomodproxy.Handler | newGoHandler | go/ | Go, at -go-base-path |
| LXD | lxdproxy.Handler | newLxdHandler | lxd/ | Lxd, at -lxd-base-path; nil when off |
| Docker | registryproxy.Handler | newDockerHandler | docker/ | Docker; nil when off |
| files | filesproxy.Handler | newFilesHandler | files/ | Files, at /files |
The gateway's ProxyEngines field is &proxyengines.Handler{Document: own}, the description this daemon serves at /pkgproxy/v1/proxy-engines. The files Handler also has Upstream and VolatileUpstream (set with -upstream-pkgproxy), Policy, Refusing and VolatileMaxSize.
Upstream settings
| Proxy engine | Upstream | Flag and default | Other settings |
|---|---|---|---|
| apt | whatever host the client's URL names | none | -apt-revalidate-ttl, -min-forced-revalidate-interval, -fill-drain-timeout |
| snap | the Snap Store API, and the CDN URLs it hands out | -snap-store-url https://api.snapcraft.io | -snap-external-url, -snap-revalidate-ttl, -fill-drain-timeout |
| Go | a module proxy | -go-upstream https://proxy.golang.org | -go-sumdb sum.golang.org (or off), -go-base-path /go, -go-revalidate-ttl |
| Docker | one registry | -docker-upstream https://registry-1.docker.io (off turns the proxy engine off) | -docker-revalidate-ttl |
| LXD | named simplestreams servers | -lxd-remote name=url, repeatable (off turns the proxy engine off) | -lxd-base-path /lxd, -lxd-revalidate-ttl, -lxd-checkpoint-interval, -lxd-tls-addr, -fill-drain-timeout |
| files | whatever URL the client names | none | -files-allow-host, -files-allow-network, -files-max-size (8G), -files-volatile-max-size (filesproxy.DefaultVolatileMaxSize, 1 MiB) |
-upstream-pkgproxy sends every proxy engine's fetches through another pkgproxy (§27); it is refused together with -snap-store-url, -go-upstream or -docker-upstream (flagsReplacedByUpstreamPkgproxy). -upstream-check-interval (0, off; at least 5m) has the store, Go and Docker upstreams checked when quiet (§28.8): snapproxy.StoreChecks and checkTargets.
Every -<engine>-revalidate-ttl defaults to -revalidate-ttl, 5 minutes. The default LXD remotes: pkgproxy-images (images.lxd.canonical.com), pkgproxy-ubuntu and pkgproxy-ubuntu-daily (cloud-images.ubuntu.com /releases, /daily), and craft-providers' craft-com.ubuntu.cloud-buildd and craft-com.ubuntu.cloud-buildd-daily (/buildd/releases, /buildd/daily).
What each proxy engine caches
| Proxy engine | Kept forever (immutable) | Revalidated after its TTL | Not cached |
|---|---|---|---|
| apt | pool/**, dists/**/by-hash/**, checked against the hashes the parsed indexes list | other dists/**: Release, InRelease, Packages, Sources | URLs without dists or pool; CONNECT tunnels |
| snap | .snap files and their dm-verity data by sha3-384; the store's download paths an anonymous answer names next to a hash (store-path/<hash>) | info, find, catalogue answers and assertions | answers to a user's macaroon that show private snaps; anything on a brand store's device session |
| Go | .mod and .zip of a version, checked against the checksum database before commit (unchecked with -go-sumdb off) | @v/list, @latest, .info | a path that is not one of the protocol's five endpoints |
| Docker | blobs and manifests by sha256 digest, checked against it | tags: a pointer to a digest per set of accepted media types, checked with a HEAD | methods other than GET and HEAD (405); tag lists and the catalog (registry 404) |
| LXD | image files by the sha256 their product catalog lists | streams/v1/index.json and product catalogs | a path no parsed catalog listed: 404, never fetched |
| files | the file by sha256 or sha512, whatever its URL | volatile files, volatile/<hex>: fetched from the origin at every request, the last copy kept | a private address outside -files-allow-network; a host outside -files-allow-host; a file over -files-max-size, or a volatile one over -files-volatile-max-size |
Shared helpers by proxy engine
| Proxy engine | fill.Group (large blobs, streamed) | flight.Group (one shared fetch) | revalidate.Group (bounded revalidation) | Stale entries on upstream failure |
|---|---|---|---|---|
| apt | pool files | dists/**, and the passthrough fallback | yes, RevalidateWait, OriginRetryInterval | yes |
| snap | .snap files | relayed metadata fetches, and assertions fetched ahead | no | its own code, with a back-off: revalidationBackoff (backoff.go) serves an expired entry without asking for Retry-After, or its TTL doubled per failure, up to maxRevalidationBackoff (1h); at most 10 000 keys |
| Go | no | .mod and .zip | yes, through revalidate.ServeDoc | yes |
| Docker | blobs | manifests | no | its own code: a stale tag pointer |
| LXD | image files | no | yes, through revalidate.ServeDoc | yes |
| files | files | no | no | volatile files: the kept copy, X-Pkgproxy-Cache: STALE, while the origin cannot answer; read into memory up to -files-volatile-max-size, not through a fill |
revalidate.Policy: Wait defaults to revalidate.DefaultWait (10s), after which Do returns ErrSlow; RetryInterval to DefaultRetryInterval (1m), within which a host that failed gets ErrFailing without a fetch; Epoch is the offline switch's, so a change of mode forgets failures. revalidate.Doc.ServedStale, when set, is told of every stale copy served, for the upstream status (§28.13); the proxy engines that serve stale with their own code call offline.Switch.ServedStale themselves.
The snap proxy engine and a store in trouble
| File | What it does |
|---|---|
session.go | While the store's auth is not contacted (offline, or its auth breaker open), or in store-overloaded when the store fails a nonce or session request, answers it with a placeholder, pkgproxy-<16 hex>-n-<32 hex> or -s-, marked X-Pkgproxy-Synthesised. Once auth answers again, a request carrying this pkgproxy's placeholder gets 401 with WWW-Authenticate: Macaroon refresh_device_session=1; while auth is still refused but the request's class is not, the placeholder is dropped. The prefix is kept in <cache-dir>/snap-session-prefix (cmd/pkgproxy/upstreams.go). |
backoff.go | revalidationBackoff, above. An answer, or a change of offline mode, forgets a key. |
repeatable.go | markRepeatable: a nonce and a refresh are marked idempotent, with a nil X-Idempotency-Key, which is not sent, and keep a body of up to 1 MiB, so net/http sends them again on a new connection when the store's edge closed the pooled one, as it does after each 429 and 408. A session request, which spends its nonce, is sent once. |
check.go | Check, StoreChecks: the idle checks of the store's classes (§28.8). |
Disabled proxy engines
newDockerHandler returns nil for -docker-upstream off, newLxdHandler nil for -lxd-remote off. RunFlag assigns them to the gateway only when non-nil: a nil *lxdproxy.Handler stored in an http.Handler field would be a non-nil interface, and the gateway's h.Lxd != nil check would call through it. With Docker off, a registry-shaped path still matches and gets 404; with LXD off, /lxd falls through to the snap prefixes and then 404.
In the documentation
Roads from here
SNAP
- Snap Store relay — the snap proxy engine from the outside
GO MODULES
- Go module proxy — the Go proxy engine from the outside
DOCKER
- Docker Hub registry mirror — the Docker proxy engine from the outside
INSIDE PKGPROXY · for contributors
- A request in code — how a request reaches a proxy engine
- internal/fill — the fills blob proxy engines stream through
- Store and keys — the store every proxy engine writes
- The source tree — the rest of the source tree