The proxy engines' packages

contributors for contributors

How each proxy engine (proxy, snapproxy, gomodproxy, registryproxy, lxdproxy, filesproxy) is built from a store, the upstream transport and shared helpers, what the Go and Docker proxy engines do inside, and how the snap proxy engine copes with a store in trouble.

On the map: inside pkgproxy

▶ Watch this deck — 7 scenes, ~5 min

01 What this topic covers

ABOUT THIS TOPIC The proxy engines' packages the part of pkgproxy that serves one kind of client 1 How a proxy engine is built 2 The revalidate package 3 Inside the Go proxy engine 4 Inside the Docker proxy engine 5 When the snap store is in trouble
About this topic, The proxy engines' packages: how a proxy engine is built, the revalidate package, inside the Go proxy engine, inside the Docker proxy engine, a store in trouble

How each proxy engine is built from a store, a transport and shared helpers, what the Go and Docker proxy engines do inside, and the snap proxy engine's code for a store in trouble.

A proxy engine is the part of pkgproxy that serves one kind of client: apt, snap, Go, Docker, LXD or files. If you have not seen A request in code, start there. You will see how the daemon command builds every proxy engine from a store, a transport and shared helpers, and the revalidate package that several of them share. Then you look inside the Go and Docker proxy engines, and at how the snap proxy engine copes with a store in trouble.

02 How a proxy engine is built

SIX HANDLERS, ONE SHAPE package proxy snapproxy gomodproxy registryproxy lxdproxy filesproxy its own packages cachekey · index · origin · apt snap gomod · sumdb client anonymous pull tokens simplestreams · lxdindex NewTransport · AddressPolicy store under -cache-dir apt/ snap/ go/ docker/ lxd/ files/ every Handler gets Store Logger · Stats upstreamTransport() apt, files: own base, same layers around it -docker-upstream off -lxd-remote off nil: left out of the gateway shared helpers the proxy engines import fill · flight · revalidate · httprange · reqlog · stats · offline
Six proxy engine packages, each with its own cache subdirectory and its own protocol packages; every Handler gets a store, the upstream transport (apt and files wrap the same layers around their own), a logger and the stats recorder; Docker and LXD can be off; shared helpers fill, flight, revalidate, httprange and reqlog underneath

newAptHandler … newFilesHandler in cmd/pkgproxy, each with fslocal.New(cacheDir/<engine>), after setUpChain. A disabled proxy engine's constructor returns nil, and the gateway field stays a nil interface. Each handler names the object a request is for with reqlog.SetObject, for the cache page's live views.

Each proxy engine is a Handler type in its own package, and the daemon command builds them one after another before it listens. Every proxy engine gets a local store of its own, rooted at a subdirectory of the cache directory named after it, with the logger and the statistics recorder. Four fetch through the shared upstream transport; apt and the files proxy engine wrap the same layers around a transport of their own. Docker and LXD can be turned off, and are then left out of the gateway. Inside, the proxy engines share helpers: fills for large blobs, flight for one shared fetch, revalidate for documents that change.

03 The revalidate package

INTERNAL/REVALIDATE apt go lxd Group.Do(ctx, p, host, key, fetch) one detached fetch per key, shared upstream host after p.Wait (DefaultWait = 10s) ErrSlow · Failed(p, host) within p.RetryInterval (1m) ErrFailing · fetch not called either way the caller serves the stale entry failure{at, epoch}: another offline.Switch epoch forgets it proxy.Handler: RevalidateWait, OriginRetryInterval
Requests for one key call revalidate.Group.Do, which shares one detached fetch; after Policy.Wait it returns ErrSlow and marks the host; within Policy.RetryInterval it returns ErrFailing without fetching; the epoch of the offline switch resets failures

revalidate.Group.Do shares one detached fetch per key, bounded by Policy.Wait (10 s) and Policy.RetryInterval (1 min). Doc.ServedStale tells the upstream status of a stale copy served. The snap and Docker proxy engines serve stale entries with their own code.

In the code, the apt, Go and LXD handlers share the revalidate package. A Group runs one revalidation per key, detached from the request and joined by every request for that key. Do waits for it up to the policy's Wait, ten seconds by default, then returns ErrSlow and records the host as failing. Within the RetryInterval, one minute, Do returns ErrFailing at once, without fetching. Each failure carries the offline switch's epoch, so a change of mode forgets it. The apt handler names the two fields RevalidateWait and OriginRetryInterval.

04 Inside the Go proxy engine

INTERNAL/GOMODPROXY internal/gateway /go internal/gomodproxy gomod.Request{Endpoint} .mod / .zip miss · one flight, detached fetch upstream store + temp file dirhash sumdb, commit then served from the store: not streamed while it downloads list · @latest · .info revalidate.ServeDoc
The gateway passes /go paths to gomodproxy; gomod parses them; mod and zip misses are one flight that writes to the store and a temp file, hashes with dirhash, checks the sumdb and commits; list, latest and info go through revalidate.ServeDoc

gomod.Request names the endpoint; Endpoint.Immutable picks the path. fetchAndStoreImmutable runs under flight, hashes with dirhash and commits after verifyAgainstSumdb; serveTTL uses revalidate.ServeDoc. See design §16.

The gateway hands every path under the go base path to the Go handler. The gomod package parses the request into a module, a version and one of the five endpoints. A go.mod or zip miss is one shared fetch, detached from the client that started it: the body goes to the store and, for a zip, to a temporary file, and is hashed with dirhash. The entry is committed only after the checksum database check, then served from the store. Unlike apt's pool files, it is not streamed while it downloads. The three changing endpoints go through the shared revalidation code.

05 Inside the Docker proxy engine

INTERNAL/REGISTRYPROXY docker /v2/ 200, no auth asked token registry-1.docker.io by digest: immutable, checked on the way in manifests/sha256/9f… once per digest blobs/sha256/4c… streamed through a fill by tag: can change tags/library/debian/_/12/… a pointer to a digest one pointer per set of accepted media types after -docker-revalidate-ttl (5m): HEAD /v2/library/debian/manifests/12 same digest: StoredAt moves on · new digest: fetch it by digest a HEAD is not counted as a pull by Docker Hub
registryproxy answers /v2/ itself and uses an anonymous token upstream; in the store a tag pointer leads to a manifest by digest, which lists blobs by digest; only the tag pointer is revalidated, with a HEAD after five minutes

tags/<name>/_/<tag>/<id> holds a digest; manifests/sha256/… and blobs/sha256/… hold immutable content. revalidateTag uses headTag; a stale pointer is served when the registry cannot be asked.

The registry proxy answers the /v2 version check itself, so Docker sends no credentials, and fetches with an anonymous pull token of its own. Everything in its store is addressed by digest. Manifests and blobs are stored once per digest, whatever repository they came through, and are checked against that digest on the way in; a blob streams through a fill. Only a tag can change. It is stored as a small pointer to a digest, one per set of accepted media types, and after five minutes it is checked with a HEAD request, which Docker Hub does not count as a pull.

06 When the snap store is in trouble

INTERNAL/SNAPPROXY session.go POST …/auth/nonces, …/auth/sessions while auth is refused: {"nonce": "pkgproxy-3f1c…-n-…"} X-Pkgproxy-Synthesised auth back: 401 WWW-Authenticate: Macaroon refresh_device_session=1 backoff.go an expired answer whose revalidation failed: served, store not asked until Retry-After, or the TTL doubled per failure, at most 1h repeatable.go nonce and refresh marked idempotent, body under 1 MiB kept: net/http resends on a new connection when the edge closed the old one check.go StoreChecks: what upstream-check-interval asks the store
Four files of snapproxy: session.go answers nonces and sessions with placeholders while the store's auth is refused, then 401 refresh_device_session; backoff.go serves an expired answer without asking for up to an hour; repeatable.go marks nonces and refreshes idempotent so net/http resends them on a new connection; check.go holds StoreChecks

Placeholders are pkgproxy-<16 hex>-n-… or -s-…, the prefix kept in <cache-dir>/snap-session-prefix. revalidationBackoff: Retry-After, or the TTL doubled per failure, up to maxRevalidationBackoff; markRepeatable; StoreChecks.

Four files of the snap proxy engine deal with a store in trouble. session.go answers nonce and session requests with placeholders while the store's sign-in is refused, and later answers a request that carries one with a 401, so snapd gets a real session. backoff.go remembers, by cache key, until when an expired answer whose revalidation failed is served without asking the store again, up to an hour. repeatable.go lets net/http send a nonce or a refresh again on a new connection. check.go holds the store's idle checks.

07 Where to go next general version

WHERE NEXT every proxy engine is a store, a transport and shared helpers, plus the protocol of its own clients you are here The proxy engines' packages internal/fill the fills blob proxy engines stream through Store and keys the store every proxy engine writes Go module proxy the Go proxy engine from the outside
Where next from The proxy engines' packages: internal/fill, Store and keys, Go module proxy

Next: internal/fill, for the fills blob proxy engines stream through; Store and keys, for the store every proxy engine writes; Go module proxy, for the Go proxy engine from outside.

You now know what every proxy engine is built from, and what the Go, Docker and snap proxy engines add for their own clients. To see the fills that the proxy engines with large blobs stream through, watch internal/fill. To see the store they all write to, watch Store and keys. To see the Go proxy engine as a developer uses it, watch Go module proxy.

Read more

Each proxy engine as the daemon builds it

RunFlag calls setUpChain (§27), then the constructors in this order, after opening the stats recorder, and before it listens. Each opens fslocal.New(-cache-dir/<dir>) and passes the logger, stats.Recorder and the offline switch.

Proxy enginePackage and typeConstructorStore dirGateway field
aptproxy.HandlernewAptHandlerapt/Apt
snapsnapproxy.HandlernewSnapHandlersnap/Snap, inside offline.MiddlewareAnswer
Gogomodproxy.HandlernewGoHandlergo/Go, at -go-base-path
LXDlxdproxy.HandlernewLxdHandlerlxd/Lxd, at -lxd-base-path; nil when off
Dockerregistryproxy.HandlernewDockerHandlerdocker/Docker; nil when off
filesfilesproxy.HandlernewFilesHandlerfiles/Files, at /files

The gateway's ProxyEngines field is &proxyengines.Handler{Document: own}, the description this daemon serves at /pkgproxy/v1/proxy-engines. The files Handler also has Upstream and VolatileUpstream (set with -upstream-pkgproxy), Policy, Refusing and VolatileMaxSize.

Upstream settings

Proxy engineUpstreamFlag and defaultOther settings
aptwhatever host the client's URL namesnone-apt-revalidate-ttl, -min-forced-revalidate-interval, -fill-drain-timeout
snapthe Snap Store API, and the CDN URLs it hands out-snap-store-url https://api.snapcraft.io-snap-external-url, -snap-revalidate-ttl, -fill-drain-timeout
Goa module proxy-go-upstream https://proxy.golang.org-go-sumdb sum.golang.org (or off), -go-base-path /go, -go-revalidate-ttl
Dockerone registry-docker-upstream https://registry-1.docker.io (off turns the proxy engine off)-docker-revalidate-ttl
LXDnamed simplestreams servers-lxd-remote name=url, repeatable (off turns the proxy engine off)-lxd-base-path /lxd, -lxd-revalidate-ttl, -lxd-checkpoint-interval, -lxd-tls-addr, -fill-drain-timeout
fileswhatever URL the client namesnone-files-allow-host, -files-allow-network, -files-max-size (8G), -files-volatile-max-size (filesproxy.DefaultVolatileMaxSize, 1 MiB)

-upstream-pkgproxy sends every proxy engine's fetches through another pkgproxy (§27); it is refused together with -snap-store-url, -go-upstream or -docker-upstream (flagsReplacedByUpstreamPkgproxy). -upstream-check-interval (0, off; at least 5m) has the store, Go and Docker upstreams checked when quiet (§28.8): snapproxy.StoreChecks and checkTargets.

Every -<engine>-revalidate-ttl defaults to -revalidate-ttl, 5 minutes. The default LXD remotes: pkgproxy-images (images.lxd.canonical.com), pkgproxy-ubuntu and pkgproxy-ubuntu-daily (cloud-images.ubuntu.com /releases, /daily), and craft-providers' craft-com.ubuntu.cloud-buildd and craft-com.ubuntu.cloud-buildd-daily (/buildd/releases, /buildd/daily).

What each proxy engine caches

Proxy engineKept forever (immutable)Revalidated after its TTLNot cached
aptpool/**, dists/**/by-hash/**, checked against the hashes the parsed indexes listother dists/**: Release, InRelease, Packages, SourcesURLs without dists or pool; CONNECT tunnels
snap.snap files and their dm-verity data by sha3-384; the store's download paths an anonymous answer names next to a hash (store-path/<hash>)info, find, catalogue answers and assertionsanswers to a user's macaroon that show private snaps; anything on a brand store's device session
Go.mod and .zip of a version, checked against the checksum database before commit (unchecked with -go-sumdb off)@v/list, @latest, .infoa path that is not one of the protocol's five endpoints
Dockerblobs and manifests by sha256 digest, checked against ittags: a pointer to a digest per set of accepted media types, checked with a HEADmethods other than GET and HEAD (405); tag lists and the catalog (registry 404)
LXDimage files by the sha256 their product catalog listsstreams/v1/index.json and product catalogsa path no parsed catalog listed: 404, never fetched
filesthe file by sha256 or sha512, whatever its URLvolatile files, volatile/<hex>: fetched from the origin at every request, the last copy kepta private address outside -files-allow-network; a host outside -files-allow-host; a file over -files-max-size, or a volatile one over -files-volatile-max-size

Shared helpers by proxy engine

Proxy enginefill.Group (large blobs, streamed)flight.Group (one shared fetch)revalidate.Group (bounded revalidation)Stale entries on upstream failure
aptpool filesdists/**, and the passthrough fallbackyes, RevalidateWait, OriginRetryIntervalyes
snap.snap filesrelayed metadata fetches, and assertions fetched aheadnoits own code, with a back-off: revalidationBackoff (backoff.go) serves an expired entry without asking for Retry-After, or its TTL doubled per failure, up to maxRevalidationBackoff (1h); at most 10 000 keys
Gono.mod and .zipyes, through revalidate.ServeDocyes
Dockerblobsmanifestsnoits own code: a stale tag pointer
LXDimage filesnoyes, through revalidate.ServeDocyes
filesfilesnonovolatile files: the kept copy, X-Pkgproxy-Cache: STALE, while the origin cannot answer; read into memory up to -files-volatile-max-size, not through a fill

revalidate.Policy: Wait defaults to revalidate.DefaultWait (10s), after which Do returns ErrSlow; RetryInterval to DefaultRetryInterval (1m), within which a host that failed gets ErrFailing without a fetch; Epoch is the offline switch's, so a change of mode forgets failures. revalidate.Doc.ServedStale, when set, is told of every stale copy served, for the upstream status (§28.13); the proxy engines that serve stale with their own code call offline.Switch.ServedStale themselves.

The snap proxy engine and a store in trouble

FileWhat it does
session.goWhile the store's auth is not contacted (offline, or its auth breaker open), or in store-overloaded when the store fails a nonce or session request, answers it with a placeholder, pkgproxy-<16 hex>-n-<32 hex> or -s-, marked X-Pkgproxy-Synthesised. Once auth answers again, a request carrying this pkgproxy's placeholder gets 401 with WWW-Authenticate: Macaroon refresh_device_session=1; while auth is still refused but the request's class is not, the placeholder is dropped. The prefix is kept in <cache-dir>/snap-session-prefix (cmd/pkgproxy/upstreams.go).
backoff.gorevalidationBackoff, above. An answer, or a change of offline mode, forgets a key.
repeatable.gomarkRepeatable: a nonce and a refresh are marked idempotent, with a nil X-Idempotency-Key, which is not sent, and keep a body of up to 1 MiB, so net/http sends them again on a new connection when the store's edge closed the pooled one, as it does after each 429 and 408. A session request, which spends its nonce, is sent once.
check.goCheck, StoreChecks: the idle checks of the store's classes (§28.8).

Disabled proxy engines

newDockerHandler returns nil for -docker-upstream off, newLxdHandler nil for -lxd-remote off. RunFlag assigns them to the gateway only when non-nil: a nil *lxdproxy.Handler stored in an http.Handler field would be a non-nil interface, and the gateway's h.Lxd != nil check would call through it. With Docker off, a registry-shaped path still matches and gets 404; with LXD off, /lxd falls through to the snap prefixes and then 404.

Roads from here

SNAP

GO MODULES

DOCKER

INSIDE PKGPROXY · for contributors