The sides' packages

contributors for contributors

How each side (proxy, snapproxy, gomodproxy, registryproxy, lxdproxy, filesproxy) is built from a store, the upstream transport and shared helpers, and what the Go and Docker sides do inside.

On the map: inside pkgproxy

▶ Watch this deck — 6 scenes, ~4 min

01 What this topic covers

ABOUT THIS TOPIC The sides' packages the part of pkgproxy that serves one kind of client 1 How a side is built 2 The revalidate package 3 Inside the Go side 4 Inside the Docker side
About this topic, The sides' packages: how a side is built, the revalidate package, inside the Go side, inside the Docker side

How each side is built from a store, a transport and shared helpers, and what the Go and Docker sides do inside.

A side is the part of pkgproxy that serves one kind of client: apt, snap, Go, Docker, LXD or files by checksum. If you have not seen A request in code, start there. You will see how the daemon command builds every side from a store, a transport and shared helpers, and the revalidate package that three of them share. Then you look inside the Go side and inside the Docker side.

02 How a side is built

SIX HANDLERS, ONE SHAPE package proxy snapproxy gomodproxy registryproxy lxdproxy filesproxy its own packages cachekey · index · origin · apt snap gomod · sumdb client anonymous pull tokens simplestreams · lxdindex NewTransport · AddressPolicy store under -cache-dir apt/ snap/ go/ docker/ lxd/ files/ every Handler gets Store Logger · Stats upstreamTransport() apt, files: own base, same layers around it -docker-upstream off -lxd-remote off nil: left out of the gateway shared helpers the sides import fill · flight · revalidate · httprange · reqlog · stats · offline
Six side packages, each with its own cache subdirectory and its own protocol packages; every Handler gets a store, the upstream transport (apt and files wrap the same layers around their own), a logger and the stats recorder; Docker and LXD can be off; shared helpers fill, flight, revalidate, httprange and reqlog underneath

newAptHandler … newFilesHandler in cmd/pkgproxy, each with fslocal.New(cacheDir/<side>). A disabled side's constructor returns nil, and the gateway field stays a nil interface. Each handler names the object a request is for with reqlog.SetObject, for the cache page's live views.

Each side is a Handler type in its own package, and the daemon command builds them one after another before it listens. Every side gets a local store of its own, rooted at a subdirectory of the cache directory named after it, with the logger and the statistics recorder. Four fetch through the shared upstream transport; apt and the files side wrap the same layers around a transport of their own. Docker and LXD can be turned off, and are then left out of the gateway. Inside, the sides share helpers: fills for large blobs, flight for one shared fetch, revalidate for documents that change.

03 The revalidate package

INTERNAL/REVALIDATE apt go lxd Group.Do(ctx, p, host, key, fetch) one detached fetch per key, shared upstream host after p.Wait (DefaultWait = 10s) ErrSlow · Failed(p, host) within p.RetryInterval (1m) ErrFailing · fetch not called either way the caller serves the stale entry failure{at, epoch}: another offline.Switch epoch forgets it proxy.Handler: RevalidateWait, OriginRetryInterval
Requests for one key call revalidate.Group.Do, which shares one detached fetch; after Policy.Wait it returns ErrSlow and marks the host; within Policy.RetryInterval it returns ErrFailing without fetching; the epoch of the offline switch resets failures

revalidate.Group.Do shares one detached fetch per key, bounded by Policy.Wait (10 s) and Policy.RetryInterval (1 min). The snap and Docker sides serve stale entries with their own code.

In the code, the apt, Go and LXD handlers share the revalidate package. A Group runs one revalidation per key, detached from the request and joined by every request for that key. Do waits for it up to the policy's Wait, ten seconds by default, then returns ErrSlow and records the host as failing. Within the RetryInterval, one minute, Do returns ErrFailing at once, without fetching. Each failure carries the offline switch's epoch, so a change of mode forgets it. The apt handler names the two fields RevalidateWait and OriginRetryInterval.

04 Inside the Go side

INTERNAL/GOMODPROXY internal/gateway /go internal/gomodproxy gomod.Request{Endpoint} .mod / .zip miss · one flight, detached fetch upstream store + temp file dirhash sumdb, commit then served from the store: not streamed while it downloads list · @latest · .info revalidate.ServeDoc
The gateway passes /go paths to gomodproxy; gomod parses them; mod and zip misses are one flight that writes to the store and a temp file, hashes with dirhash, checks the sumdb and commits; list, latest and info go through revalidate.ServeDoc

gomod.Request names the endpoint; Endpoint.Immutable picks the path. fetchAndStoreImmutable runs under flight, hashes with dirhash and commits after verifyAgainstSumdb; serveTTL uses revalidate.ServeDoc. See design §16.

The gateway hands every path under the go base path to the Go handler. The gomod package parses the request into a module, a version and one of the five endpoints. A go.mod or zip miss is one shared fetch, detached from the client that started it: the body goes to the store and, for a zip, to a temporary file, and is hashed with dirhash. The entry is committed only after the checksum database check, then served from the store. Unlike apt's pool files, it is not streamed while it downloads. The three changing endpoints go through the shared revalidation code.

05 Inside the Docker side

INTERNAL/REGISTRYPROXY docker /v2/ 200, no auth asked token registry-1.docker.io by digest: immutable, checked on the way in manifests/sha256/9f… once per digest blobs/sha256/4c… streamed through a fill by tag: can change tags/library/debian/_/12/… a pointer to a digest one pointer per set of accepted media types after -docker-revalidate-ttl (5m): HEAD /v2/library/debian/manifests/12 same digest: StoredAt moves on · new digest: fetch it by digest a HEAD is not counted as a pull by Docker Hub
registryproxy answers /v2/ itself and uses an anonymous token upstream; in the store a tag pointer leads to a manifest by digest, which lists blobs by digest; only the tag pointer is revalidated, with a HEAD after five minutes

tags/<name>/_/<tag>/<id> holds a digest; manifests/sha256/… and blobs/sha256/… hold immutable content. revalidateTag uses headTag; a stale pointer is served when the registry cannot be asked.

The registry proxy answers the /v2 version check itself, so Docker sends no credentials, and fetches with an anonymous pull token of its own. Everything in its store is addressed by digest. Manifests and blobs are stored once per digest, whatever repository they came through, and are checked against that digest on the way in; a blob streams through a fill. Only a tag can change. It is stored as a small pointer to a digest, one per set of accepted media types, and after five minutes it is checked with a HEAD request, which Docker Hub does not count as a pull.

06 Where to go next general version

WHERE NEXT every side is a store, a transport and shared helpers, plus the protocol of its own clients you are here The sides' packages internal/fill the fills blob sides stream through Store and keys the store every side writes Go module proxy the Go side from the outside
Where next from The sides' packages: internal/fill, Store and keys, Go module proxy

Next: internal/fill, for the fills blob sides stream through; Store and keys, for the store every side writes; Go module proxy, for the Go side from outside.

You now know what every side is built from, and what the Go and Docker sides add for their own clients. To see the fills that the sides with large blobs stream through, watch internal/fill. To see the store they all write to, watch Store and keys. To see the Go side as a developer uses it, watch Go module proxy.

Read more

Each side as the daemon builds it

RunFlag calls the constructors in this order, after opening the stats recorder, and before it listens. Each opens fslocal.New(-cache-dir/<dir>) and passes the logger, stats.Recorder and the offline switch.

SidePackage and typeConstructorStore dirGateway field
aptproxy.HandlernewAptHandlerapt/Apt
snapsnapproxy.HandlernewSnapHandlersnap/Snap, inside offline.MiddlewareAnswer
Gogomodproxy.HandlernewGoHandlergo/Go, at -go-base-path
LXDlxdproxy.HandlernewLxdHandlerlxd/Lxd, at -lxd-base-path; nil when off
Dockerregistryproxy.HandlernewDockerHandlerdocker/Docker; nil when off
filesfilesproxy.HandlernewFilesHandlerfiles/Files, at /files

Upstream settings

SideUpstreamFlag and defaultOther settings
aptwhatever host the client's URL namesnone-apt-revalidate-ttl, -min-forced-revalidate-interval, -fill-drain-timeout
snapthe Snap Store API, and the CDN URLs it hands out-snap-store-url https://api.snapcraft.io-snap-external-url, -snap-revalidate-ttl, -fill-drain-timeout
Goa module proxy-go-upstream https://proxy.golang.org-go-sumdb sum.golang.org (or off), -go-base-path /go, -go-revalidate-ttl
Dockerone registry-docker-upstream https://registry-1.docker.io (off turns the side off)-docker-revalidate-ttl
LXDnamed simplestreams servers-lxd-remote name=url, repeatable (off turns the side off)-lxd-base-path /lxd, -lxd-revalidate-ttl, -lxd-checkpoint-interval, -lxd-tls-addr, -fill-drain-timeout
fileswhatever URL the client namesnone-files-allow-host, -files-allow-network, -files-max-size (8G)

Every -<side>-revalidate-ttl defaults to -revalidate-ttl, 5 minutes. The default LXD remotes: pkgproxy-images (images.lxd.canonical.com), pkgproxy-ubuntu and pkgproxy-ubuntu-daily (cloud-images.ubuntu.com /releases, /daily), and craft-providers' craft-com.ubuntu.cloud-buildd and craft-com.ubuntu.cloud-buildd-daily (/buildd/releases, /buildd/daily).

What each side caches

SideKept forever (immutable)Revalidated after its TTLNot cached
aptpool/**, dists/**/by-hash/**, checked against the hashes the parsed indexes listother dists/**: Release, InRelease, Packages, SourcesURLs without dists or pool; CONNECT tunnels
snap.snap files by sha3-384info, find, catalogue answers and assertionsanswers to a user's macaroon that show private snaps; anything on a brand store's device session
Go.mod and .zip of a version, checked against the checksum database before commit (unchecked with -go-sumdb off)@v/list, @latest, .infoa path that is not one of the protocol's five endpoints
Dockerblobs and manifests by sha256 digest, checked against ittags: a pointer to a digest per set of accepted media types, checked with a HEADmethods other than GET and HEAD (405); tag lists and the catalog (registry 404)
LXDimage files by the sha256 their product catalog listsstreams/v1/index.json and product catalogsa path no parsed catalog listed: 404, never fetched
filesthe file by sha256 or sha512, whatever its URLnothinga private address outside -files-allow-network; a host outside -files-allow-host; a file over -files-max-size

Shared helpers by side

Sidefill.Group (large blobs, streamed)flight.Group (one shared fetch)revalidate.Group (bounded revalidation)Stale entries on upstream failure
aptpool filesdists/**, and the passthrough fallbackyes, RevalidateWait, OriginRetryIntervalyes
snap.snap filesrelayed metadata fetches, and assertions fetched aheadnoits own code
Gono.mod and .zipyes, through revalidate.ServeDocyes
Dockerblobsmanifestsnoits own code: a stale tag pointer
LXDimage filesnoyes, through revalidate.ServeDocyes
filesfilesnononot applicable

revalidate.Policy: Wait defaults to revalidate.DefaultWait (10s), after which Do returns ErrSlow; RetryInterval to DefaultRetryInterval (1m), within which a host that failed gets ErrFailing without a fetch; Epoch is the offline switch's, so a change of mode forgets failures.

Disabled sides

newDockerHandler returns nil for -docker-upstream off, newLxdHandler nil for -lxd-remote off. RunFlag assigns them to the gateway only when non-nil: a nil *lxdproxy.Handler stored in an http.Handler field would be a non-nil interface, and the gateway's h.Lxd != nil check would call through it. With Docker off, a registry-shaped path still matches and gets 404; with LXD off, /lxd falls through to the snap prefixes and then 404.

Roads from here

GO MODULES

DOCKER

INSIDE PKGPROXY · for contributors