pkgproxy design: Docker

The Docker Hub registry mirror. Part of the pkgproxy design, whose index says which file holds each § number.

• 19. Docker Hub registry mirror
  • 19.1 Scope: anonymous Docker Hub pulls
  • 19.2 Why the mirror lives at /v2/, not under a base path
  • 19.3 Authentication: pkgproxy holds the token, the client never does
  • 19.4 Cache layout and policy
  • 19.5 Client enablement
  • 19.6 Flags and explicit boundaries
  • 19.6.1 Stats

19. Docker Hub registry mirror

The handler is internal/registryproxy; pkgproxyctl setup docker configures a client.

19.1 Scope: anonymous Docker Hub pulls

A Docker daemon’s registry-mirrors setting makes it try a mirror before Docker Hub, for docker.io images only — it has no effect on ghcr.io, quay.io or any other registry, so those are out of scope here (they would need per-registry containerd hosts.toml / podman registries.conf configuration, a different client-side mechanism). And only anonymous pulls of public images are supported. Together these keep the design small: one fixed upstream, no credentials anywhere, and no question of whether serving cached content to a client is authorized — everything cached is public.

Pushes, tags/list and the catalog are not served (404). The referrers API (/v2/<name>/referrers/<digest>) is relayed to the upstream registry uncached, since the set of referrers of a digest grows as signatures and attestations are pushed; a registry without the API answers 404 and the client falls back to the referrers tag schema. Otherwise: docker pull and podman pull need only the version check, manifests and blobs.

19.2 Why the mirror lives at /v2/, not under a base path

Unlike the go and LXD sides, whose clients are given a URL with a path, Docker validates each mirror as a bare scheme://host[:port] and rejects one with a path component. So the registry API is served at the root, sharing the /v2/ prefix with the snap store (/v2/snaps/..., /v2/assertions/..., /v2/download/...). registryproxy.Matches tells them apart by shape: the version check itself (/v2/), or a path ending in /manifests/<ref> / /blobs/<digest> / /referrers/<digest> after a grammar-valid repository name. No snap store path has such a component, and any other /v2/... path goes to the snap side. The gateway checks Docker after the go and LXD sides and before snap, and sends snap only its explicit prefixes (/v2, plus /api/v1 and /v1, which snapd also sends to the store URL it is pointed at): a path no side claims (a disabled side’s base path, a typo) gets a 404 from the gateway itself instead of being relayed to the snap store.

19.3 Authentication: pkgproxy holds the token, the client never does

Docker Hub requires a Bearer token even for public images. pkgproxy answers GET /v2/ with 200 itself, so the client concludes this mirror needs no authentication and never sends credentials (or asks us for a token). On a cache miss pkgproxy handles the registry’s 401 + WWW-Authenticate: Bearer realm=...,service=...,scope=... challenge the standard way, requesting an anonymous token from the named realm (auth.docker.io) and caching it per repository until it expires. The realm comes from the configured upstream’s own response, so it is as trusted as the upstream. The http.Client drops the Authorization header whenever a redirect (blob downloads are redirected to a CDN) leaves the registry’s host.

19.4 Cache layout and policy

Blobs and manifests are keyed by digest alone, not by repository: the digest fully determines the bytes, so two repositories (or tags) sharing a layer share one entry. A consequence is that a cached digest is served under any repository name, including one that does not exist upstream or does not hold that digest: a request for /v2/<name>/blobs/<digest> is answered from the cache without asking whether <name> has it. A real registry would say BLOB_UNKNOWN there, but the bytes are the ones the digest names either way, and since only public content is cached (§19.1) nothing is revealed that the client could not pull from Hub itself; asking the registry on every hit would cost the round trip caching is meant to save. The _ path component in tag keys cannot be a repository name component (those start alphanumeric), so keys of different repositories cannot collide on the filesystem. <id> is a hash of the set of media types the client’s Accept headers accept, because a registry may answer one tag with a different manifest (index vs. single-platform, schema 2 vs. OCI) depending on what the client accepts. It is the set — split on commas, trimmed, lower-cased, parameters dropped, q=0 types excluded, sorted and deduplicated — not the raw header text: dockerd, containerd and podman spell the same set differently (one header per type or one comma-separated header, different orders and spacing), registries choose by membership in the set rather than by order or weight, and a pointer missed because of spelling costs a full manifest GET against Hub’s pull rate limit. The client’s headers are still sent upstream as they came.

Tags are resolved with HEAD. A tag with no pointer, or an expired one, is resolved with HEAD /v2/<name>/manifests/<tag> and its Docker-Content-Digest; Docker Hub does not count HEAD requests against the anonymous pull rate limit (the main practical reason to cache at all), so the common “tag unchanged” case costs no pull, and neither does a tag that resolves to a manifest already cached (pulled by digest, through another tag, or for another Accept set). Only a digest not cached is fetched with a GET, by digest. A HEAD that names no digest, or fails other than by saying the tag is unknown, falls back to a GET of the tag on a first resolution. If the registry cannot be reached during revalidation, the stale resolution is served with a warning, as the other sides do. If it reports the tag gone with a 404, the client gets a 404 and falls back to Docker Hub itself. A 401 or 403 is ambiguous: to an anonymous client a repository that was deleted, one that was made private, and a token the registry turned down all look the same, so a tag that resolved before keeps being served from its stale pointer (with the warning, and asked again on every request until the registry answers otherwise). With nothing cached, a 401 or 403 is a 404 to the client.

Blob streaming. A GET miss on a blob starts a fill, the same design and code as the apt side’s pool/** fills (§8.1; internal/fill): the download runs detached from any client, and every GET for the blob while it runs, Range resumes included, follows the bytes as they are written. If the fill cannot be tailed (no Content-Length upstream), requests wait for it and are served from the Store. A HEAD on a miss is answered from the registry’s own HEAD without caching. Content that does not hash to its digest is never committed, and the blob’s last byte is held back from tailing readers until the digest has been verified and the blob committed: a corrupt blob therefore never reaches a client as a complete body (its response is cut short instead), and a client that has read a whole blob knows it is cached. Unlike the apt fill there is no drain: where apt hands the clients already following a fill the origin’s unverified bytes (§8 step 7), here they get a cut-short response. Manifest and tag fetches, which are shared between concurrent requests by singleflight, are likewise detached from the context of the request that started them.

19.5 Client enablement

pkgproxyctl setup docker prints guidance on a terminal, and a script for sudo sh otherwise: choose /var/snap/docker/current/config/daemon.json (Docker snap) or /etc/docker/daemon.json, create it if absent or merge in place with sed -z -E (pkgproxy first, other settings and mirrors kept; no jq, which Ubuntu Core lacks in a form that can write the file; the result is checked and the file left alone if the layout was not understood), restart the daemon. -undo removes only pkgproxy’s entry, and the registry-mirrors key when no mirror is left in it, so that undoing a setup that added the only mirror leaves no empty list behind. Docker accepts an http:// mirror, so no TLS is involved. The daemon restart (root) is why, unlike setup lxd-remote (§17.13), this command needs root rather than refusing it.

19.6 Flags and explicit boundaries

-docker-upstream names the one upstream (off disables the side), and -docker-revalidate-ttl is how long a tag pointer is trusted (§18).

Not done: non-Hub registries, authenticated pulls, garbage collection.

19.6.1 Stats

Blob serves are recorded in internal/stats as type "docker" (§13.5), keyed <repository>@<digest> — Docker’s own digest-reference syntax — so the report names the image a layer was pulled for rather than a bare, anonymous blobs/sha256/<hex> cache key. A layer shared by several images is therefore counted once per repository it is pulled through, even though it is stored once; the bytes-saved total still adds up, since each hit is counted exactly once, under the repository that asked for it. Hit/fetch classification follows §13.2: a cached blob is a hit, the request that started a fill is a fetch, and a request that joined another’s fill is a hit. Bytes saved are what the hits sent, as for every kind: a HEAD answered from the cache is a hit that saves nothing, and a Range request saves the range it got. A HEAD for a blob that is not cached is answered by the registry and not counted. Manifests and tag resolution are not recorded: they are a few KB each and would only crowd layers out of a bytes-saved ranking. pkgproxyctl stats shortens the digest in the KEY column to 12 hex digits, the form docker images shows.