Docker Hub registry mirror

docker for users for administrators for CI operators

How pkgproxy becomes a registry mirror for Docker Hub at the root of its address, holds the registry token for anonymous pulls, rechecks tags and keeps manifests and layers by digest forever.

On the map: docker

▶ Watch this deck — 8 scenes, ~4 min

01 What this topic covers

ABOUT THIS TOPIC Docker Hub registry mirror Docker Hub pulls through pkgproxy, with no credentials anywhere 1 A registry mirror for Docker Hub 2 pkgproxy holds the token 3 Tags are rechecked, digests are not 4 Layers stream while they are checked 5 The mirror's settings and limits
The topic title, Docker Hub registry mirror, and its five scenes in order: the mirror, the token, tags and digests, layers, the settings

pkgproxy mirrors Docker Hub for anonymous pulls: it holds the token, rechecks tags, and keeps manifests and layers by digest.

The Docker Hub registry mirror lets Docker pull Docker Hub images through pkgproxy, with no credentials anywhere. First, pkgproxy becomes Docker's registry mirror. Then you see how it holds the registry token itself, why tags are rechecked and digests are not, and how layers stream to Docker while they are checked. Administrators and CI operators also see the mirror's settings and limits.

02 A registry mirror for Docker Hub

REGISTRY MIRROR $ pkgproxyctl setup docker | sudo sh # /etc/docker/daemon.json { "registry-mirrors": ["http://pkgproxy.local:3142"], …other settings kept… } dockerd /v2/… at the root pkgproxy registry-1.docker.io other registries: direct, as before ghcr.io, quay.io…
pkgproxyctl setup docker adds pkgproxy to registry-mirrors in daemon.json; dockerd asks pkgproxy at /v2/ for Docker Hub images, which fetches from registry-1.docker.io; images from other registries such as ghcr.io go directly

pkgproxyctl setup docker | sudo sh edits /etc/docker/daemon.json (or the Docker snap's) and restarts the daemon. registry-mirrors takes no path, so the API is at /v2/. Only docker.io images use it.

Docker's daemon can try a registry mirror before Docker Hub. pkgproxyctl setup docker, piped to sudo sh, puts pkgproxy's address first in the registry-mirrors list in daemon.json, keeping the other settings, and restarts Docker. Docker accepts a mirror only as a bare address with no path, so the registry API is served at the root, under /v2, next to the snap relay. The mirror applies to Docker Hub images only. Images from other registries are pulled directly, as before.

03 pkgproxy holds the token

THE TOKEN dockerd pkgproxy registry auth.docker.io GET /v2/ 200: no login needed GET manifest 401 · Bearer realm=… anonymous token? kept until it expires again, with the token 200 anonymous pulls of public images only
Sequence: dockerd's GET /v2/ is answered 200 by pkgproxy; pkgproxy's request to the registry gets 401 with a Bearer challenge; pkgproxy gets an anonymous token from auth.docker.io and retries with it, getting 200

GET /v2/ is answered 200 by pkgproxy. On a miss, the registry's 401 Bearer challenge is met with an anonymous token, cached per repository. Pushes and authenticated pulls are not supported.

Docker Hub wants a token even for public images. pkgproxy answers Docker's version check itself, saying that no login is needed, so Docker never sends credentials. When pkgproxy has to fetch, the registry answers 401 with a challenge. pkgproxy asks the token service it names, auth.docker.io, for an anonymous token for that repository, keeps the token until it expires, and asks again with it. Only anonymous pulls of public images are supported, so everything cached is something anyone could pull.

04 Tags are rechecked, digests are not

TAGS AND DIGESTS library/debian:12 points at manifests/sha256/4f3a… trusted for 5 minutes, then HEAD …/manifests/12 not counted as a pull blobs/sha256/9c2e… blobs/sha256/b7d0… by digest: checked against it · kept forever manifests/sha256/a81d… another image a layer used by both is stored once tag: -docker-revalidate-ttl 5m
The tag debian:12 points at a manifest digest and is rechecked with HEAD after five minutes; the manifest and its layers are stored by sha256 digest, kept forever, and a layer shared by two images is stored once

A tag pointer (tags/<name>/_/<tag>/…) is trusted for -docker-revalidate-ttl (default: -revalidate-ttl, 5m), then resolved with HEAD. Manifests and blobs are keyed manifests/sha256/<hex> and blobs/sha256/<hex>, verified and immutable.

A tag such as debian:12 is only a pointer to a manifest digest, and it moves. pkgproxy keeps the digest a tag last resolved to for five minutes, the docker-revalidate-ttl flag, then asks again with a HEAD request, which Docker Hub does not count against its anonymous pull limit. Manifests and layers are named by their digest, so they are checked against it, kept forever, and shared by every repository and tag that uses them.

05 Layers stream while they are checked

LAYERS pull 1 pull 2 one fill blobs/sha256/9c2e… registry bytes as they arrive last byte held until the digest matches digest matches last byte sent · the layer is cached digest differs not kept · the response ends short a client that read the whole layer knows it is cached
Two pulls follow one layer download from the registry; pkgproxy holds back the last byte until the digest matches; a match releases it and the layer is cached, a mismatch cuts the response short

A blob miss is a fill shared by every pull. Its last byte is held back until the sha256 digest matches and the blob is committed; a mismatch is not cached and its responses end short.

A layer that is not cached starts one download, and every pull of it, resumed ones included, follows the bytes as they are written. pkgproxy holds back the last byte until the layer matches its digest and is committed. A layer that does not match is not kept, and the response is cut short, so Docker never receives a corrupt layer as complete. A client that has read a whole layer knows that it is cached.

06 The mirror's settings and limits for administrators for CI operators

SETTINGS -docker-upstream https://registry-1.docker.io # the default: Docker Hub -docker-upstream off # no mirror $ sudo snap set pkgproxy docker-revalidate-ttl=10m default 5m served: pulls manifests · blobs not served: push · tags/list · catalog referrers: relayed, uncached <cache-dir>/docker/ nothing deleted on its own
The Docker flags: -docker-upstream defaults to registry-1.docker.io and off disables the mirror; -docker-revalidate-ttl defaults to five minutes; pulls are served, pushes, tag lists and the catalog are not; the cache lives in the docker directory

-docker-upstream (default https://registry-1.docker.io, or off, after which registry paths get a 404) and -docker-revalidate-ttl (default: -revalidate-ttl, 5m; snap option docker-revalidate-ttl). Only pulls are served; the cache is <cache-dir>/docker.

The docker-upstream flag names the one registry pkgproxy fetches from, registry-1.docker.io by default. Set to off, it turns the mirror off. The docker-revalidate-ttl flag sets how long a tag is trusted, five minutes by default, and is also a snap option. The mirror serves pulls only: pushes, tag lists and the catalog are not served, and referrers are relayed to the registry uncached. Layers and manifests are kept in the docker directory of the cache, and nothing there is deleted on its own.

07 Where to go next for CI operators in place of the general version

WHERE NEXT Docker pulls Docker Hub images through pkgproxy, and each layer comes from Docker Hub only once you are here Docker Hub registry mirror pkgproxy for CI build machines pulling the same images Statistics layers counted per image, with the bytes saved
You are here, Docker Hub registry mirror; next: pkgproxy for CI, Statistics

Next: pkgproxy for CI, for build machines pulling the same images; Statistics, for layers counted per image.

Docker now pulls Docker Hub images through pkgproxy, and each layer comes from Docker Hub only once. To see build machines pulling the same images from one cache, watch pkgproxy for CI. To see layers counted per image, with the bytes they saved, watch Statistics.

08 Where to go next general version

WHERE NEXT Docker pulls Docker Hub images through pkgproxy, and each layer comes from Docker Hub only once you are here Docker Hub registry mirror Immutable or not why some things are kept forever and others rechecked Snap Store relay the other side under /v2 pkgproxyctl setup apt, snapd, Go and LXD on the same cache
You are here, Docker Hub registry mirror; next: Immutable or not, Snap Store relay, pkgproxyctl setup

Next: Immutable or not, for what is kept and what is rechecked; Snap Store relay, the other side under /v2; pkgproxyctl setup, for the other clients.

Docker now pulls Docker Hub images through pkgproxy, and each layer comes from Docker Hub only once. To see why some things are kept forever and others rechecked, watch Immutable or not. To see the other side that shares /v2, watch Snap Store relay. To point apt, snapd, Go and LXD at the same cache, watch pkgproxyctl setup.

Read more

Paths answered

The mirror answers at the root of port 3142. These shapes are the registry's; any other path under /v2 belongs to the Snap Store relay. Only GET and HEAD are served; any other method is a 405.

PathWhat pkgproxy doesCache policy
/v2/Answers 200 itself, with an empty JSON body: no login needednot cached; never asks the registry
/v2/<name>/manifests/<tag>Resolves the tag to a manifest digest with a HEAD to the registry, then serves that manifestthe tag's digest is revalidated after docker-revalidate-ttl; the manifest is kept forever
/v2/<name>/manifests/sha256:<hex>Serves the manifest, fetched once and checked against its digestkept forever
/v2/<name>/blobs/sha256:<hex>Serves the layer or configuration blob; a miss is one download shared by every pull, Range includedkept forever, checked while it streams in; a HEAD on a miss asks the registry and caches nothing
/v2/<name>/referrers/sha256:<hex>Relays the request to the registrynot cached
/v2/<name>/tags/list, /v2/_catalog404not served

Manifests and blobs are shared by every repository and tag that names their digest. A cached digest is served under any repository name without asking the registry. With docker-upstream set to off, every path above is a 404.

When the registry answers badly

Registry's answer while resolving a tagClient gets
Cannot be reachedThe stale manifest, if the tag resolved before, with a warning in the log
404404: the tag is gone, and the client falls back to Docker Hub
401 or 403, the tag resolved beforeThe stale manifest, asked again on every request
401 or 403, nothing cached404
A failed fetch of a manifest or blob not cached502

for administrators for CI operators

Settings

SettingFlagEnvironmentSnap optionDefault
The registry to fetch from, or off to turn the mirror off-docker-upstreamPKGPROXY_DOCKER_UPSTREAMflag or environment onlyhttps://registry-1.docker.io
How long a tag's digest is trusted-docker-revalidate-ttlPKGPROXY_DOCKER_REVALIDATE_TTLdocker-revalidate-ttlrevalidate-ttl, 5m

for administrators for CI operators

On disk

Under the docker directory of the cache (/var/snap/pkgproxy/common/cache/docker in the snap, /var/cache/pkgproxy/docker otherwise), each object with its metadata in an extended attribute of its file:

docker/objects/
  blobs/sha256/<hex>              layers and configuration blobs
  manifests/sha256/<hex>          manifests and image indexes, with their media type
  tags/<name>/_/<tag>/<id>        the digest a tag last resolved to, per set of accepted media types

Nothing there is deleted on its own. To drop it all, stop pkgproxy (sudo snap stop pkgproxy), remove the directory and start it again. See The store.

Roads from here

CORE · the path every side shares

SNAP

SETUP AND THE COMMAND LINE

OPERATIONS