pkgproxy design: files by checksum

Caching large files that have no package protocol of their own, by their checksum. Part of the pkgproxy design, whose index says which file holds each § number.

• 24. Files by checksum
  • 24.1 The endpoint
  • 24.2 The cache key is the content
  • 24.3 Fetching
  • 24.4 Limits
  • 24.5 Statistics
  • 24.6 What is not done

24. Files by checksum

internal/filesproxy caches large files that no other side would: the cloud images image-garden boots (.qcow2, .img, .raw.xz), hundreds of MB each. One pkgproxy on a fast local network then serves every image to every CI machine after the first download.

Neither of the other ways in works for them. A forward-proxy GET of an arbitrary http:// URL reaches the apt side, which caches only dists/ and pool/ paths and streams anything else through (§5). An https:// URL, which most images and checksum files are, is a CONNECT tunnel pkgproxy never sees into (§1).

So the client names the origin URL, and pkgproxy is the HTTP(S) client to the origin. That is not the HTTPS caching §1 rules out: pkgproxy intercepts nobody’s TLS, it verifies the origin’s certificate against the system CAs as any client does, and it keeps only bytes that match the checksum the client gave.

The issue is gitlab.com/zygoon/pkgproxy#95.

24.1 The endpoint

GET /files/v1/fetch?url=<percent-encoded URL>&sha256=<hex>
GET /files/v1/fetch?url=<percent-encoded URL>&sha512=<hex>

with an optional &size=<bytes>, which a client usually knows too: from the same data files as the checksum, or from the origin’s own listing.

The gateway routes /files to the side; /files/v1/fetch is its one endpoint, and any other path under /files is a 404. A request is refused with 400 when:

• url is missing, is not an absolute http or https URL with a host, or has a user name or password (authenticated origins are not supported);
• neither sha256 nor sha512 is given, or both are, or the value is not hexadecimal of the right length;
• size is given and is not a positive number.

A request whose size is larger than -files-max-size is refused with 403 without contacting the origin (§24.4). On a hit, a size other than the cached file’s is a 400: the checksum names that file, so the client’s idea of it is wrong.

The checksum is required. Every image image-garden knows of has one, published upstream (SHA256SUMS, Gentoo’s <image URL>.sha256) or pinned in its data files, so the cache never has to decide whether a URL still names the same bytes. A client that finds the current image with a discovery program (Gentoo, Amazon Linux) runs it itself, fetching the small files it reads directly, and then asks pkgproxy for the final URL with its checksum. pkgproxy runs no discovery programs.

The response carries Content-Length, Accept-Ranges, X-Pkgproxy-Origin-URL, the URL the bytes were downloaded from after redirects, and X-Pkgproxy-Cache: hit from a complete entry, miss while the file downloads, whether this request started the download or joined it. On a hit the origin URL is that of the request that fetched the file, which may not be this one’s; the bytes are the same.

HEAD answers from the cache only: a miss is a 404, as saying how large the file is would mean asking the origin, and a client that wants the file asks with GET anyway.

24.2 The cache key is the content

The entry is sha256/<hex> or sha512/<hex> in the side’s own store, -cache-dir/files, whatever the URL. Mirrors, redirects and URLs that move to a new file, such as Ubuntu’s current/, cannot serve stale or wrong bytes: a new image has a new checksum, so its request misses and fetches it. The same file at two URLs, two mirrors or a current/ URL and its dated twin, is stored once. A file asked for once by SHA-256 and once by SHA-512 is stored twice; that is rare enough not to matter.

An entry is immutable and never revalidated. Its metadata records the origin URL (store.ObjectMeta.OriginURL).

24.3 Fetching

A miss joins or starts the file’s fill (internal/fill, as apt’s pool files and LXD’s images do, §8.1): the download runs detached from any client, every request for the file follows it as it is written, Range requests included, so wget --continue works, and ten CI jobs that start together download the image once.

The fill holds back the file’s last byte until the checksum matches (fill.Spec.HoldLastByte). A file that does not match is not kept, and every response following it ends short, so a client that checks the checksum, as image-garden does, or only the length, rejects it and downloads it directly. Unlike apt’s (§8 step 7), there is no uncached second fetch: the checksum is all the client has to tell a good file from a bad one. Nor is there a drain, so -fill-drain-timeout does not apply.

The size, when the client gives it, is checked as the checksum is. An origin that announces another Content-Length is not read at all, and a body of another length is not kept, its responses ending short. It also lets the requests follow an origin that sends no Content-Length as it downloads; without it, they wait for the whole file, and are answered from the cache. The response still waits for the origin’s headers before it starts, even with the size known (fill.Spec.Size is not used): that keeps a fetch that fails early a 502, which a client falls back on, rather than a 200 that ends short.

The client follows up to 10 redirects, to http and https URLs only, which download.fedoraproject.org needs to pick a mirror. It fetches through the offline switch’s transport, so offline mode refuses a miss with the 503 every side gives (§21), and serves hits. A failed fetch is a 502, a refused one a 403 (see §24.4); the client then downloads the file directly.

24.4 Limits

fetch makes pkgproxy download a URL a client names, so it has limits:

• Hosts. Any host by default. Images come from many mirrors, and redirectors send clients to hosts nobody can list in advance; a list that allows nothing until it is filled in would also protect little, as pkgproxy already tunnels CONNECT to any host on port 443 and streams any http:// URL through. -files-allow-host narrows it to host names, or domains starting with . for every host under them. It is checked against the URL the client names, not those a redirect leads to. A host not allowed is a 403.
• No private addresses. The side’s transport (NewTransport) refuses to connect to loopback, private (RFC 1918 and unique local), link-local, unspecified, multicast, “this network” (0.0.0.0/8) and carrier-grade NAT (100.64.0.0/10) addresses, unless -files-allow-network allows the network. The check runs in the dialer’s ControlContext, on the address actually connected to, so it covers every redirect and whatever DNS answers. A refused connection is a 403. With an HTTP proxy set in the environment, it is the proxy that is connected to and checked.
• Size. -files-max-size, 8 GiB by default, refuses a file whose Content-Length is larger without reading it, and cuts off one that sends more than that.

Nothing is evicted: the files cache grows, as every other side’s does, until eviction is designed for the whole cache (#96).

24.5 Statistics

The side records each request with stats.RecordFilesServe, in a files bucket of /pkgproxy/v1/stats, keyed <file name>@<algorithm>:<hex>: the file name is the last element of the requested URL’s path, as Docker’s blobs are keyed by repository. A file is content, so its hits count as a download saved. pkgproxyctl stats shows the bucket in its total, and the digest shortened as Docker’s.

The cache page (§20.7) shows the files store as a kind of its own, Files by checksum: its files grouped by the name they were served under most (stats.Recorder.FileNames), one leaf per checksum, and those never served under a name as files of unknown name. On the map (§20.8) a file’s activity is found by its store key, and the live views (§20.9) draw its requests, the cache fetching it from the host of the URL its client named.

24.6 What is not done

• Running discovery programs, or caching the small files they read.
• Mirroring directory trees.
• Authenticated origins.
• Signature verification (GPG over SHA256SUMS): the client verifies, and passes pkgproxy only the checksum it has verified.
• A setup script or setup page section: clients opt in by URL.