Files by checksum

files for users for administrators for CI operators

How a client asks pkgproxy for a large file by URL and checksum, why the checksum is the cache key, how only matching bytes are kept, and the limits on what pkgproxy will fetch.

On the map: files

▶ Watch this deck — 8 scenes, ~5 min

01 What this topic covers

ABOUT THIS TOPIC Files by checksum large downloads with no package format, cached by their checksum 1 Name the file and its checksum 2 The checksum is the key 3 Only matching bytes are kept 4 Limits on what it fetches
The topic title, Files by checksum, and its four scenes in order: naming the file and its checksum, the checksum as the key, only matching bytes kept, the limits

A client asks pkgproxy for a large file by URL and checksum. The checksum is the cache key, and only bytes that match it are kept.

Files by checksum is how pkgproxy caches large downloads that have no package format, such as the cloud images a CI job boots. First, a client names the file's URL and its checksum. Then you see why the checksum, not the URL, is the cache key, and how only bytes that match it are kept. Administrators and CI operators also see the limits on what pkgproxy fetches.

02 Name the file and its checksum

FILES BY CHECKSUM CI job $ curl -o noble.img "http://pkgproxy.local:3142/files/v1/fetch\ ?url=https%3A%2F%2Fcloud-images.ubuntu.com%2Fnoble%2Fcurrent%2F… &sha256=<hex>&size=<bytes>" # size is optional /files/… pkgproxy https cloud-images.ubuntu.com certificate checked no TLS intercepted X-Pkgproxy-Origin-URL: https://cloud-images… X-Pkgproxy-Cache: hit
A CI machine asks pkgproxy's /files/v1/fetch for a cloud image URL with its sha256; pkgproxy fetches it over HTTPS from cloud-images.ubuntu.com, checking the certificate, and answers with headers naming the origin URL and whether it was a hit

GET /files/v1/fetch?url=…&sha256=… (or sha512, optional size). pkgproxy is the HTTPS client and verifies the origin's certificate. X-Pkgproxy-Origin-URL and X-Pkgproxy-Cache describe the answer.

Some large downloads, such as the cloud images a CI job boots, have no package protocol of their own. A client asks the fetch endpoint under /files for the file's URL, HTTP or HTTPS, together with its SHA-256 or SHA-512, and its size when it knows it. pkgproxy downloads the file itself, as an ordinary HTTPS client that checks the server's certificate. It intercepts nobody's TLS. The response says where the bytes came from, and whether they were already cached.

03 The checksum is the key

THE KEY files/sha256/3b5c… …/noble/current/…img · sha256=3b5c… …/noble/<date>/…img · sha256=3b5c… stored once later, current/ names a newer image …/noble/current/…img · sha256=e94a… files/sha256/e94a… a miss: downloaded never rechecked · a moved URL cannot get the old bytes
Two URLs for the same image with the same sha256 lead to one cache entry; a newer image under the current URL has a new checksum and gets its own entry

The key is sha256/<hex> or sha512/<hex> in <cache-dir>/files, whatever the URL. Entries are immutable; a new file has a new checksum and misses.

The cache entry is named by the checksum, not by the URL. The same image at two addresses, such as Ubuntu's current directory and its dated twin, or two mirrors, is stored once. When a URL starts naming a newer image, the client asks with the new checksum, misses, and the new file is downloaded. So a cached file never needs rechecking, and a URL that now names a newer file is not answered with the old one.

04 Only matching bytes are kept

ONLY MATCHING BYTES CI job 1 CI job 2 CI job 3 one fill sha256/3b5c… origin image, as it downloads last byte held until the checksum matches checksum matches last byte sent · the file is kept checksum differs not kept · every response ends short · client downloads it directly a size the client gave is checked the same way
Three CI jobs follow one download of an image; pkgproxy holds back the last byte until the checksum matches; a match is kept, a mismatch is dropped and every response ends short

A miss is a fill shared by every request, Range included. The last byte is held until the checksum, and the size if given, match; otherwise nothing is cached and every response ends short.

On a miss, one download starts, and every request for the file follows it as it is written, so ten CI jobs that start together download the image once, and a resumed download works. pkgproxy holds back the last byte until the checksum matches. A file that does not match is not kept, and every response following it ends short, so the client rejects it and downloads the file directly. A size the client gave is checked the same way.

05 Limits on what it fetches for administrators for CI operators

LIMITS pkgproxy downloads whatever URL a client names, so: $ sudo snap set pkgproxy files-allow-host=cloud-images.ubuntu.com,.fedoraproject.org # empty, the default: any host 127.0.0.1 · 192.168.0.1 · 10.0.0.1 not connected to checked on every redirect $ sudo snap set pkgproxy files-allow-network=192.168.0.0/24 files-max-size=8G # the default · 0 means no limit 403 refused · 502 failed either way the client downloads the file directly
Three limits on file fetches: an optional host allow list, no connections to private addresses unless a network is allowed, and a maximum size of 8 GiB by default; refusals are 403

files-allow-host (empty: any host), files-allow-network (empty: no private networks) and files-max-size (8G, 0 for no limit), as snap options or flags. The private-address check runs on every connection, redirects included.

A client chooses the URL, so the fetch endpoint has limits. Any host is allowed until the files-allow-host setting lists host names, or domains that start with a dot. pkgproxy does not connect to loopback, private, link local or similar addresses, after any redirect, unless the files-allow-network setting allows that network. Files larger than the files-max-size setting, 8 GiB by default, are refused. A refused host, address or size the client gave is a 403; a failed download, or one the origin says is too large, is a 502, and the client then downloads the file itself.

06 Where to go next for administrators in place of the general version

WHERE NEXT a large file now comes from the internet once, checked against its checksum, and every later request gets it from pkgproxy you are here Files by checksum Settings the files settings and how to set them The cache page files as a kind of their own, grouped by name
You are here, Files by checksum; next: Settings, The cache page

Next: Settings, for the files settings; The cache page, where files by checksum are a kind of their own, grouped by name.

A large file such as a cloud image now comes from the internet once, checked against its checksum, and every later request gets it from pkgproxy. To see the files settings and how to set them, watch Settings. To see the cached files as a kind of their own on the live map of the cache, grouped by the name they were served under most, watch The cache page.

07 Where to go next for CI operators in place of the general version

WHERE NEXT a large file now comes from the internet once, checked against its checksum, and every later request gets it from pkgproxy you are here Files by checksum pkgproxy for CI many CI machines behind one cache Offline mode cached files served with no network
You are here, Files by checksum; next: pkgproxy for CI, Offline mode

Next: pkgproxy for CI, for many machines behind one cache; Offline mode, for files served with no network.

A large file such as a cloud image now comes from the internet once, checked against its checksum, and every later request gets it from pkgproxy. To put many CI machines behind one cache, so that ten jobs booting the same image download it once, watch pkgproxy for CI. To see cached files served with no network, watch Offline mode.

08 Where to go next general version

WHERE NEXT a large file now comes from the internet once, checked against its checksum, and every later request gets it from pkgproxy you are here Files by checksum Fetch once one download shared by every client that asks Offline mode cached files served with no network The store where the files are kept on disk
You are here, Files by checksum; next: Fetch once, Offline mode, The store

Next: Fetch once, for one download shared by every client; Offline mode, for files served with no network; The store, for where they are kept.

A large file such as a cloud image now comes from the internet once, checked against its checksum, and every later request gets it from pkgproxy. To see how one download is shared by every client that asks while it is written, watch Fetch once. To see cached files served with no network, watch Offline mode. To see where the files are kept on disk, watch The store.

Read more

The request

GET  /files/v1/fetch?url=<URL>&sha256=<hex>[&size=<bytes>]
GET  /files/v1/fetch?url=<URL>&sha512=<hex>[&size=<bytes>]
HEAD the same

Any other path under /files is a 404, and any method but GET and HEAD a 405.

ParameterRequiredValue
urlyesAn absolute http:// or https:// URL with a host, percent-encoded. No user name or password.
sha256one of the two64 hexadecimal digits.
sha512one of the two128 hexadecimal digits. Give one checksum, not both.
sizenoThe file's size in bytes, a positive number. Checked like the checksum, and lets clients follow a download whose server sends no length.

The response

HeaderValue
Content-Typeapplication/octet-stream
Content-LengthThe file's size
Accept-Rangesbytes: Range requests and resumed downloads work, on a hit and while the file downloads
X-Pkgproxy-Cachehit from a complete cached file; miss while the file downloads, whether this request started the download or joined it
X-Pkgproxy-Origin-URLThe URL the bytes were downloaded from, after redirects. On a hit it is the URL of the request that fetched the file.

Status codes

StatusWhen
200 or 206The file, from the cache or as it downloads.
400A missing or malformed parameter; or, on a hit, a size other than the cached file's.
403The host is not allowed by files-allow-host; size is larger than files-max-size; or a connection would go to a private address files-allow-network does not allow, after any redirect.
404A HEAD for a file that is not cached (HEAD never contacts the origin).
405A method other than GET or HEAD.
502The download failed, after asking the origin again for up to upstream-retry-for (see Fetch once): the origin could not be fetched, more than 10 redirects, a file larger than files-max-size, or bytes that do not match the checksum or size.
503Offline mode, for a file that is not cached; cached files are still served.

A download that fails after its headers were sent ends short instead, so the client rejects it. On any error, the client downloads the file directly.

for administrators for CI operators

Settings

SettingFlagEnvironmentDefaultMeaning
files-allow-host-files-allow-hostPKGPROXY_FILES_ALLOW_HOSTempty: any hostHosts clients may name, separated by commas: a host name, or a domain starting with . for every host under it. Empty or * allows every host. Checked on the URL the client names, not on redirects.
files-allow-network-files-allow-networkPKGPROXY_FILES_ALLOW_NETWORKempty: nonePrivate networks, in CIDR notation and separated by commas, that files may be fetched from.
files-max-size-files-max-sizePKGPROXY_FILES_MAX_SIZE8GLargest file fetched, such as 8G or 500M; 0 means no limit.

All three are also snap options: sudo snap set pkgproxy files-max-size=20G. The private addresses refused unless allowed are loopback, private and unique local, link-local, unspecified, multicast, 0.0.0.0/8 and 100.64.0.0/10. A proxy pkgproxy fetches through (see Settings) is used wherever it is, on loopback or the local network too, as for apt; a client that names the proxy's own address as a file's URL is fetched from directly, and checked like any other.

for administrators for CI operators

On disk

Files are kept in the files directory of the cache, as files/objects/sha256/<hex> or files/objects/sha512/<hex>, each with its metadata, the origin URL among it, in an extended attribute of the file. The cache is /var/snap/pkgproxy/common/cache in the snap and /var/cache/pkgproxy otherwise. Files are never revalidated or deleted; see The store.

In the documentation

Roads from here

CORE · the path every side shares

OPERATIONS