Files by checksum
files for users for administrators for CI operators
How a client asks pkgproxy for a large file by URL and checksum, why the checksum is the cache key, how only matching bytes are kept, and the limits on what pkgproxy will fetch.
On the map: files
01 What this topic covers
A client asks pkgproxy for a large file by URL and checksum. The checksum is the cache key, and only bytes that match it are kept.
Files by checksum is how pkgproxy caches large downloads that have no package format, such as the cloud images a CI job boots. First, a client names the file's URL and its checksum. Then you see why the checksum, not the URL, is the cache key, and how only bytes that match it are kept. Administrators and CI operators also see the limits on what pkgproxy fetches.
02 Name the file and its checksum
GET /files/v1/fetch?url=…&sha256=… (or
sha512, optional size). pkgproxy is the HTTPS
client and verifies the origin's certificate. X-Pkgproxy-Origin-URL
and X-Pkgproxy-Cache describe the answer.
Some large downloads, such as the cloud images a CI job boots, have no package protocol of their own. A client asks the fetch endpoint under /files for the file's URL, HTTP or HTTPS, together with its SHA-256 or SHA-512, and its size when it knows it. pkgproxy downloads the file itself, as an ordinary HTTPS client that checks the server's certificate. It intercepts nobody's TLS. The response says where the bytes came from, and whether they were already cached.
03 The checksum is the key
The key is sha256/<hex> or sha512/<hex>
in <cache-dir>/files, whatever the URL. Entries are
immutable; a new file has a new checksum and misses.
The cache entry is named by the checksum, not by the URL. The same image at two addresses, such as Ubuntu's current directory and its dated twin, or two mirrors, is stored once. When a URL starts naming a newer image, the client asks with the new checksum, misses, and the new file is downloaded. So a cached file never needs rechecking, and a URL that now names a newer file is not answered with the old one.
04 Only matching bytes are kept
A miss is a fill shared by every request, Range included. The last byte is held until the checksum, and the size if given, match; otherwise nothing is cached and every response ends short.
On a miss, one download starts, and every request for the file follows it as it is written, so ten CI jobs that start together download the image once, and a resumed download works. pkgproxy holds back the last byte until the checksum matches. A file that does not match is not kept, and every response following it ends short, so the client rejects it and downloads the file directly. A size the client gave is checked the same way.
05 Limits on what it fetches for administrators for CI operators
files-allow-host (empty: any host),
files-allow-network (empty: no private networks) and
files-max-size (8G, 0 for no limit),
as snap options or flags. The private-address check runs on every
connection, redirects included.
A client chooses the URL, so the fetch endpoint has limits. Any host is allowed until the files-allow-host setting lists host names, or domains that start with a dot. pkgproxy does not connect to loopback, private, link local or similar addresses, after any redirect, unless the files-allow-network setting allows that network. Files larger than the files-max-size setting, 8 GiB by default, are refused. A refused host, address or size the client gave is a 403; a failed download, or one the origin says is too large, is a 502, and the client then downloads the file itself.
06 Where to go next for administrators in place of the general version
Next: Settings, for the files settings; The cache page, where files by checksum are a kind of their own, grouped by name.
A large file such as a cloud image now comes from the internet once, checked against its checksum, and every later request gets it from pkgproxy. To see the files settings and how to set them, watch Settings. To see the cached files as a kind of their own on the live map of the cache, grouped by the name they were served under most, watch The cache page.
07 Where to go next for CI operators in place of the general version
Next: pkgproxy for CI, for many machines behind one cache; Offline mode, for files served with no network.
A large file such as a cloud image now comes from the internet once, checked against its checksum, and every later request gets it from pkgproxy. To put many CI machines behind one cache, so that ten jobs booting the same image download it once, watch pkgproxy for CI. To see cached files served with no network, watch Offline mode.
08 Where to go next general version
Next: Fetch once, for one download shared by every client; Offline mode, for files served with no network; The store, for where they are kept.
A large file such as a cloud image now comes from the internet once, checked against its checksum, and every later request gets it from pkgproxy. To see how one download is shared by every client that asks while it is written, watch Fetch once. To see cached files served with no network, watch Offline mode. To see where the files are kept on disk, watch The store.
Read more
The request
GET /files/v1/fetch?url=<URL>&sha256=<hex>[&size=<bytes>] GET /files/v1/fetch?url=<URL>&sha512=<hex>[&size=<bytes>] HEAD the same
Any other path under /files is a 404, and any method but GET and HEAD a 405.
| Parameter | Required | Value |
|---|---|---|
url | yes | An absolute http:// or https:// URL with a host, percent-encoded. No user name or password. |
sha256 | one of the two | 64 hexadecimal digits. |
sha512 | one of the two | 128 hexadecimal digits. Give one checksum, not both. |
size | no | The file's size in bytes, a positive number. Checked like the checksum, and lets clients follow a download whose server sends no length. |
The response
| Header | Value |
|---|---|
Content-Type | application/octet-stream |
Content-Length | The file's size |
Accept-Ranges | bytes: Range requests and resumed downloads work, on a hit and while the file downloads |
X-Pkgproxy-Cache | hit from a complete cached file; miss while the file downloads, whether this request started the download or joined it |
X-Pkgproxy-Origin-URL | The URL the bytes were downloaded from, after redirects. On a hit it is the URL of the request that fetched the file. |
Status codes
| Status | When |
|---|---|
| 200 or 206 | The file, from the cache or as it downloads. |
| 400 | A missing or malformed parameter; or, on a hit, a size other than the cached file's. |
| 403 | The host is not allowed by files-allow-host; size is larger than files-max-size; or a connection would go to a private address files-allow-network does not allow, after any redirect. |
| 404 | A HEAD for a file that is not cached (HEAD never contacts the origin). |
| 405 | A method other than GET or HEAD. |
| 502 | The download failed, after asking the origin again for up to upstream-retry-for (see Fetch once): the origin could not be fetched, more than 10 redirects, a file larger than files-max-size, or bytes that do not match the checksum or size. |
| 503 | Offline mode, for a file that is not cached; cached files are still served. |
A download that fails after its headers were sent ends short instead, so the client rejects it. On any error, the client downloads the file directly.
for administrators for CI operators
Settings
| Setting | Flag | Environment | Default | Meaning |
|---|---|---|---|---|
files-allow-host | -files-allow-host | PKGPROXY_FILES_ALLOW_HOST | empty: any host | Hosts clients may name, separated by commas: a host name, or a domain starting with . for every host under it. Empty or * allows every host. Checked on the URL the client names, not on redirects. |
files-allow-network | -files-allow-network | PKGPROXY_FILES_ALLOW_NETWORK | empty: none | Private networks, in CIDR notation and separated by commas, that files may be fetched from. |
files-max-size | -files-max-size | PKGPROXY_FILES_MAX_SIZE | 8G | Largest file fetched, such as 8G or 500M; 0 means no limit. |
All three are also snap options: sudo snap set pkgproxy files-max-size=20G. The private addresses refused unless allowed are loopback, private and unique local, link-local, unspecified, multicast, 0.0.0.0/8 and 100.64.0.0/10. A proxy pkgproxy fetches through (see Settings) is used wherever it is, on loopback or the local network too, as for apt; a client that names the proxy's own address as a file's URL is fetched from directly, and checked like any other.
for administrators for CI operators
On disk
Files are kept in the files directory of the cache, as files/objects/sha256/<hex> or files/objects/sha512/<hex>, each with its metadata, the origin URL among it, in an extended attribute of the file. The cache is /var/snap/pkgproxy/common/cache in the snap and /var/cache/pkgproxy otherwise. Files are never revalidated or deleted; see The store.
In the documentation
Roads from here
CORE · the path every side shares
- Fetch once — one download shared while it is written
- The store — where the files end up
- Offline mode — hits served while offline
OPERATIONS
- pkgproxy for CI — many CI machines behind one cache
- Settings — the files settings
- Statistics — files counted in the statistics
- The cache page — files on the live map of the cache