apt cache keys and hash checks
apt for administrators
An apt request becomes a key of host and archive prefix plus the path from pool or dists onward, and each download is checked against the hash its index listed before it is cached.
On the map: apt
01 What this topic covers
How an apt URL becomes a cache key, and how each download is checked against the hash its index lists.
pkgproxy keys every apt download by its archive and path, and checks it against the hash apt's own indexes list before keeping it. Watch A forward proxy first if you have not. Here you see how the host, port and prefix name the archive, and why the prefix is part of the key. Then the hashes learned from apt's own traffic, why a download that does not match is not cached, and what is not checked.
02 Host, port and prefix name the archive
The archive id is the host (with its port, unless it is 80) plus the path prefix before pool/
or dists/; the key is the archive id and the path from there
on. A URL with neither segment is passed through uncached.
pkgproxy splits an apt URL at the first path segment named pool or dists. Everything before it, the host, with its port when that is not eighty, and any path prefix, is the archive's identity. Everything from pool or dists onward is the path inside the archive. The key joins the two. Empty segments are dropped, so a doubled slash names the same archive. A URL with neither segment has no key, and is passed through without being cached.
03 Why the prefix is part of the key
deb.debian.org/debian and
deb.debian.org/debian-security are separate archives, as are KDE
Neon's /unstable and /release. Mirrors on
different hosts are cached separately: mirror aliasing is not built yet.
The prefix is kept because one host can serve several archives. deb.debian.org carries both /debian and /debian-security. KDE Neon's editions, /unstable and /release on one host, share suite names, yet are different archives. Dropping the prefix would mix them. Two hosts are always two archives: a mirror such as ftp.de.debian.org is cached apart from deb.debian.org, because collapsing mirrors needs an alias table that is not built yet.
04 Hashes learned on the way through
InRelease lists the hashes of the index files;
Packages lists Filename and SHA256
for each package. Both are parsed as they pass through. A
by-hash URL names its own hash.
pkgproxy checks downloads against hashes it learns from apt's own traffic; it does not crawl the archive separately. When apt fetches a suite's InRelease, pkgproxy reads its SHA-256 list for the Packages files. When apt fetches a Packages file, it reads each package's file name and hash. The pool download that follows is hashed while it is written, and committed only if the hash matches. A by-hash URL carries its hash in the path, so it needs no lookup.
05 A mismatch is never cached
A conflicting hash for a known pool path is logged and ignored. Content that fails its hash check is never committed; clients already receiving it get the origin's bytes unmodified, the rest a passthrough fetch.
The first hash learned for a pool path wins. If a later index names a different hash for the same file, pkgproxy logs a conflict and keeps the first; a hash in a new algorithm is merged in. A download that does not match its expected hash is not cached. Clients already receiving it still get the mirror's bytes unchanged, since apt checks them itself, and clients that had received nothing yet are answered by a passthrough fetch, also uncached.
06 What is not checked
InRelease, Release and Release.gpg
are cached unverified; apt verifies the signature. .xz indexes
are not parsed, so a pool file listed only there is cached without a hash
check, with a warning in the log.
Some downloads have no hash to check. InRelease and Release sit at the top of the chain: nothing lists their hash, so they are cached as they come, and apt checks their signatures. Packages files compressed with xz are cached and served but not parsed, since pkgproxy has no xz reader. A pool file listed only in such a file, such as one apt found through a by-hash Packages.xz, is cached unchecked, and the log notes the gap.
07 Where to go next general version
Next: pool and dists, for what a key does on a hit; The store, for where downloads are written; Fetch once, for one fetch per key.
You now know how each apt download gets its key, and why one that fails its hash is not kept. To see what each kind of key does on a hit, watch pool and dists. To see where keyed downloads are written, watch The store. To see one fetch shared by every request for a key, watch Fetch once.
Read more
In the documentation
Roads from here
CORE · the path every side shares
- The store — where keyed downloads are written
- Fetch once — one fetch per key
APT
- pool forever and dists rechecked — what each kind of key does on a hit
- apt through a forward proxy — how apt requests reach pkgproxy
FILES
- Files by checksum — another download kept only when its hash matches