apt cache keys and hash checks

apt for administrators

An apt request becomes a key of host and archive prefix plus the path from pool or dists onward, and each download is checked against the hash its index listed before it is cached.

On the map: apt

▶ Watch this deck — 7 scenes, ~4 min

01 What this topic covers

ABOUT THIS TOPIC Cache keys every apt download is keyed by archive and path, and checked against its hash 1 Host, port and prefix name the archive 2 Why the prefix is part of the key 3 Hashes learned on the way through 4 A mismatch is never cached 5 What is not checked
The topic title, Cache keys, and its 5 scenes in order: Host, port and prefix name the archive, Why the prefix is part of the key, Hashes learned on the way through, A mismatch is never cached, What is not checked

How an apt URL becomes a cache key, and how each download is checked against the hash its index lists.

pkgproxy keys every apt download by its archive and path, and checks it against the hash apt's own indexes list before keeping it. Watch A forward proxy first if you have not. Here you see how the host, port and prefix name the archive, and why the prefix is part of the key. Then the hashes learned from apt's own traffic, why a download that does not match is not cached, and what is not checked.

02 Host, port and prefix name the archive

ARCHIVE ID GET http://archive.ubuntu.com/ubuntu/dists/noble/InRelease archive.ubuntu.com/ubuntu archive id: the host (and a port other than 80), the prefix dists/noble/InRelease the path from dists/ or pool/ onward archive.ubuntu.com/ubuntu/dists/noble/InRelease the cache key http://archive.ubuntu.com//ubuntu/dists/… the same archive id http://deb.debian.org/debian/README no pool/ or dists/: passed through, not cached
An Ubuntu URL is split at dists/ into the archive id archive.ubuntu.com/ubuntu and the path dists/noble/InRelease, joined into the cache key; a doubled slash gives the same archive id; a URL with neither pool nor dists is passed through

The archive id is the host (with its port, unless it is 80) plus the path prefix before pool/ or dists/; the key is the archive id and the path from there on. A URL with neither segment is passed through uncached.

pkgproxy splits an apt URL at the first path segment named pool or dists. Everything before it, the host, with its port when that is not eighty, and any path prefix, is the archive's identity. Everything from pool or dists onward is the path inside the archive. The key joins the two. Empty segments are dropped, so a doubled slash names the same archive. A URL with neither segment has no key, and is passed through without being cached.

03 Why the prefix is part of the key

THE PREFIX deb.debian.org/debian deb.debian.org/debian-security one host, two archives origin.archive.neon.kde.org/unstable origin.archive.neon.kde.org/release same suite names, different archives without the prefix, each pair would share one set of keys deb.debian.org/debian ftp.de.debian.org/debian two hosts, two archives: each cached on its own mirror aliasing, one archive id for several hosts: not built yet
Pairs of archive ids: deb.debian.org/debian and /debian-security on one host; KDE Neon's /unstable and /release on one host; deb.debian.org/debian and ftp.de.debian.org/debian on two hosts, cached separately because mirror aliasing is not built yet

deb.debian.org/debian and deb.debian.org/debian-security are separate archives, as are KDE Neon's /unstable and /release. Mirrors on different hosts are cached separately: mirror aliasing is not built yet.

The prefix is kept because one host can serve several archives. deb.debian.org carries both /debian and /debian-security. KDE Neon's editions, /unstable and /release on one host, share suite names, yet are different archives. Dropping the prefix would mix them. Two hosts are always two archives: a mirror such as ftp.de.debian.org is cached apart from deb.debian.org, because collapsing mirrors needs an alias table that is not built yet.

04 Hashes learned on the way through

LEARNED HASHES learned hashes held in memory, filled by apt's own requests dists/trixie/InRelease main/binary-amd64/Packages.gz SHA256 3a7b… dists/trixie/main/binary-amd64/Packages.gz pool/main/h/hello/hello_2.10-3_amd64.deb SHA256 9c1e… pool/main/h/hello/hello_2.10-3_amd64.deb hashed while written: 9c1e… matches, committed dists/…/binary-amd64/by-hash/SHA256/3a7b… the hash is in the URL: no lookup
apt's InRelease and Packages requests pass through pkgproxy, which records the hashes they list; the pool download is hashed while written and committed when it matches; a by-hash URL carries its own hash

InRelease lists the hashes of the index files; Packages lists Filename and SHA256 for each package. Both are parsed as they pass through. A by-hash URL names its own hash.

pkgproxy checks downloads against hashes it learns from apt's own traffic; it does not crawl the archive separately. When apt fetches a suite's InRelease, pkgproxy reads its SHA-256 list for the Packages files. When apt fetches a Packages file, it reads each package's file name and hash. The pool download that follows is hashed while it is written, and committed only if the hash matches. A by-hash URL carries its hash in the path, so it needs no lookup.

05 A mismatch is never cached

FIRST HASH WINS pool/main/h/hello/hello_2.10-3_amd64.deb SHA256 9c1e… kept a later Packages: SHA256 4d02… conflict: logged, first one kept a later Packages: SHA512 … a new algorithm: merged in mirror hash check mismatch not cached clients already receiving get the mirror's bytes unchanged clients with nothing yet a passthrough fetch, uncached
A pool path keeps the first SHA256 learned; a later different SHA256 is logged as a conflict, a SHA512 is merged in. Below, a download that fails the hash check is not cached; clients already receiving it get the bytes, the others a passthrough fetch

A conflicting hash for a known pool path is logged and ignored. Content that fails its hash check is never committed; clients already receiving it get the origin's bytes unmodified, the rest a passthrough fetch.

The first hash learned for a pool path wins. If a later index names a different hash for the same file, pkgproxy logs a conflict and keeps the first; a hash in a new algorithm is merged in. A download that does not match its expected hash is not cached. Clients already receiving it still get the mirror's bytes unchanged, since apt checks them itself, and clients that had received nothing yet are answered by a passthrough fetch, also uncached.

06 What is not checked

GAPS dists/trixie/InRelease main/…/Packages.gz parsed for pool hashes pool/…/hello_2.10-3_amd64.deb checked against its hash nothing lists its hash: cached as it comes apt checks its signature main/…/by-hash Packages.xz cached, not parsed pool/…/other_1.0-1_amd64.deb no expected hash: cached unchecked the log notes it: caching an apt file without a known hash to check it against
InRelease at the top is cached as it comes and checked by apt's signature verification; a Packages.gz below it is parsed and its pool files are checked; a by-hash Packages.xz is cached but not parsed, so its pool file is cached unchecked and the log notes it

InRelease, Release and Release.gpg are cached unverified; apt verifies the signature. .xz indexes are not parsed, so a pool file listed only there is cached without a hash check, with a warning in the log.

Some downloads have no hash to check. InRelease and Release sit at the top of the chain: nothing lists their hash, so they are cached as they come, and apt checks their signatures. Packages files compressed with xz are cached and served but not parsed, since pkgproxy has no xz reader. A pool file listed only in such a file, such as one apt found through a by-hash Packages.xz, is cached unchecked, and the log notes the gap.

07 Where to go next general version

WHERE NEXT each apt download is keyed by its archive and path, and one that fails its hash is not kept you are here Cache keys pool and dists what each kind of key does on a hit The store where keyed downloads are written Fetch once one fetch shared by every request for a key
You are here, Cache keys; next: pool and dists, The store, Fetch once

Next: pool and dists, for what a key does on a hit; The store, for where downloads are written; Fetch once, for one fetch per key.

You now know how each apt download gets its key, and why one that fails its hash is not kept. To see what each kind of key does on a hit, watch pool and dists. To see where keyed downloads are written, watch The store. To see one fetch shared by every request for a key, watch Fetch once.

Read more

Roads from here

CORE · the path every side shares

APT

FILES