pool forever and dists rechecked

apt for users for administrators for CI operators

Pool files and by-hash indexes are cached forever; the other dists files are served for a short TTL and then revalidated, and a cached copy is served when the mirror fails.

On the map: apt

▶ Watch this deck — 6 scenes, ~4 min

01 What this topic covers

ABOUT THIS TOPIC pool and dists packages from pool are kept forever; indexes from dists are rechecked 1 Three kinds of path 2 A fleet runs apt update for administrators and CI operators 3 When the mirror fails 4 dists is stored first, pool streams
The topic title, pool and dists, and its 4 scenes in order: Three kinds of path, A fleet runs apt update, When the mirror fails, dists is stored first, pool streams

Pool files are kept forever, dists files are rechecked after a short window, and a failing mirror is covered by the cached copy.

Apt downloads two kinds of file: packages from pool, which pkgproxy keeps forever, and indexes from dists, which it checks again with the mirror after a few minutes. If you have not seen A forward proxy yet, start there. Here you see the three kinds of path, and fifty machines running apt update together. Then what happens when the mirror fails, and why a dists file is stored before it is served while a pool file streams.

02 Three kinds of path

POLICY BY PATH pool/** kept forever fetched once · hash checked · never rechecked dists/**/by-hash/SHA256/… kept forever the hash is in the name dists/** InRelease, Packages… rechecked served for the TTL, then a conditional GET anything else passed through not cached
Four rows: pool files kept forever; by-hash files under dists kept forever; other dists files rechecked after a TTL; anything else passed through and not cached

pool/** and dists/**/by-hash/** are immutable: cached forever, never revalidated. Other dists/** files are revalidated after a TTL. Anything else is passed through uncached.

The path decides what happens. Pool files never change once published, so they are fetched once, checked against their hash and kept forever without asking the mirror again. So are files under by-hash, even inside dists, because the hash is in their name. Every other file under dists, InRelease, Release, Packages and the rest, changes with each archive update and is rechecked. A path outside pool and dists is passed through and not cached.

03 A fleet runs apt update for administrators for CI operators

A FLEET host 1 host 2 host 3 … host 50 pkgproxy dists/trixie/InRelease first 5 minutes: every request a hit -apt-revalidate-ttl 5m 1 conditional GET deb.debian.org 304 Not Modified a 304 restarts the five minutes; the file is not rewritten Cache-Control: no-cache checked under 30 seconds ago: still served from the copy -min-forced-revalidate-interval 30s
Many hosts ask pkgproxy for the same InRelease; within five minutes all are answered from the copy; after that one conditional GET goes to the mirror and a 304 restarts the clock; a no-cache request inside thirty seconds of the last check is still served from the copy

-apt-revalidate-ttl (default: -revalidate-ttl, 5m) is how long a dists/** entry is served without asking the mirror; -min-forced-revalidate-interval (default 30s) bounds what a client's Cache-Control: no-cache can force. Both are also snap options.

When fifty machines run apt update, each asks for the same InRelease. Inside the -apt-revalidate-ttl, five minutes by default, pkgproxy answers all of them from its copy. After that, the first request sends one conditional GET to the mirror and the others join it; a 304 restarts the clock without rewriting the file. When apt asks for a fresh copy with no-cache, pkgproxy still serves its own if it checked within the -min-forced-revalidate-interval, thirty seconds by default.

04 When the mirror fails

MIRROR DOWN apt pkgproxy recheck InRelease mirror down · stalled · 5xx cached InRelease served from the copy a warning is logged 10 s a request waits at most this long, then gets the copy 1 min a mirror that failed is not asked again for this long 404 the mirror answered 404 or 410: the file is gone 502 known to be superseded, and cannot be fetched again
The mirror is down while an index is rechecked, so pkgproxy serves its cached copy to apt and logs a warning; a request waits at most ten seconds; a failed mirror is left alone for a minute; a 404 is passed on and a known-superseded entry gets 502

A failed revalidation serves the stale copy (outcome=stale, with a warning). The wait is bounded at 10 seconds and a failed origin is not retried for a minute. 404/410 become 404; a known-superseded entry that cannot be refetched gets 502.

If the mirror is down, stalls, or answers with a server error while an index is being rechecked, pkgproxy serves the copy it has and logs a warning, so apt update keeps working. A request waits at most ten seconds for the check, then gets the copy while the check carries on. A mirror that failed is left alone for a minute. Two failures are passed on: a 404 or 410 from the mirror reaches apt as a 404, and an entry the index already knows is superseded gets a 502 when it cannot be fetched again.

05 dists is stored first, pool streams

TWO FETCH PATHS dists: fetch, store, parse, serve fetch whole write to disk parse hashes serve from disk small files: nothing reaches apt before it is stored and checked pool: one fill, streamed mirror fill written and hashed every waiting client headers at once, then the bytes as they arrive either way, concurrent requests for one key share one fetch
Top lane: a dists file is fetched whole, written to disk, parsed for hashes and then served from disk. Bottom lane: a pool file is a fill whose bytes stream to every waiting client while they are written and hashed

A dists/** miss, by-hash included, is fetched, stored, parsed and then served from the store. A pool/** miss is a fill that streams to its clients while it writes.

Misses take one of two paths. A dists file is small: pkgproxy fetches it whole, writes it to the store, parses it for hashes, and only then serves it from disk, so a hash mismatch never leaves apt with part of a file. A pool file can be hundreds of megabytes. Its miss is a fill: the response headers go out at once, and the bytes stream to every waiting client while they are written and hashed. Either way, concurrent requests share one fetch.

06 Where to go next general version

WHERE NEXT each package comes from the mirror once, and apt update keeps working when the mirror fails you are here pool and dists Immutable or not the same rule on every side Fetch once how a pool miss streams to every client Offline mode apt update without the internet
You are here, pool and dists; next: Immutable or not, Fetch once, Offline mode

Next: Immutable or not, for the same rule on every side; Fetch once, for how a pool miss streams; Offline mode, for apt update without the internet.

Each package now comes from the mirror once, and apt update keeps working when the mirror fails. To see the same rule on every side, watch Immutable or not. To see how a pool miss streams to every client, watch Fetch once. To run apt update without the internet, watch Offline mode.

Read more

Path patterns

The first path segment named pool or dists makes a URL an archive path, whatever comes before it; there is no list of archives to configure.

PathKindCache policyChecked against
…/pool/…packages and sourceskept forever, never recheckedthe SHA-256 from a Packages or Sources file already read; unchecked, with a warning, when none is known
…/dists/…/by-hash/ALGO/HEXan index named by its hashkept forever, never recheckedthe hash in its own name
…/dists/…/InRelease, Release, Release.gpgthe signed root of a suiterechecked after -apt-revalidate-ttlnothing: apt checks the signature
any other …/dists/… file: Packages*, Sources*, Contents*, Translation-*, *.diff/Indexindexesrechecked after -apt-revalidate-ttlthe hash Release lists for it, when known
a path with no pool or dists segmentanything elsepassed through, not cached—
a directory: a trailing slash, the bare dists or pool, or one the mirror redirects to a trailing slash or answers with HTMLa directory listingrelayed, not cached—
a . or .. segment anywhere, or an empty one (//) from dists/ or pool/ onnot the object's own namerelayed, not cached—
any segment ending in .meta.jsoncould clash with pkgproxy's metadata filesrelayed, not cached—
https:// sources, through CONNECTencryptedtunnelled, not cached—

A listing that an older pkgproxy cached as a file where a directory belongs is removed once the mirror answers a file beneath it, never on the strength of a request alone; until then a request beneath it is relayed.

Compressed .xz indexes are cached and served but not read, so a pool file listed only in one is cached unchecked. HTTPS sources are covered in HTTPS apt sources, and how the cache key is built in apt cache keys and hash checks.

A rechecked index, step by step

SituationWhat pkgproxy does
cached and checked within -apt-revalidate-ttlserves it, asks nothing
older than thatsends one If-Modified-Since request for every client waiting on it; a 304 keeps the copy and restarts the clock
no Last-Modified was recordedfetches it again in full
a newer Release lists a different hash for the cached copyfetches it again in full, even within the TTL
the mirror fails, stalls, or answers 5xx or another unexpected statusserves the cached copy, warns in the log, and the access log says outcome=stale
the check takes more than 10 secondsserves the cached copy while the check carries on
the mirror failed a check within the last minuteserves the cached copy without asking it
the mirror answers 404 or 410404 to apt
the copy is known to be superseded and cannot be fetched again502 to apt

A miss on an index is fetched whole, stored, read for hashes and only then served. A miss on a pool file streams to every waiting client while it is written; see Fetch once.

A client asking for a fresh copy

apt sends these headers for Acquire::http::No-Cache and Acquire::http::Max-Age.

Request headerEffect on a cached index within its TTL
Cache-Control: no-cachechecked with the mirror first, unless checked within -min-forced-revalidate-interval
Cache-Control: max-age=Nchecked first if the last check is older than N seconds, with the same bound
Pragma: no-cache, with no Cache-Controlthe same as no-cache

A forced check is a conditional request: a 304 keeps the copy, so it never repairs wrong content; pkgproxyctl cache invalidate does. Pool files and by-hash files are never rechecked, whatever the client asks. In offline mode no check is forced.

Status codes

SituationStatus apt sees
a request line that is not an absolute http:// URL, as apt sends to a proxy400
a method other than GET, HEAD or CONNECT405
CONNECT to a port other than 443403
a passed-through request or CONNECT to a loopback address, or to this machine at a port pkgproxy listens on403
the mirror answers 404 or 410404
a downloaded file fails its hash checkthe mirror's bytes, unchanged and not cached
the mirror cannot be reached and nothing is cached502

for administrators for CI operators

Settings

OptionEnvironment variableSnap optionDefaultMeaning
-apt-revalidate-ttlPKGPROXY_APT_REVALIDATE_TTLapt-revalidate-ttl-revalidate-ttl, 5mhow long an index is served before it is checked again
-revalidate-ttlPKGPROXY_REVALIDATE_TTLrevalidate-ttl5mthe same for every side that has no setting of its own
-min-forced-revalidate-intervalPKGPROXY_MIN_FORCED_REVALIDATE_INTERVALmin-forced-revalidate-interval30show recently checked an index must be for a client's no-cache to be ignored
-fill-drain-timeoutPKGPROXY_FILL_DRAIN_TIMEOUTfill-drain-timeout10mhow long a pool file that failed its hash check keeps going to the clients already receiving it
-upstream-idle-timeoutPKGPROXY_UPSTREAM_IDLE_TIMEOUTupstream-idle-timeout1mhow long a mirror may send nothing before the request fails; 0 waits forever

The 10-second wait and the one-minute pause after a failure are fixed. Cached files live under apt/objects/HOST/PATH in the cache directory, such as apt/objects/archive.ubuntu.com/ubuntu/pool/main/a/apt/apt_2.8.3_amd64.deb; see The store.

In the documentation

Roads from here

CORE · the path every side shares

APT

OPERATIONS