pool forever and dists rechecked
apt for users for administrators for CI operators
Pool files and by-hash indexes are cached forever; the other dists files are served for a short TTL and then revalidated, and a cached copy is served when the mirror fails.
On the map: apt
01 What this topic covers
Pool files are kept forever, dists files are rechecked after a short window, and a failing mirror is covered by the cached copy.
Apt downloads two kinds of file: packages from pool, which pkgproxy keeps forever, and indexes from dists, which it checks again with the mirror after a few minutes. If you have not seen A forward proxy yet, start there. Here you see the three kinds of path, and fifty machines running apt update together. Then what happens when the mirror fails, and why a dists file is stored before it is served while a pool file streams.
02 Three kinds of path
pool/** and dists/**/by-hash/** are immutable:
cached forever, never revalidated. Other dists/** files are
revalidated after a TTL. Anything else is passed through uncached.
The path decides what happens. Pool files never change once published, so they are fetched once, checked against their hash and kept forever without asking the mirror again. So are files under by-hash, even inside dists, because the hash is in their name. Every other file under dists, InRelease, Release, Packages and the rest, changes with each archive update and is rechecked. A path outside pool and dists is passed through and not cached.
03 A fleet runs apt update for administrators for CI operators
-apt-revalidate-ttl (default: -revalidate-ttl, 5m) is how long a
dists/** entry is served without asking the mirror;
-min-forced-revalidate-interval (default 30s)
bounds what a client's Cache-Control: no-cache can force. Both
are also snap options.
When fifty machines run apt update, each asks for the same InRelease. Inside the -apt-revalidate-ttl, five minutes by default, pkgproxy answers all of them from its copy. After that, the first request sends one conditional GET to the mirror and the others join it; a 304 restarts the clock without rewriting the file. When apt asks for a fresh copy with no-cache, pkgproxy still serves its own if it checked within the -min-forced-revalidate-interval, thirty seconds by default.
04 When the mirror fails
A failed revalidation serves the stale copy (outcome=stale,
with a warning). The wait is bounded at 10 seconds and a failed origin is
not retried for a minute. 404/410 become
404; a known-superseded entry that cannot be refetched gets
502.
If the mirror is down, stalls, or answers with a server error while an index is being rechecked, pkgproxy serves the copy it has and logs a warning, so apt update keeps working. A request waits at most ten seconds for the check, then gets the copy while the check carries on. A mirror that failed is left alone for a minute. Two failures are passed on: a 404 or 410 from the mirror reaches apt as a 404, and an entry the index already knows is superseded gets a 502 when it cannot be fetched again.
05 dists is stored first, pool streams
A dists/** miss, by-hash included, is fetched,
stored, parsed and then served from the store. A pool/** miss
is a fill that streams to its clients while it writes.
Misses take one of two paths. A dists file is small: pkgproxy fetches it whole, writes it to the store, parses it for hashes, and only then serves it from disk, so a hash mismatch never leaves apt with part of a file. A pool file can be hundreds of megabytes. Its miss is a fill: the response headers go out at once, and the bytes stream to every waiting client while they are written and hashed. Either way, concurrent requests share one fetch.
06 Where to go next general version
Next: Immutable or not, for the same rule on every side; Fetch once, for how a pool miss streams; Offline mode, for apt update without the internet.
Each package now comes from the mirror once, and apt update keeps working when the mirror fails. To see the same rule on every side, watch Immutable or not. To see how a pool miss streams to every client, watch Fetch once. To run apt update without the internet, watch Offline mode.
Read more
Path patterns
The first path segment named pool or dists makes a URL an archive path, whatever comes before it; there is no list of archives to configure.
| Path | Kind | Cache policy | Checked against |
|---|---|---|---|
…/pool/… | packages and sources | kept forever, never rechecked | the SHA-256 from a Packages or Sources file already read; unchecked, with a warning, when none is known |
…/dists/…/by-hash/ALGO/HEX | an index named by its hash | kept forever, never rechecked | the hash in its own name |
…/dists/…/InRelease, Release, Release.gpg | the signed root of a suite | rechecked after -apt-revalidate-ttl | nothing: apt checks the signature |
any other …/dists/… file: Packages*, Sources*, Contents*, Translation-*, *.diff/Index | indexes | rechecked after -apt-revalidate-ttl | the hash Release lists for it, when known |
a path with no pool or dists segment | anything else | passed through, not cached | — |
a directory: a trailing slash, the bare dists or pool, or one the mirror redirects to a trailing slash or answers with HTML | a directory listing | relayed, not cached | — |
a . or .. segment anywhere, or an empty one (//) from dists/ or pool/ on | not the object's own name | relayed, not cached | — |
any segment ending in .meta.json | could clash with pkgproxy's metadata files | relayed, not cached | — |
https:// sources, through CONNECT | encrypted | tunnelled, not cached | — |
A listing that an older pkgproxy cached as a file where a directory belongs is removed once the mirror answers a file beneath it, never on the strength of a request alone; until then a request beneath it is relayed.
Compressed .xz indexes are cached and served but not read, so a pool file listed only in one is cached unchecked. HTTPS sources are covered in HTTPS apt sources, and how the cache key is built in apt cache keys and hash checks.
A rechecked index, step by step
| Situation | What pkgproxy does |
|---|---|
cached and checked within -apt-revalidate-ttl | serves it, asks nothing |
| older than that | sends one If-Modified-Since request for every client waiting on it; a 304 keeps the copy and restarts the clock |
no Last-Modified was recorded | fetches it again in full |
a newer Release lists a different hash for the cached copy | fetches it again in full, even within the TTL |
the mirror fails, stalls, or answers 5xx or another unexpected status | serves the cached copy, warns in the log, and the access log says outcome=stale |
| the check takes more than 10 seconds | serves the cached copy while the check carries on |
| the mirror failed a check within the last minute | serves the cached copy without asking it |
the mirror answers 404 or 410 | 404 to apt |
| the copy is known to be superseded and cannot be fetched again | 502 to apt |
A miss on an index is fetched whole, stored, read for hashes and only then served. A miss on a pool file streams to every waiting client while it is written; see Fetch once.
A client asking for a fresh copy
apt sends these headers for Acquire::http::No-Cache and Acquire::http::Max-Age.
| Request header | Effect on a cached index within its TTL |
|---|---|
Cache-Control: no-cache | checked with the mirror first, unless checked within -min-forced-revalidate-interval |
Cache-Control: max-age=N | checked first if the last check is older than N seconds, with the same bound |
Pragma: no-cache, with no Cache-Control | the same as no-cache |
A forced check is a conditional request: a 304 keeps the copy, so it never repairs wrong content; pkgproxyctl cache invalidate does. Pool files and by-hash files are never rechecked, whatever the client asks. In offline mode no check is forced.
Status codes
| Situation | Status apt sees |
|---|---|
a request line that is not an absolute http:// URL, as apt sends to a proxy | 400 |
a method other than GET, HEAD or CONNECT | 405 |
CONNECT to a port other than 443 | 403 |
a passed-through request or CONNECT to a loopback address, or to this machine at a port pkgproxy listens on | 403 |
the mirror answers 404 or 410 | 404 |
| a downloaded file fails its hash check | the mirror's bytes, unchanged and not cached |
| the mirror cannot be reached and nothing is cached | 502 |
for administrators for CI operators
Settings
| Option | Environment variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-apt-revalidate-ttl | PKGPROXY_APT_REVALIDATE_TTL | apt-revalidate-ttl | -revalidate-ttl, 5m | how long an index is served before it is checked again |
-revalidate-ttl | PKGPROXY_REVALIDATE_TTL | revalidate-ttl | 5m | the same for every side that has no setting of its own |
-min-forced-revalidate-interval | PKGPROXY_MIN_FORCED_REVALIDATE_INTERVAL | min-forced-revalidate-interval | 30s | how recently checked an index must be for a client's no-cache to be ignored |
-fill-drain-timeout | PKGPROXY_FILL_DRAIN_TIMEOUT | fill-drain-timeout | 10m | how long a pool file that failed its hash check keeps going to the clients already receiving it |
-upstream-idle-timeout | PKGPROXY_UPSTREAM_IDLE_TIMEOUT | upstream-idle-timeout | 1m | how long a mirror may send nothing before the request fails; 0 waits forever |
The 10-second wait and the one-minute pause after a failure are fixed. Cached files live under apt/objects/HOST/PATH in the cache directory, such as apt/objects/archive.ubuntu.com/ubuntu/pool/main/a/apt/apt_2.8.3_amd64.deb; see The store.
In the documentation
- Settings: the revalidate TTLs
- Removing a cached file early
Roads from here
CORE · the path every side shares
- Immutable or revalidated — the same rule on every side
- Fetch once — how a pool miss streams to every client
- Offline mode — apt update without the internet
APT
- apt cache keys and hash checks — how the key and its policy are chosen
OPERATIONS
- Settings — changing the revalidate TTL
- pkgproxy for CI — many machines behind one cache