Immutable or revalidated

core for everyone

Which cached entries are kept forever and which are rechecked after a short TTL on every side, how a recheck is made, what a client asking for a fresh copy changes, and what is served when the upstream fails.

On the map: core

▶ Watch this deck — 7 scenes, ~5 min

01 What this topic covers

ABOUT THIS TOPIC Immutable or not files that never change are kept forever; lists that change are rechecked 1 Two kinds of entry on every side 2 A short TTL, then a recheck 3 One window, overridden per side for administrators and CI operators 4 When apt asks for a fresh copy 5 When the recheck fails
The topic title, Immutable or not, and its 5 scenes in order: Two kinds of entry on every side, A short TTL, then a recheck, One window, overridden per side, When apt asks for a fresh copy, When the recheck fails

Which entries are kept forever and which are rechecked after a short window, on every side.

Some files never change once published, and pkgproxy keeps them forever; lists that change are checked again with the upstream after a short time. It helps to have seen A request first. Here you see the two kinds of entry on every side, the short window and the recheck that follows it, and how each side can have its own window. Then what changes when apt asks for a fresh copy, and what is served when the recheck fails.

02 Two kinds of entry on every side

IMMUTABLE OR NOT apt snap go docker lxd files kept forever, never checked again pool/** dists/**/by-hash/** snap files, keyed by their sha3-384 @v/v1.2.3.mod @v/v1.2.3.zip blobs and manifests, by digest image files a catalog lists with a hash every file, keyed by its checksum rechecked after a TTL InRelease Release Packages* snap information, most assertions @v/list @latest @v/v1.2.3.info tags: the digest a tag names streams/v1/index.json images.json nothing named by hash or version: forever · lists that move: rechecked
A table with one row per side, apt, snap, go, docker, lxd and files; the left column lists what is kept forever, the right column what is rechecked after a TTL

Content addressed by hash or by version is immutable: fetched once, verified and cached forever. Indexes, version lists and tags are mutable and revalidated after a TTL. Files by checksum have no mutable part.

Every side of pkgproxy sorts what it caches into two kinds. Content named by its hash or by a fixed version never changes: an apt package in the pool, a snap file, a Go module's zip, a Docker layer or manifest fetched by digest, an LXD image file, and every file fetched by checksum. These are kept forever and never checked again. Lists that move are rechecked after a short time: apt's Release and Packages indexes, snap information, Go version lists, Docker tags, and LXD's image lists.

03 A short TTL, then a recheck

TTL, THEN A RECHECK fetched dists/trixie/InRelease 5 minutes: every request is a hit hit · hit · hit · hit a fleet's apt update costs one check GET …/InRelease If-Modified-Since: … mirror 304 Not Modified keep the copy, 5 more minutes 200 OK replace the copy 5 more minutes of hits Go, LXD and snap lists are fetched again in full · a Docker tag is resolved with HEAD
A time line for one apt index: five minutes of hits from disk, then a conditional request upstream answered 304, which keeps the copy and starts another five minutes; a 200 would replace it

Inside the TTL a mutable entry is a hit. After it, apt sends If-Modified-Since: 304 renews the entry, 200 replaces it. The other sides fetch again; Docker resolves a tag with HEAD.

A list that moves is stored with the time it was fetched. For five minutes by default, every request for it is a hit, answered from disk, so a fleet running apt update together costs the upstream one check. After that, apt's indexes are rechecked with a conditional request carrying the cached copy's Last-Modified date. A 304 keeps the copy and starts its five minutes again; a 200 replaces it. Go and LXD lists and snap information are fetched again in full, and a Docker tag is resolved again with a HEAD request.

04 One window, overridden per side for administrators for CI operators

ONE WINDOW, OVERRIDDEN PER SIDE -revalidate-ttl 5m every side -apt-revalidate-ttl -snap-revalidate-ttl -go-revalidate-ttl -lxd-revalidate-ttl -docker-revalidate-ttl apt snap go lxd docker unset: -revalidate-ttl unset: -revalidate-ttl unset: -revalidate-ttl unset: -revalidate-ttl unset: -revalidate-ttl 0 or less is refused, by snap set and when pkgproxy starts max-age from the store wins longer window: fewer requests upstream, older lists offline mode: every window frozen, nothing rechecked
One global revalidate TTL of five minutes applies to every side; five per-side settings take the global value unless set; zero or less is refused; the snap store's max-age wins; offline mode freezes every window

-revalidate-ttl (5m) applies to every side; -apt-revalidate-ttl, -snap-revalidate-ttl, -go-revalidate-ttl, -lxd-revalidate-ttl and -docker-revalidate-ttl override it for one side each, with snap options of the same names. Zero is refused. A snap store max-age takes precedence; offline mode makes every window endless.

One setting, the -revalidate-ttl, sets the window for every side, five minutes by default. Each side can override it with its own flag and snap option: the apt, snap, go, LXD and docker revalidate TTLs. A side whose own setting is not set takes the global value. A window must be longer than zero, and pkgproxy refuses zero before it starts. The snap side also keeps an answer for the max age the Snap Store announces with it, and uses its TTL for answers that announce none. A longer window means fewer requests upstream and older lists. Offline mode freezes every window.

05 When apt asks for a fresh copy

A CLIENT ASKS FOR FRESH apt update Cache-Control: max-age=0 pkgproxy check, not download If-Modified-Since mirror checked in the last 30 seconds? served from disk anyway forced checks of one file share one request upstream a 304 keeps the copy: a check only confirms $ pkgproxyctl cache invalidate http://deb.debian.org/debian/dists/trixie/InRelease
apt sends Cache-Control max-age=0; pkgproxy checks the index upstream with a conditional request unless it checked it in the last thirty seconds; several forced checks share one request; pkgproxyctl cache invalidate removes a file

Cache-Control: no-cache, max-age=N or Pragma: no-cache force a conditional GET, except within -min-forced-revalidate-interval (30s) of the last check. A 304 keeps the entry; pkgproxyctl cache invalidate removes one.

Apt update asks for a fresh copy of each index: it sends Cache Control, max age zero. pkgproxy treats that as a reason to check, not to download again: an index inside its five minutes gets a conditional request upstream before it is served. To bound the traffic clients can cause, an index checked within the last thirty seconds is served from disk anyway, and forced checks of one file share one request. A check only confirms what is cached. To throw a cached apt file away, use pkgproxyctl cache invalidate.

06 When the recheck fails

THE MIRROR IS DOWN apt update recheck mirror no answer · 5xx stale cached copy served anyway, with a warning in the log waits at most 10 s, then the copy a failing host is skipped for 1 minute apt, Go and LXD · the recheck carries on in the background 404 or 410 upstream: apt gets a 404 fails its hash check: passed through, not cached
A recheck to a mirror that is down: after ten seconds the request is answered from the stale cached copy while the recheck carries on; the host is skipped for a minute; a 404 or 410 is passed on as a 404

A failed revalidation serves the stale entry with a warning. apt, Go and LXD wait at most 10 s for it and skip a failing host for 1 min. For apt, an origin 404/410 is relayed as 404; a hash mismatch is passed through uncached.

When a recheck fails, because the mirror is unreachable, stalls, or answers with a 5xx, the cached copy is served anyway and a warning is logged, so apt update keeps working while the mirror is down. For apt, Go and LXD, a request waits at most ten seconds for the recheck, then gets the cached copy while the recheck carries on, and a host that failed is left alone for a minute. Two answers are not covered up: a 404 or 410 from the mirror reaches apt as a 404, and content that fails its hash check is not cached.

07 Where to go next general version

WHERE NEXT a package is kept forever, an index is checked again after a few minutes, and an upstream that is down still leaves you the cached copy you are here Immutable or not Offline mode when nothing may be rechecked pool and dists apt's files in detail Stalled upstreams when an upstream counts as stalled
You are here, Immutable or not; next: Offline mode, pool and dists, Stalled upstreams

Next: Offline mode, for freshness frozen; pool and dists, for apt in detail; Stalled upstreams, for when an upstream counts as stalled.

Now you know why a package is kept forever while an index is checked again after a few minutes, and why an upstream that is down still leaves you the cached copy. To see what happens when nothing may be checked again, watch Offline mode. To see apt's pool and dists in detail, watch pool and dists. To see when an upstream counts as stalled, watch Stalled upstreams.

Read more

Every side's entries

Kept forever means fetched once, checked against its hash where one is known, and never asked about again. Rechecked means served from the cache for the window in the last column, then checked with the upstream. Relayed means passed between client and upstream without being kept.

apt

EntryPolicyWindow
anything under pool/, and dists/…/by-hash/…kept forever, hash checkednone
Release, InRelease, Release.gpgrechecked with If-Modified-Sinceapt-revalidate-ttl
every other file under dists/ (Packages, Sources, Contents, Translation, diff indexes)rechecked with If-Modified-Sinceapt-revalidate-ttl
a URL with no pool or dists path segmentrelayednone
an https:// source, through a CONNECT tunnelrelayed, unseennone

An index with no Last-Modified from the mirror, or one a newer index already shows to be superseded, is fetched again in full instead of conditionally.

Snap Store

EntryPolicyWindow
snap files, by their SHA3-384kept forevernone
snap-revision assertions, and account, snap-declaration, store or validation-set assertions pinned to a sequence numberkept forevernone
every other assertion, account-key includedrecheckedsnap-revalidate-ttl
snap info, find, categories, sections, names, repairs, and the old v1 search and detailsrecheckedthe store's max-age, else snap-revalidate-ttl
refresh and install requestsrelayed; each result recorded for answering offlinenone
download redirects without a hash, such as resourcesfollowed and streamed, not keptnone
logins, device sessions and serials, cohorts, metadatarelayednone
any other store pathrelayednone

The store's max-age wins whether it is shorter or longer than the window; a max-age of zero counts as none.

Go modules

EntryPolicyWindow
.mod and .zip of a versionkept forever, after the checksum database checknone
list, @latest and .infofetched again in fullgo-revalidate-ttl
a 404 from the upstreamnot keptnone

Docker Hub

EntryPolicyWindow
layers and configs, by digestkept forever, digest checkednone
manifests and indexes, by digestkept forever, digest checkednone
the digest a tag resolves to, per set of accepted media typesrechecked with a HEADdocker-revalidate-ttl
a HEAD of a layer not yet cachedanswered from the registry's HEAD, not keptnone
referrersrelayednone

LXD images

EntryPolicyWindow
streams/v1/index.json and every other .json catalogfetched again in fulllxd-revalidate-ttl
an image file whose path a catalog listedkept forever, hash checkednone
a path no catalog listedanswered 404, never fetchednone

LXD's image API, on the HTTPS port, reads the same catalogs and files, so it follows the same policy.

Files by checksum

Every file is kept forever under its checksum and never rechecked; there is nothing to revalidate.

for administrators for CI operators

The windows

SettingDefaultWindow for
revalidate-ttl5mevery side whose own setting is unset
apt-revalidate-ttlrevalidate-ttlapt indexes
snap-revalidate-ttlrevalidate-ttlsnap details and assertions without a max-age
go-revalidate-ttlrevalidate-ttlGo version lists, @latest, .info
docker-revalidate-ttlrevalidate-ttlDocker tags
lxd-revalidate-ttlrevalidate-ttlLXD catalogs
min-forced-revalidate-interval30sapt only: how long after a check a client's request for a fresh copy is still served from the cache

Each is a -flag, a PKGPROXY_ variable and a snap option of the same name; see Settings. A window must be longer than zero: pkgproxy refuses 0 at start and in -check-config. In offline mode every window is endless; see Offline mode.

A client asking for a fresh copy

SideCache-Control: no-cache, max-age=N, Pragma: no-cache
apta conditional request upstream before serving, unless the index was checked within min-forced-revalidate-interval; requests forcing one index share one check
snap, Go, Docker, LXD, filesignored: the window decides

A Pragma: no-cache counts only when there is no Cache-Control. A check confirms or replaces an index; it never deletes one. To throw a cached apt file away, use pkgproxyctl cache invalidate.

When a recheck fails

SideUpstream unreachable, stalled or answering 5xxUpstream says the entry is gone
aptcached copy served with a warning; waits at most 10 s, host skipped for 1 min404 or 410 relayed as 404
Go, LXDcached copy served with a warning; waits at most 10 s, host skipped for 1 mincached copy served with a warning, as for any answer other than 200
snapcached copy served with a warning, also on a 429 from the storeupstream's answer relayed
Dockerstale tag served with a warning; a 401 or 403 also keeps a tag that resolved before404, and the client falls back to Docker Hub

An outcome of stale in the access log marks each such answer.

In the documentation

Roads from here

CORE · the path every side shares

APT

DOCKER

OPERATIONS