pkgproxy design: apt

How pkgproxy proxies and caches apt archives: what apt asks for, the proxy’s shape, normalization, the caching policy, the main abstractions and the fetch path. Part of the pkgproxy design, whose index says which file holds each § number.

• 2. How apt actually talks to an archive
  • 2.1 What a client is configured with
  • 2.2 Directory layout of an archive
  • 2.3 Release/InRelease fields we need
  • 2.4 Packages/Sources stanza fields we need
  • 2.5 HTTP behavior apt actually relies on
• 3. Proxy shape: forward proxy, not mirror substitution
• 4. Normalization: collapsing mirrors to one cache entry
  • 4.1 Path normalization for the cache key
  • 4.2 Content-hash addressing for pool files (the strong guarantee)
  • 4.3 Index files (dists/**)
• 5. Caching policy summary
• 6. Data model
• 7. Go interfaces
  • 7.1 Storage backend
  • 7.2 Origin fetcher
  • 7.3 Cache key resolution / normalization
  • 7.4 Index cache (Release/Packages hash lookups)
  • 7.5 Proxy handler wiring
• 8. Fetch path (immutable, cache miss)
  • 8.1 Cache misses on pool/** stream; they never wait for the whole file

2. How apt actually talks to an archive

This section is the protocol analysis the rest of the design rests on. Sourced from the Debian Repository Format reference, the sources.list(5) and apt-transport-http(1) manpages, and apt’s own methods/http.cc.

2.1 What a client is configured with

A source entry (one-line sources.list or deb822 *.sources) names the type (deb fetches Packages, deb-src fetches Sources), the archive root URI, the suites (a codename such as bookworm, or a path relative to the root for a “flat” repo) and the components (main, contrib, …). The proxy never sees this configuration, only the HTTP requests it leads to.

Proxy config: apt honors the http_proxy env var and Acquire::http::Proxy (globally or per-host via Acquire::http::Proxy::<host>). This is the integration point we use: point apt at us as a normal forward proxy, and let it keep talking to whatever mirror hostnames are in its sources.list. See §3 for why this beats the alternative of rewriting sources.list to name our host directly.

2.2 Directory layout of an archive

<root>/
  dists/<suite>/
    InRelease                        # inline-signed meta-index (preferred)
    Release                          # unsigned meta-index
    Release.gpg                      # detached signature for Release
    <component>/
      binary-<arch>/
        Packages, Packages.gz, Packages.xz
        Packages.diff/Index          # pdiff meta-index (not fetched, v1)
      source/
        Sources, Sources.gz, Sources.xz
      i18n/
        Translation-en.bz2, Translation-en.xz, ...
      Contents-<arch>.gz
    by-hash/                         # present per-index-dir when
      SHA256/<hex>                   # Acquire-By-Hash: yes
  pool/<component>/<first-letter(s)>/<source-name>/
    <pkg>_<version>_<arch>.deb
    <pkg>_<version>.dsc
    <pkg>_<version>.orig.tar.xz
    <pkg>_<version>.debian.tar.xz

Two path families matter for caching, and they have opposite mutability:

• pool/** — content-addressed by construction. The filename encodes name+version+arch, and Debian archive policy never reuses one. Once published, a pool file is immutable. Cache forever, no revalidation.
• dists/** (and top-level Release*) — mutable. Packages, Sources, Release etc. are re-published on every archive push (multiple times a day). Must revalidate, except the by-hash/<algo>/<hex> variants, which are themselves immutable (see below).

2.3 Release/InRelease fields we need

Suite: bookworm
Codename: bookworm
Date: Sat, 02 Jul 2016 05:20:50 +0000
Valid-Until: <optional expiry>
Acquire-By-Hash: yes
SHA256:
 <hex>  <size>  main/binary-amd64/Packages.xz
 <hex>  <size>  main/binary-amd64/Packages
 <hex>  <size>  main/source/Sources.xz
 ...

The per-file SHA256/MD5Sum lines are the important part: they give us, for free, from a file apt is downloading anyway, a verified hash for every index file in the suite. We use this as a secondary integrity check and as the key space for by-hash (§4.2).

Acquire-By-Hash: yes means the origin also serves every index at <dir>/by-hash/<ALGO>/<hex>. Those paths are immutable (the hash is in the URL) and can be cached forever exactly like pool files, even though they live under dists/. Modern apt fetches Packages/Sources almost only through these paths when the archive offers them, as Debian and Ubuntu do.

2.4 Packages/Sources stanza fields we need

Package: apt
Filename: pool/main/a/apt/apt_2.6.1_amd64.deb
Size: 1362292
MD5sum: 2519c8c1afd27e70cf4ac10a5fa46e32
SHA256: 3183eff291d1e9d905e78a6b467bbfb90b20fc2808d50b5e91bf55158b4c18b

Filename is the path a client will GET next; SHA256 (or SHA512) is the hash we verify the fetched bytes against before admitting them to the cache. We do not need to fully model Debian control-file syntax — only extract Filename/Size/hash fields per stanza (see §6).

2.5 HTTP behavior apt actually relies on

From methods/http.cc and apt-transport-http(1):

The practical upshot: apt already implements the client half of HTTP caching correctly (conditional requests, range resume). Our job is mostly to be a well-behaved HTTP/1.1 origin + a well-behaved HTTP/1.1 client to the real mirror, plus the caching and normalization logic in between.

3. Proxy shape: forward proxy, not mirror substitution

Two ways to insert ourselves:

1. Forward proxy. apt keeps deb.debian.org, ftp.us.debian.org etc. in its sources.list; we’re configured via Acquire::http::Proxy / http_proxy. Requests arrive with an absolute-URI request line (GET http://deb.debian.org/debian/pool/... HTTP/1.1) naming the real origin host.
2. Mirror substitution. We rewrite sources.list to point at our own host (http://cache.local/debian/...) fronting one fixed upstream.

Chosen: forward proxy. It is the only shape that lets one cache instance transparently absorb requests to many different mirror hostnames — which is exactly the “collapse two mirrors into one backing store file” requirement — without editing every machine’s sources.list per-mirror. It also requires zero apt-side config beyond the two proxy lines, and it degrades safely: if our proxy is down, apt’s proxy config simply fails over per normal apt/system behavior (or the admin unsets the env var).

Because caching HTTPS content is out of scope, CONNECT gets a blind, uncached byte-for-byte tunnel, restricted to port 443 so this stays “let HTTPS apt sources through” rather than a general-purpose TCP relay (pkgproxy already relays plaintext http:// to any host and port via PolicyPassthroughOnly, §4, so this does not change the threat model). We deliberately do not terminate TLS or MITM the tunneled traffic.

4. Normalization: collapsing mirrors to one cache entry

Two mirrors serving the same archive (e.g. deb.debian.org and ftp.de.debian.org) are not guaranteed byte-identical at every path — sync timing differs — but should converge, and Debian mirroring policy is that they carry the same content once settled. We use two complementary normalization strategies:

4.1 Path normalization for the cache key

Strip scheme; the archive id is the host plus whatever path prefix precedes dists/ or pool/ (empty segments dropped), and the canonical path is everything from dists/ or pool/ onward. The prefix is part of the identity, not noise: deb.debian.org hosts both /debian and /debian-security, and KDE Neon’s editions (<host>/unstable/..., <host>/release/...) share Suite/Codename and pool-relative Filename: values but are disjoint archives (no shared pool file names or hashes), so dropping the prefix would alias them while keeping it costs no storage.

Two hosts not listed as aliases of each other are separate archives (separate key namespace), which is the safe default: we never silently merge content from mirrors we haven’t been told are equivalent. Collapsing mirrors of one archive (deb.debian.org/debian and ftp.de.debian.org/debian) under one archive id therefore needs an alias table (host+prefix → canonical archive id). It is not built yet (not required for v1); today the archive id is always the request’s own host and prefix. The intended design:

• Hostname heuristics for well-known regional naming, such as Ubuntu’s <cc>.archive.ubuntu.com clamping to archive.ubuntu.com. Config-free, but a guess.
• Release/InRelease identity for everything else: two hosts serving a dists/<suite>/Release with matching Suite/Codename/Date (ideally the full hash-line set) are the same archive, a much stronger signal than hostname shape. Checked lazily, on the first request for a new host.

Either source only proposes an alias; a mismatch means treat as separate, not an error, since mirrors may be at different sync points. Confirmed aliases are persisted, not re-derived on every restart.

4.2 Content-hash addressing for pool files (the strong guarantee)

Path normalization alone trusts that two paths-with-the-same-suffix really are the same bytes. For pool/** we don’t have to trust that: every Packages/Sources stanza gives us a SHA256 for the file at that Filename. Once we’ve parsed the index (which we cache and re-fetch per §4.3), we verify every pool file we fetch against that hash before admitting it to the cache. This means:

• Two mirrors that happen to disagree on a pool path are caught, not silently merged — hash mismatch is a hard error (log + fail the fetch, never cache). The first known hash for a pool path wins; a later index naming a different one is logged as a conflict, not taken, while a hash in a newly seen algorithm is merged in.
• by-hash/<algo>/<hex> requests (§2.3) are trivially content-addressed — the hash is literally in the URL, so it needs no index lookup.
• Any other dists/** file with a hash already learned from its Release is checked against it by the same rule.

We use path-normalized keys with hash verification on write, not content-addressed storage (sha256/<hex> with a path→hash index, which would store a package that appears under several pool paths once): a simpler, human-inspectable store layout, with the same safety property and no second indirection layer. The Store interface (§7.1) keeps switching the key scheme an internal change.

Known gaps, accepted:

• InRelease (and Release) are the trust root: nothing lists their hash, so they are cached unverified. apt verifies their signatures.
• .xz indices are cached and served but not parsed (Go’s standard library has no xz reader), so a pool file listed only in an .xz Packages, such as one apt fetched by hash, has no expected hash and is cached unverified, logged as a gap.

4.3 Index files (dists/**)

Same path normalization, but cached as mutable entries: revalidate against origin using If-Modified-Since/ETag we recorded on the last fetch, honor origin 304, and additionally apply a short local freshness TTL (-revalidate-ttl, default 5 minutes) so a burst of apt update across a fleet hits the origin once, not once per host. On a cache hit inside the TTL we skip revalidation entirely; outside it we revalidate before serving from disk.

A client’s Cache-Control: no-cache (= max-age=0), a max-age=N older than the entry’s last origin check, or Pragma: no-cache without Cache-Control, turns a hit inside the TTL into a conditional GET to the origin before serving. This only ever checks: a 304 keeps the entry, so it does not heal wrong content (pkgproxyctl cache invalidate does). To bound the origin traffic any client can cause, an entry checked within -min-forced-revalidate-interval is served from cache regardless, and concurrent forced requests share one fetch.

If a revalidation fails — the origin is unreachable, stalls, or answers 5xx or another unexpected status — the cached entry is served anyway, with a warning logged: a mirror outage must not take apt update down with it. Two bounds keep an outage from costing each request its own timeout. A request that has a cached copy to fall back on waits at most RevalidateWait for the revalidation, then serves the copy while the revalidation carries on in the background (it is detached from the request, §7.5). And an origin host that failed, or ran past RevalidateWait, is not asked again for OriginRetryInterval: entries it serves are answered from the cache. Any other answer from the host clears that, and so does a change of the offline mode in effect (§21). Both are Handler fields with defaults in internal/revalidate, which the gomod and LXD sides share. Three failures are not papered over: an origin 404 or 410 is relayed as a 404 (the file is gone, e.g. a removed suite), a hash mismatch takes the passthrough fallback (§8 step 7), and an entry the index already knows to be superseded gets a 502, since serving content known to be wrong would only surface in apt as a confusing hash-sum mismatch.

5. Caching policy summary

Everything not matching a known archive root (stray requests, non-repo paths) is proxied without caching.

6. Data model

internal/apt extracts only the caching-relevant fields: from Release/InRelease, the suite identity and the per-file hashes; from each Packages/Sources stanza, Filename, Size and hashes. Stanzas are RFC 822-style (Key: value, folded continuation lines); there is no full Debian control-file model. InRelease’s OpenPGP clearsign armour is stripped, not verified, so it parses like Release. A malformed Date or Valid-Until leaves that field unset rather than discarding the hashes.

7. Go interfaces

7.1 Storage backend

The core abstraction, store.Store (internal/store): local disk (fslocal) today, S3-compatible object storage later, and an in-memory memstore that lets tests exercise the real visibility contract without fslocal’s disk layout.

• Bodies are io.Reader/io.Writer streams, never []byte, so multi-hundred-MB .deb files are never fully buffered in memory.
• Writes go through Create and become visible only on Commit: a local-disk implementation writes a temp file and rename(2)s it, an S3 one would complete a multipart upload. A reader never observes a half-written object, so a failed origin fetch never appears as a hit.
• OpenRange is part of the interface because apt’s resume (§2.5) needs ranges on cached entries, not just passthrough.
• UpdateMeta replaces metadata without touching the body, so a 304 revalidation is a metadata-only write.

7.2 Origin fetcher

origin.Fetcher (internal/origin) is a thin interface over the HTTP client, so the proxy core is testable without a network and retry and timeout policy live in one place.

7.3 Cache key resolution / normalization

cachekey.Resolver (internal/cachekey) encapsulates §4: it turns a request URL into a canonical key and the policy that applies to it.

• PolicyImmutable: pool/** and by-hash/**.
• PolicyRevalidate: dists/** metadata.
• PolicyPassthroughOnly: anything unrecognized.

The key is <archive id>/<path>, e.g. deb.debian.org/debian/pool/main/a/apt/apt_2.6.1_amd64.deb, so two distinct archives never collide even if some path suffix matches. For a by-hash path the resolver also returns the hash from the URL.

7.4 Index cache (Release/Packages hash lookups)

index.Lookup (internal/index) gives the expected hash for a fetch before it is admitted to the store (§4.2). It is populated as a side effect of proxying Release/Packages/Sources — parse on the way through, update the lookup, serve the bytes unchanged — never by a separate crawl, so the proxy that queries it also writes it. A by-hash index body has no file name, so it is classified by directory shape (binary-<arch>/by-hash/... is Packages, source/by-hash/... is Sources), and gzip is recognized by its magic bytes.

7.5 Proxy handler wiring

proxy.Handler (internal/proxy) implements forward-proxy semantics (absolute-URI request line, r.URL.Host set) rather than a httputil.ReverseProxy pointed at one host, since it must handle arbitrary origin hosts per request (§3). Concurrent requests for one key share one origin fetch, so a thundering herd (50 machines apt update-ing at once) costs one fetch. Because the fetch is shared, it runs under a context detached from the request that started it (context.WithoutCancel): one client giving up must not fail the others. The same holds for every other side’s shared fetches.

8. Fetch path (immutable, cache miss)

1. Client GETs http://deb.debian.org/debian/pool/main/a/apt/apt_2.6.1_amd64.deb.
2. Resolver.Resolve → key deb.debian.org/debian/pool/main/a/apt/apt_2.6.1_amd64.deb, PolicyImmutable.
3. Store.Stat misses.
4. One goroutine is elected to fetch and the others join it (a fill for pool/**, §8.1; a shared fetch for dists/**).
5. Index.ExpectedHash looked up for this path (from the last cached Packages for whichever suite covers it — resolved at parse time, not fetch time, since a pool path can appear in multiple suites’ indices; any one matching hash is sufficient).
6. Origin.Fetch streams the body. Response is simultaneously (a) hashed, (b) written to Store.Create’s writer, (c) made available to clients — see §8.1 for how, and why that is not a direct write to the one client that happened to ask first. A dists/** file is instead stored, parsed for the index and then served from the store: it is small, and this keeps a hash mismatch from leaving clients with partly streamed state.
7. On EOF: if hash matches expected (or no expected hash was known yet — logged as a gap, still cached), Commit with Immutable: true. If it mismatches, nothing is cached, and the clients already receiving the object finish receiving the origin’s bytes, unmodified — we only refuse to cache bad content, we don’t corrupt what the client gets beyond what the origin already sent. (The client’s own index may be newer than ours; apt does its own verification.) Clients that got nothing yet are served by a passthrough fetch: for dists/** one buffered fetch shared by all of them, each answered with its own Range; for a pool/** fill that could not be tailed, one streamed passthrough per client, since a buffer per client would cost hundreds of MB and leave each silent until the download ends.
8. The fill leaves the in-progress table only after Commit, so a request that finds no fill in progress and no cached object really is a miss.

Cache hit path: Store.Stat/Open/OpenRange directly, no origin round-trip, Range requests honored from the stored object.

8.1 Cache misses on pool/** stream; they never wait for the whole file

A client that sees nothing, not even headers, while a large package downloads gives up after its own timeout, and a fetch bound to that client would be cancelled with it, so a slow file could never be cached. A pool/** miss is therefore a fill (internal/fill, shared by every side that serves large blobs):

• It runs on its own goroutine under a context detached from any client (context.WithoutCancel), so it finishes, and caches, regardless of who leaves. The first retry is a hit.
• As soon as the origin’s response headers arrive, the proxy answers — 200 with the origin’s Content-Length/Last-Modified, flushed immediately — and then follows the download. Time to headers is the origin’s, not the download’s.
• Bytes are tailed from the ObjectWriter itself: a writer may implement io.ReaderAt over what it has written so far (fslocal reads its temp file, memstore its buffer), and readers wait on a progress signal. When Commit closes the temp file, a reader that has not finished switches to the committed object. A response that runs to the end of the object ends only once the fill has committed it (or abandoned it, below), so a client that has read a whole object finds it cached when it asks again.
• Every request for the key joins the same fill — the one that triggered it and any arriving later, including Range/If-Range resumes (answered 206 from the tail) and conditional requests — so concurrency costs one origin fetch.
• If the origin dies mid-body, tailing readers get an error, which cuts their connection short (they were promised more bytes than will come), and nothing is cached.
• On a hash mismatch the fill leaves the in-progress table, waits (bounded by -fill-drain-timeout; §18) for the requests that joined it to finish, and only then discards: slow clients still receive the origin’s bytes unmodified (§8 step 7). A request counts as joined from the moment it finds the fill in the table (under the table’s lock), not when it starts reading, which is what makes this race-free; a fill with nothing to tail releases its joiners at once. A request arriving after the drop starts a new fill.

A fill that cannot be tailed — origin response without Content-Length, or a Store whose writer is not an io.ReaderAt — waits for the fill, then serves what it committed. dists/** (including by-hash) keeps the fetch-then-serve path: those files are small, are parsed for the index, and revalidated.

A fill whose origin stops sending fails after -upstream-idle-timeout (§18). Bytes sent to a client before verification completes cannot be taken back — the same property as §8 step 7.