pkgproxySnap StoreSource
Contents · Stats

pkgproxy design: stats

The read-only stats API, and the default socket path both sides agree on. Part of the pkgproxy design, whose index says which file holds each § number.

13. Stats API

A read-only JSON REST API (internal/statsapi) reports what the cache is actually doing: one top-N-by-bytes-saved list of cache entries across every proxy, each tagged with its type — individually cached objects for apt, go modules, LXD and Docker (there are far too many distinct objects cached over a long-running instance to summarize any other way), and one rollup per snap name for snaps.

This section records the design and why it is shaped this way. The wire format, as clients see it, is documented in statsapi.md; how the handler, the windowed query and pkgproxyctl stats behave in detail, with code and test references, is in statsapi-internals.md.

13.1 A second listener, not a third gateway route

The API is served on its own listener, -stats-socket-path (always on; the default path is described in §15), rather than being multiplexed onto the main listener the way the proxy sides share one via internal/gateway. Two reasons:

  • internal/gateway’s dispatch relies on request shape (and fixed path prefixes) to tell the sides apart; a reporting route on the same listener would claim a shape that untrusted client traffic can reach.
  • A read-only reporting surface and a mutating admin surface have different risk profiles, and neither belongs next to untrusted apt/snapd traffic. The admin API is on a third socket of its own (§16.9), so this one stays read-only by construction.

13.2 Recording: hits, fetches, and joined requests

internal/stats.Recorder tracks two counters per cache key (per sha3-384 hash for snap blobs): Hits (served without this request contacting the origin) and Fetches (this request did contact the origin, whether that produced a fresh body or a 304 revalidation). A request that joins another’s in-flight fetch, fill or revalidation counts as a hit: the question is “did this request avoid a fetch”, not “was a fetch involved anywhere”. The request that started it counts as a fetch, however many joined it. singleflight’s own “shared” bit cannot tell the two apart, as it is true for the starter too once anyone joined, so internal/flight reports joined instead: whether this caller waited for another’s call.

Each hit or fetch records the object’s size, from the metadata already in hand, and how many of its bytes the response carries: httprange.BodyLength, worked out from the request and the very validators the response is served with, so it is known before the body is sent. That is the whole object for a GET, the range for a 206, and nothing for a HEAD, a 304 or a 416. A hit adds it to the key’s Saved, which is what bytes-saved reports: a HEAD, or a client’s revalidation answered with 304 from the cache, is a hit that saved nothing, and a client that downloads by Range (some fetch everything that way) saves what it got. A client that hangs up halfway still counts in full, as the bytes are counted before they are sent. Counting is deliberately not a store.Store concern (a hit counter on ObjectMeta): that would rewrite metadata on every hit and push a proxy-level concern into a backend-agnostic interface an S3 backend (§7.1) would also have to support. The handlers feed internal/stats explicitly.

Not every hit is a download saved. Content (apt’s pool/** and dists/**/by-hash/**, Go modules’ .zip and .mod, LXD’s image files, snap and Docker blobs) is fetched once and served unchanged, so a hit spares the client the download. Metadata (apt’s other dists/** files, Go modules’ list, @latest and .info, LXD’s *.json index and catalogs) is revalidated as it ages, and a client that has it mostly asks only whether it changed, so a hit spares little. Both are recorded alike and ranked together, and stats.IsContent(kind, key) tells them apart by key when a report is made: the checkpoint has nothing to say about it, so one written by any version is classified the same way. What the cache saved, on the cache page and in pkgproxyctl stats, adds up content only; each also shows the metadata served, apart and briefly.

13.3 Snap name/revision attribution: a JSON-walk side channel, not a cache key

Snap blobs are cached purely content-addressed (sha3-384/<hex>, §11.4), so nothing about the key or its metadata carries a snap’s name. The only place a name and a hash appear together is a relayed API response, which the rewriter already walks (§11.3); the walk also carries the nearest enclosing "name" and "revision" down to each download it reports. This is a permissive, best-effort heuristic, not a per-endpoint schema, consistent with the rewriter leaving unknown fields alone. A mis-attribution only affects reporting, never caching: a blob never associated with a name is still counted, under the empty "" name.

13.4 Persistence: periodic checkpoint, not per-request durability

Counts are kept in memory and written to <cache-dir>/stats.json every -stats-checkpoint-interval, as a plain JSON file: not a store.Store object (a single mutable snapshot has no content-hash or immutability semantics), and not written per request (a fleet-wide apt update can hit one pool file many times a second). A clean shutdown checkpoints once more; an unclean exit can lose up to one interval of counts, an accepted trade-off since the counts are not durability-critical. The file has a version. A file without one (version 0) has no Saved, and is read as every hit having sent the whole object: size times hits.

13.5 Endpoints

Every route of pkgproxy’s own API, here and on the admin socket (§16.9), is under /pkgproxy/v1/ (statsapi.Prefix, adminapi.Prefix). A bare /v1/ is what the proxied protocols use (the snap store serves /v1/, Docker /v2/), so the prefix names pkgproxy and keeps a route unambiguous wherever it is served. The bare /v1/... paths answer 404.

Read-only by construction: net/http.ServeMux’s method-prefixed patterns ("GET /path") return 405 for any method other than GET (and the HEAD a GET pattern also accepts), so there’s no separate check needed to keep this a read-only surface. The full request and response reference is statsapi.md; statsapi-internals.md §3–§5 has the parsing and edge cases.

  • GET /pkgproxy/v1/stats is one list of cache entries across every proxy, ranked by bytes saved, each with a type field so a client can render or filter by proxy without separate endpoints. Apt, go module, LXD and Docker entries are one per cached object (see §17.3 for the LXD key shape and §19.6.1 for Docker’s). Snap entries are one rollup per snap name, because snap blobs are cached by hash and a name spans many revisions (§13.3). ?since= windows it (§13.6). Each entry also says whether it is content or metadata (content, see below).
  • GET /pkgproxy/v1/info reports the daemon’s listen addresses, whether it serves the setup page, its build version (for pkgproxyctl version), and the offline state (§21), so that pkgproxyctl setup prints the ports the daemon actually uses (§18) and pkgproxyctl open finds the setup page (§20.5). It isn’t a stat, but it is read-only, and this socket is the one pkgproxyctl can reach without privileges; unlike a state file it can’t go stale. The offline state is here because the admin socket that controls it is not meant for everyone (gitlab.com/zygoon/pkgproxy#66).

13.6 Windowed reports: -r/?since= and the per-key bucket ladder

pkgproxyctl stats -r <since> and the API’s ?since= answer “what changed since this point in time” for any key, so that one workload’s cost (an apt-get install on freshly imaged machines) can be read without subtracting all-time totals by hand.

Two designs are rejected:

  • Periodic full-snapshot ring buffer (diff the closest checkpoint against the live counters): resolution is capped by the sampling interval, and the interval/retention trade-off is arbitrary.
  • Full per-event access trace ({time, key} per hit or fetch): exact, but it grows without bound and needs its own rotation and retention.

Instead each cache key (reports have always been per object, and a windowed one should be too) has a fixed-size, multi-resolution bucket ladder, the RRDtool/collectd pattern: recent activity at fine resolution, consolidated into coarser fixed-size tiers as it ages (the tiers are ladderTiers in internal/stats/ladder.go). Buckets are rolled lazily, on the next touch of the key, not on a ticker: each bucket stores its own start time, so a quiet key costs nothing. A query reads one tier, never mixing two (a coarser tier already holds what a finer one still keeps), plus the finer tiers’ open buckets, and a since older than the ladder reaches is clamped and the clamped time returned, never silently answered for a narrower window. statsapi-internals.md §4.3 has the query in detail.

Because a ladder is fixed size, it persists through §13.4’s checkpoint as more fields in the same JSON, with no new file, rotation or replay. As the checkpoint marshals a copy of the maps outside the lock, that copy must deep-copy each ladder, not share its slices with the live map.

15. A well-defined default stats socket path, for both sides

Both -stats-socket-path (the daemon) and -socket-path (pkgproxyctl stats) are optional and resolve through the same defaultStatsSocketPath() when omitted, so neither packaging nor an operator has to keep two paths in step by hand:

  • Inside a snap (SNAP_COMMON set): $SNAP_COMMON/stats.sock. That directory is the snap’s own and reachable by every app in it with no plug, whether the snap is this project’s or another one bundling pkgproxy, and a bundled pkgproxyctl sees the same SNAP_COMMON as its daemon.
  • Otherwise: /run/pkgproxy/stats.sock, the FHS/systemd RuntimeDirectory= convention: runtime state is ephemeral and separate from the persistent cache. The daemon creates /run/pkgproxy if needed; unprivileged, without it pre-created, startup fails with a clear permission error. It is a system-daemon default, and a local test run points -cache-dir and -stats-socket-path somewhere writable.