Stats API internals

Developer notes for the read-only stats API. The client-facing reference, which is what a dashboard author reads, is statsapi.md; keep it in step with any change described here. The reasons behind the design (a separate listener, how hits are classified, snap name attribution, checkpointing, the bucket ladder and the alternatives it replaced) are in design/stats.md §13 and §15. This document covers how the code behaves today and where to find it.

1. Where the code lives

2. Listener and socket path

The daemon always serves the stats API; there is no way to turn it off. -stats-socket-path (or PKGPROXY_STATS_SOCKET_PATH, through applyEnvDefaults in cmd/pkgproxy/envflags.go) picks the path. When it is empty, RunFlag fills it from defaultStatsSocketPath():

• $SNAP_COMMON/stats.sock when SNAP_COMMON is set. For the pkgproxy snap that is /var/snap/pkgproxy/common/stats.sock; snap/snapcraft.yaml deliberately passes no -stats-socket-path. The flag is not one of the snap set keys in snap/local/bin/pkgproxy-env.
• /run/pkgproxy/stats.sock otherwise.

pkgproxyctl stats -socket-path resolves through the same function, so the two sides agree without configuration (design/stats.md §15 has the reasoning).

The listener comes from listenAt(ctx, socketPath, ""), shared with the main and admin listeners (design.md §12): it creates the parent directory (0755), removes a stale socket file, and clears the umask around net.Listen, so the socket is 0777 and any local user can connect. The http.Server around it sets only ReadHeaderTimeout (from -read-header-timeout) and a BaseContext; there is no authentication.

The mutating admin API is on a different socket (-admin-socket-path, internal/adminapi) so that this one stays read-only by construction; see the defaultAdminSocketPath doc comment in statssocket.go.

3. Routing and methods

statsapi.Handler holds a *stats.Recorder and an Info, and builds its http.ServeMux lazily, once (sync.Once), with two patterns:

GET /pkgproxy/v1/stats  -> serveStats   (statsapi.StatsPath)
GET /pkgproxy/v1/info   -> serveInfo    (statsapi.InfoPath)

Both paths are built from statsapi.Prefix, /pkgproxy/v1, which the admin socket’s routes share (adminapi.Prefix; a test keeps the two equal). The prefix names pkgproxy so that no route reads as one of the proxied protocols’ own: the snap store serves /v1/ and Docker /v2/. pkgproxyctl builds its requests from the same constants.

Consequences of using method-prefixed ServeMux patterns, none of which the handler code implements itself:

• A GET pattern also matches HEAD, so HEAD returns 200 with headers and no body.
• Any other method on a registered path gets 405 Method Not Allowed, with Allow: GET, HEAD and the standard body Method Not Allowed\n.
• Any other path, including /pkgproxy/v1/stats/, /pkgproxy/v1/stats/snaps and the bare /v1/stats and /v1/info, gets 404 page not found\n.

Successful responses go through writeJSON: Content-Type: application/json (httpheader.JSON), compact JSON from json.Encoder with a trailing newline. The encode error is discarded; by the time it could happen the status line is already written.

There is no 500 path. Recorder methods are safe on a nil receiver (they return nothing) and take their own lock, so a handler without a recorder serves empty reports.

4. GET /pkgproxy/v1/stats

4.1 Query parameters

limit: parsed with strconv.Atoi. Only a positive integer replaces defaultStatsLimit (200); anything else, including 0, a negative number or garbage, silently falls back to 200. There is no upper bound. (The report functions treat limit <= 0 as unbounded, but the API never passes one.)

since: parseSince reads it with strconv.ParseInt(v, 10, 64).

• Missing or empty: ok == false, the all-time report.
• Not a base-10 int64 (including out of range): 400 Bad Request, written by http.Error with the text invalid since="<value>": <strconv error>. The message is built with l10n.D.Errorf and has a Polish translation in pl.po, but Error() returns the English text and the handler has no request locale to translate with, so clients always see English.
• Otherwise time.Unix(sec, 0). Negative values and times in the future are accepted; the first is clamped (below), the second yields no entries.

4.2 Response shape

Without since: Recorder.TopByBytesSaved(limit), a bare []stats.Entry. With since: Recorder.TopSinceByBytesSaved(since, limit), wrapped in statsapi.SinceReport{Since, Entries}. In both cases a nil slice is replaced with an empty one so the JSON is [], never null.

SinceReport.Since is the time.Time actually used. time.Time’s JSON form is RFC 3339 with a fraction only when it is non-zero, in the time’s zone. time.Unix returns local time, so an unclamped value is whole seconds in the daemon’s local zone; a clamped value is computed from the recorder’s clock (time.Now() in the daemon) and carries nanoseconds, for example 2026-08-24T10:32:51.065531098+02:00.

stats.Entry fields and JSON keys:

Keys per type, as recorded by the handlers:

• apt: resolved.Key from internal/proxy, the normalized cache key with the archive host, e.g. deb.debian.org/debian/pool/main/h/hello/hello_2.10-3_amd64.deb. Index files under dists/** are recorded the same way as pool/** files; content is false for them unless they are under by-hash/ (cachekey.KeyPolicy is PolicyImmutable).
• gomod: the module proxy object key, e.g. golang.org/x/text/@v/v0.3.0.zip, for all five endpoints; content is true for .zip and .mod.
• lxd: <remote>/<path> (design/lxd.md §17.3), for image files and for the *.json index and catalogs, whose content is false.
• docker: statsKey in internal/registryproxy/blob.go, <repository>@<digest>; a layer shared by several images is counted under each repository (design/docker.md §19.6.1).
• snap: the name from Recorder.AssociateSnap, fed by the JSON walk in internal/snapproxy (design/stats.md §13.3).

Entries with zero hits in the reported period are dropped (appendEntries, appendSnapEntries and their Since variants). sortAndLimitByBytesSaved sorts by BytesSaved descending, then Type and Key ascending, so the order is stable, and then truncates.

4.3 Windowed reports

Each key’s counters carry a Trace *ladder, touched on every recorded serve. ladderTiers is

1-minute buckets x30 rows   -> the last 30m
1-hour buckets   x48 rows   -> the last 2d
1-day buckets    x60 rows   -> the last 60d

and TopSinceByBytesSaved answers a query in three steps:

1. Clamp. clampSince moves since forward to ladderHorizon(now) (now minus the coarsest tier’s span, 60 days) if it is older. The clamp is the same for every key. The clamped value is what is returned as SinceReport.Since. Only the windowed view loses precision: the all-time counters are never pruned.
2. Pick one tier. ladder.sum chooses the finest tier whose horizon (now - Resolution*Rows) is not after since, and sums only that tier’s rows and open bucket whose end is after since. It never mixes two tiers’ rows, since a coarser tier already holds cascaded totals of what the finer one still keeps.
3. Add the finer tiers’ open buckets. A bucket cascades into its parent only when it closes, so every finer tier’s open bucket is activity the chosen tier does not have yet. Leaving it out would drop exactly the most recent requests whenever since forces a coarser tier.

Buckets are aligned to their resolution (alignedStart) and a bucket counts when its end is after since, so the effective start is since rounded down to the chosen tier’s resolution. The returned Since is not rounded; it is the (possibly clamped) requested time.

For snaps the windowed rollup still picks SizeBytes by the global latest touch, not the latest touch inside the window.

How the ladder is rolled, persisted with the checkpoint, and cloned for Checkpoint() is in design/stats.md §13.6.

5. GET /pkgproxy/v1/info

statsapi.Info is {"addr": ..., "lxd_tls_addr": ..., "setup_page": ..., "version": ...}, filled once at startup by pkgproxyCmd.info(lxdTLS): Addr is -addr as given (not the resolved listener address), LXDTLSAddr is -lxd-tls-addr, or offSentinel ("off") when no LXD TLS server was started, SetupPage is -setup-page, and Version is pkgproxyVersion() (cmd/pkgproxy/version.go): the main module version the go command records in the binary, or (devel) and the VCS revision. Offline is the one field filled per request: serveInfo copies Info and sets it from Handler.Offline.State() (internal/offline), the mode and the breakers read under one lock, upstreams sorted by host and never null. A nil Handler.Offline leaves it out. The admin socket’s GET /pkgproxy/v1/offline builds its own adminapi.OfflineState from the same snapshot.

It has four consumers. pkgproxyctl version (cmd/pkgproxy/cmd_version.go) decodes only version, prints it next to its own, and says the daemon is another build only when both versions are known and differ; a missing field means a daemon from before 0.9.0. pkgproxyctl open (cmd/pkgproxy/cmd_open.go) decodes it into its own openInfo, whose SetupPage is a *bool so that a daemon from before the field (as during a snap refresh) counts as serving the page; it retries for up to 5 s while the socket is missing or refuses connections, and turns Addr into a loopback URL (design/setup-page.md §20.5). The third is applyDaemonInfo in cmd/pkgproxy/cmd_setup.go, used by the pkgproxyctl setup subcommands. It returns early when -addr (and -lxd-tls-addr, where the subcommand has it) were given explicitly, always dials defaultStatsSocketPath() (setup has no -socket-path), ignores every error so setup still works with no daemon running, and only replaces a value the daemon reported as non-empty. The third is pkgproxyctl stats, which reads it after /pkgproxy/v1/stats for its text report only (not -json), ignores any error, and prints the mode and the open upstreams when the mode is not never. Why this lives on the stats socket is in design/stats.md §13.5.

6. The pkgproxyctl client

6.1 Transport

ctlClient{socketPath} in cmd_stats.go is shared with the admin-API commands (cmd_cache.go’s doJSON, cmd_offline.go), which point it at the admin socket instead.

• httpClient() returns an http.Client whose Transport.DialContext ignores the network and address and always dials unix to socketPath. Request URLs use the placeholder host http://pkgproxy-stats.
• ctlDialTimeout (5s) is both the dialer timeout and the client’s overall timeout, so a wrong or stale socket fails fast.
• getJSON(ctx, path, v) issues the GET. A transport error becomes cannot reach pkgproxy's stats socket (<path>): <err>; any status but 200 becomes cannot use pkgproxy's stats socket: <path> answered HTTP <code> (the body is not read); otherwise the body is decoded into the shared stats.Entry or statsapi.SinceReport types, with no parallel client-side structs.

6.2 pkgproxyctl stats

Flags: -socket-path, -limit (default defaultCtlLimit, 20, smaller than the API’s 200 because it sizes a terminal table), -json, and -r.

-r goes through parseSinceFlag(raw, now), which tries, in order: Unix seconds (strconv.ParseInt), RFC 3339 (time.RFC3339), and a Go duration meaning “ago” (time.ParseDuration, so 30m, 2h, but not 1d). Failure is invalid -r "<raw>": not a Unix timestamp, RFC3339 timestamp, or duration, before any request is made. The result is sent as ?limit=<n>&since=<unix-seconds>; without -r only ?limit=<n> is sent.

-json re-encodes what it received with two-space indentation: a flat array, or {"since": ..., "entries": [...]} with -r.

Otherwise renderReport prints:

• the title, and with -r a “Showing changes since” line in local time. If SinceReport.Since is more than one second after the requested time (the daemon clamped it), it adds “(history only reaches back this far — your requested <ts> was older)”;
• the total bytes saved, broken down by type, adding up only the entries whose Content is true, and, when the entries include metadata, a line with the metadata served, not counted in that total;
• one table (TYPE, KEY, SIZE, HITS, FETCHES, SAVED) with a TOTAL row, or noHitsYetMsg when there are no entries. A metadata entry’s SAVED is in parentheses, and the TOTAL row’s SAVED leaves it out, as the total above does; its HITS and FETCHES count every entry.

An entry from a daemon before 0.9.0 has no content field; stats.Entry.UnmarshalJSON fills it in with stats.IsContent, so the totals come out as a newer daemon’s would.

entryKeyLabel shortens KEY for display only: apt shows path.Base of the key, docker shortens @sha256:<hex> to 12 hex digits (shortDigestKey), an empty snap name shows as (unknown), and other types are unchanged. formatBytes uses 1024-based units labelled KB, MB, and so on.

7. Tests

internal/statsapi/statsapi_test.go:

• TestServeStats_ReturnsJSONArrayAcrossAllTypes: one array, every type.
• TestServeStats_EmptyIsJSONArrayNotNull: [], not null.
• TestServeStats_DefaultAndExplicitLimit: 200 by default, ?limit=.
• TestServeStats_SinceReturnsWindowedReport: the SinceReport envelope.
• TestServeStats_InvalidSinceIs400.
• TestNonGETMethodsAreRejected: 405 for other methods.
• TestUnknownPathIs404.
• TestServeInfo.

internal/stats/ladder_test.go covers the ladder, including TestLadder_CoarserTierIncludesUncascadedFinerOpenBuckets (step 3 above), TestLadder_SinceOlderThanHorizonIsClamped, TestLadder_KeyWithNoActivityInWindowIsOmitted and TestCheckpoint_ConcurrentWithRecordDoesNotRaceOnLadder (run under -race). internal/stats/stats_test.go covers the all-time report: order, limit, snap rollups, the "" bucket, nil-receiver safety and checkpoints.

cmd/pkgproxy/cmd_stats_test.go covers the client: parseSinceFlag, renderReport (including the clamp note and the empty placeholder), ctlClient.getJSON (round trip, non-200, missing socket), and the pkgproxyctl stats command end to end against a fake daemon on a socket (statsDaemon), with -json, -r and a bad -r.

8. Changing the API

The API has no formal compatibility promise (see the Stability section of statsapi.md), but scripts do read it. When changing it:

• Prefer adding fields, types or endpoints over renaming or removing them. Clients are told to ignore what they don’t know.
• Update statsapi.md, this document, and design/stats.md §13 when the design changes, in the same change.
• Add a NEWS entry for anything a client can notice.
• Keep pkgproxyctl stats decoding the shared stats and statsapi types rather than its own copies, so the two cannot drift.