Statistics
operations for everyone
pkgproxy counts hits, fetches and bytes saved for every cached file, separates content from metadata, keeps the counts across restarts and serves them on a read-only stats socket to pkgproxyctl stats and your scripts.
On the map: operations
01 What this topic covers
pkgproxy counts what every cached file saved and reports it through pkgproxyctl stats and a read-only socket.
Statistics are pkgproxy's count of how often each cached file was served and how much downloading that saved. If your clients do not use pkgproxy yet, watch pkgproxyctl setup first. You see hits and fetches, then content and metadata, then the report that pkgproxyctl stats prints. Administrators also see the stats socket the numbers come from, and how the counts outlive a restart.
02 A hit or a fetch, for every file
Per cache key: hits, fetches, the size, and
bytes-saved, what the hits sent. A joined request is a hit;
the one that started the fetch is a fetch.
pkgproxy counts, for each cached file, how often a request was served without going upstream, and how often a request did go-upstream. The first request for a package is a fetch. A request that joins it while it downloads counts as a hit, since it avoided a download, and so does every later one served from disk. Each hit adds the bytes it sent to what the file saved: the whole file for a download, the part a range request asked for, and nothing for a HEAD or a not modified answer.
03 Content saved, metadata served
Each entry has content: true or false, decided by its key
when the report is made. The totals add only content.
Not every hit is a download saved. Content, such as a package from apt's pool, a Go module's zip, an LXD image or a snap, is fetched once and served unchanged, so a hit spares the client the whole download. Metadata, such as apt's Release and Packages files or a Go module's version list, is rechecked as it ages, and a client that already has it mostly asks only whether it changed. Both are counted and ranked together, but what the cache saved adds up content only; metadata is shown apart, in parentheses.
04 pkgproxyctl stats
pkgproxyctl stats, -r 1h for the last hour,
-limit N (20 by default), -json for the
entries as JSON.
pkgproxyctl stats prints the report: the total the cache saved, split by kind of download, then the entries that saved the most, twenty by default, with each one's size, hits, fetches and bytes saved. Run it after an apt update; if the numbers grow, apt is using the proxy. The -r flag limits the report to recent changes, such as the last hour, and the -json flag prints the raw entries for scripts.
05 A read-only API on its own socket for administrators
-stats-socket-path, by default
$SNAP_COMMON/stats.sock in the snap and
/run/pkgproxy/stats.sock otherwise. The reference is
docs/statsapi.md.
The numbers come from a read only JSON API on a UNIX socket of its own, not on the port the clients use, so client traffic cannot reach it and nothing on the network can. Any HTTP client that speaks to a UNIX socket can read it. The stats endpoint lists the entries ranked by bytes saved, and takes a limit and a since time; the info endpoint reports the listen addresses, the version and the offline state. Any method other than GET or HEAD is answered with 405.
06 Counts kept across restarts for administrators
-stats-checkpoint-interval (snap option
stats-checkpoint-interval, default 30s) writes
<cache-dir>/stats.json. A crash loses at most one
interval.
Counts are kept in memory and written to stats.json in the cache directory every thirty seconds by default, and once more on a clean shutdown; a crash loses at most one interval of counts. For windowed reports, each entry keeps a small fixed ladder of buckets: sixty of a minute, forty eight of an hour and sixty of a day. A since time is answered from the finest tier that reaches back far enough, and one older than sixty days is moved forward, and the answer says so. The ladder is saved in the same file.
07 Where to go next for administrators in place of the general version
Next: The cache page, for the counts as a live map; The access log, for one line per request; Immutable or not, for why metadata is rechecked.
You can now see how much downloading pkgproxy saved, file by file, with pkgproxyctl stats. To see the same counts drawn as a live map of the cache, watch The cache page. For one line per request instead of totals, watch The access log. To see why metadata is rechecked and content is not, watch Immutable or not.
08 Where to go next for CI operators in place of the general version
Next: pkgproxy for CI, to measure one build; The cache page, to see a build's downloads live; The access log, for one line per request.
You can now see how much downloading pkgproxy saved, file by file, with pkgproxyctl stats. To measure what one build cost, watch pkgproxy for CI. To see a build's downloads live on a map of the cache, watch The cache page. For one line per request, watch The access log.
09 Where to go next general version
Next: Fetch once, for why a joined request is a hit; Immutable or not, for why metadata is rechecked; pkgproxyctl, for the other commands.
You can now see how much downloading pkgproxy saved, file by file, with pkgproxyctl stats. To see why a request that joins a download counts as a hit, watch Fetch once. To see why metadata is rechecked and content is not, watch Immutable or not. For the other commands, watch pkgproxyctl.
Read more
pkgproxyctl stats
| Option | Default | Does |
|---|---|---|
-limit N | 20 | show at most N entries, those that saved the most |
-r WHEN | all time | only changes since WHEN: a Unix timestamp, an RFC 3339 time, or how long ago, such as 30m or 2h |
-json | off | print the entries as JSON instead of a report |
-socket-path PATH | the stats socket below | where to ask |
The stats socket
| How pkgproxy runs | Socket |
|---|---|
| the pkgproxy snap | /var/snap/pkgproxy/common/stats.sock |
| bundled in another snap | /var/snap/<snap>/common/stats.sock |
| on its own | /run/pkgproxy/stats.sock |
Moved with -stats-socket-path or PKGPROXY_STATS_SOCKET_PATH; it is not a snap option. Every local user can read it; nothing can write through it. Ask it with any HTTP client that speaks to a UNIX socket; the host name in the URL is ignored:
curl --unix-socket /run/pkgproxy/stats.sock 'http://localhost/pkgproxy/v1/stats?limit=10' curl --unix-socket /run/pkgproxy/stats.sock "http://localhost/pkgproxy/v1/stats?since=$(date -d '1 hour ago' +%s)" curl --unix-socket /run/pkgproxy/stats.sock http://localhost/pkgproxy/v1/info
| Path | Method | Answers |
|---|---|---|
/pkgproxy/v1/stats | GET, HEAD | the entries that saved the most, biggest first |
/pkgproxy/v1/info | GET, HEAD | the listen addresses, the version and the offline state |
| Status | When |
|---|---|
200 | a JSON answer |
400 | since is not a whole number |
404 | any other path, a trailing / included |
405 | any method but GET or HEAD |
Errors are plain text.
/pkgproxy/v1/stats
| Parameter | Default | Meaning |
|---|---|---|
limit=N | 200 | at most N entries; anything but a positive whole number means the default |
since=T | all time | count only from T, in Unix seconds; empty is the same as absent |
Without since the answer is an array of entries; with it, an object {"since": …, "entries": […]} whose since is the start really used, in the daemon's time zone. An empty answer is [] or "entries":[], never null. Only entries hit at least once appear.
| Field | Meaning |
|---|---|
type | apt, snap, gomod, lxd, docker or files |
key | what was downloaded; see below |
content | true for content, false for metadata |
size-bytes | its size |
hits | times served without going upstream, joined requests included |
fetches | times pkgproxy went upstream for it, to download or to recheck |
bytes-saved | what the hits sent: the whole file for a GET, the range asked for, nothing for a HEAD or a 304 |
Keys and content, by type
type | key | Content | Metadata |
|---|---|---|---|
apt | archive and path, such as deb.debian.org/debian/pool/main/h/hello/hello_2.10-3_amd64.deb | under pool/, and dists/…/by-hash/… | every other file under dists/ |
gomod | module file, such as golang.org/x/text/@v/v0.3.0.zip | .zip, .mod | list, @latest, .info |
lxd | <remote>/<path>, such as pkgproxy-ubuntu/streams/v1/index.json | image files | anything ending in .json |
docker | <repository>@<digest>; a layer pulled through two repositories shows twice | every entry | none counted |
files | <file name>@<algorithm>:<hex>, the name being the URL's last part | every entry | none counted |
snap | the snap's name, one entry for all its revisions; "" for downloads not matched to a name | every entry | none counted |
For a snap, size-bytes is the revision served last, fetches counts every download of the snap, cached or not, and bytes-saved adds up all revisions. What the cache saved is the sum of bytes-saved over the entries whose content is true; the ranking puts both kinds together.
How far back since reaches
since within | Counts start at the beginning of its |
|---|---|
| the last hour | minute |
| the last 2 days | hour |
| the last 60 days | day |
| further back | moved forward to 60 days ago |
All-time counts are never trimmed.
/pkgproxy/v1/info
| Field | Meaning |
|---|---|
addr | the client address, as given to -addr (:3142) |
lxd_tls_addr | the HTTPS address for LXD, as given to -lxd-tls-addr (:3143), or off |
setup_page | whether / serves the setup page; absent before 0.9.0, read as true |
version | the daemon's version, as pkgproxyctl version shows it |
offline.mode | never, store-overloaded or offline, read at the time of asking |
offline.automatic | while offline with the machine: no-network or airplane-mode |
offline.configured_mode | while offline with the machine: the mode that applies again |
offline.upstreams | hosts being left alone, each with host, state (open or half-open) and retry_in_seconds |
pkgproxyctl setup, open, stats and version read this endpoint to find the daemon's ports, its setup page, its offline mode and its build. See Offline mode.
for administrators for CI operators
Kept on disk
| Setting | Flag | Snap option | Default |
|---|---|---|---|
| how often counts are saved | -stats-checkpoint-interval | stats-checkpoint-interval | 30s |
| where the socket is | -stats-socket-path | flag or environment only | see above |
The counts and their history are saved to stats.json in the cache directory, /var/snap/pkgproxy/common/cache/stats.json in the snap, at every interval and on a clean shutdown. A crash loses what changed since the last save.
In the documentation
Roads from here
CORE · the path every side shares
- Fetch once — the shared fetch a joined request waits for
- Immutable or revalidated — why metadata is rechecked and content is not
SETUP AND THE COMMAND LINE
- pkgproxyctl — the stats socket and the other commands
OPERATIONS
- The cache page — the same counts drawn as a live map
- The access log — one log line per request instead of totals
- pkgproxy for CI — measuring what one build cost