Statistics

operations for everyone

pkgproxy counts hits, fetches and bytes saved for every cached file, separates content from metadata, keeps the counts across restarts and serves them on a read-only stats socket to pkgproxyctl stats and your scripts.

On the map: operations

▶ Watch this deck — 9 scenes, ~5 min

01 What this topic covers

ABOUT THIS TOPIC Statistics how much downloading each cached file saved 1 hit or fetch 2 content, metadata 3 pkgproxyctl stats 4 the stats socket 5 restarts
About this topic: Statistics, for everyone; the last two scenes for administrators; its scenes: hit or fetch, content, metadata, pkgproxyctl stats, the stats socket, restarts

pkgproxy counts what every cached file saved and reports it through pkgproxyctl stats and a read-only socket.

Statistics are pkgproxy's count of how often each cached file was served and how much downloading that saved. If your clients do not use pkgproxy yet, watch pkgproxyctl setup first. You see hits and fetches, then content and metadata, then the report that pkgproxyctl stats prints. Administrators also see the stats socket the numbers come from, and how the counts outlive a restart.

02 A hit or a fetch, for every file

COUNTING deb.debian.org/debian/pool/main/h/hello/hello_2.10-3_amd64.deb client 1 goes upstream upstream fetch client 2 joins the fetch while it downloads hit client 3, later disk hit SIZEHITSFETCHESSAVED 52.5 KB21105.0 KB saved: what the hits sent a GET saves the whole file · a Range request the part it got · a HEAD or a 304 is a hit that saved nothing
Three requests for one package: the first fetches it from upstream, the second joins that fetch and the third is served from disk; both are hits; the entry counts two hits, one fetch and twice the size saved

Per cache key: hits, fetches, the size, and bytes-saved, what the hits sent. A joined request is a hit; the one that started the fetch is a fetch.

pkgproxy counts, for each cached file, how often a request was served without going upstream, and how often a request did go-upstream. The first request for a package is a fetch. A request that joins it while it downloads counts as a hit, since it avoided a download, and so does every later one served from disk. Each hit adds the bytes it sent to what the file saved: the whole file for a download, the part a range request asked for, and nothing for a HEAD or a not modified answer.

03 Content saved, metadata served

SAVED, OR ONLY SERVED content fetched once, served unchanged: a hit is a download saved apt: pool/… dists/…/by-hash/… go: …/@v/v0.35.0.zip ….mod lxd: image files snap, docker and files: all of it metadata rechecked as it ages: a hit saves little apt: dists/…/InRelease Packages.xz go: …/@v/list @latest ….info lxd: *.json indexes and catalogs Total bandwidth saved: 214.0 KB (snaps: 0 B, apt: 210.0 KB, go modules: 4.0 KB, …) Metadata served from the cache, not counted as saved: 3.0 KB (in parentheses below) apt InRelease 1.0 KB 3 2 (3.0 KB)
Two columns: content such as apt pool files and by-hash indexes, Go zip and mod files, LXD images, snaps, Docker blobs and files counts as saved; metadata such as apt's other dists files, Go list and info files and LXD json counts apart; the report shows both

Each entry has content: true or false, decided by its key when the report is made. The totals add only content.

Not every hit is a download saved. Content, such as a package from apt's pool, a Go module's zip, an LXD image or a snap, is fetched once and served unchanged, so a hit spares the client the whole download. Metadata, such as apt's Release and Packages files or a Go module's version list, is rechecked as it ages, and a client that already has it mostly asks only whether it changed. Both are counted and ranked together, but what the cache saved adds up content only; metadata is shown apart, in parentheses.

04 pkgproxyctl stats

PKGPROXYCTL STATS $ pkgproxyctl stats pkgproxy cache usage Total bandwidth saved: 214.0 KB (snaps: 0 B, apt: 210.0 KB, go modules: 4.0 KB, …) Metadata served from the cache, not counted as saved: 3.0 KB CACHE ENTRIES (top 20 by bytes saved) TYPEKEYSIZEHITSFETCHESSAVED apthello_2.10-3_amd64.deb52.5 KB41210.0 KB gomodgolang.org/x/mod/@v/v0.35.0.zip2.0 KB214.0 KB aptInRelease1.0 KB32(3.0 KB) TOTAL94214.0 KB after apt update: growing numbers mean apt uses the proxy $ pkgproxyctl stats -r 1h the last hour $ pkgproxyctl stats -json for scripts
The output of pkgproxyctl stats: the total saved split by kind, the metadata served, and a table of the top entries with type, key, size, hits, fetches and saved; below, the -r and -json options

pkgproxyctl stats, -r 1h for the last hour, -limit N (20 by default), -json for the entries as JSON.

pkgproxyctl stats prints the report: the total the cache saved, split by kind of download, then the entries that saved the most, twenty by default, with each one's size, hits, fetches and bytes saved. Run it after an apt update; if the numbers grow, apt is using the proxy. The -r flag limits the report to recent changes, such as the last hour, and the -json flag prints the raw entries for scripts.

05 A read-only API on its own socket for administrators

THE STATS API pkgproxy apt, snapd, go … :3142 scripts, pkgproxyctl stats.sock a UNIX socket, not on the network $ curl --unix-socket /run/pkgproxy/stats.sock 'http://localhost/pkgproxy/v1/stats?limit=10' [{"type":"apt","key":"deb.debian.org/debian/pool/…","content":true,"hits":1,…}] GET /pkgproxy/v1/stats ranked by bytes saved · ?limit= ?since= GET /pkgproxy/v1/info listen addresses · version · offline state POST /pkgproxy/v1/stats 405 Method Not Allowed read only by construction
Clients reach pkgproxy on port 3142; scripts and pkgproxyctl reach it on the stats socket; curl over the socket reads /pkgproxy/v1/stats and /pkgproxy/v1/info; other methods get 405

-stats-socket-path, by default $SNAP_COMMON/stats.sock in the snap and /run/pkgproxy/stats.sock otherwise. The reference is docs/statsapi.md.

The numbers come from a read only JSON API on a UNIX socket of its own, not on the port the clients use, so client traffic cannot reach it and nothing on the network can. Any HTTP client that speaks to a UNIX socket can read it. The stats endpoint lists the entries ranked by bytes saved, and takes a limit and a since time; the info endpoint reports the listen addresses, the version and the offline state. Any method other than GET or HEAD is answered with 405.

06 Counts kept across restarts for administrators

KEPT ACROSS RESTARTS counters in memory every 30s and on a clean shutdown <cache-dir>/stats.json a crash loses at most one interval of counts per entry: 60 buckets of a minute · the last hour 48 buckets of an hour · the last 2 days 60 buckets of a day · the last 60 days ?since= reads the finest tier that reaches back; older than 60 days is moved forward the ladder is saved in the same stats.json
Counters in memory are written to stats.json in the cache directory every 30 seconds and on a clean shutdown; each entry keeps 60 minute buckets, 48 hour buckets and 60 day buckets for windowed reports

-stats-checkpoint-interval (snap option stats-checkpoint-interval, default 30s) writes <cache-dir>/stats.json. A crash loses at most one interval.

Counts are kept in memory and written to stats.json in the cache directory every thirty seconds by default, and once more on a clean shutdown; a crash loses at most one interval of counts. For windowed reports, each entry keeps a small fixed ladder of buckets: sixty of a minute, forty eight of an hour and sixty of a day. A since time is answered from the finest tier that reaches back far enough, and one older than sixty days is moved forward, and the answer says so. The ladder is saved in the same file.

07 Where to go next for administrators in place of the general version

WHERE NEXT see how much each cached file saved you are here Statistics The cache page the same counts, as a live map The access log one line per request instead of totals Immutable or not why metadata is rechecked and content is not
Where next from Statistics: The cache page, the same counts, as a live map; The access log, one line per request instead of totals; Immutable or not, why metadata is rechecked and content is not

Next: The cache page, for the counts as a live map; The access log, for one line per request; Immutable or not, for why metadata is rechecked.

You can now see how much downloading pkgproxy saved, file by file, with pkgproxyctl stats. To see the same counts drawn as a live map of the cache, watch The cache page. For one line per request instead of totals, watch The access log. To see why metadata is rechecked and content is not, watch Immutable or not.

08 Where to go next for CI operators in place of the general version

WHERE NEXT see how much each cached file saved you are here Statistics pkgproxy for CI measure what one build cost The cache page a build's downloads, live The access log one line per request
Where next from Statistics: pkgproxy for CI, measure what one build cost; The cache page, a build's downloads, live; The access log, one line per request

Next: pkgproxy for CI, to measure one build; The cache page, to see a build's downloads live; The access log, for one line per request.

You can now see how much downloading pkgproxy saved, file by file, with pkgproxyctl stats. To measure what one build cost, watch pkgproxy for CI. To see a build's downloads live on a map of the cache, watch The cache page. For one line per request, watch The access log.

09 Where to go next general version

WHERE NEXT see how much each cached file saved you are here Statistics Fetch once why a joined request is a hit Immutable or not why metadata is rechecked and content is not pkgproxyctl the other commands
Where next from Statistics: Fetch once, why a joined request is a hit; Immutable or not, why metadata is rechecked and content is not; pkgproxyctl, the other commands

Next: Fetch once, for why a joined request is a hit; Immutable or not, for why metadata is rechecked; pkgproxyctl, for the other commands.

You can now see how much downloading pkgproxy saved, file by file, with pkgproxyctl stats. To see why a request that joins a download counts as a hit, watch Fetch once. To see why metadata is rechecked and content is not, watch Immutable or not. For the other commands, watch pkgproxyctl.

Read more

pkgproxyctl stats

OptionDefaultDoes
-limit N20show at most N entries, those that saved the most
-r WHENall timeonly changes since WHEN: a Unix timestamp, an RFC 3339 time, or how long ago, such as 30m or 2h
-jsonoffprint the entries as JSON instead of a report
-socket-path PATHthe stats socket belowwhere to ask

The stats socket

How pkgproxy runsSocket
the pkgproxy snap/var/snap/pkgproxy/common/stats.sock
bundled in another snap/var/snap/<snap>/common/stats.sock
on its own/run/pkgproxy/stats.sock

Moved with -stats-socket-path or PKGPROXY_STATS_SOCKET_PATH; it is not a snap option. Every local user can read it; nothing can write through it. Ask it with any HTTP client that speaks to a UNIX socket; the host name in the URL is ignored:

curl --unix-socket /run/pkgproxy/stats.sock 'http://localhost/pkgproxy/v1/stats?limit=10'
curl --unix-socket /run/pkgproxy/stats.sock "http://localhost/pkgproxy/v1/stats?since=$(date -d '1 hour ago' +%s)"
curl --unix-socket /run/pkgproxy/stats.sock http://localhost/pkgproxy/v1/info
PathMethodAnswers
/pkgproxy/v1/statsGET, HEADthe entries that saved the most, biggest first
/pkgproxy/v1/infoGET, HEADthe listen addresses, the version and the offline state
StatusWhen
200a JSON answer
400since is not a whole number
404any other path, a trailing / included
405any method but GET or HEAD

Errors are plain text.

/pkgproxy/v1/stats

ParameterDefaultMeaning
limit=N200at most N entries; anything but a positive whole number means the default
since=Tall timecount only from T, in Unix seconds; empty is the same as absent

Without since the answer is an array of entries; with it, an object {"since": …, "entries": […]} whose since is the start really used, in the daemon's time zone. An empty answer is [] or "entries":[], never null. Only entries hit at least once appear.

FieldMeaning
typeapt, snap, gomod, lxd, docker or files
keywhat was downloaded; see below
contenttrue for content, false for metadata
size-bytesits size
hitstimes served without going upstream, joined requests included
fetchestimes pkgproxy went upstream for it, to download or to recheck
bytes-savedwhat the hits sent: the whole file for a GET, the range asked for, nothing for a HEAD or a 304

Keys and content, by type

typekeyContentMetadata
aptarchive and path, such as deb.debian.org/debian/pool/main/h/hello/hello_2.10-3_amd64.debunder pool/, and dists/…/by-hash/…every other file under dists/
gomodmodule file, such as golang.org/x/text/@v/v0.3.0.zip.zip, .modlist, @latest, .info
lxd<remote>/<path>, such as pkgproxy-ubuntu/streams/v1/index.jsonimage filesanything ending in .json
docker<repository>@<digest>; a layer pulled through two repositories shows twiceevery entrynone counted
files<file name>@<algorithm>:<hex>, the name being the URL's last partevery entrynone counted
snapthe snap's name, one entry for all its revisions; "" for downloads not matched to a nameevery entrynone counted

For a snap, size-bytes is the revision served last, fetches counts every download of the snap, cached or not, and bytes-saved adds up all revisions. What the cache saved is the sum of bytes-saved over the entries whose content is true; the ranking puts both kinds together.

How far back since reaches

since withinCounts start at the beginning of its
the last hourminute
the last 2 dayshour
the last 60 daysday
further backmoved forward to 60 days ago

All-time counts are never trimmed.

/pkgproxy/v1/info

FieldMeaning
addrthe client address, as given to -addr (:3142)
lxd_tls_addrthe HTTPS address for LXD, as given to -lxd-tls-addr (:3143), or off
setup_pagewhether / serves the setup page; absent before 0.9.0, read as true
versionthe daemon's version, as pkgproxyctl version shows it
offline.modenever, store-overloaded or offline, read at the time of asking
offline.automaticwhile offline with the machine: no-network or airplane-mode
offline.configured_modewhile offline with the machine: the mode that applies again
offline.upstreamshosts being left alone, each with host, state (open or half-open) and retry_in_seconds

pkgproxyctl setup, open, stats and version read this endpoint to find the daemon's ports, its setup page, its offline mode and its build. See Offline mode.

for administrators for CI operators

Kept on disk

SettingFlagSnap optionDefault
how often counts are saved-stats-checkpoint-intervalstats-checkpoint-interval30s
where the socket is-stats-socket-pathflag or environment onlysee above

The counts and their history are saved to stats.json in the cache directory, /var/snap/pkgproxy/common/cache/stats.json in the snap, at every interval and on a clean shutdown. A crash loses what changed since the last save.

Roads from here

CORE · the path every side shares

SETUP AND THE COMMAND LINE

  • pkgproxyctl — the stats socket and the other commands

OPERATIONS