The cache page
operations for administrators for CI operators
A page for the machine that runs pkgproxy and the networks it trusts that draws the cache as a 3D map of disk space and use, shows what is being downloaded right now on a wall dashboard and a byte flow, and lists every cached item.
On the map: operations
01 What this topic covers
The cache page draws what fills the cache, what is used and what is being downloaded now, for the machine that runs pkgproxy and the networks it trusts.
The cache page is a web page that shows what pkgproxy holds, what is used, and what it is downloading right now. If you have not seen Statistics, start there. You see who may open the page and how to show it to your network. Then come the map, where a block's ground is its size on disk and its height is how much it is used, the wall dashboard, and the byte flow with the details.
02 A page for the machine that runs it
http://localhost:3142/pkgproxy/cache, over TCP from this
machine, without Forwarded, Via,
X-Forwarded-* or X-Real-IP. pkgproxy's apt proxy
refuses to connect to this machine's loopback or its own ports, and what it
fetches carries Via, so it cannot relay the page.
The cache page shows what pkgproxy holds and what it is downloading right now. Open it in a browser on the machine that runs pkgproxy, at localhost, port 3142, or follow the line about the cache on the setup page there. What the cache holds is what your machines download, so by default only a local visitor sees it. A request from another machine, or one carrying the headers a proxy adds, gets a 404, as for any unknown path.
03 Showing it to your network
trusted-networks (-trusted-networks): CIDR
prefixes separated by commas, empty by default. A refused request gets
the 404 of any unknown path. Give visitors the machine's address or
public-url: a short host name may be answered by any machine
on the link (mDNS, LLMNR).
pkgproxy often runs on a machine without a screen. To watch the cache page from elsewhere, set the trusted-networks option to the address ranges of the networks you trust. A browser on those networks then sees the page, its live views and the cache line on the setup page, as one on the machine does. Some requests are still refused, from any network: those that came through a reverse proxy, over a UNIX socket, or for a name that is not the machine's. A range of every address is refused too, and pkgproxy logs the networks it trusts when it starts.
04 Ground is disk space
Ground is what the item takes on disk, as du counts it. A
kind's smallest items fold into one "N more items" block. The layout is
fixed once the page loads, so nothing moves.
The map draws every item in the cache as a block, and the ground a block takes is its size on disk. Blocks are laid out by kind, then by group: apt by archive and source package, snaps by name and revision, Go by module and version, LXD by image, Docker blobs by repository, and files fetched by checksum by their name. The sizes come from a walk of the cache directory, run only while someone looks, and at most once a minute.
05 Height is use
Hits in: All time, Last week, Last day or Last hour. Drag to turn and tilt, scroll to zoom, and click a block to zoom into its kind, then its group. Beside the map, a table of every item can be sorted and searched.
On that ground, a block stands as tall as how often it was served in the last hour, day, week or all time, on a log scale, so that a few popular items do not flatten the rest. Indexes and other metadata stay flat. The colour of its top fades with how long ago it was last served, and its sides show when within the window. The page asks for new counts every ten seconds, and a block served since then lights up.
06 The wall dashboard
Machines are named by their host names, looked up from their addresses; virtual machines and snaps on a socket are "Socket clients". Lines are as wide as the bytes they carried and fade over six seconds. "Turntable" spins the map slowly.
The page opens with the wall dashboard, made for a screen on a wall. The map stands in the middle, and each machine that asks for something is a sphere in front of it. A request is a line from the sphere down to its item's block. A hit comes back down that line. A fetch goes on up to its upstream across the top, such as Ubuntu, Debian, the Snap Store, Docker Hub, LXD or Go, and machines that join it merge into one stem. Tunnels go off to their hosts in the top corners.
07 The byte flow and the details
Views: Wall dashboard, Map, Byte flow and Details; #map,
#flow or #details opens one. At most four pages
watch live at once. Without JavaScript the page shows a flat treemap and
its tables.
The byte flow draws the same requests from left to right: the upstreams, the cache split by kind, and the machines, with a band to each as wide as what it got lately. A line swells as its bytes arrive. Pointing at a machine fades the others and shows the part of the cache it used. Every figure is in the details: what the cache holds, its free space, what it saved, what each machine got, and a table of every item, each row leading to its block on the map.
08 Where to go next general version
Next: Statistics, for the same counts from a script; The access log, for one request after the fact; pkgproxy for CI, for a build farm's screen.
You can now see at a glance what fills your cache, what is used and what is being downloaded right now. To read the same counts from a script, watch Statistics. To follow one request after the fact, watch The access log. To put the dashboard in front of a build farm, watch pkgproxy for CI.
Read more
Where it is
| Path, on the main port | What it answers |
|---|---|
/pkgproxy/cache | the cache page: HTML for a browser, the same figures as plain text otherwise |
/pkgproxy/cache/activity | what the page polls for its heights and hits |
/pkgproxy/cache/events | the stream of requests behind the wall dashboard and the byte flow |
/ | the setup page, with a line about the cache and a link to the page |
All of them are served only with setup-page on. Every answer with the cache's figures is sent with Cache-Control: no-store.
Who may open it
| Setting | Snap option | Flag | Default | Meaning |
|---|---|---|---|---|
| Trusted networks | trusted-networks | -trusted-networks | (empty) | Networks, in CIDR notation and separated by commas (a single address is one machine), whose visitors see the cache page, its live views and the setup page's cache line as a visitor on the machine does |
sudo snap set pkgproxy trusted-networks=192.168.0.0/24,fd00::/8
A request sees the page only when all of these hold; anyone else gets the 404 of any unknown path, and the setup page shows them no line about the cache:
- it comes from the machine itself, or from an address in a trusted network;
- it reached pkgproxy over TCP, not over a UNIX socket, as LXD containers and other snaps do;
- it carries none of the headers a reverse proxy adds:
Forwarded,Via,X-Forwarded-*,X-Real-IP,True-Client-IP,CF-Connecting-IP; - it names the machine in
Host: an IP address,localhost, the machine's host name or the host ofpublic-url. Another web site whose name someone points at the machine cannot read the page.
0.0.0.0/0 and ::/0 are refused, so that opening the page to everyone is never a typo; a wide network such as 0.0.0.0/1 is still accepted. pkgproxy logs the networks it trusts when it starts.
The machine's names count as much as its addresses. Set public-url only to a name you control. Its short host name is trusted too, and on a network where any machine can answer for a short name (mDNS or LLMNR, as Windows and macOS use), a visitor reaching pkgproxy by that name trusts those answers: give visitors its address or public-url instead.
In the documentation
Roads from here
FILES
- Files by checksum — files are a kind of their own
SETUP AND THE COMMAND LINE
- The setup page — the page that links to it
OPERATIONS
- Statistics — the numbers behind the page
- The access log — the same requests one line each
- pkgproxy for CI — a screen for a build farm
- Settings — the trusted-networks setting and the others