The cache page

operations for administrators for CI operators

A page for the machine that runs pkgproxy and the networks it trusts that draws the cache as a 3D map of disk space and use, shows what is being downloaded right now on a wall dashboard and a byte flow, and lists every cached item.

On the map: operations

▶ Watch this deck — 8 scenes, ~5 min

01 What this topic covers

ABOUT THIS TOPIC The cache page what pkgproxy holds and what it is downloading right now 1 this machine 2 your network 3 ground: disk 4 height: use 5 wall dashboard 6 byte flow, details
About this topic: The cache page, for administrators and CI operators; its scenes: this machine, your network, ground: disk, height: use, wall dashboard, byte flow, details

The cache page draws what fills the cache, what is used and what is being downloaded now, for the machine that runs pkgproxy and the networks it trusts.

The cache page is a web page that shows what pkgproxy holds, what is used, and what it is downloading right now. If you have not seen Statistics, start there. You see who may open the page and how to show it to your network. Then come the map, where a block's ground is its size on disk and its height is how much it is used, the wall dashboard, and the byte flow with the details.

02 A page for the machine that runs it

THIS MACHINE BY DEFAULT pkgproxy the cache page what is cached · what moves now a browser on this machine localhost:3142/pkgproxy/cache the setup page, here: size · free space · See what is cached. what the cache holds is what your machines download: by default, local visitors only another machine through a proxy Via · Forwarded · X-Real-IP 404, as for any unknown path
By default, a browser on the machine that runs pkgproxy opens localhost:3142/pkgproxy/cache and gets the cache page, also linked from the setup page there; a request from another machine, or one carrying proxy headers such as Via or Forwarded, gets a 404

http://localhost:3142/pkgproxy/cache, over TCP from this machine, without Forwarded, Via, X-Forwarded-* or X-Real-IP. pkgproxy's apt proxy refuses to connect to this machine's loopback or its own ports, and what it fetches carries Via, so it cannot relay the page.

The cache page shows what pkgproxy holds and what it is downloading right now. Open it in a browser on the machine that runs pkgproxy, at localhost, port 3142, or follow the line about the cache on the setup page there. What the cache holds is what your machines download, so by default only a local visitor sees it. A request from another machine, or one carrying the headers a proxy adds, gets a 404, as for any unknown path.

03 Showing it to your network

TRUSTED NETWORKS pkgproxy no screen $ sudo snap set pkgproxy trusted-networks=192.168.0.0/24 a browser on 192.168.0.0/24 the cache page its live views the setup page's cache line through a reverse proxy over a UNIX socket for a name not the machine's 404, from any network as for any unknown path 0.0.0.0/0 · ::/0 refused the networks it trusts are logged at start
sudo snap set pkgproxy trusted-networks=192.168.0.0/24; a browser on that network then gets the cache page, its live views and the setup page's cache line from a headless pkgproxy; requests through a reverse proxy, over a UNIX socket or for another name still get a 404; 0.0.0.0/0 and ::/0 are refused and the networks are logged at start

trusted-networks (-trusted-networks): CIDR prefixes separated by commas, empty by default. A refused request gets the 404 of any unknown path. Give visitors the machine's address or public-url: a short host name may be answered by any machine on the link (mDNS, LLMNR).

pkgproxy often runs on a machine without a screen. To watch the cache page from elsewhere, set the trusted-networks option to the address ranges of the networks you trust. A browser on those networks then sees the page, its live views and the cache line on the setup page, as one on the machine does. Some requests are still refused, from any network: those that came through a reverse proxy, over a UNIX socket, or for a name that is not the machine's. A range of every address is refused too, and pkgproxy logs the networks it trusts when it starts.

04 Ground is disk space

GROUND IS DISK SPACE snap apt docker lxd go ground = size on disk, seen from above trixie bookworm by kind, then by group apt · archive, source package snap · name, revision go · module, version lxd · image docker · repository files · the name it was served under a walk of the cache directory: only while someone looks, at most once a minute
The cache seen from above: one area per kind, snap, apt, docker, lxd, go and files, each as large as it is on disk, apt split into archives; beside it how each kind is grouped; below, sizes come from a walk of the cache directory while someone looks, at most once a minute

Ground is what the item takes on disk, as du counts it. A kind's smallest items fold into one "N more items" block. The layout is fixed once the page loads, so nothing moves.

The map draws every item in the cache as a block, and the ground a block takes is its size on disk. Blocks are laid out by kind, then by group: apt by archive and source package, snaps by name and revision, Go by module and version, LXD by image, Docker blobs by repository, and files fetched by checksum by their name. The sizes come from a walk of the cache directory, run only while someone looks, and at most once a minute.

05 Height is use

HEIGHT IS USE height: how often it was served in the window hits in All time Last week Last day Last hour on a log scale: a few popular items do not flatten the rest indexes: flat top: when last served hour · day · week · earlier · never sides: bottom is the window's start, top is now every 10 s: new counts · a block served since lights up
Blocks seen from the side: one tall, one middling, one short and an index lying flat; a choice of window, all time, last week, last day or last hour; tops faded by when last served; a block's side shaded by when within the window; the page polls every 10 seconds

Hits in: All time, Last week, Last day or Last hour. Drag to turn and tilt, scroll to zoom, and click a block to zoom into its kind, then its group. Beside the map, a table of every item can be sorted and searched.

On that ground, a block stands as tall as how often it was served in the last hour, day, week or all time, on a log scale, so that a few popular items do not flatten the rest. Indexes and other metadata stay flat. The colour of its top fades with how long ago it was last served, and its sides show when within the window. The page asks for new counts every ten seconds, and a block served since then lights up.

06 The wall dashboard

WALL DASHBOARD build-1 build-2 laptop Socket clients hit Ubuntu Debian Snap Store Docker Hub LXD Go fetch joined: one stem tunnel host
The wall dashboard: the upstreams across the top, the map of blocks in the middle, machines as spheres in front; one request goes from a sphere to a block, a hit comes back down, two requests join into one fetch that goes up to Debian, and a tunnel goes off to a host in the top corner

Machines are named by their host names, looked up from their addresses; virtual machines and snaps on a socket are "Socket clients". Lines are as wide as the bytes they carried and fade over six seconds. "Turntable" spins the map slowly.

The page opens with the wall dashboard, made for a screen on a wall. The map stands in the middle, and each machine that asks for something is a sphere in front of it. A request is a line from the sphere down to its item's block. A hit comes back down that line. A fetch goes on up to its upstream across the top, such as Ubuntu, Debian, the Snap Store, Docker Hub, LXD or Go, and machines that join it merge into one stem. Tunnels go off to their hosts in the top corners.

07 The byte flow and the details

BYTE FLOW AND DETAILS Debian Snap Store Go apt snap go build-1 laptop pointing at a machine shows only its part Details holds · free space · saved, in all and today · from the cache · machines now · each machine's share ITEMSIZESERVEDSAVED core24 · 105574 MB12888 MB trixie · linux-signed-amd64105 MB3315 MB a row leads to its block on the map
The byte flow: upstreams on the left, the cache by kind in the middle, machines on the right, bands as wide as what each got; below, the details: the page's figures and a table of every item with its size, hits served and bytes saved

Views: Wall dashboard, Map, Byte flow and Details; #map, #flow or #details opens one. At most four pages watch live at once. Without JavaScript the page shows a flat treemap and its tables.

The byte flow draws the same requests from left to right: the upstreams, the cache split by kind, and the machines, with a band to each as wide as what it got lately. A line swells as its bytes arrive. Pointing at a machine fades the others and shows the part of the cache it used. Every figure is in the details: what the cache holds, its free space, what it saved, what each machine got, and a table of every item, each row leading to its block on the map.

08 Where to go next general version

WHERE NEXT what fills the cache and what it downloads now you are here The cache page Statistics the same counts, from a script The access log one line per request, after the fact pkgproxy for CI the dashboard in front of a build farm
Where next from The cache page: Statistics, the same counts, from a script; The access log, one line per request, after the fact; pkgproxy for CI, the dashboard in front of a build farm

Next: Statistics, for the same counts from a script; The access log, for one request after the fact; pkgproxy for CI, for a build farm's screen.

You can now see at a glance what fills your cache, what is used and what is being downloaded right now. To read the same counts from a script, watch Statistics. To follow one request after the fact, watch The access log. To put the dashboard in front of a build farm, watch pkgproxy for CI.

Read more

Where it is

Path, on the main portWhat it answers
/pkgproxy/cachethe cache page: HTML for a browser, the same figures as plain text otherwise
/pkgproxy/cache/activitywhat the page polls for its heights and hits
/pkgproxy/cache/eventsthe stream of requests behind the wall dashboard and the byte flow
/the setup page, with a line about the cache and a link to the page

All of them are served only with setup-page on. Every answer with the cache's figures is sent with Cache-Control: no-store.

Who may open it

SettingSnap optionFlagDefaultMeaning
Trusted networkstrusted-networks-trusted-networks(empty)Networks, in CIDR notation and separated by commas (a single address is one machine), whose visitors see the cache page, its live views and the setup page's cache line as a visitor on the machine does

sudo snap set pkgproxy trusted-networks=192.168.0.0/24,fd00::/8

A request sees the page only when all of these hold; anyone else gets the 404 of any unknown path, and the setup page shows them no line about the cache:

  • it comes from the machine itself, or from an address in a trusted network;
  • it reached pkgproxy over TCP, not over a UNIX socket, as LXD containers and other snaps do;
  • it carries none of the headers a reverse proxy adds: Forwarded, Via, X-Forwarded-*, X-Real-IP, True-Client-IP, CF-Connecting-IP;
  • it names the machine in Host: an IP address, localhost, the machine's host name or the host of public-url. Another web site whose name someone points at the machine cannot read the page.

0.0.0.0/0 and ::/0 are refused, so that opening the page to everyone is never a typo; a wide network such as 0.0.0.0/1 is still accepted. pkgproxy logs the networks it trusts when it starts.

The machine's names count as much as its addresses. Set public-url only to a name you control. Its short host name is trusted too, and on a network where any machine can answer for a short name (mDNS or LLMNR, as Windows and macOS use), a visitor reaching pkgproxy by that name trusts those answers: give visitors its address or public-url instead.

Roads from here

FILES

SETUP AND THE COMMAND LINE

OPERATIONS