Cache page code

contributors for contributors

How the cache page is built from a lazy scan of the cache directory (internal/cacheusage), the stats recorder's activity summed onto the map's blocks, and the access log's live events streamed by a hub that never blocks a request, to local visitors and trusted networks only.

On the map: inside pkgproxy

▶ Watch this deck — 8 scenes, ~6 min

01 What this topic covers

ABOUT THIS TOPIC Cache page code what the cache holds, and the requests being served right now 1 What the page is built from 2 A scan, laid out once 3 Activity summed onto blocks 4 Requests the access log observes 5 A hub that never blocks a request 6 This machine and the networks it trusts
About this topic, Cache page code: the sources, the scan, the activity, the events, the hub, this machine and the networks it trusts

How the cache page is built from a scan of the cache, the recorder's activity and the access log's live events, and who may see it.

The cache page shows an administrator what the cache holds and which requests are being served right now. If you have not seen A request in code, start there. You will see the three sources it is built from, and the scan that lays out its map once. Then you see how activity is summed onto blocks, which events the access log reports, and the hub that streams them without blocking a request. Last comes who may open the page.

02 What the page is built from

THREE SOURCES, THREE PATHS withSetupPage · local and trusted visitors only setupPageCache what the page fetches cacheusage.Monitor · Scan what takes the disk stats.Recorder.Activity how often and when each object was served reqlog observers → liveHub each request, as it happens GET /pkgproxy/cache the page and its layout, once GET /pkgproxy/cache/activity polled every 10 s, every minute while hidden GET /pkgproxy/cache/events Server-Sent Events, kept open
Three sources on the left, cacheusage.Monitor with its scan, stats.Recorder.Activity and the reqlog observers feeding liveHub, each leading to one path on the right: /pkgproxy/cache with the layout once, /pkgproxy/cache/activity polled every 10 seconds, /pkgproxy/cache/events as Server-Sent Events; local visitors only

withSetupPage serves /pkgproxy/cache, /pkgproxy/cache/activity and /pkgproxy/cache/events to local and trusted visitors from setupPageCache: a cacheusage.Monitor, stats.Recorder.Activity and the liveHub.

The cache page is served by the setup page's handler in the daemon command, and only to a visitor on this machine or on a network it trusts. It is built from three sources. A scan of the cache directory, in the cacheusage package, measures what takes the disk. The statistics recorder's activity says how often and when each object was served. The request log's observers report each request as it happens. The page loads once with its layout, then polls the activity every ten seconds, every minute while hidden, and keeps a stream of live events open.

03 A scan, laid out once

MEASURE, CLASSIFY, LAY OUT -cache-dir/ apt/objects/… snap/objects/… go/ docker/ lxd/ files/ Scan: allocated blocks of each object on disk, no symlinks followed Monitor.Status(ctx) last walk older than 1m: walk, wait up to 1s otherwise the older report; nothing walked while nobody looks Classify(kind, key, names) apt deb.debian.org/debian hello kind · group: the archive · leaf: the source package Report.Blocks (BlockLimits) → newCacheMapData → treemap.Squarify, once snap apt docker lxd go files
The cache directory with one store per kind is walked by Scan; Monitor.Status walks again when the last report is over a minute old and waits up to a second; Classify puts an apt key in kind apt, group deb.debian.org/debian and leaf hello; the report's blocks are laid out once as a treemap of kinds

cacheusage.Scan sums allocated blocks, as du does; Monitor.Status walks when the last report is older than DefaultMaxAge (1 min) and waits DefaultWait (1 s). Report.Blocks is cut to BlockLimits, five times the SVG treemap's, which is what a browser without JavaScript gets.

pkgproxy does not keep a running total of the cache; Scan measures it when the page asks. It walks the cache directory and sums the blocks each file takes on disk. The monitor walks lazily: a page view starts a walk when the last is over a minute old, and waits for it up to a second, or shows the older report. Classify puts each key in its kind, a group and a leaf, such as an archive and a source package. The report keeps the largest of each within a budget of blocks, and the daemon lays them out once, with the treemap package.

04 Activity summed onto blocks

ACTIVITY ONTO BLOCKS snapshotActivity: copy under the lock every request takes it; the series are summed after type Activity struct { Type, Key string Content bool Last time.Time Hits, Saved Windows Minutes []int64 // 60 Hours []int64 // 24 Days []int64 // 60 } Windows: Hour, Day, Week, All via library/debian via library/ubuntu blobs/sha256/ab12… newCacheActivity: sum by block hello its leaf 12 more group's rest rest kind's rest one entry per block at most GET /pkgproxy/cache/activity every 10 s · computed at most every 5 s, shared
The recorder's lock held only to copy totals; the Activity struct with hits and saved bytes per window and series per minute, hour and day; a registry blob served through two repositories merged into one key; keys summed onto a leaf's block, a group's rest or a kind's rest; the page polling /pkgproxy/cache/activity every 10 seconds

stats.Recorder.Activity returns one Activity per store key; newCacheActivity sums them by block, by the same group and leaf IDs the layout gives, falling back to the group's rest, then the kind's. activityTTL is 5 s.

The heights and colours come from the recorder. Activity copies each key's totals under the recorder's lock, which every request takes, and sums the series after releasing it: hits and bytes saved in the last hour, day, week and all time, the minute it was last served, and hits per minute, hour and day. A registry blob or a file counted under several names is merged into one. The server then sums each key onto the block the map draws it on, or onto a rest. The page polls this every ten seconds; it is computed at most every five, and shared.

05 Requests the access log observes

BEGIN, OUTCOME, END reqlog.Wrap(handler, logger, observers...) both listeners · liveHub.observe reqlog.SetObject(r, stats.TypeApt, resolved.Key) reqlog.SetObjectFrom(r, stats.TypeFiles, key, host) once the key is known StepBegin kind, key, origin, remote address StepOutcome each SetOutcome StepEnd final outcome, status, bytes a panic ends it with outcome error CONNECT, once its tunnel is open kind tunnel · key host:port no object: no events such as an API call relayed to the snap store
reqlog.Wrap with observers on both listeners; a side calls reqlog.SetObject, or SetObjectFrom for files; the request then begins, gets an event per outcome and ends with its final outcome, status and bytes, or error on a panic; an open CONNECT tunnel begins as kind tunnel; a request with no object has no events

reqlog.Wrap(next, logger, observers...) calls each Observer with StepBegin, StepOutcome and StepEnd, from the goroutine serving the request, so an observer must not block. reqlog.KindTunnel names a tunnel, its key the host and port.

The live views come from the access log. reqlog.Wrap takes observers, and the cache page's hub is one, on both listeners. A side names the object a request is for with SetObject, its kind and store key, as soon as it knows the key; the files side adds the host with SetObjectFrom. That request begins. Each outcome its handler records is an event, and it ends with its final outcome, status and bytes, or with error if the handler panics. An open CONNECT tunnel begins as kind tunnel. A request without an object has no events.

06 A hub that never blocks a request

THE LIVE HUB request request request liveHub encode, offer page · queue of 512 page · queue of 512 page · queue of 512 page · queue of 512 full: lost 37 a page that fell behind is told how many it lost nobody watching: nothing encoded 5th page: busy, retry: 30000 192.168.1.20reverse lookup in the background, at most 2 sbuilder-3 @a client over a UNIX socket listenerSocket clients
Request goroutines call liveHub.observe, which encodes each event and offers it to the queue of 512 of each of at most four watching pages; a page that falls behind is told how many events it lost; a fifth page gets busy and retries in 30 seconds; client names come from background reverse lookups, and socket clients share one name

liveHub.send never waits: a full queue (liveBuffer) counts the event as lost. Lookups take at most 2 s, 8 at a time, and are kept 10 minutes; past 1,024 clients the old ones are forgotten. An idle stream says it is alive every 20 s.

Observers run on the goroutines that serve the requests, so the live hub only encodes an event and offers it to each watching page's queue of 512. A page that falls behind loses events, and is told how many once it has read what was queued before them. Nothing is encoded while nobody watches. At most four pages watch at once; another gets a stream that says busy, ends, and retries after thirty seconds. Client names come from reverse lookups in the background, and clients on a socket are Socket clients.

07 This machine and the networks it trusts

LOCAL AND TRUSTED VISITORS ONLY browser on this machine setupPageCache.mayView(r) the cache page activity and events too a trusted network -trusted-networks over TCP, not a UNIX socket loopback, own or trusted address no Forwarded, Via, X-Forwarded-*… Host: an IP, localhost, host name or the host of -public-url the Host check stops DNS rebinding a client of apt's proxy GET http://127.0.0.1:3142/… 403: dialguard and it would carry Via: 1.1 pkgproxy anyone else: 404
A browser on this machine or on a trusted network passes setupPageCache.mayView, which requires TCP, a loopback, own or trusted address, no forwarding header and a Host that names this machine, and gets the cache page; anyone else gets the gateway's 404; a request relayed through pkgproxy's own forward proxy is refused by the dial guard with 403 and would carry Via

setupPageCache.mayView checks the listener, forwardingHeaders, the unmapped remote address against loopback, ownAddrs and -trusted-networks, and the Host (namesMachine) against DNS rebinding. The flag refuses 0.0.0.0/0 and ::/0; anyone else is passed on to the gateway.

What the cache holds shows what the machine downloads, so the page, its activity and its events go to a local visitor, or to one from a network the -trusted-networks flag names. One check decides for both: a request over TCP, not through a socket listener, from a loopback, own or trusted address, without a header a proxy adds, such as Forwarded or Via, and for a Host that is an address, localhost, the host name or the public-url's host. The forward proxy cannot relay the page: the apt side's dialguard refuses this machine, and what it fetches carries Via. Anyone else gets the gateway's 404.

08 Where to go next general version

WHERE NEXT the map comes from a scan, its activity from the recorder, and its live events from the access log you are here Cache page code The cache page what an administrator sees on it A request in code where the events start Store and keys the keys the blocks are sorted by
Where next from Cache page code: The cache page, A request in code, Store and keys

Next: The cache page, for what an administrator sees; A request in code, for where the events start; Store and keys, for the keys the blocks are sorted by.

You now know where each part of the cache page comes from: the map from a scan, its activity from the recorder, and its live events from the access log. To see the page as an administrator uses it, watch The cache page. To follow those events back to where a request is handled, watch A request in code. To see the keys the blocks are sorted by, watch Store and keys.

Read more

HTTP endpoints

All on the -addr listener, served by withSetupPage (only with -setup-page), for GET and HEAD, to a local visitor or one from -trusted-networks only. Anyone else's request goes on to the gateway and gets its 404. The handler name in the access log is setup-page.

PathConstantAnswerHeaders
/pkgproxy/cachecachePagePathThe cache page: HTML (cachepage.html.tmpl, with the map's JSON and cachepage.js) when Accept prefers it, otherwise the same figures as plain text through guideCache-Control: no-store, Vary: Accept, Accept-Language, Host, Content-Language
/pkgproxy/cache/activitycacheActivityPathapplication/json: a cacheActivity, or null when the recorder has no activityCache-Control: no-store
/pkgproxy/cache/eventscacheEventsPathtext/event-stream: one liveEvent per data: line; a HEAD gets the headers only; passed to the gateway when the hub is nilCache-Control: no-store
/setupPagePathThe setup page; for a local or trusted visitor it also carries a cacheSummary and a link to the cache pageCache-Control: no-cache, no-store for a local or trusted visitor

Who may view

setupPageCache.mayView(r) is true only when all of these hold, for the page, its activity and its events alike:

  • no forwarding header: Forwarded, Via, X-Forwarded-For, X-Forwarded-Host, X-Forwarded-Proto, X-Forwarded-Port, X-Forwarded-Server, X-Real-Ip, True-Client-Ip, Cf-Connecting-Ip;
  • the local address is TCP, not a UNIX socket listener;
  • the remote address, unmapped and without its zone, is loopback, one of this machine's interface addresses (isOwnAddr) or in a trusted network (isTrustedAddr);
  • Host is an IP address, localhost, this machine's name or the host of -public-url (namesMachine), against DNS rebinding.

-trusted-networks (trustedNetworksFlag; snap option trusted-networks; empty by default) takes CIDR prefixes separated by commas, a single address as a host prefix, through parseNetworks, which also takes an IPv4-mapped network as the IPv4 network it maps. A prefix of length 0 (0.0.0.0/0, ::/0) is refused. When any is set, the daemon logs showing the cache page to visitors from trusted networks with networks once at start. newSetupPageCache gets the prefixes and -public-url's host.

cacheActivity: what the page polls

{"now": <Unix seconds>, "leaves": [<leafActivity>…]}. The series end with the minute, hour and UTC day now is in.

JSON keyFieldMeaning
gGroupThe group's ID, or the kind alone for what of a kind is not drawn as a group
idLeafThe leaf's ID; absent for a group's or a kind's rest
cContentContent rather than indexes or metadata
lLastLast served, Unix seconds
hHitsHits in the last hour, day, week, all time
sSavedBytes saved in the same four windows
mi, ho, dMinutes, Hours, DaysSparse series: pairs of an index and a count, over 60 minutes, 24 hours and 60 days

IDs join a kind, then a label and a name per level, with \x1f (groupID, leafID): the same IDs the map's blocks carry, so the page finds a leaf's block however it was laid out.

The map's JSON

cacheMapData, embedded in the page: kinds (kind, name, size, r, rest), groups (k, id, name, full, size, r, rest), blocks (k, g, id, name, size, r, meta), text (translated strings), activity (the first cacheActivity), live (cacheLiveData: events, upstreams, logos, text; absent without the hub), size, free. r is a rectangle's corners in a ground of cacheMapWorld (1000) units, to a hundredth.

liveEvent: the event stream

tSent whenOther keys
hithe stream startsat
busyalready liveMaxWatchers pages watch; preceded by retry: 30000, and the stream endsnone
breqlog.StepBeginid, at, c (client address, or socket), n (name, once known), k (kind), g and l (block IDs) or h (a tunnel's host), u (upstream host)
oreqlog.StepOutcomeid, at, o
ereqlog.StepEndid, at, o (final), b (bytes)
na client's reverse lookup found a namec, n
lostthe queue had been full and has just drainedlost: how many events were dropped

An idle stream gets : keepalive every liveKeepalive. Nothing is encoded while no page watches.

Limits and intervals

NameValueWhat it bounds
liveBuffer512Events queued per watching page before they are lost
liveMaxWatchers4Pages watching at once (a browser opens at most 6 connections to one host)
liveBusyRetry30sWhen a page told busy tries again
liveKeepalive20sKeepalive comment on an idle stream
liveLookupTimeout2sOne reverse lookup of a client
liveMaxLookups8Lookups at once
liveNameTTL10mHow long a client's name is kept
liveMaxClients1024Clients' names kept before the old ones are forgotten
liveNamesTTL5sHow often the names of snaps, registry blobs and files are read again from the recorder
activityTTL5sHow long one encoded activity answer is shared
poll() in cachepage.js10s, 60s while the tab is hiddenHow often the page fetches the activity
cacheusage.DefaultMaxAge1mAge of the last scan after which a page view starts another
cacheusage.DefaultWait1sHow long a page view waits for a scan in progress
MapBudget, MinKindBudget, MaxGroups, MinShareOf300, 32, 48, 8MapLimits, the SVG treemap a page without JavaScript shows: at most 800 rectangles
BlockLimits5 × MapLimitsReport.Blocks, what the 3D map and the activity use

internal/cacheusage

NameWhat it is
Scan(ctx, dir, names) (*Report, error)Walks the cache directory, no symlinks, no other file systems, summing allocated blocks as du does; each object is counted with its .meta.json file, which pkgproxy writes beside it for 0.11
Tally, NewTally, Tally.Add, Tally.ReportThe summing behind Scan
ReportTotal; Kinds within MapLimits, Blocks within BlockLimits, largest first
Kind, Group, Leaf, RestA kind of download, an archive, snap, module, image, repository or file name, its members, and what a level leaves out (Count, Size)
LimitsBudget, MinKindBudget, MaxGroups
Monitor, NewMonitor, Monitor.Status, Monitor.LastThe last report, rescanned in the background under the daemon's context; Status has Report, Err, Age, Measuring
Classify(kind, key, names) (group, leaf Ident)A store key to its group and leaf
Ident, LabelA name, or a label standing in for one: LabelIndexes, LabelUnknownSnap, LabelSnapMetadata, LabelUnknownRepo, LabelRegistryMetadata, LabelOther, LabelUnknownFile
NamesWhat the recorder knows of objects kept by hash: Snaps, DockerRepos, Files
DiskSpace(dir) (Space, error)Free (for unprivileged users) and total space of the file system
AptDir … FilesDir, KindOtherThe store subdirectories, and the kind of everything else under the cache directory
KindGroupLeaf
aptarchive, <host>/<prefix>source package of a pool file; dists/** under LabelIndexes
snapsnap namerevision; everything but blobs under LabelSnapMetadata
gomodmodule pathversion (its .info, .mod and .zip)
lxd<remote>/<image path>serial
dockerrepository the blob was served throughblob; manifests and tags under LabelRegistryMetadata
filesname it was served under<algo>:<short hex>

Roads from here

OPERATIONS

INSIDE PKGPROXY · for contributors