Cache page code
contributors for contributors
How the cache page is built from a lazy scan of the cache directory (internal/cacheusage), the stats recorder's activity summed onto the map's blocks, and the access log's live events streamed by a hub that never blocks a request, to local visitors and trusted networks only.
On the map: inside pkgproxy
01 What this topic covers
How the cache page is built from a scan of the cache, the recorder's activity and the access log's live events, and who may see it.
The cache page shows an administrator what the cache holds and which requests are being served right now. If you have not seen A request in code, start there. You will see the three sources it is built from, and the scan that lays out its map once. Then you see how activity is summed onto blocks, which events the access log reports, and the hub that streams them without blocking a request. Last comes who may open the page.
02 What the page is built from
withSetupPage serves /pkgproxy/cache,
/pkgproxy/cache/activity and
/pkgproxy/cache/events to local and trusted visitors from
setupPageCache: a cacheusage.Monitor,
stats.Recorder.Activity and the liveHub.
The cache page is served by the setup page's handler in the daemon command, and only to a visitor on this machine or on a network it trusts. It is built from three sources. A scan of the cache directory, in the cacheusage package, measures what takes the disk. The statistics recorder's activity says how often and when each object was served. The request log's observers report each request as it happens. The page loads once with its layout, then polls the activity every ten seconds, every minute while hidden, and keeps a stream of live events open.
03 A scan, laid out once
cacheusage.Scan sums allocated blocks, as du
does; Monitor.Status walks when the last report is older than
DefaultMaxAge (1 min) and waits DefaultWait (1 s).
Report.Blocks is cut to BlockLimits, five times
the SVG treemap's, which is what a browser without JavaScript gets.
pkgproxy does not keep a running total of the cache; Scan measures it when the page asks. It walks the cache directory and sums the blocks each file takes on disk. The monitor walks lazily: a page view starts a walk when the last is over a minute old, and waits for it up to a second, or shows the older report. Classify puts each key in its kind, a group and a leaf, such as an archive and a source package. The report keeps the largest of each within a budget of blocks, and the daemon lays them out once, with the treemap package.
04 Activity summed onto blocks
stats.Recorder.Activity returns one Activity per
store key; newCacheActivity sums them by block, by the same
group and leaf IDs the layout gives, falling back to the group's rest,
then the kind's. activityTTL is 5 s.
The heights and colours come from the recorder. Activity copies each key's totals under the recorder's lock, which every request takes, and sums the series after releasing it: hits and bytes saved in the last hour, day, week and all time, the minute it was last served, and hits per minute, hour and day. A registry blob or a file counted under several names is merged into one. The server then sums each key onto the block the map draws it on, or onto a rest. The page polls this every ten seconds; it is computed at most every five, and shared.
05 Requests the access log observes
reqlog.Wrap(next, logger, observers...) calls each
Observer with StepBegin,
StepOutcome and StepEnd, from the goroutine
serving the request, so an observer must not block.
reqlog.KindTunnel names a tunnel, its key the host and port.
The live views come from the access log. reqlog.Wrap takes observers, and the cache page's hub is one, on both listeners. A side names the object a request is for with SetObject, its kind and store key, as soon as it knows the key; the files side adds the host with SetObjectFrom. That request begins. Each outcome its handler records is an event, and it ends with its final outcome, status and bytes, or with error if the handler panics. An open CONNECT tunnel begins as kind tunnel. A request without an object has no events.
06 A hub that never blocks a request
liveHub.send never waits: a full queue
(liveBuffer) counts the event as lost. Lookups take at most
2 s, 8 at a time, and are kept 10 minutes; past 1,024 clients the old
ones are forgotten. An idle stream says it is alive every 20 s.
Observers run on the goroutines that serve the requests, so the live hub only encodes an event and offers it to each watching page's queue of 512. A page that falls behind loses events, and is told how many once it has read what was queued before them. Nothing is encoded while nobody watches. At most four pages watch at once; another gets a stream that says busy, ends, and retries after thirty seconds. Client names come from reverse lookups in the background, and clients on a socket are Socket clients.
07 This machine and the networks it trusts
setupPageCache.mayView checks the listener,
forwardingHeaders, the unmapped remote address against
loopback, ownAddrs and -trusted-networks, and
the Host (namesMachine) against DNS rebinding.
The flag refuses 0.0.0.0/0 and ::/0; anyone
else is passed on to the gateway.
What the cache holds shows what the machine downloads, so the page, its activity and its events go to a local visitor, or to one from a network the -trusted-networks flag names. One check decides for both: a request over TCP, not through a socket listener, from a loopback, own or trusted address, without a header a proxy adds, such as Forwarded or Via, and for a Host that is an address, localhost, the host name or the public-url's host. The forward proxy cannot relay the page: the apt side's dialguard refuses this machine, and what it fetches carries Via. Anyone else gets the gateway's 404.
08 Where to go next general version
Next: The cache page, for what an administrator sees; A request in code, for where the events start; Store and keys, for the keys the blocks are sorted by.
You now know where each part of the cache page comes from: the map from a scan, its activity from the recorder, and its live events from the access log. To see the page as an administrator uses it, watch The cache page. To follow those events back to where a request is handled, watch A request in code. To see the keys the blocks are sorted by, watch Store and keys.
Read more
HTTP endpoints
All on the -addr listener, served by withSetupPage (only with -setup-page), for GET and HEAD, to a local visitor or one from -trusted-networks only. Anyone else's request goes on to the gateway and gets its 404. The handler name in the access log is setup-page.
| Path | Constant | Answer | Headers |
|---|---|---|---|
/pkgproxy/cache | cachePagePath | The cache page: HTML (cachepage.html.tmpl, with the map's JSON and cachepage.js) when Accept prefers it, otherwise the same figures as plain text through guide | Cache-Control: no-store, Vary: Accept, Accept-Language, Host, Content-Language |
/pkgproxy/cache/activity | cacheActivityPath | application/json: a cacheActivity, or null when the recorder has no activity | Cache-Control: no-store |
/pkgproxy/cache/events | cacheEventsPath | text/event-stream: one liveEvent per data: line; a HEAD gets the headers only; passed to the gateway when the hub is nil | Cache-Control: no-store |
/ | setupPagePath | The setup page; for a local or trusted visitor it also carries a cacheSummary and a link to the cache page | Cache-Control: no-cache, no-store for a local or trusted visitor |
Who may view
setupPageCache.mayView(r) is true only when all of these hold, for the page, its activity and its events alike:
- no forwarding header:
Forwarded,Via,X-Forwarded-For,X-Forwarded-Host,X-Forwarded-Proto,X-Forwarded-Port,X-Forwarded-Server,X-Real-Ip,True-Client-Ip,Cf-Connecting-Ip; - the local address is TCP, not a UNIX socket listener;
- the remote address, unmapped and without its zone, is loopback, one of this machine's interface addresses (
isOwnAddr) or in a trusted network (isTrustedAddr); Hostis an IP address,localhost, this machine's name or the host of-public-url(namesMachine), against DNS rebinding.
-trusted-networks (trustedNetworksFlag; snap option trusted-networks; empty by default) takes CIDR prefixes separated by commas, a single address as a host prefix, through parseNetworks, which also takes an IPv4-mapped network as the IPv4 network it maps. A prefix of length 0 (0.0.0.0/0, ::/0) is refused. When any is set, the daemon logs showing the cache page to visitors from trusted networks with networks once at start. newSetupPageCache gets the prefixes and -public-url's host.
cacheActivity: what the page polls
{"now": <Unix seconds>, "leaves": [<leafActivity>…]}. The series end with the minute, hour and UTC day now is in.
| JSON key | Field | Meaning |
|---|---|---|
g | Group | The group's ID, or the kind alone for what of a kind is not drawn as a group |
id | Leaf | The leaf's ID; absent for a group's or a kind's rest |
c | Content | Content rather than indexes or metadata |
l | Last | Last served, Unix seconds |
h | Hits | Hits in the last hour, day, week, all time |
s | Saved | Bytes saved in the same four windows |
mi, ho, d | Minutes, Hours, Days | Sparse series: pairs of an index and a count, over 60 minutes, 24 hours and 60 days |
IDs join a kind, then a label and a name per level, with \x1f (groupID, leafID): the same IDs the map's blocks carry, so the page finds a leaf's block however it was laid out.
The map's JSON
cacheMapData, embedded in the page: kinds (kind, name, size, r, rest), groups (k, id, name, full, size, r, rest), blocks (k, g, id, name, size, r, meta), text (translated strings), activity (the first cacheActivity), live (cacheLiveData: events, upstreams, logos, text; absent without the hub), size, free. r is a rectangle's corners in a ground of cacheMapWorld (1000) units, to a hundredth.
liveEvent: the event stream
t | Sent when | Other keys |
|---|---|---|
hi | the stream starts | at |
busy | already liveMaxWatchers pages watch; preceded by retry: 30000, and the stream ends | none |
b | reqlog.StepBegin | id, at, c (client address, or socket), n (name, once known), k (kind), g and l (block IDs) or h (a tunnel's host), u (upstream host) |
o | reqlog.StepOutcome | id, at, o |
e | reqlog.StepEnd | id, at, o (final), b (bytes) |
n | a client's reverse lookup found a name | c, n |
lost | the queue had been full and has just drained | lost: how many events were dropped |
An idle stream gets : keepalive every liveKeepalive. Nothing is encoded while no page watches.
Limits and intervals
| Name | Value | What it bounds |
|---|---|---|
liveBuffer | 512 | Events queued per watching page before they are lost |
liveMaxWatchers | 4 | Pages watching at once (a browser opens at most 6 connections to one host) |
liveBusyRetry | 30s | When a page told busy tries again |
liveKeepalive | 20s | Keepalive comment on an idle stream |
liveLookupTimeout | 2s | One reverse lookup of a client |
liveMaxLookups | 8 | Lookups at once |
liveNameTTL | 10m | How long a client's name is kept |
liveMaxClients | 1024 | Clients' names kept before the old ones are forgotten |
liveNamesTTL | 5s | How often the names of snaps, registry blobs and files are read again from the recorder |
activityTTL | 5s | How long one encoded activity answer is shared |
poll() in cachepage.js | 10s, 60s while the tab is hidden | How often the page fetches the activity |
cacheusage.DefaultMaxAge | 1m | Age of the last scan after which a page view starts another |
cacheusage.DefaultWait | 1s | How long a page view waits for a scan in progress |
MapBudget, MinKindBudget, MaxGroups, MinShareOf | 300, 32, 48, 8 | MapLimits, the SVG treemap a page without JavaScript shows: at most 800 rectangles |
BlockLimits | 5 × MapLimits | Report.Blocks, what the 3D map and the activity use |
internal/cacheusage
| Name | What it is |
|---|---|
Scan(ctx, dir, names) (*Report, error) | Walks the cache directory, no symlinks, no other file systems, summing allocated blocks as du does; each object is counted with its .meta.json file, which pkgproxy writes beside it for 0.11 |
Tally, NewTally, Tally.Add, Tally.Report | The summing behind Scan |
Report | Total; Kinds within MapLimits, Blocks within BlockLimits, largest first |
Kind, Group, Leaf, Rest | A kind of download, an archive, snap, module, image, repository or file name, its members, and what a level leaves out (Count, Size) |
Limits | Budget, MinKindBudget, MaxGroups |
Monitor, NewMonitor, Monitor.Status, Monitor.Last | The last report, rescanned in the background under the daemon's context; Status has Report, Err, Age, Measuring |
Classify(kind, key, names) (group, leaf Ident) | A store key to its group and leaf |
Ident, Label | A name, or a label standing in for one: LabelIndexes, LabelUnknownSnap, LabelSnapMetadata, LabelUnknownRepo, LabelRegistryMetadata, LabelOther, LabelUnknownFile |
Names | What the recorder knows of objects kept by hash: Snaps, DockerRepos, Files |
DiskSpace(dir) (Space, error) | Free (for unprivileged users) and total space of the file system |
AptDir … FilesDir, KindOther | The store subdirectories, and the kind of everything else under the cache directory |
| Kind | Group | Leaf |
|---|---|---|
| apt | archive, <host>/<prefix> | source package of a pool file; dists/** under LabelIndexes |
| snap | snap name | revision; everything but blobs under LabelSnapMetadata |
| gomod | module path | version (its .info, .mod and .zip) |
| lxd | <remote>/<image path> | serial |
| docker | repository the blob was served through | blob; manifests and tags under LabelRegistryMetadata |
| files | name it was served under | <algo>:<short hex> |
In the documentation
Roads from here
OPERATIONS
- The cache page — the page as an administrator sees it
- Statistics — the recorder the activity is read from
INSIDE PKGPROXY · for contributors
- A request in code — the reqlog.Wrap the events come from
- Store and keys — the store keys the blocks are sorted by
- The source tree — the rest of the source tree
- How pkgproxy is tested — how a change is tested