pkgproxy design: upstream status
Whether each upstream answers now, and how stable it has been over time,
and how pkgproxy shows that on its pages, in pkgproxyctl and in the
stats API. Part of the pkgproxy design, whose index says
which file holds each § number. Being built for 0.12.2
(gitlab.com/zygoon/pkgproxy#169).
28. Upstream status
The breakers of store-overloaded
(§21) say which upstreams pkgproxy is
leaving alone right now, and nothing else: a closed breaker is an
upstream that answers or one that was never asked, and nothing is kept
once a breaker closes. During a partial outage of the snap store, the
questions are which part of it is down and since when; over a year, how
stable the store, the archives and the registries have been. Both were
answered from logs by hand (gitlab.com/zygoon/pkgproxy#153).
28.1 Where the status comes from
From what clients ask, and from checks (§28.8). offline.Switch sees
every upstream request where it leaves pkgproxy,
roundTripper.RoundTrip, names its breaker (host and endpoint class),
and sorts its outcome in observe. The recorder sits there, in every
mode, so every proxy engine is covered. Two things reach it from
elsewhere: the snap side reports an answer it synthesised in the store’s
place (gitlab.com/zygoon/pkgproxy#155), which never goes upstream, and
each side marks its requests’ context with its proxy engine (apt, snap,
Go, Docker, LXD, files), which the recorder keeps with the counts.
Tunnels are not recorded. Switch.Dial still refuses them while their
host’s breaker is open, but what a client tunnels to over CONNECT is
its own business, and a list of those hosts would be a record of where
it went (§28.10).
A series is a breaker: a host, or a host and an endpoint class of
the snap store, named as UpstreamState.Name names it
(api.snapcraft.io find). Hosts are an open set (PPAs, files by
checksum, LXD remotes), so at most 64 series are kept. The hosts pkgproxy
is configured with (the store, the Go proxy, the Docker registry, the
LXD remotes) always have a series, so that they can be shown as never
contacted. Any other host takes a free series, or the one of the
unconfigured host contacted least recently; that series’ history stays
in the files until retention (§28.7) drops it, and is shown again if
the host comes back. Eviction by recency is the first version, not a
promise: it keeps the series for hosts that are in use, which is what
the pages are for.
Status is never made up from nothing: a series no request has reached recently has no recent data, which is a state of its own, not a good one. By default pkgproxy does not contact upstreams by itself, as that loads an origin it is trying to spare, and would make “answering” mean “answers pkgproxy’s check” rather than “answers snapd”. The breaker’s half-open probe is a real client request, recorded as any other. Checks can be turned on (§28.8).
28.2 States
Each series is, now, in one of these states, shown as a word and an icon, never by colour alone:
- answering: the last request was answered, within the freshness window, 15 minutes;
- slow: answering, but the median latency of the last 15 minutes is
over the class’s threshold:
find3 s,refreshandcatalog2 s,info,assertions,authanddownload1 s, a host’s own breaker 2 s; - failing: the last requests failed, and the breaker is still closed;
- left alone: the breaker is open (“next attempt in 4m10s”), or half open (“being tried again now”), in the words of §21. With the two levels of gitlab.com/zygoon/pkgproxy#174 it also says which broke: the whole host (“api.snapcraft.io: not answering at all”) or one of its endpoint classes (“api.snapcraft.io catalog”);
- offline: the switch is offline; the page says so once, and the rows keep their last known facts;
- no recent data: no request within the freshness window (“not contacted since 3 h ago, last answered OK”), or never contacted.
A row also says since when the state holds, and the last failure: when,
its kind (§28.6), and for the store whether its edge or a backend sent
it (the x-view-name of §21).
28.3 Pages
/pkgproxy/upstreams, for local and trusted visitors only, as the cache page is (§20.7), and a 404 for anyone else. Rendered on the server, inline SVG, no JavaScript needed.- Overview: a row per host, troubled ones first, then by request volume, at most 12 and the rest folded. A host with classes has one row, in the state of its worst class with data, linking to its detail. A host left alone as a whole is drawn apart from a host with one class left alone (gitlab.com/zygoon/pkgproxy#174). Each row has a 24-hour strip, the availability and the coverage (§28.5).
- Snap store detail,
/pkgproxy/upstreams/api.snapcraft.io: a table of the classes (state, availability, coverage, p50 and p95 latency, last failure), and under it a colour strip for every class on one time axis, for 24 hours, 30 days or a year. Failures that line up across classes are the store down; failures in one row are one backend. The store’s CDN hosts, whichdownloadredirects to, each have a row of their own. Then the incidents of the last 30 days: a run of breaker openings with no answer in between is one incident (“catalog left alone 09:14 to 13:58, 31 attempts, 429 from edge”). - Plain text (
Accept: text/plain) gives the same tables.
- The setup page: every visitor sees an abbreviated history, a row
per proxy engine (apt, snap, Go, Docker, LXD, files) with the snap
store as its own row, each with 30 daily cells, the availability and
the state now, from the counts kept per engine (§28.6). No other host
is named. Local and trusted visitors also get a line (“2 of 9 left
alone”) linking to
/pkgproxy/upstreams. - The cache page: a link to
/pkgproxy/upstreamsin its top bar.
28.4 The mode banner
Every page pkgproxy serves (the setup page, the cache page, the upstream
status page) shows at its top whether anything is off, so that a glance
says so. Anything raises it: one CDN node or one PPA left alone turns it
amber too. It flags; the details, and whether they matter, are on
/pkgproxy/upstreams.
- nothing, while pkgproxy is online and every breaker is closed;
- degraded, amber:
store-overloadedwith a breaker not closed, naming up to three upstreams left alone, whether a whole host or one of its classes, and the last failure’s kind (“Degraded for 4 h 47 m. Leaving alone: api.snapcraft.io catalog (429 from the store’s edge). Everything else answers.”); - offline, grey: the mode
offline, set by hand or automatically, with the reason (“Offline for 12 m. This machine has no network.”). Grey, not red: offline is often chosen, and red is for failures in the strips.
The duration runs from when the current episode began: the switch going
offline, or the first breaker opening after all were closed. Public
visitors see the state, the duration and a reason without host names;
local and trusted visitors see the names and a link to the details. On
the setup page the banner takes the place of the notice of
§20.6.
pkgproxyctl stats and pkgproxyctl upstreams print the same line
first.
28.5 The strips
A strip is a row of buckets: 96 of 15 minutes for 24 hours, 120 of 6 hours for 30 days, 365 days for a year (a calendar of weeks on a wide screen, of months on a phone). Each bucket is one of three values:
- ok: requests were answered and none failed (green);
- failed: some failed, or the series was down (below) for part of it, shaded by the share: up to 5 % light amber, up to 50 % amber, over 50 % red;
- no data: no request, and not down (an empty dotted outline).
Patterns lie over them: a diagonal hatch where the breaker was open, a cross-hatch where pkgproxy was offline, dots where pkgproxy was stopped, and a green hatch where only a check answered.
Availability is a share of time, not of requests. Counting requests would leave out what an open breaker refused, so that a day-long outage would weigh only the few requests that opened the breaker and the probes after it. A series is down from the failure that opened its breaker until the next answer, to a client or to a check; between those, nothing says it came back. Availability is the share of covered time it was not down; coverage is the share of time with a request, a refusal or down time. Without checks, down time runs until the next client asks, which can be long after the upstream came back; with checks (§28.8) it ends within a check interval. The two are always shown together (“99.6 %, cov 31 %”), and so are refusals (“1 204 answered from the cache while left alone”).
Every strip is an <svg role="img"> with a summary title, and each cell
has a title with its counts; a table of the buckets with failures is
under it, in <details>. The colours are tokens of the shared style,
light and dark, with a contrast of 3:1 at least.
28.6 What is recorded
Each request is added into the open buckets of its series, never stored on its own:
- its outcome:
ok: any other answer, 404 and 304 included;failed: a transport error, a timeout, or a 408, 429, 500, 502, 503 or 504. 408 and 500 count here though not for the breaker, which ignores them as being about one request;refused: the switch did not let it out;synthesised: answered by pkgproxy in the store’s place, reported by the snap side;check: a check’s answer (§28.8); a failed check isfailed;cancelled: the client hung up, counted but left out of everything else;- a failure of a retry (
offline.WithRetry) counts as failed, though it does not move the breaker;
- its proxy engine, from the request’s context: the outcomes are counted for each engine that used the series, as one host can serve several (an Ubuntu archive through apt and through files);
- the error kind:
http-408,http-429,http-500,http-502,http-503,http-504,timeout,reset,eof,dns,connect,tlsorother, from the error chain; - the store’s view, the
x-view-nameof the answer, or none for its edge, for answers and failures alike; - the latency to the response headers, on the monotonic clock, into a histogram of ten buckets up to 0.1, 0.25, 0.5, 1, 2, 4, 8, 16, 32 s and more; p50 and p95 are read from it.
A bucket holds those counts: outcomes by engine, failures by kind, up to eight views with their answered and failed counts, the latency histogram, and the seconds the series was down, the breaker open or the switch offline in it.
Events are rare and kept as they come:
breaker: a breaker changing between closed, open and half open, with its level, how long, and the failure that opened it;mode: the mode in effect changing, and why (by hand, no network, airplane mode);daemon: pkgproxy starting and stopping cleanly. After a crash there is no stop event, and the time of the last checkpoint stands for it, so a gap reads “pkgproxy was stopped” within one checkpoint interval.
Each series also keeps, now: the last answer, the last failure (when, kind, status, view), the last request, and since when its state holds. The pages read the states from these, applying the freshness window when they read, so that nothing needs a timer.
The request path looks up the series and adds into its open buckets
under s.mu, which report already holds. A bucket is closed at its
boundary under the same lock, so no addition lands in a bucket that is
being written. Nothing there allocates or does I/O; events are appended
where the switch already logs them.
28.7 Buckets, retention and storage
Append, do not rewrite. What is closed is never written again: it is appended once, and later only dropped whole. What is rewritten is small and of a fixed size, whatever the uptime.
Tiers. Each coarser tier is made as the finer one closes, so a finer tier is dropped at the end of its retention without aggregating anything then:
- 15 minutes, for 48 hours;
- 1 hour (UTC), for 90 days, which the 30-day strip’s 6-hour cells are read from;
- 1 day, for 10 years;
- events, for as long as the days.
A day is closed at the machine’s local midnight, from the 15-minute buckets, which are still kept then: every time zone in use is a whole number of 15-minute steps from UTC, so a day in any of them is a whole number of buckets. A day records its UTC start and its length, 23 or 25 hours around a change of daylight saving time, so that it reads the same if the machine’s zone changes later. A visitor in another zone sees the machine’s days. Hours stay UTC, as no zone needs them otherwise.
With retention capped, what a series holds levels off at about 2 200 hours and 3 650 days, whatever pkgproxy’s uptime: storage reaches a constant size, and only what ages out of a tier is dropped. The figures are defaults, constants until someone needs them changed.
Files. Under <cache-dir>/upstreams/, next to the statistics and for
the same reasons
(§13.4):
live: only what is open, the current 15-minute bucket, hour and day of each series, each series’ now and the events not yet written, with aversion; written with the statistics’ checkpoint (every 30 s), to a temporary file that is renamed. It does not grow with history;quarters-<date>: the closed 15-minute buckets of that UTC day, appended and synced, removed once older than 48 hours;<year>/hours-<month>: the closed hours and the events of that UTC month, appended and synced, removed once older than 90 days, its events moved to the year’s;<year>/days: the closed days and the events of that year, appended and synced, removed once older than 10 years.
A crash loses at most the counts of one checkpoint interval, never a
closed bucket: closed buckets are appended and synced before live
stops holding them, and a bucket found in both is taken from the
appended file. A torn last record is skipped. A clock that jumps back
never reopens a closed bucket, and one that jumps forward closes those
in between with no data.
One interface, several formats. The recorder writes through an interface, not a format: load what is open, checkpoint it, append the closed buckets and events of a tier, drop a tier’s records older than a time, and read a series’ buckets of a tier between two times. It is implemented three times, in this order:
- JSON lines: a JSON object per record, which is what pkgproxy writes until the others exist. Easy to read and to fix by hand, and to test.
- Binary: a header with a magic number and a version, then fixed-size records, so that a record is found by its offset and a torn tail is a length that is not a multiple of the record size. Smaller than JSON, with nothing to parse on start.
- SQL, in the state database (§28.11).
Moving from one to the next is a one-time import, by key, which can be repeated after a crash; the old files are kept until it is done. A JSON line stays the format of an export.
Dumping. pkgproxyctl upstreams dump prints the records as JSON
lines whatever the format. Whether it asks the daemon or opens the files
itself is open. An API works from anywhere the stats socket reaches and
includes the open buckets, but is more API to keep. Opening the files
works with the daemon stopped and needs no API, but needs read access to
the cache directory and has to cope with a record being appended as it
reads.
28.8 Checks
-upstream-check-interval, and the snap option
upstream-check-interval, is 0, off, by default; otherwise a duration
of at least 5 minutes. When it is on, pkgproxy sends a check, a cheap
request of its own, to a series of a configured host that no client
request has reached within the longer of that interval and the series’
own (below). It also checks a series whose breaker is due to be tried
again, when no client request is about to be its probe, so that an
upstream that came back is seen to. It never checks while the mode
offline is in effect, chosen or automatic: nothing is to go out then.
A check is an upstream request like any other: an answer is a check,
a failure is failed and counts towards opening the breaker, and a
Retry-After holds off that series’ checks for as long as it asks.
One request per host says nothing of the classes that fail apart from each other (gitlab.com/zygoon/pkgproxy#153), so the snap store is checked per class, each with a request that changes nothing at the store and an interval of its own. The store throttles some sooner than others: asked for its categories more than once a day, it asks pkgproxy to slow down.
| series | check | at most every |
|---|---|---|
store info |
GET /v2/snaps/info/hello |
30 min |
store find |
GET /v2/snaps/find?name=hello |
1 h |
store assertions |
GET /v2/assertions/account/canonical |
30 min |
store catalog |
GET /v2/snaps/categories |
1 day |
store refresh, auth, download |
none | |
| the Go proxy | GET / |
30 min |
| a Docker registry | GET /v2/ (a 401 is an answer) |
30 min |
| an LXD remote | GET /streams/v1/index.json |
30 min |
refresh is a POST about a device, auth makes a nonce or a session at
the store, and the store counts a download as one: none of them has a
request that changes nothing, so they are not checked, and say “no
recent data” when no client asked. The table is a starting point, to be
changed as the store’s limits are learned.
28.9 API and pkgproxyctl
On the stats socket, read only, so that pkgproxyctl needs no
privileges:
GET /pkgproxy/v1/upstreams: the mode, the freshness window, and for each series its state (answering,slow,failing,left-alone,being-tried,offline,no-data,never), since when, its breaker (closed,open,half-open: closed is said, so that “not listed” no longer means two things) and, with gitlab.com/zygoon/pkgproxy#174, which level of it is open (hostorclass), the last answer, request and failure, and the last 15 minutes’ counts and latencies;GET /pkgproxy/v1/upstreams/history?host=&class=&step=&since=&until=: the buckets of one series in columns, with zeros for no requests; the down seconds are a column of their own, so that availability and coverage are read from the same answer;GET /pkgproxy/v1/upstreams/events?since=&host=: the events, and the incidents made of them;/pkgproxy/v1/info: itsofflineobject gainssince(when the mode in effect began) anddegraded_since(when the current run of breakers not closed began), for the banner.
The snap store’s series are its eight breakers: find, info,
refresh, assertions, catalog, auth, download and the host’s
own.
pkgproxyctl upstreams prints the overview, and -r chooses its
window; pkgproxyctl offline status points to it. docs/statsapi.md
documents the endpoints.
28.10 Privacy
No client address, path, query or header is kept but the host, the
class, the proxy engine and the store’s view name, which is cut to
[A-Za-z0-9._-] and 64 bytes. Nothing is kept of tunnels: not their
hosts, not their counts. The documentation of each client’s setup says
so, so that nobody expects otherwise.
Which hosts pkgproxy fetches from for a client (PPAs, registries, LXD remotes, files by URL) can be private too: the details are for local and trusted visitors and the stats socket, the public summary is by proxy engine, and the retention of §28.7 bounds how long a host is kept.
28.11 The SQL state database
With the state database of gitlab.com/zygoon/pkgproxy#106, the storage interface of §28.7 gets its third implementation: a table of series, one of buckets keyed by series, tier and start, one of failures by kind and view, and one of events. The checkpoint becomes a transaction, and dropping a tier’s old records a delete by start.
28.12 Several nodes
Not solved here, but not to be ruled out. Nodes that serve one site share their cache coherently: a blob one of them fetched, the others serve. Their upstream status needs far less: each node records its own series, appended to files of its own (named for the node), and the pages merge them as they read, minutes late at worst. No node rewrites another’s records, so the rule of §28.7, append and do not rewrite, holds across nodes too.
Checks must not multiply with the nodes, which would load the origin the design is trying to spare. With two tiers (gitlab.com/zygoon/pkgproxy#180) only the tier that talks to the origin checks it, and the others see it through that tier. Peers behind one address need one of them chosen to check a series at a time, by a lease or a fixed assignment; which is open.