Upstream status
operations for administrators for CI operators
How pkgproxy records how each upstream server answers, from its clients' requests and optional checks, and shows it in a banner on every page, on the upstream status pages, on the setup page and in pkgproxyctl upstreams.
On the map: operations
01 What this topic covers
pkgproxy records how each upstream server answers, from its clients'
requests and, when turned on, its own checks, and shows it on every page,
on the status pages and in pkgproxyctl upstreams.
pkgproxy keeps a history of how each server it downloads from answers. First you see where that history comes from and where it is kept. Then the banner every page shows while something is wrong, and the status pages for each server. Then how to read a strip of that history, the checks pkgproxy can send a server nobody asks, and how much pkgproxy answered itself while a server was in trouble.
02 A history of every server
Recorded from clients' requests, per server and, for the snap store, per
part. Kept under <cache-dir>/upstreams/ and never
deleted. States: answering, answering slowly, failing, left alone, being
tried again, pkgproxy offline, no recent requests, never asked.
pkgproxy keeps a history of how each server it downloads from answers: the snap store, the Go proxy, Docker Hub, the apt mirrors and every other host. It learns this from the requests its clients make, so nothing extra goes out. The snap store is kept part by part, such as search, details, refreshes and sign-in, as each part has a breaker of its own. The history is written under the upstreams directory of the cache, an hour and a day at a time, and is never thrown away. A server nobody asked lately shows as no recent requests, not as answering.
03 The banner on every page
Degraded (amber) names up to three servers or parts left alone and the
last failure's kind; offline (grey) says why. Nothing is shown while
pkgproxy is online and every breaker is closed. Public visitors see no
host names. pkgproxyctl stats opens with the same line.
While something is wrong, every page pkgproxy serves says so in a banner at the top. Amber means degraded: in store-overloaded mode, pkgproxy is leaving a server alone after it kept failing, and answers from its cache meanwhile. Grey means offline, chosen by hand or because the machine has no network; grey, not red, as offline is often chosen. The banner says for how long, and why, such as the store's catalogue being asked to slow down. pkgproxyctl stats starts with the same line.
04 The status pages
/pkgproxy/upstreams and /pkgproxy/upstreams/<host>,
for this machine and trusted-networks, linked from the bar
of every page. The setup page shows anyone a 30-day summary; "Other
servers" appears only once three or more were asked something.
The page /pkgproxy/upstreams shows each server now, and over twenty four hours, thirty days or a year. Each server links to a page of its own. The snap store's lists each of its parts, what their requests came to, who answered them, and the times each was left alone in the last thirty days. Like the cache page, these pages are for this machine and trusted networks alone. Everyone else sees, on the setup page, the last thirty days summed up for the snap store, Go modules, Docker Hub and other servers, with no other server named.
05 Reading a strip
All answered: full height, green; up to 5 % failed: 3/4; up to 50 %: 1/2; over 50 %: 1/4, red. Patterns mark cells with no requests. Figures: "99.6% answered · data for 31% of the time". Hover or tap a cell for its record.
Each server's history is drawn as a strip of cells: fifteen minutes each over a day, six hours over thirty days, a day over a year. A cell with requests is as tall as the share of them answered, so taller is better, and its colour says the same. A cell with no requests has a pattern that says why: left alone, pkgproxy offline or not running, only checks answered, or nothing asked. Beside the strip are two figures, shown together: the share of requests answered, and the share of the time with data. A hundred percent of three requests is not a quiet day of good answers.
06 Checks while clients are quiet
upstream-check-interval (0, off; at least
5m). A check is counted apart from clients' requests, and
also tries again a part left alone once its time is up. Never while
pkgproxy is offline.
A server nobody uses shows no recent requests, which says nothing about whether it would answer. With the upstream-check-interval setting, such as thirty minutes, pkgproxy sends a quiet server one small request of its own. The Go proxy and Docker Hub are checked every interval, the snap store's details and assertions at most every thirty minutes, its search every hour and its catalogue once a day. The store's sign-in, refreshes and downloads, and every other server, are never checked: none has a request known to change nothing. The setting is off by default, and at least five minutes.
07 What pkgproxy answered itself
"Last 30 days: left alone or offline for 4h12m; pkgproxy answered 1284
requests itself." pkgproxyctl upstreams (-json
for the stats API's answer) has the columns LEFT ALONE/OFFLINE 24H,
ANSWERED BY PKGPROXY 24H and CHECKED.
When a server has trouble, the point of a cache is that its clients notice less of it. So the pages say how much: for each server, how long it was left alone or pkgproxy was offline over the window shown, and how many requests pkgproxy answered itself meanwhile, from its cache or in the snap store's place. pkgproxyctl upstreams prints the same for the last twenty four hours, a row for each server and each part of the snap store, with its state, its recent answers and how often it is checked.
08 Where to go next
Next: Offline mode, for the breakers that leave a server alone; The cache page, for the status on the live map; Two tiers, for a pkgproxy behind another.
Instead of guessing why a download failed, you can now see how each server has answered, and how much pkgproxy answered for it. To see what leaves a server alone in the first place, watch Offline mode. To see the status on the live map of the cache, watch The cache page. To put a pkgproxy behind another one, watch Two tiers.
Read more
Where the status is shown
| Where | Who | What |
|---|---|---|
| The banner at the top of every page | everyone; host names only for this machine and trusted networks | Degraded (amber) or offline (grey), for how long and why. Nothing while online with every breaker closed. |
| The status element in every page's bar | everyone, at their level of detail | Online, Degraded or Offline, re-read every 30 seconds while the page is open. |
/ (the setup page) | everyone | How the upstream servers have answered over the last 30 days: Snap Store, Go modules, Docker Hub and, once three or more were asked, Other servers. No other host is named. |
/pkgproxy/upstreams | this machine and trusted-networks | Each server now and over 24 hours, 30 days or a year. |
/pkgproxy/upstreams/<host> | this machine and trusted-networks | One server; for the snap store, each part, who answered (its edge or a backend), and the times left alone in the last 30 days. |
| The cache page's live views | this machine and trusted-networks | A dot on each upstream's mark, in the shape and colour of its worst part's state. |
pkgproxyctl upstreams | the stats socket | The same as a table, for the last 24 hours. -json prints the stats API's answer. |
pkgproxyctl stats | the stats socket | Starts with the banner's line. |
The pages answer text/plain too, with the same tables.
States
| Word | Meaning |
|---|---|
| answering | The last request was answered, within the last 15 minutes. |
| answering slowly | Answering, but the median time to the response headers over 15 minutes is over the part's threshold: search 3 s, refresh and catalogue 2 s, the rest 1 s, a whole host 2 s. |
| failing | The last requests failed; the breaker is still closed. |
| left alone | The breaker is open: requests are not sent ("next try in 4m10s"). |
| being tried again | The breaker is half open. |
| pkgproxy offline | Offline mode is offline. The server may well be fine. |
| no recent requests | Nothing was asked in the last 15 minutes. Not an outage. |
| never asked | Nothing was ever asked. |
Breakers open only in offline-mode=store-overloaded; see Offline mode. Failures are named by kind: http-408, http-429, http-500, http-502, http-503, http-504, timeout, reset, eof, dns, connect, tls or other. On the pages each is underlined and says what it means on hover or keyboard focus.
Strips
| Cell | Drawn as |
|---|---|
| All requests answered | green, full height |
| Up to 5 % failed | light amber, 3/4 height |
| Up to 50 % failed | amber, 1/2 height |
| Over 50 % failed | red, 1/4 height |
| Only pkgproxy's checks answered | green horizontal stripes |
| Left alone | a red diagonal hatch |
| pkgproxy offline | a grey cross-hatch |
| pkgproxy not running | grey dots |
| Nothing asked | a faint grey block |
A cell is 15 minutes over 24 hours, 6 hours over 30 days (a calendar of weeks), and a day over a year (a calendar of months). The figures beside a strip are the share of clients' requests answered, of those answered or failed, and the share of the time with data. Under each server, and in a host's table of parts: how long it was left alone or pkgproxy offline, and how many requests pkgproxy answered itself, from a stale copy in its cache or with a placeholder or recorded answer in the snap store's place.
for administrators for CI operators
Checks
| Setting | Flag | Default | Meaning |
|---|---|---|---|
upstream-check-interval | -upstream-check-interval | 0: off | How long a server nobody asked goes unasked before pkgproxy sends it a check of its own. At least 5m. |
| Server | Check | At most every |
|---|---|---|
| snap store details | GET /v2/snaps/info/hello | 30 min, or the interval if longer |
| snap store assertions | GET /v2/assertions/account/canonical | 30 min, or the interval if longer |
| snap store search | GET /v2/snaps/find?name=hello | 1 h, or the interval if longer |
| snap store catalogue | GET /v2/snaps/categories | 1 day |
| the Go proxy | GET / | the interval |
| the Docker registry | GET /v2/ (a 401 counts as an answer) | the interval |
| store sign-in, refreshes, downloads; every other host | none |
A check is counted apart from clients' requests. A part left alone is not checked; one being tried again is. Nothing is checked while pkgproxy is offline. Each status page says under its heading what is checked, and how often.
for administrators for CI operators
pkgproxyctl upstreams
$ pkgproxyctl upstreams Offline mode: store-overloaded. All times are UTC. Checks: off; status comes from clients' requests only (upstream-check-interval). SERVER STATE SINCE RECENT LEFT ALONE/OFFLINE 24H ANSWERED BY PKGPROXY 24H CHECKED LAST FAILURE api.snapcraft.io catalog left alone, next try in 4m10s 09:14 0 of 1 answered 4h48m 1284 off 14:01 http-429 from the store's edge api.snapcraft.io find answering slowly 09:12 12 of 14 answered - - off -
RECENT counts the last 15 to 30 minutes. CHECKED is the check interval, off, or never for a part that is never checked. Options: -socket-path (the stats socket), -json.
for administrators for CI operators
The stats API
On the stats socket:
| Path | Answer |
|---|---|
GET /pkgproxy/v1/upstreams | Every server and part now: state, breaker, last answer, request and failure, the last 15 minutes' counts and latencies, the last 24 hours' time left alone or offline and answers by pkgproxy, and, with checks on, check_interval_seconds and check_every_seconds. |
GET /pkgproxy/v1/upstreams/history?host=&class=&step=&since=&until= | The buckets of one series in which something happened. |
GET /pkgproxy/v1/upstreams/events?since=&host= | Breaker and mode changes, starts and stops. |
GET /pkgproxy/v1/info | Its offline object has since and, while something is left alone, degraded_since. |
for administrators for CI operators
On disk
<cache-dir>/upstreams/
live.json what is open now, and the last 48 hours' 15-minute
buckets; rewritten at most every 5 minutes
<year>/hours-<MM>.jsonl the hours and events of a UTC month,
gzipped (.jsonl.gz) once the year is over
<year>/days.jsonl the days and events of a year
Closed buckets are appended and synced; nothing is deleted. A crash loses at most five minutes of the open buckets' counts. A live.json that cannot be read is moved aside, to live.json.bad; pkgproxy starts all the same.
In the documentation
Roads from here
CORE · the path every proxy engine shares
- Offline mode — the breakers that leave a server alone
SETUP AND THE COMMAND LINE
- The setup page — the 30-day summary everyone sees
- pkgproxyctl — pkgproxyctl upstreams and stats
OPERATIONS
- The cache page — the live map with a status dot per server
- Settings — upstream-check-interval among the settings
- Two tiers — a pkgproxy behind another