Upstream status

operations for administrators for CI operators

How pkgproxy records how each upstream server answers, from its clients' requests and optional checks, and shows it in a banner on every page, on the upstream status pages, on the setup page and in pkgproxyctl upstreams.

On the map: operations

▶ Watch this deck — 8 scenes, ~5 min

01 What this topic covers

ABOUT THIS TOPIC Upstream status how each server pkgproxy downloads from has answered 1 A history of every server 2 The banner on every page 3 The status pages 4 Reading a strip 5 Checks while clients are quiet 6 What pkgproxy answered itself
The topic title, Upstream status, and its six scenes in order: a history of every server, the banner on every page, the status pages, reading a strip, checks while clients are quiet, what pkgproxy answered itself

pkgproxy records how each upstream server answers, from its clients' requests and, when turned on, its own checks, and shows it on every page, on the status pages and in pkgproxyctl upstreams.

pkgproxy keeps a history of how each server it downloads from answers. First you see where that history comes from and where it is kept. Then the banner every page shows while something is wrong, and the status pages for each server. Then how to read a strip of that history, the checks pkgproxy can send a server nobody asks, and how much pkgproxy answered itself while a server was in trouble.

02 A history of every server

UPSTREAM STATUS pkgproxy snap store Go proxy Docker Hub apt mirrors and every other host clients notes how each answers the snap store, part by part: find info refresh auth … <cache-dir>/upstreams/ an hour and a day at a time · never thrown away nobody asked lately: no recent requests, not answering
Clients' requests pass through pkgproxy to the snap store, the Go proxy, Docker Hub and apt mirrors; pkgproxy records how each answers, the snap store part by part, into the upstreams directory of its cache, an hour and a day at a time

Recorded from clients' requests, per server and, for the snap store, per part. Kept under <cache-dir>/upstreams/ and never deleted. States: answering, answering slowly, failing, left alone, being tried again, pkgproxy offline, no recent requests, never asked.

pkgproxy keeps a history of how each server it downloads from answers: the snap store, the Go proxy, Docker Hub, the apt mirrors and every other host. It learns this from the requests its clients make, so nothing extra goes out. The snap store is kept part by part, such as search, details, refreshes and sign-in, as each part has a breaker of its own. The history is written under the upstreams directory of the cache, an hour and a day at a time, and is never thrown away. A server nobody asked lately shows as no recent requests, not as answering.

03 The banner on every page

THE BANNER any page pkgproxy serves: the setup page, the cache page, the status pages Degraded for 4h49m. Left alone: api.snapcraft.io catalog (http-429 from the store's edge). Other servers are asked as usual. amber: leaving a server alone, answering from the cache Offline for 12m. This machine has no network. Only what is cached is served. grey, not red: offline is often chosen $ pkgproxyctl stats Degraded for 4h49m. Left alone: api.snapcraft.io catalog (http-429 …
A page served by pkgproxy with a degraded banner at the top: degraded for 4h49m, the snap store's catalog left alone after http-429 from the store's edge; below it, the offline banner: this machine has no network; and pkgproxyctl stats opening with the same line

Degraded (amber) names up to three servers or parts left alone and the last failure's kind; offline (grey) says why. Nothing is shown while pkgproxy is online and every breaker is closed. Public visitors see no host names. pkgproxyctl stats opens with the same line.

While something is wrong, every page pkgproxy serves says so in a banner at the top. Amber means degraded: in store-overloaded mode, pkgproxy is leaving a server alone after it kept failing, and answers from its cache meanwhile. Grey means offline, chosen by hand or because the machine has no network; grey, not red, as offline is often chosen. The banner says for how long, and why, such as the store's catalogue being asked to slow down. pkgproxyctl stats starts with the same line.

04 The status pages

STATUS PAGES /pkgproxy/upstreams 24 hours · 30 days · a year api.snapcraft.io proxy.golang.org registry-1.docker.io archive.ubuntu.com …/api.snapcraft.io find · info · refresh assertions · catalog · auth who answered: edge or backend times left alone, 30 days catalog: 09:14 to 13:58 for this machine and trusted-networks alone, like the cache page the setup page, for everyone: last 30 days Snap Store Go modules Other servers no other server named
/pkgproxy/upstreams lists each server with a strip, over 24 hours, 30 days or a year; the snap store's own page lists its parts and when each was left alone; these pages are for this machine and trusted networks; the setup page shows everyone a 30-day summary naming no other server

/pkgproxy/upstreams and /pkgproxy/upstreams/<host>, for this machine and trusted-networks, linked from the bar of every page. The setup page shows anyone a 30-day summary; "Other servers" appears only once three or more were asked something.

The page /pkgproxy/upstreams shows each server now, and over twenty four hours, thirty days or a year. Each server links to a page of its own. The snap store's lists each of its parts, what their requests came to, who answered them, and the times each was left alone in the last thirty days. Like the cache page, these pages are for this machine and trusted networks alone. Everyone else sees, on the setup page, the last thirty days summed up for the snap store, Go modules, Docker Hub and other servers, with no other server named.

05 Reading a strip

A STRIP 24 hours: 96 cells of 15 minutes · 30 days: 6 hours · a year: a day taller is better: the share of requests answered no requests, drawn by why: left alone · pkgproxy offline · not running only checks answered · nothing asked 99.6% answered · data for 31% of the time 100% of three requests is not a quiet day of good answers: the second figure says so
A strip of cells: full-height cells where every request was answered, shorter ones where some failed, patterned cells where there were no requests; beside it, 99.6% answered and data for 31% of the time

All answered: full height, green; up to 5 % failed: 3/4; up to 50 %: 1/2; over 50 %: 1/4, red. Patterns mark cells with no requests. Figures: "99.6% answered · data for 31% of the time". Hover or tap a cell for its record.

Each server's history is drawn as a strip of cells: fifteen minutes each over a day, six hours over thirty days, a day over a year. A cell with requests is as tall as the share of them answered, so taller is better, and its colour says the same. A cell with no requests has a pattern that says why: left alone, pkgproxy offline or not running, only checks answered, or nothing asked. Beside the strip are two figures, shown together: the share of requests answered, and the share of the time with data. A hundred percent of three requests is not a quiet day of good answers.

06 Checks while clients are quiet

CHECKS nobody asked lately: no recent requests, which says nothing either way $ sudo snap set pkgproxy upstream-check-interval=30m checked GET /Go proxy · every interval GET /v2/Docker Hub · every interval store details, assertions30m store search1h store catalogueonce a day never checked store sign-in, refreshes, downloads apt mirrors, LXD, files, every other host no request known to change nothing off by default · at least 5m
With upstream-check-interval=30m, pkgproxy sends a quiet server one small request: the Go proxy and Docker Hub every interval, the snap store's details and assertions every 30 minutes, search every hour, catalogue once a day; the store's sign-in, refreshes and downloads and every other server never

upstream-check-interval (0, off; at least 5m). A check is counted apart from clients' requests, and also tries again a part left alone once its time is up. Never while pkgproxy is offline.

A server nobody uses shows no recent requests, which says nothing about whether it would answer. With the upstream-check-interval setting, such as thirty minutes, pkgproxy sends a quiet server one small request of its own. The Go proxy and Docker Hub are checked every interval, the snap store's details and assertions at most every thirty minutes, its search every hour and its catalogue once a day. The store's sign-in, refreshes and downloads, and every other server, are never checked: none has a request known to change nothing. The setting is off by default, and at least five minutes.

07 What pkgproxy answered itself

ANSWERED BY PKGPROXY a server in trouble; its clients notice less of it api.snapcraft.io Last 30 days: left alone or offline for 4h12m; pkgproxy answered 1284 requests itself $ pkgproxyctl upstreams Offline mode: store-overloaded. All times are UTC. SERVERSTATELEFT ALONEANSWEREDCHECKED api.snapcraft.io catalogleft alone, next try in 4m10s4h48m128424h api.snapcraft.io findanswering slowly--1h proxy.golang.orgnever asked--30m # columns shortened; also SINCE, RECENT and LAST FAILURE
Under a server's strip: last 30 days, left alone or offline for 4h12m, pkgproxy answered 1284 requests itself; pkgproxyctl upstreams prints a row per server and per part of the snap store with its state, time left alone, answers by pkgproxy and how often it is checked

"Last 30 days: left alone or offline for 4h12m; pkgproxy answered 1284 requests itself." pkgproxyctl upstreams (-json for the stats API's answer) has the columns LEFT ALONE/OFFLINE 24H, ANSWERED BY PKGPROXY 24H and CHECKED.

When a server has trouble, the point of a cache is that its clients notice less of it. So the pages say how much: for each server, how long it was left alone or pkgproxy was offline over the window shown, and how many requests pkgproxy answered itself meanwhile, from its cache or in the snap store's place. pkgproxyctl upstreams prints the same for the last twenty four hours, a row for each server and each part of the snap store, with its state, its recent answers and how often it is checked.

08 Where to go next

WHERE NEXT you can see how each server has answered, and how much pkgproxy answered for it you are here Upstream status Offline mode the breakers that leave a server alone The cache page the status on the live map Two tiers a pkgproxy behind another
You are here, Upstream status; next: Offline mode, The cache page, Two tiers

Next: Offline mode, for the breakers that leave a server alone; The cache page, for the status on the live map; Two tiers, for a pkgproxy behind another.

Instead of guessing why a download failed, you can now see how each server has answered, and how much pkgproxy answered for it. To see what leaves a server alone in the first place, watch Offline mode. To see the status on the live map of the cache, watch The cache page. To put a pkgproxy behind another one, watch Two tiers.

Read more

Where the status is shown

WhereWhoWhat
The banner at the top of every pageeveryone; host names only for this machine and trusted networksDegraded (amber) or offline (grey), for how long and why. Nothing while online with every breaker closed.
The status element in every page's bareveryone, at their level of detailOnline, Degraded or Offline, re-read every 30 seconds while the page is open.
/ (the setup page)everyoneHow the upstream servers have answered over the last 30 days: Snap Store, Go modules, Docker Hub and, once three or more were asked, Other servers. No other host is named.
/pkgproxy/upstreamsthis machine and trusted-networksEach server now and over 24 hours, 30 days or a year.
/pkgproxy/upstreams/<host>this machine and trusted-networksOne server; for the snap store, each part, who answered (its edge or a backend), and the times left alone in the last 30 days.
The cache page's live viewsthis machine and trusted-networksA dot on each upstream's mark, in the shape and colour of its worst part's state.
pkgproxyctl upstreamsthe stats socketThe same as a table, for the last 24 hours. -json prints the stats API's answer.
pkgproxyctl statsthe stats socketStarts with the banner's line.

The pages answer text/plain too, with the same tables.

States

WordMeaning
answeringThe last request was answered, within the last 15 minutes.
answering slowlyAnswering, but the median time to the response headers over 15 minutes is over the part's threshold: search 3 s, refresh and catalogue 2 s, the rest 1 s, a whole host 2 s.
failingThe last requests failed; the breaker is still closed.
left aloneThe breaker is open: requests are not sent ("next try in 4m10s").
being tried againThe breaker is half open.
pkgproxy offlineOffline mode is offline. The server may well be fine.
no recent requestsNothing was asked in the last 15 minutes. Not an outage.
never askedNothing was ever asked.

Breakers open only in offline-mode=store-overloaded; see Offline mode. Failures are named by kind: http-408, http-429, http-500, http-502, http-503, http-504, timeout, reset, eof, dns, connect, tls or other. On the pages each is underlined and says what it means on hover or keyboard focus.

Strips

CellDrawn as
All requests answeredgreen, full height
Up to 5 % failedlight amber, 3/4 height
Up to 50 % failedamber, 1/2 height
Over 50 % failedred, 1/4 height
Only pkgproxy's checks answeredgreen horizontal stripes
Left alonea red diagonal hatch
pkgproxy offlinea grey cross-hatch
pkgproxy not runninggrey dots
Nothing askeda faint grey block

A cell is 15 minutes over 24 hours, 6 hours over 30 days (a calendar of weeks), and a day over a year (a calendar of months). The figures beside a strip are the share of clients' requests answered, of those answered or failed, and the share of the time with data. Under each server, and in a host's table of parts: how long it was left alone or pkgproxy offline, and how many requests pkgproxy answered itself, from a stale copy in its cache or with a placeholder or recorded answer in the snap store's place.

for administrators for CI operators

Checks

SettingFlagDefaultMeaning
upstream-check-interval-upstream-check-interval0: offHow long a server nobody asked goes unasked before pkgproxy sends it a check of its own. At least 5m.
ServerCheckAt most every
snap store detailsGET /v2/snaps/info/hello30 min, or the interval if longer
snap store assertionsGET /v2/assertions/account/canonical30 min, or the interval if longer
snap store searchGET /v2/snaps/find?name=hello1 h, or the interval if longer
snap store catalogueGET /v2/snaps/categories1 day
the Go proxyGET /the interval
the Docker registryGET /v2/ (a 401 counts as an answer)the interval
store sign-in, refreshes, downloads; every other hostnone

A check is counted apart from clients' requests. A part left alone is not checked; one being tried again is. Nothing is checked while pkgproxy is offline. Each status page says under its heading what is checked, and how often.

for administrators for CI operators

pkgproxyctl upstreams

$ pkgproxyctl upstreams
Offline mode: store-overloaded. All times are UTC.
Checks: off; status comes from clients' requests only (upstream-check-interval).
SERVER                    STATE                          SINCE  RECENT             LEFT ALONE/OFFLINE 24H  ANSWERED BY PKGPROXY 24H  CHECKED  LAST FAILURE
api.snapcraft.io catalog  left alone, next try in 4m10s  09:14  0 of 1 answered    4h48m                   1284                      off      14:01 http-429 from the store's edge
api.snapcraft.io find     answering slowly               09:12  12 of 14 answered  -                       -                         off      -

RECENT counts the last 15 to 30 minutes. CHECKED is the check interval, off, or never for a part that is never checked. Options: -socket-path (the stats socket), -json.

for administrators for CI operators

The stats API

On the stats socket:

PathAnswer
GET /pkgproxy/v1/upstreamsEvery server and part now: state, breaker, last answer, request and failure, the last 15 minutes' counts and latencies, the last 24 hours' time left alone or offline and answers by pkgproxy, and, with checks on, check_interval_seconds and check_every_seconds.
GET /pkgproxy/v1/upstreams/history?host=&class=&step=&since=&until=The buckets of one series in which something happened.
GET /pkgproxy/v1/upstreams/events?since=&host=Breaker and mode changes, starts and stops.
GET /pkgproxy/v1/infoIts offline object has since and, while something is left alone, degraded_since.

for administrators for CI operators

On disk

<cache-dir>/upstreams/
  live.json                   what is open now, and the last 48 hours' 15-minute
                              buckets; rewritten at most every 5 minutes
  <year>/hours-<MM>.jsonl     the hours and events of a UTC month,
                              gzipped (.jsonl.gz) once the year is over
  <year>/days.jsonl           the days and events of a year

Closed buckets are appended and synced; nothing is deleted. A crash loses at most five minutes of the open buckets' counts. A live.json that cannot be read is moved aside, to live.json.bad; pkgproxy starts all the same.

Roads from here

CORE · the path every proxy engine shares

SETUP AND THE COMMAND LINE

OPERATIONS