pkgproxy design: offline mode
Offline mode: serving what is cached, with nothing sent upstream. Part of the pkgproxy design, whose index says which file holds each § number.
21. Offline mode
A switch (internal/offline.Switch) makes pkgproxy really offline: what is
cached keeps being served, nothing is ever revalidated, and nothing at all
goes upstream, whatever a client asks for. It is meant for a machine that
has lost its network, and for a build or test that must see exactly what
was cached.
The guarantee is made at two places. Upstream traffic leaves pkgproxy
only through cmd.upstreamTransport(), the http.RoundTripper every side
fetches with, and through proxy.Handler.Dial, the dialer of a CONNECT
tunnel. While the switch is on, Switch.Transport and Switch.Dial refuse
with offline.ErrOffline before any DNS lookup or connection. The offline
test harness (internal/gateway/offline_test.go) replaces
http.DefaultTransport with one that fails the test on any use, so a code
path that reaches upstream around those two (a nil client defaulting to
http.DefaultClient, say) is caught when it is added, not just refused.
Freshness is frozen. Every side’s ttl() returns offline.Frozen while
the switch is on, so a cached entry is never old enough to revalidate, and
apt’s Cache-Control: max-age=0 (which apt update sends) no longer forces
a revalidation. Immutable entries were never revalidated. Entries are
served as hits, not as “stale after a failed revalidation”, so the journal
is not filled with one warning per request.
What the cache cannot answer fails fast. offline.Middleware, around
the gateway, turns the 502 every side gives for a failed upstream fetch
into a 503 with a plain-text “pkgproxy is offline” body: the upstream is
not at fault, nothing was attempted, and the client may try again once the
switch is off. It has no Retry-After (on any side, the snap side’s 424
too): asking again cannot help until the mode changes, and apt 3 honours one
on a 503 by waiting up to 30 s, three times, at every miss; only
store-overloaded sends one. A CONNECT is refused the same way. Every
502 is rewritten, not only one from a refused fetch: while the switch is
on nothing is fetched, so a side’s 502 can only mean that the upstream
could not be reached. The body is English like every HTTP error body,
because HTTP clients carry no locale
(§22); it
is not translated.
The snap side answers differently: snapd retries every 5xx for about half a
minute (its defaultRetryStrategy) at each of its requests, and shows
nothing of a body it does not parse, so an install that cannot succeed would
stall that long over and over. What is refused to snapd is a 424 (the
request failed because the store, a dependency, was not contacted) with the
store’s own JSON error list (pkgproxy-offline), which snapd does not
retry, decodes for assertions, and reports with its message, without
mistaking it for the assertion being absent (snapproxy.RefusedAnswer;
offline.MiddlewareAnswer; the message is English,
§22).
One exception to “whatever is cached is served”. An apt dists/** entry
that the index already knows to be superseded
(§4.3, the check against the apt index: a
Release fetched earlier expects other hashes) is not served from the cache,
online or off, since apt would reject it with a hash-sum mismatch. It forces
an unconditional fetch, which offline is refused, so the client gets the
503 although stale bytes are cached. Serving it anyway would only move the
failure into apt.
Modes. -offline-mode is one of never (the default: go online as
needed), offline (everything above) and store-overloaded: stay online
until an upstream shows trouble (a transport error, a timeout, a 429 or a
5xx), then leave that upstream alone for a while, as offline mode would,
and probe it again. The store’s load balancer answers a bare 429 for a
request rate it dislikes, and every client of a proxy shares one address, so
what looks like an overloaded store may be ours to ease.
In store-overloaded mode each upstream host (not port) has a circuit
breaker in the very Transport and Dial wrappers that make offline mode
a guarantee, so every side has it. It opens after three consecutive failures,
or at once on a 429/503 with Retry-After (honoured, up to ten minutes),
for 30 s, doubled each time it opens again without the upstream having
answered in between, up to 10 min. A failure is a transport error, a timeout,
or a 429, 502, 503 or 504: not a 500 or 501, which say something
is wrong with that request, and must not stop the others to the same host. An
unreachable host (a dial that fails) is a failure; a host that can be
connected to is not an answer, so a tunnel that connects does not close a
breaker that HTTP requests opened. Once the time is up, one request is let
through as a probe: an answer closes the breaker, a failure reopens it for
twice as long (the doubling stops at the cap, so a long outage cannot
overflow it). Only the probe decides: the request reserving it gets a token,
and an answer from a request that was already in flight when the breaker
opened, success or failure, neither closes the breaker nor counts as the
probe’s. A request the client abandoned says nothing about the upstream.
While open, requests are refused with ErrOffline, so no traffic goes out;
what the cache can answer is served (an entry whose revalidation is refused
is served stale), and what it cannot is a 503 whose Retry-After is the
time left. An open breaker does not freeze the freshness windows, only
offline does. Each transition is logged (opening at warning level,
recovery at info level).
Which 502 becomes a 503. In offline mode every 502 does. In
store-overloaded mode only one whose own upstream request was refused:
Middleware puts a record in the request’s context, Transport and Dial
note a refusal in it (with the time until that host’s probe), and the
Retry-After is the shortest among the hosts that request was refused. A
502 for a host whose breaker is closed stays a 502 while another host’s
breaker is open. The record travels in the context the sides hand to their
shared fetches (context.WithoutCancel keeps values), so the leader of a
shared fetch gets the 503. A request that only waited for another
request’s shared fetch, which was the one refused, notes the refusal itself
when the error it got is ErrOffline (offline.NoteRefused; the snap side
does, the others see a plain 502 in that tiny window).
Switching. -offline-mode sets the start mode, and -offline is
short for -offline-mode offline. Given both, from flags or PKGPROXY_*
variables, they must name the same mode, or the daemon refuses to start,
rather than let the order of the flags, or the alphabetical order the
environment is applied in, decide. GET/PUT /pkgproxy/v1/offline on the admin
socket reads and sets the mode at run time ({"mode": ...}); the answer
lists the upstreams a breaker is not closed for. Requests in flight
finish normally. Changing the mode in effect, by hand or with the
host’s network, forgets the breakers and the hosts whose revalidation
failed (§4.3), which are asked again at
once: a failure from before the change is no news of the upstream after
it. Turning offline
“off” means mode never, not “back to the previous mode”: there is no
breaker memory to return to. pkgproxyctl offline is the front end, and
the stats socket’s /pkgproxy/v1/info reports the mode and the upstreams too,
read-only.
Persistence. In pkgproxy’s own snap (SNAP_NAME is pkgproxy; a
snap that bundles pkgproxy has other options and is left alone) a PUT
also runs snapctl set offline-mode=<mode>, so the run-time switch and
snap set are one setting. The mode is written even for never, the
default, so that an explicit choice survives a change of the default.
Run from an app rather than a hook, snapctl gets an ephemeral context
from snapd, whose config transaction is committed when the snapctl
request ends; no configure hook runs, so there is no restart and no loop
with the hook. When the hook itself switches the mode, the daemon’s write
repeats the value the hook is committing; config transactions merge on
commit, so the two agree. The mode applies whether or not snapctl
succeeds; a failure is logged and returned as persist_error, and
pkgproxyctl reports it and exits 1. Answers carry restart_mode, the
mode after a restart, and persistent; outside the snap nothing keeps a
run-time switch, and pkgproxyctl says so.
Automatic offline. -auto-offline puts offline in effect while
the host is offline, over whatever mode is configured; the configured mode
is kept and reported beside the reason (no-network or airplane-mode).
GET /pkgproxy/v1/offline reports the trigger as auto_offline,
spelled as the flag and the snap option spell it, off included.
The default is network on Linux and off elsewhere: defaults matter,
and a laptop losing its network should be served from the cache with no
setup; the trigger is safe on by default because an explicit switch wins
until the next edge and the settle time of 5 s absorbs blips.
network means no usable default route, watched over rtnetlink, which the
network plug allows. A route, not rfkill, decides: rfkill is wrong for
airplane mode with a cable plugged in or Wi-Fi turned back on in flight,
and misses KDE’s airplane mode, which blocks no radio; rfkill only names
the reason. airplane-mode follows GNOME’s rule (at least one radio, none
unblocked), read from the rfkill uevent files, which any snap may read;
a uevent socket would need network-observe, so they are polled. A change
must hold for 5 s before it counts. Only edges move the switch:
an explicit switch ends the override until the host next goes offline,
except the configure hook re-applying offline-mode, which keeps it. The
override is never persisted, and outside Linux any value but off is
refused.
What does not work offline is whatever is never cached: apt over HTTPS
(a tunnel), a registry or go module not seen before, a find for a search
that was never made through the proxy while online (it is refused with a 424,
not answered with an empty list, which snap find would print as “No matching
snaps”; a local search over the cached catalogue was considered and rejected:
it lacks the publisher, revision and type snapd needs, and the store ranks by
full text and popularity), and any snap action that was never answered through
the proxy while online.
snap install, snap download and snap refresh of a snap that was, work:
see §11.7.