Offline mode
core for everyone
The three offline modes, what is served and what fails while nothing goes upstream, the per-host circuit breaker of store-overloaded, switching at run time without a restart, and going offline with the machine.
On the map: core
01 What this topic covers
The three offline modes, what works and what fails without the internet, and how the mode is switched.
Offline mode is how pkgproxy serves your machines from its cache when the internet is gone, or when a build must see only what is cached. If you have not seen Immutable or not, start there. Here you see the three modes, what is served and what is refused, and why only what was cached online works. Then how a troubled upstream is left alone, switching modes without a restart, and going offline with the machine.
02 Three modes
-offline-mode is store-overloaded (default),
never or offline. -offline
is short for -offline-mode offline; given both, they must
agree.
pkgproxy has three offline modes, set with the offline mode flag or snap option. store-overloaded, the default, stays online, but leaves an upstream that shows trouble alone for a while and serves what is cached meanwhile. Never goes online as needed, and waits on an upstream it cannot reach until the connection gives up. Offline serves only what is cached and sends nothing at all upstream, whatever a client asks for: for a machine that has lost its network, or a build that must see exactly what was cached. The offline flag is short for offline mode offline.
03 Served from the cache, or refused at once
Hits are served with every TTL frozen. Upstream requests and
CONNECT dials are refused before they start. A miss is a
503 (pkgproxy is offline, no
Retry-After); snapd gets a 424.
While offline, every cached entry is served as a hit. Freshness is frozen: no entry is ever old enough to recheck, and apt's request for a fresh copy no longer forces one. Nothing goes upstream; a request is refused before any DNS lookup or connection. What the cache cannot answer fails at once with a 503 saying pkgproxy is offline, without Retry After, since asking again cannot help. Snapd gets a 424 instead, which it does not retry, and a tunnel for an HTTPS source is refused.
04 Only what was cached online
Never-cached content fails offline: HTTPS apt sources, new modules and
images, unseen snap searches. A dists/** entry the index knows
is superseded is refused online or off. Snaps seen online install and
refresh, unless the device needs a new store session (see Snap Store
relay).
Offline mode can only serve what was cached while online. Apt sources over HTTPS go through a tunnel and are never cached. A Go module or Docker image never fetched before fails, and so does a snap search that was never made through pkgproxy. An apt index that a newer Release has already superseded is not served either, since apt would reject it with a hash sum mismatch. Installing, downloading or refreshing a snap that was fetched through pkgproxy while online works, but only on a device that does not need a new store session first.
05 Leaving a troubled upstream alone for administrators for CI operators
One breaker per host: open after 3 consecutive failures, or at once on
429/503 with Retry-After; 30 s,
doubled up to 10 min; then one probe. A 500 or
501 is not a failure.
store-overloaded mode gives each upstream host a circuit breaker. It opens after three consecutive failures, which are transport errors, timeouts, and 429, 502, 503 or 504 answers, or at once when a 429 or 503 says when to retry. While it is open the host is left alone, for thirty seconds at first, doubling each time it opens again, up to ten minutes. What is cached is served, stale if need be, and a miss gets a 503 whose Retry After is the time left. Then one probe request decides whether it closes.
06 Switching without a restart
pkgproxyctl offline (status), on,
off and mode MODE, or snap set pkgproxy
offline-mode=…: no restart either way. In the snap the switch is
saved as the option.
pkgproxyctl offline shows the mode, and offline on, offline off and offline mode switch it at run time, through the admin socket, without a restart. Requests in flight finish normally. In the snap, snap set offline mode does the same, and both change one setting: the daemon writes a run time switch back as the snap option, so it survives a restart. Outside the snap a switch lasts until the next restart. Any change of mode forgets which hosts failed, so they are asked again at once.
07 Going offline with the machine
-auto-offline is network (default on Linux),
airplane-mode or off (the only value elsewhere).
Changes settle for 5 s; offline-mode is not changed.
On Linux, pkgproxy also goes offline by itself while its machine is offline. The auto-offline setting chooses the trigger. Network, the default, means no usable default route, which airplane mode, a pulled cable or Wi-Fi out of reach all lead to. Airplane mode means every radio is off. Off turns it off. A change must hold for five seconds. The configured mode is kept and applies again when the network returns, and an explicit switch wins until the machine next goes offline.
08 Where to go next for CI operators in place of the general version
Next: pkgproxy for CI, for builds that see only the cache; Immutable or
not, for what offline freezes; pkgproxyctl, for the offline
command.
Without the internet, your builds still get everything pkgproxy cached, and anything else fails at once. To run builds that must see exactly what was cached, watch pkgproxy for CI. To see the windows that offline mode freezes, watch Immutable or not. To switch it from the command line, watch pkgproxyctl.
09 Where to go next general version
Next: Immutable or not, for what offline freezes;
pkgproxyctl, for the offline command; Snap Store relay, for
snaps without a network and what still needs the store.
Without the internet, your machines still get everything pkgproxy cached, and anything else fails at once. To see the windows that offline mode freezes, watch Immutable or not. To switch it from the command line, watch pkgproxyctl. To see snaps installed and refreshed with no network, and what still needs the store, watch Snap Store relay.
Read more
Modes
offline-mode | Upstreams | Windows | What the cache cannot answer |
|---|---|---|---|
store-overloaded (default) | contacted, except a host whose breaker is open | as configured | from a host left alone: 503 with Retry-After, the time until its next try; snapd gets 424 |
never | contacted as needed; one that cannot be reached is waited on until the connection gives up | as configured | fetched |
offline | never contacted, no DNS lookup or connection | endless: every cached entry is a hit | 503 without Retry-After; snapd gets 424 |
The breaker of store-overloaded, one per upstream host: it opens after 3 failures in a row (a transport error, a timeout, 429, 502, 503, 504), or at once on a 429 or 503 with Retry-After (honoured up to 10 min). It stays open 30 s, doubling each time it opens again, up to 10 min; then one probe request decides. A 500 or 501 is not a failure. The default was never up to pkgproxy 0.12.0; a mode chosen with snap set or pkgproxyctl offline is kept across the change.
What each side does while offline
| Side | Still served | Refused |
|---|---|---|
| apt | cached pool/ files and dists/ indexes, as they were | anything not cached; an index a newer Release shows to be superseded; https:// sources, whose tunnels are refused |
| Snap Store | cached snap info, assertions and snap files; install, download and refresh of snaps that were answered online; searches made online | searches never made online and actions never answered online, as 424; every request of a device that has a serial but no store session yet, cached or not (see Snap Store relay) |
| Go modules | cached .mod, .zip, list, @latest and .info; the "not found" answers the go command gets while it looks for the module that holds a package, so go install example.com/tool@latest works for a tool installed through pkgproxy before | modules and versions never fetched |
| Docker Hub | tags resolved before, cached manifests and layers | images never pulled |
| LXD images | cached catalogs and image files, through lxc and through LXD's image API | images never fetched |
| Files by checksum | files already fetched | files never fetched |
What a refused client sees
| Client | Status | Body |
|---|---|---|
| apt, go, Docker, LXD, curl | 503 | pkgproxy is offline: the content is not in the cache |
the same, in store-overloaded | 503, Retry-After | pkgproxy is not contacting the upstream for a while, as it looked overloaded: the content is not in the cache |
| snapd | 424 | a store error list: pkgproxy is offline and the content is not in its cache |
snapd, in store-overloaded | 424 | pkgproxy is giving the store a rest, as it looked overloaded, and the content is not in its cache |
The bodies are in English whatever the daemon's language. snapd does not retry a 424, where it would retry a 5xx for half a minute.
Going offline with the machine
auto-offline | pkgproxy is offline while | Reported reason |
|---|---|---|
network (default on Linux) | the machine has no usable default route | no-network |
airplane-mode | the machine has at least one radio and every radio is off | airplane-mode |
off (the only value outside Linux) | never by itself | none |
A change counts once it has held for 5 s. The configured offline-mode is kept and applies again once the machine is back. A switch by hand wins until the machine next goes offline.
Switching
| Command | Does |
|---|---|
pkgproxyctl offline or pkgproxyctl offline status | shows the mode in effect, why if automatic, and each upstream being left alone with its next try |
pkgproxyctl offline on | mode offline |
pkgproxyctl offline off | mode never (not the mode before, nor the default store-overloaded) |
pkgproxyctl offline mode MODE | mode never, store-overloaded or offline |
pkgproxyctl offline mode -keep-automatic MODE | sets the mode for when the machine is back, staying offline while it is not; the snap's configure hook uses it |
sudo snap set pkgproxy offline-mode=MODE | the same setting, without a restart |
sudo snap set pkgproxy auto-offline=off | stop going offline with the machine |
Each command takes -socket-path, the admin socket. In the snap a switch is saved as the offline-mode option and survives a restart; outside it a switch lasts until the daemon restarts, and pkgproxyctl says so. Any change of mode forgets the breakers and the hosts whose recheck failed.
for administrators for CI operators
Settings
| Setting | Flag | Variable | Snap option | Default |
|---|---|---|---|---|
| start mode | -offline-mode | PKGPROXY_OFFLINE_MODE | offline-mode | store-overloaded |
short for offline | -offline | PKGPROXY_OFFLINE | flag or environment only | off |
| going offline with the machine | -auto-offline | PKGPROXY_AUTO_OFFLINE | auto-offline | network on Linux, else off |
Given both -offline and -offline-mode, they must name the same mode, or pkgproxy refuses to start.
for administrators for CI operators
The admin API
GET and PUT of /pkgproxy/v1/offline on the admin socket read and set the mode; pkgproxyctl offline is its front end. The stats socket's /pkgproxy/v1/info reports the same, read-only.
| Field | Meaning |
|---|---|
mode | the mode in effect: never, store-overloaded or offline |
configured_mode | while offline automatically: the mode that applies again |
automatic | while offline automatically: no-network or airplane-mode |
auto_offline | the auto-offline setting |
keep_automatic | in a PUT: keep an automatic override |
upstreams | hosts whose breaker is not closed: host, state (open or half-open), retry_in_seconds |
restart_mode | the mode after a restart |
persistent | whether a switch survives a restart |
persist_error | from a PUT: why the mode could not be saved |
In the documentation
Roads from here
CORE · the path every side shares
- Immutable or revalidated — what offline freezes
- Stalled upstreams — what counts as a failing upstream
SNAP
- Snap Store relay — snaps installed offline
SETUP AND THE COMMAND LINE
- pkgproxyctl — the offline command
OPERATIONS
- Settings — the offline-mode snap option and the configure hook
- pkgproxy for CI — builds that must see what was cached