Offline mode

core for everyone

The three offline modes, what is served and what fails while nothing goes upstream, the per-host circuit breaker of store-overloaded, switching at run time without a restart, and going offline with the machine.

On the map: core

▶ Watch this deck — 9 scenes, ~5 min

01 What this topic covers

ABOUT THIS TOPIC Offline mode serving your machines from the cache when the internet is gone 1 Three modes 2 Served from the cache, or refused 3 Only what was cached online 4 Leaving a troubled upstream alone for administrators and CI operators 5 Switching without a restart 6 Going offline with the machine
The topic title, Offline mode, and its 6 scenes in order: Three modes, Served from the cache, or refused, Only what was cached online, Leaving a troubled upstream alone, Switching without a restart, Going offline with the machine

The three offline modes, what works and what fails without the internet, and how the mode is switched.

Offline mode is how pkgproxy serves your machines from its cache when the internet is gone, or when a build must see only what is cached. If you have not seen Immutable or not, start there. Here you see the three modes, what is served and what is refused, and why only what was cached online works. Then how a troubled upstream is left alone, switching modes without a restart, and going offline with the machine.

02 Three modes

OFFLINE MODES -offline-mode store-overloaded | never | offline store-overloaded the default · online, with care pkgproxy upstreams a troubled one left alone never waits on an unreachable one pkgproxy upstreams online as needed offline no network · exact builds pkgproxy upstreams only the cache, nothing sent -offline = -offline-mode offline
Three modes side by side: store-overloaded, the default, leaves a troubled upstream alone for a while; never goes online as needed; offline serves only the cache and contacts nothing

-offline-mode is store-overloaded (default), never or offline. -offline is short for -offline-mode offline; given both, they must agree.

pkgproxy has three offline modes, set with the offline mode flag or snap option. store-overloaded, the default, stays online, but leaves an upstream that shows trouble alone for a while and serves what is cached meanwhile. Never goes online as needed, and waits on an upstream it cannot reach until the connection gives up. Offline serves only what is cached and sends nothing at all upstream, whatever a client asks for: for a machine that has lost its network, or a build that must see exactly what was cached. The offline flag is short for offline mode offline.

03 Served from the cache, or refused at once

WHILE OFFLINE cached request pkgproxy cache hit every TTL frozen a request for fresh forces no recheck upstream no DNS, no connection a miss 503 Service Unavailable pkgproxy is offline: … no Retry-After: asking again cannot help 424 Failed Dependency for snapd, which does not retry it a CONNECT tunnel is refused too
While offline, a cached request is served from disk with its freshness frozen; a miss is refused before anything goes upstream and answered 503, or 424 for snapd

Hits are served with every TTL frozen. Upstream requests and CONNECT dials are refused before they start. A miss is a 503 (pkgproxy is offline, no Retry-After); snapd gets a 424.

While offline, every cached entry is served as a hit. Freshness is frozen: no entry is ever old enough to recheck, and apt's request for a fresh copy no longer forces one. Nothing goes upstream; a request is refused before any DNS lookup or connection. What the cache cannot answer fails at once with a 503 saying pkgproxy is offline, without Retry After, since asking again cannot help. Snapd gets a 424 instead, which it does not retry, and a tunnel for an HTTPS source is refused.

04 Only what was cached online

WHAT OFFLINE CAN SERVE cached while online nothing else works offline apt indexes and packages already fetched Go module versions already fetched Docker tags and layers already pulled fails offline apt sources over HTTPS: a tunnel, never cached a Go module never fetched before a Docker image never pulled before a snap search never made through pkgproxy an apt index a newer Release superseded snap install, download, refresh of a seen snap not yet: a device that needs a new store session first
Two columns: what works offline, content fetched while online, including snaps; and what fails: HTTPS apt sources, modules and images never fetched, snap searches never made, and superseded apt indexes; snaps only on a device that needs no new store session

Never-cached content fails offline: HTTPS apt sources, new modules and images, unseen snap searches. A dists/** entry the index knows is superseded is refused online or off. Snaps seen online install and refresh, unless the device needs a new store session (see Snap Store relay).

Offline mode can only serve what was cached while online. Apt sources over HTTPS go through a tunnel and are never cached. A Go module or Docker image never fetched before fails, and so does a snap search that was never made through pkgproxy. An apt index that a newer Release has already superseded is not served either, since apt would reject it with a hash sum mismatch. Installing, downloading or refreshing a snap that was fetched through pkgproxy while online works, but only on a device that does not need a new store session first.

05 Leaving a troubled upstream alone for administrators for CI operators

STORE-OVERLOADED closed one breaker per host a failure: transport error, timeout 429 502 503 504 3 in a row, or at once on 429/503 + Retry-After open: left alone 30s → 60s → … 10m cached: served, stale if need be 503 on a miss Retry-After: time left one probe answer: closed failure: open, twice as long a 500 or 501 is not a failure: it says something about that request only an open breaker does not freeze freshness; only offline mode does
A circuit breaker for one upstream host: closed, then open after three failures for 30 seconds doubling up to 10 minutes, serving the cache and answering misses 503 with Retry-After, then a single probe that closes or reopens it

One breaker per host: open after 3 consecutive failures, or at once on 429/503 with Retry-After; 30 s, doubled up to 10 min; then one probe. A 500 or 501 is not a failure.

store-overloaded mode gives each upstream host a circuit breaker. It opens after three consecutive failures, which are transport errors, timeouts, and 429, 502, 503 or 504 answers, or at once when a 429 or 503 says when to retry. While it is open the host is left alone, for thirty seconds at first, doubling each time it opens again, up to ten minutes. What is cached is served, stale if need be, and a miss gets a 503 whose Retry After is the time left. Then one probe request decides whether it closes.

06 Switching without a restart

SWITCHING $ pkgproxyctl offline $ pkgproxyctl offline on $ pkgproxyctl offline off $ pkgproxyctl offline mode store-overloaded admin socket pkgproxy no restart $ sudo snap set pkgproxy offline-mode=offline offline-mode snap option: one setting saved outside the snap: lasts until pkgproxy restarts, and pkgproxyctl says so a change of mode forgets which hosts failed: they are asked again at once
pkgproxyctl offline commands and snap set offline-mode both reach the running daemon without a restart; in the snap, the daemon saves the mode as the snap option

pkgproxyctl offline (status), on, off and mode MODE, or snap set pkgproxy offline-mode=…: no restart either way. In the snap the switch is saved as the option.

pkgproxyctl offline shows the mode, and offline on, offline off and offline mode switch it at run time, through the admin socket, without a restart. Requests in flight finish normally. In the snap, snap set offline mode does the same, and both change one setting: the daemon writes a run time switch back as the snap option, so it survives a restart. Outside the snap a switch lasts until the next restart. Any change of mode forgets which hosts failed, so they are asked again at once.

07 Going offline with the machine

AUTO-OFFLINE the machine network no default route network network the default: no usable default route airplane mode, a pulled cable, Wi-Fi out of reach airplane-mode every radio is off off offline only when you say so pkgproxy configured mode offline configured mode again 5 s later pkgproxyctl offline off wins until the machine next goes offline
The machine's network drops; five seconds later pkgproxy goes offline by itself, and when the network returns it goes back to the configured mode; below, the three triggers network, airplane-mode and off

-auto-offline is network (default on Linux), airplane-mode or off (the only value elsewhere). Changes settle for 5 s; offline-mode is not changed.

On Linux, pkgproxy also goes offline by itself while its machine is offline. The auto-offline setting chooses the trigger. Network, the default, means no usable default route, which airplane mode, a pulled cable or Wi-Fi out of reach all lead to. Airplane mode means every radio is off. Off turns it off. A change must hold for five seconds. The configured mode is kept and applies again when the network returns, and an explicit switch wins until the machine next goes offline.

08 Where to go next for CI operators in place of the general version

WHERE NEXT without the internet, your builds still get what pkgproxy cached, and anything else fails at once you are here Offline mode pkgproxy for CI builds that must see exactly what was cached Immutable or not the windows offline mode freezes pkgproxyctl switching it from the command line
You are here, Offline mode; next: pkgproxy for CI, Immutable or not, pkgproxyctl

Next: pkgproxy for CI, for builds that see only the cache; Immutable or not, for what offline freezes; pkgproxyctl, for the offline command.

Without the internet, your builds still get everything pkgproxy cached, and anything else fails at once. To run builds that must see exactly what was cached, watch pkgproxy for CI. To see the windows that offline mode freezes, watch Immutable or not. To switch it from the command line, watch pkgproxyctl.

09 Where to go next general version

WHERE NEXT without the internet, your machines still get what pkgproxy cached, and anything else fails at once you are here Offline mode Immutable or not the windows offline mode freezes pkgproxyctl switching it from the command line Snap Store relay snaps with no network, and what still needs the store
You are here, Offline mode; next: Immutable or not, pkgproxyctl, Snap Store relay

Next: Immutable or not, for what offline freezes; pkgproxyctl, for the offline command; Snap Store relay, for snaps without a network and what still needs the store.

Without the internet, your machines still get everything pkgproxy cached, and anything else fails at once. To see the windows that offline mode freezes, watch Immutable or not. To switch it from the command line, watch pkgproxyctl. To see snaps installed and refreshed with no network, and what still needs the store, watch Snap Store relay.

Read more

Modes

offline-modeUpstreamsWindowsWhat the cache cannot answer
store-overloaded (default)contacted, except a host whose breaker is openas configuredfrom a host left alone: 503 with Retry-After, the time until its next try; snapd gets 424
nevercontacted as needed; one that cannot be reached is waited on until the connection gives upas configuredfetched
offlinenever contacted, no DNS lookup or connectionendless: every cached entry is a hit503 without Retry-After; snapd gets 424

The breaker of store-overloaded, one per upstream host: it opens after 3 failures in a row (a transport error, a timeout, 429, 502, 503, 504), or at once on a 429 or 503 with Retry-After (honoured up to 10 min). It stays open 30 s, doubling each time it opens again, up to 10 min; then one probe request decides. A 500 or 501 is not a failure. The default was never up to pkgproxy 0.12.0; a mode chosen with snap set or pkgproxyctl offline is kept across the change.

What each side does while offline

SideStill servedRefused
aptcached pool/ files and dists/ indexes, as they wereanything not cached; an index a newer Release shows to be superseded; https:// sources, whose tunnels are refused
Snap Storecached snap info, assertions and snap files; install, download and refresh of snaps that were answered online; searches made onlinesearches never made online and actions never answered online, as 424; every request of a device that has a serial but no store session yet, cached or not (see Snap Store relay)
Go modulescached .mod, .zip, list, @latest and .info; the "not found" answers the go command gets while it looks for the module that holds a package, so go install example.com/tool@latest works for a tool installed through pkgproxy beforemodules and versions never fetched
Docker Hubtags resolved before, cached manifests and layersimages never pulled
LXD imagescached catalogs and image files, through lxc and through LXD's image APIimages never fetched
Files by checksumfiles already fetchedfiles never fetched

What a refused client sees

ClientStatusBody
apt, go, Docker, LXD, curl503pkgproxy is offline: the content is not in the cache
the same, in store-overloaded503, Retry-Afterpkgproxy is not contacting the upstream for a while, as it looked overloaded: the content is not in the cache
snapd424a store error list: pkgproxy is offline and the content is not in its cache
snapd, in store-overloaded424pkgproxy is giving the store a rest, as it looked overloaded, and the content is not in its cache

The bodies are in English whatever the daemon's language. snapd does not retry a 424, where it would retry a 5xx for half a minute.

Going offline with the machine

auto-offlinepkgproxy is offline whileReported reason
network (default on Linux)the machine has no usable default routeno-network
airplane-modethe machine has at least one radio and every radio is offairplane-mode
off (the only value outside Linux)never by itselfnone

A change counts once it has held for 5 s. The configured offline-mode is kept and applies again once the machine is back. A switch by hand wins until the machine next goes offline.

Switching

CommandDoes
pkgproxyctl offline or pkgproxyctl offline statusshows the mode in effect, why if automatic, and each upstream being left alone with its next try
pkgproxyctl offline onmode offline
pkgproxyctl offline offmode never (not the mode before, nor the default store-overloaded)
pkgproxyctl offline mode MODEmode never, store-overloaded or offline
pkgproxyctl offline mode -keep-automatic MODEsets the mode for when the machine is back, staying offline while it is not; the snap's configure hook uses it
sudo snap set pkgproxy offline-mode=MODEthe same setting, without a restart
sudo snap set pkgproxy auto-offline=offstop going offline with the machine

Each command takes -socket-path, the admin socket. In the snap a switch is saved as the offline-mode option and survives a restart; outside it a switch lasts until the daemon restarts, and pkgproxyctl says so. Any change of mode forgets the breakers and the hosts whose recheck failed.

for administrators for CI operators

Settings

SettingFlagVariableSnap optionDefault
start mode-offline-modePKGPROXY_OFFLINE_MODEoffline-modestore-overloaded
short for offline-offlinePKGPROXY_OFFLINEflag or environment onlyoff
going offline with the machine-auto-offlinePKGPROXY_AUTO_OFFLINEauto-offlinenetwork on Linux, else off

Given both -offline and -offline-mode, they must name the same mode, or pkgproxy refuses to start.

for administrators for CI operators

The admin API

GET and PUT of /pkgproxy/v1/offline on the admin socket read and set the mode; pkgproxyctl offline is its front end. The stats socket's /pkgproxy/v1/info reports the same, read-only.

FieldMeaning
modethe mode in effect: never, store-overloaded or offline
configured_modewhile offline automatically: the mode that applies again
automaticwhile offline automatically: no-network or airplane-mode
auto_offlinethe auto-offline setting
keep_automaticin a PUT: keep an automatic override
upstreamshosts whose breaker is not closed: host, state (open or half-open), retry_in_seconds
restart_modethe mode after a restart
persistentwhether a switch survives a restart
persist_errorfrom a PUT: why the mode could not be saved

Roads from here

CORE · the path every side shares

SNAP

SETUP AND THE COMMAND LINE

OPERATIONS

  • Settings — the offline-mode snap option and the configure hook
  • pkgproxy for CI — builds that must see what was cached