Two tiers

operations for administrators for CI operators

How one pkgproxy fetches everything through another with upstream-pkgproxy, as CI jobs' pkgproxies do through one next to the runners, what each kind of download does then, and what happens when the upstream is down or a loop forms.

On the map: operations

▶ Watch this deck — 8 scenes, ~5 min

01 What this topic covers

ABOUT THIS TOPIC Two tiers one pkgproxy fetching everything through another 1 One pkgproxy through another 2 What the upstream runs 3 Each kind of download 4 When something is down 5 A network you trust 6 Loops
The topic title, Two tiers, and its six scenes in order: one pkgproxy through another, what the upstream runs, each kind of download, when something is down, a network you trust, loops

upstream-pkgproxy sends what one pkgproxy fetches to another, which caches it for every pkgproxy behind it.

A pkgproxy can fetch everything through another pkgproxy, rather than from the mirrors, stores and registries themselves. First you see why, and how to set it. Then how the upstream says what it runs, and what each kind of download does through it. Then what happens while the upstream or an origin is in trouble, why the network between them must be trusted, and how loops are found.

02 One pkgproxy through another

TWO TIERS job 1 pkgproxy job 2 pkgproxy job 3 pkgproxy each with an empty cache pkgproxy next to the runners origins once cache offline-mode=store-overloaded serves what it has while an origin is down $ sudo snap set pkgproxy upstream-pkgproxy=http://cache.example:3142 replaces snap-store-url, go-upstream and docker-upstream
Three CI jobs, each with its own pkgproxy and an empty cache, fetch through one pkgproxy next to the runners, set with upstream-pkgproxy; that one fetches from the origins once and keeps serving in store-overloaded mode

snap set pkgproxy upstream-pkgproxy=http://cache.example:3142 (flag -upstream-pkgproxy). Refused together with -snap-store-url, -go-upstream or -docker-upstream. Unset, pkgproxy fetches from the origins.

In CI, every job may run its own pkgproxy, in a fresh virtual machine, with an empty cache. Each would download the same files again. Instead, run one pkgproxy next to the runners, and set upstream-pkgproxy on each job's pkgproxy to its address. Everything the job fetches then goes through the one next to the runners, which caches it for every job after. With store-overloaded mode on that one, it keeps serving what it has while an origin is down. The setting replaces the snap-store-url, go-upstream and docker-upstream settings.

03 What the upstream runs

PROXY ENGINES GET /pkgproxy/v1/proxy-engines apt enabled forward snap enabled root go enabled /go docker off lxd enabled /lxd/… files enabled /files/v1/fetch files-volatile /files/v1/volatile # no cargo: older than the downstream pkgproxy at start, every 10 minutes runsfetched through the upstream turned offrefused not listedfetched from its origin upstream-pkgproxy.json saved, for a restart while the upstream is unreachable
The upstream pkgproxy answers /pkgproxy/v1/proxy-engines with each proxy engine and its path; the downstream reads it at start and every ten minutes: what runs goes through the upstream, what is turned off is refused, what is not listed goes to its origin; the description is saved in the cache

GET /pkgproxy/v1/proxy-engines on the main port, readable by anyone who can use pkgproxy. An upstream too old to describe itself, but answering as a pkgproxy does, is taken to run what 0.12 ran. Saved as <cache-dir>/upstream-pkgproxy.json.

Every pkgproxy says which of its proxy engines it runs, and where: apt, snaps, Go, Docker, LXD, files and volatile files, at /pkgproxy/v1/proxy-engines. The other pkgproxy reads that when it starts and every ten minutes. A proxy engine the upstream runs is fetched through it. One it has turned off is refused. One it does not list, as an older upstream does not list what is newer, is fetched from its origin. The description is saved in the cache, so a restart while the upstream is unreachable still knows where to send each request.

04 Each kind of download

EACH KIND OF DOWNLOAD pkgproxy upstream origins apt http:// apt https:// tunnel: neither caches it snaps /v2/download/sha3-384/… Go modules and each tier asks sum.golang.org itself files within both tiers' limits: the stricter decides breakers, statistics and logs still name the origins
Through the upstream: apt over HTTP while HTTPS tunnels go direct, snaps by their hash, Go modules checked against sum.golang.org by each pkgproxy, files within the limits of both; breakers and logs still name the origins

apt: http:// through the upstream, https:// tunnels direct. Go: sum.golang.org asked by each tier; a lookup that fails is logged and the module served. Files: each tier's files-allow-host, files-allow-network and size limits apply.

Each kind of download keeps its origin, and only the hop changes. Apt uses the upstream as its HTTP proxy, but HTTPS sources still go to their servers through tunnels neither pkgproxy can cache. Snaps are asked of the upstream by their hash. Each pkgproxy checks the Go modules it caches against the checksum database itself, so the one behind needs to reach sum.golang.org too. Files by checksum, and volatile files, are fetched within the limits of both package proxies, so the stricter one decides. The breakers, statistics and logs still name the origins.

05 When something is down

WHEN SOMETHING IS DOWN pkgproxy upstream origin cached: served not cached: 503 marked refused not counted as trouble, not asked again: one missing file does not cut off the rest pkgproxy upstream down: requests fail not bypassed to the origins, so the outage shows auto-offline=network # the default on Linux no network = no route to the upstream, which may be on the local network
An origin down: the upstream serves what it has and marks a refusal for what it lacks, which the pkgproxy behind does not count as trouble; the upstream down: requests fail until it is back; auto-offline follows the route to the upstream

Offline mode's refusals carry X-Pkgproxy-Refused; they do not open the downstream's breakers and are not retried. With auto-offline=network, the route to the upstream decides.

When an origin is down, the upstream serves what it has cached. For what it lacks, its refusal is marked as one, so the pkgproxy behind it does not take it for an upstream in trouble, nor ask again: one missing file does not cut it off from everything else. When the upstream itself is down, it is not bypassed: requests fail until it is back, so its outage is not hidden. And when pkgproxy goes offline with the machine, no network means no route to the upstream, which may be on the local network.

06 A network you trust

A NETWORK YOU TRUST clients pkgproxy upstream http:// http:// sent on to the upstream with each request: snapd's store credentials and device sessions Docker clients' registry tokens anyone who can read the network can read them both hops on a network you trust as much, such as the private network of the CI runners
Clients send credentials to pkgproxy over plain HTTP; pkgproxy passes them to its upstream over plain HTTP: snapd's store credentials and device sessions, Docker registry tokens; keep both hops on a trusted network

The hop carries snapd's macaroons and device sessions and Docker registry tokens, in plain HTTP. Use it only on a network trusted as much as the clients' own.

The hop between two package proxies is plain HTTP, as is the one between pkgproxy and its clients. It carries what the clients send for the origins: snapd's store credentials and device sessions, and the registry tokens of Docker clients. Anyone who can read that network can read them. So put the two package proxies on a network you trust as much as the one between pkgproxy and its clients, such as the private network of the CI runners.

07 Loops

LOOPS upstream-pkgproxy=http://127.0.0.1:3142 # its own address and port: refuses to start upstream-pkgproxy=http://127.0.0.1:3150 # another pkgproxy at another port: fine pkgproxy A pkgproxy B through a name or a NAT X-Pkgproxy-Via: A X-Pkgproxy-Via: A, B A sees itself: 508 Loop Detected B fetches that kind of download from its origin instead
pkgproxy refuses to start with itself as upstream; another pkgproxy on the same machine at another port is fine; in a ring, requests carry X-Pkgproxy-Via, and the pkgproxy that sees itself answers 508 Loop Detected, so the sender fetches from the origin

Each request to an upstream carries X-Pkgproxy-Via; a loop is answered 508 Loop Detected with X-Pkgproxy-Loop, and logged. The description's random instance also tells a pkgproxy that it is its own upstream.

A pkgproxy refuses to start when its upstream is itself: an address of its own machine, at a port it listens on. Another pkgproxy on the same machine, at another port, is a fine upstream. A loop through a name or a NAT is found in the requests: each request to an upstream carries the instances of the package proxies it went through. A pkgproxy that sees itself there, or more than eight of them, answers 508, Loop Detected, and the one that sent the request fetches that kind of download from its origin instead.

08 Where to go next

WHERE NEXT each file comes from the internet once, to the pkgproxy next to the runners you are here Two tiers pkgproxy for CI the pkgproxy next to the runners Upstream status how the upstream has answered Offline mode serving while an origin is down
You are here, Two tiers; next: pkgproxy for CI, Upstream status, Offline mode

Next: pkgproxy for CI, for the pkgproxy next to the runners; Upstream status, for how the upstream has answered; Offline mode, for serving while an origin is down.

Instead of every job's pkgproxy downloading the same files, each file now comes from the internet once, to the pkgproxy next to the runners. To set up that pkgproxy and its runners, watch pkgproxy for CI. To see how the upstream has been answering, watch Upstream status. To keep serving while an origin is down, watch Offline mode.

Read more

The setting

SettingFlagEnvironmentDefaultMeaning
upstream-pkgproxy-upstream-pkgproxyPKGPROXY_UPSTREAM_PKGPROXYunsetAnother pkgproxy, as http://host:port, to fetch everything through.

It cannot be combined with -snap-store-url, -go-upstream or -docker-upstream. -go-sumdb stays each pkgproxy's own. pkgproxy refuses to start when the upstream is itself: an address of this machine, loopback included, at a port it listens on.

What the upstream says it runs

$ curl http://cache.example:3142/pkgproxy/v1/proxy-engines
{"schema":1,"instance":"7b2f...","proxy-engines":{
  "apt":{"enabled":true,"kind":"forward"},
  "snap":{"enabled":true,"kind":"root"},
  "go":{"enabled":true,"kind":"path","path":"/go"},
  "docker":{"enabled":false,"kind":"root"},
  "lxd":{"enabled":true,"kind":"path","remotes":{"pkgproxy-ubuntu":"/lxd/pkgproxy-ubuntu"}},
  "files":{"enabled":true,"kind":"path","path":"/files/v1/fetch"},
  "files-volatile":{"enabled":true,"kind":"path","path":"/files/v1/volatile"}}}

Every pkgproxy serves this on its main port, to anyone who can use it. It names nothing of the daemon's version or state. Every answer of a proxy engine carries X-Pkgproxy-Engine: <name>.

The upstreamThe proxy engine is
runs itfetched through the upstream, at its path
has it turned offrefused
does not list itfetched from its origin
answers in a newer formatfollowed as last read, or as this pkgproxy runs
is 0.12 or 0.12.1, which do not describe themselvestaken to run apt, snap, Go, Docker, LXD and files at the default paths
cannot be reachedfollowed as saved in <cache-dir>/upstream-pkgproxy.json, or as this pkgproxy runs; requests fail until it answers

The description is read at start and every ten minutes, and again, at most once a minute, after a request to the upstream fails. Nothing is read while pkgproxy is offline. /pkgproxy/v1/info on the stats socket reports upstream_pkgproxy: the URL, where the description came from (upstream, saved, baseline, newer, loop or unreachable), and the target of each proxy engine.

Each proxy engine through the upstream

Proxy engineThrough the upstream
aptThe upstream is the HTTP proxy for http:// sources. https:// sources still go to their servers through tunnels.
snapThe store's API at the upstream's root; snaps by their hash, /v2/download/sha3-384/<hex>.
GoUnder the upstream's /go. Each pkgproxy checks modules against sum.golang.org itself; a lookup that fails is logged and the module served.
DockerAt the upstream's root.
LXDAt the upstream's path for each image server it serves by the same name; others from their origin.
filesThe upstream's /files/v1/fetch. Each pkgproxy applies its own files-allow-host, files-allow-network and files-max-size; the checksum is checked again.
volatile filesThe upstream's /files/v1/volatile, within both pkgproxies' limits; its STALE copy is passed on.

Breakers, statistics and logs key on the origin, not the upstream. Freshness adds up: an entry the upstream serves for up to its TTL is then served for up to this pkgproxy's own. pkgproxyctl cache invalidate forgets what one tier has, not the other's.

Refusals and outages

  • A refusal of the upstream's offline mode (a 503, or the snap store's 424) carries X-Pkgproxy-Refused: <mode>. It counts as an answer for this pkgproxy's breakers and is not retried.
  • An upstream that is down is not bypassed: requests fail until it is back.
  • With auto-offline=network, no network means no route to the upstream's address.
  • Upgrade the upstream first: an upstream from before 0.12.2 does not mark its refusals, which then count against this pkgproxy's breakers.

Loops

Every request to an upstream carries X-Pkgproxy-Via, the instances of the pkgproxies it went through. A pkgproxy that finds its own instance there, or more than 8, answers 508 Loop Detected with X-Pkgproxy-Loop: <instance>. The pkgproxy that sent it logs the loop once and fetches that proxy engine from its origin until it next reads the description.

Security

The hop between the two is plain HTTP and carries what clients send for the origins: snapd's macaroons and device sessions, and Docker registry tokens. Use it on a network you trust as much as the clients' own.

Roads from here

CORE · the path every proxy engine shares

FILES

OPERATIONS