Two tiers
operations for administrators for CI operators
How one pkgproxy fetches everything through another with upstream-pkgproxy, as CI jobs' pkgproxies do through one next to the runners, what each kind of download does then, and what happens when the upstream is down or a loop forms.
On the map: operations
01 What this topic covers
upstream-pkgproxy sends what one pkgproxy fetches to another,
which caches it for every pkgproxy behind it.
A pkgproxy can fetch everything through another pkgproxy, rather than from the mirrors, stores and registries themselves. First you see why, and how to set it. Then how the upstream says what it runs, and what each kind of download does through it. Then what happens while the upstream or an origin is in trouble, why the network between them must be trusted, and how loops are found.
02 One pkgproxy through another
snap set pkgproxy upstream-pkgproxy=http://cache.example:3142
(flag -upstream-pkgproxy). Refused together with
-snap-store-url, -go-upstream or
-docker-upstream. Unset, pkgproxy fetches from the origins.
In CI, every job may run its own pkgproxy, in a fresh virtual machine, with an empty cache. Each would download the same files again. Instead, run one pkgproxy next to the runners, and set upstream-pkgproxy on each job's pkgproxy to its address. Everything the job fetches then goes through the one next to the runners, which caches it for every job after. With store-overloaded mode on that one, it keeps serving what it has while an origin is down. The setting replaces the snap-store-url, go-upstream and docker-upstream settings.
03 What the upstream runs
GET /pkgproxy/v1/proxy-engines on the main port, readable by
anyone who can use pkgproxy. An upstream too old to describe itself, but
answering as a pkgproxy does, is taken to run what 0.12 ran. Saved as
<cache-dir>/upstream-pkgproxy.json.
Every pkgproxy says which of its proxy engines it runs, and where: apt, snaps, Go, Docker, LXD, files and volatile files, at /pkgproxy/v1/proxy-engines. The other pkgproxy reads that when it starts and every ten minutes. A proxy engine the upstream runs is fetched through it. One it has turned off is refused. One it does not list, as an older upstream does not list what is newer, is fetched from its origin. The description is saved in the cache, so a restart while the upstream is unreachable still knows where to send each request.
04 Each kind of download
apt: http:// through the upstream, https://
tunnels direct. Go: sum.golang.org asked by each tier; a
lookup that fails is logged and the module served. Files: each tier's
files-allow-host, files-allow-network and size
limits apply.
Each kind of download keeps its origin, and only the hop changes. Apt uses the upstream as its HTTP proxy, but HTTPS sources still go to their servers through tunnels neither pkgproxy can cache. Snaps are asked of the upstream by their hash. Each pkgproxy checks the Go modules it caches against the checksum database itself, so the one behind needs to reach sum.golang.org too. Files by checksum, and volatile files, are fetched within the limits of both package proxies, so the stricter one decides. The breakers, statistics and logs still name the origins.
05 When something is down
Offline mode's refusals carry X-Pkgproxy-Refused; they do
not open the downstream's breakers and are not retried. With
auto-offline=network, the route to the upstream decides.
When an origin is down, the upstream serves what it has cached. For what it lacks, its refusal is marked as one, so the pkgproxy behind it does not take it for an upstream in trouble, nor ask again: one missing file does not cut it off from everything else. When the upstream itself is down, it is not bypassed: requests fail until it is back, so its outage is not hidden. And when pkgproxy goes offline with the machine, no network means no route to the upstream, which may be on the local network.
06 A network you trust
The hop carries snapd's macaroons and device sessions and Docker registry tokens, in plain HTTP. Use it only on a network trusted as much as the clients' own.
The hop between two package proxies is plain HTTP, as is the one between pkgproxy and its clients. It carries what the clients send for the origins: snapd's store credentials and device sessions, and the registry tokens of Docker clients. Anyone who can read that network can read them. So put the two package proxies on a network you trust as much as the one between pkgproxy and its clients, such as the private network of the CI runners.
07 Loops
Each request to an upstream carries X-Pkgproxy-Via; a loop
is answered 508 Loop Detected with X-Pkgproxy-Loop,
and logged. The description's random instance also tells a
pkgproxy that it is its own upstream.
A pkgproxy refuses to start when its upstream is itself: an address of its own machine, at a port it listens on. Another pkgproxy on the same machine, at another port, is a fine upstream. A loop through a name or a NAT is found in the requests: each request to an upstream carries the instances of the package proxies it went through. A pkgproxy that sees itself there, or more than eight of them, answers 508, Loop Detected, and the one that sent the request fetches that kind of download from its origin instead.
08 Where to go next
Next: pkgproxy for CI, for the pkgproxy next to the runners; Upstream status, for how the upstream has answered; Offline mode, for serving while an origin is down.
Instead of every job's pkgproxy downloading the same files, each file now comes from the internet once, to the pkgproxy next to the runners. To set up that pkgproxy and its runners, watch pkgproxy for CI. To see how the upstream has been answering, watch Upstream status. To keep serving while an origin is down, watch Offline mode.
Read more
The setting
| Setting | Flag | Environment | Default | Meaning |
|---|---|---|---|---|
upstream-pkgproxy | -upstream-pkgproxy | PKGPROXY_UPSTREAM_PKGPROXY | unset | Another pkgproxy, as http://host:port, to fetch everything through. |
It cannot be combined with -snap-store-url, -go-upstream or -docker-upstream. -go-sumdb stays each pkgproxy's own. pkgproxy refuses to start when the upstream is itself: an address of this machine, loopback included, at a port it listens on.
What the upstream says it runs
$ curl http://cache.example:3142/pkgproxy/v1/proxy-engines
{"schema":1,"instance":"7b2f...","proxy-engines":{
"apt":{"enabled":true,"kind":"forward"},
"snap":{"enabled":true,"kind":"root"},
"go":{"enabled":true,"kind":"path","path":"/go"},
"docker":{"enabled":false,"kind":"root"},
"lxd":{"enabled":true,"kind":"path","remotes":{"pkgproxy-ubuntu":"/lxd/pkgproxy-ubuntu"}},
"files":{"enabled":true,"kind":"path","path":"/files/v1/fetch"},
"files-volatile":{"enabled":true,"kind":"path","path":"/files/v1/volatile"}}}
Every pkgproxy serves this on its main port, to anyone who can use it. It names nothing of the daemon's version or state. Every answer of a proxy engine carries X-Pkgproxy-Engine: <name>.
| The upstream | The proxy engine is |
|---|---|
| runs it | fetched through the upstream, at its path |
| has it turned off | refused |
| does not list it | fetched from its origin |
| answers in a newer format | followed as last read, or as this pkgproxy runs |
| is 0.12 or 0.12.1, which do not describe themselves | taken to run apt, snap, Go, Docker, LXD and files at the default paths |
| cannot be reached | followed as saved in <cache-dir>/upstream-pkgproxy.json, or as this pkgproxy runs; requests fail until it answers |
The description is read at start and every ten minutes, and again, at most once a minute, after a request to the upstream fails. Nothing is read while pkgproxy is offline. /pkgproxy/v1/info on the stats socket reports upstream_pkgproxy: the URL, where the description came from (upstream, saved, baseline, newer, loop or unreachable), and the target of each proxy engine.
Each proxy engine through the upstream
| Proxy engine | Through the upstream |
|---|---|
| apt | The upstream is the HTTP proxy for http:// sources. https:// sources still go to their servers through tunnels. |
| snap | The store's API at the upstream's root; snaps by their hash, /v2/download/sha3-384/<hex>. |
| Go | Under the upstream's /go. Each pkgproxy checks modules against sum.golang.org itself; a lookup that fails is logged and the module served. |
| Docker | At the upstream's root. |
| LXD | At the upstream's path for each image server it serves by the same name; others from their origin. |
| files | The upstream's /files/v1/fetch. Each pkgproxy applies its own files-allow-host, files-allow-network and files-max-size; the checksum is checked again. |
| volatile files | The upstream's /files/v1/volatile, within both pkgproxies' limits; its STALE copy is passed on. |
Breakers, statistics and logs key on the origin, not the upstream. Freshness adds up: an entry the upstream serves for up to its TTL is then served for up to this pkgproxy's own. pkgproxyctl cache invalidate forgets what one tier has, not the other's.
Refusals and outages
- A refusal of the upstream's offline mode (a 503, or the snap store's 424) carries
X-Pkgproxy-Refused: <mode>. It counts as an answer for this pkgproxy's breakers and is not retried. - An upstream that is down is not bypassed: requests fail until it is back.
- With
auto-offline=network, no network means no route to the upstream's address. - Upgrade the upstream first: an upstream from before 0.12.2 does not mark its refusals, which then count against this pkgproxy's breakers.
Loops
Every request to an upstream carries X-Pkgproxy-Via, the instances of the pkgproxies it went through. A pkgproxy that finds its own instance there, or more than 8, answers 508 Loop Detected with X-Pkgproxy-Loop: <instance>. The pkgproxy that sent it logs the loop once and fetches that proxy engine from its origin until it next reads the description.
Security
The hop between the two is plain HTTP and carries what clients send for the origins: snapd's macaroons and device sessions, and Docker registry tokens. Use it on a network you trust as much as the clients' own.
In the documentation
Roads from here
CORE · the path every proxy engine shares
- Offline mode — store-overloaded on the upstream
FILES
- Files by checksum — files and volatile files through both
OPERATIONS
- pkgproxy for CI — the pkgproxy next to the runners
- Hosted CI runners — a pkgproxy inside each job
- Upstream status — how the upstream has answered
- Settings — upstream-pkgproxy among the settings