pkgproxy design: an upstream pkgproxy
Two tiers: a pkgproxy that fetches everything through another. Part of the pkgproxy design, whose index says which file holds each § number.
27. An upstream pkgproxy
-upstream-pkgproxy http://host:port (snap set pkgproxy upstream-pkgproxy=...) makes a pkgproxy, the downstream, send what its
proxy engines fetch to another pkgproxy, the upstream, instead of the
origins (gitlab.com/zygoon/pkgproxy#147). The case it is made for is CI:
an infrastructure pkgproxy runs next to the runners, in
offline-mode=store-overloaded, and the pkgproxy under test in the spread
suite uses it as its upstream. What the upstream has cached is then a LAN
away, and it is served while an origin is in trouble, as far as the
upstream has it. Unset, a pkgproxy fetches from the origins, as it always
did.
The option names one upstream for all the proxy engines. It replaces their
own upstream options, so -snap-store-url, -go-upstream and
-docker-upstream are refused with it. -go-sumdb is not an upstream to
fetch from but what checks what is fetched, and it stays the downstream’s
own (§27.3). A downstream
serves the proxy engines its own settings turn on, at its own paths: what
the upstream runs decides only where each is fetched from.
The terms: apt, snap, Go, Docker, LXD, files, files-volatile and cargo are the proxy engines of a pkgproxy.
27.1 What each proxy engine says: /pkgproxy/v1/proxy-engines
Every pkgproxy describes itself at GET /pkgproxy/v1/proxy-engines, on
its main listener (internal/proxyengines). It is the only route of
pkgproxy’s own API there; the others stay on the stats and admin sockets
(§13.1). It says what is served where, and nothing of the daemon’s version
or state, as anyone who can use pkgproxy can read it.
{"schema":1,"instance":"7b2f...","proxy-engines":{
"apt":{"enabled":true,"kind":"forward"},
"snap":{"enabled":true,"kind":"root"},
"go":{"enabled":true,"kind":"path","path":"/go"},
"docker":{"enabled":false,"kind":"root"},
"lxd":{"enabled":true,"kind":"path","remotes":{"pkgproxy-ubuntu":"/lxd/pkgproxy-ubuntu"}},
"files":{"enabled":true,"kind":"path","path":"/files/v1/fetch"},
"files-volatile":{"enabled":true,"kind":"path","path":"/files/v1/volatile"}}}
schemais the format’s version; a downstream does not use another. A newer one is that of a newer upstream: the downstream says so in its log, follows what the upstream said before in a format it reads (saved or in memory), or else what it runs itself, as for an upstream that cannot be reached, and reads the description again every ten minutes, not sooner (sourcenewer). Answers withoutX-Pkgproxy-Engineare no answers from it, as from any upstream that describes itself.instanceis random for each run of the daemon: a downstream whose upstream says its own instance is talking to itself (§27.5).kindsays how a downstream reaches a proxy engine:forward, as an HTTP proxy (apt);root, at the root (snap and Docker, whose clients need it there);path, underpath, or, for LXD, each image server at its own path inremotes.
The downstream reads only absolute paths of the upstream’s own
(proxyengines.ValidPath: no other host, no . or ..), and leaves out
what it does not know, a kind or a proxy engine newer than itself: a newer
upstream reads as an older one.
Every answer of a proxy engine on the main listener carries
X-Pkgproxy-Engine: <name>, which the gateway sets before the proxy
engine runs; the gateway’s own answers, such as its 404 for a path no
proxy engine serves, carry none.
27.2 Following the description
internal/chain reads the description when the daemon starts, and every
ten minutes after. A request to the upstream that fails, or that is
answered without X-Pkgproxy-Engine, asks for it again, at most once a
minute. Each reading replaces what is followed in one swap, which every
request reads once, so a proxy engine never changes its target in the
middle of a request. What the upstream says decides, for each proxy
engine and each of LXD’s image servers:
- one the upstream runs is fetched through it, at its path;
- one the upstream has turned off is refused;
- one the upstream does not list is fetched from its origin. That is how a downstream newer than its upstream works: an upstream from before cargo sends cargo to crates.io.
An upstream that answers /pkgproxy/v1/proxy-engines with a 404 of its
own gateway is too old to describe itself, if it answers as a pkgproxy
does where every pkgproxy from 0.12 on has a route: /files/v1/fetch,
asked without a query, is a 400 of plain text (http.Error’s). It is then
taken to run what 0.12 ran, at the default paths (chain.Baseline): apt,
snap, Go, Docker, LXD with its default image servers, and files. A server
that answers both with a 404, or anything but such a 400, is not a
pkgproxy, as far as can be told: a wrong URL, say, to some web server. It
is logged as such and followed as an upstream that cannot be reached
(below), and its answers, which name no proxy engine, are not taken as
answers.
An upstream older than the description differs from a newer one in two more ways, which nothing can tell from its answers. Its 404s are taken as they come, as its proxy engines’ answers, and Go caches them as negative answers for its TTL: a route it does not serve, as Baseline assumes it does, gives a negative answer that the origin would not have. And it does not mark what offline mode refuses (§27.4): its 503 counts against the downstream’s breaker of the origin, as an answer of an origin in trouble would, and enough of them leave that origin alone on the downstream too, for what the upstream has cached as well. Upgrade the upstream first.
An upstream that cannot be reached is a different matter. The downstream
uses the description it saved in -cache-dir
(upstream-pkgproxy.json, with the upstream’s URL, so that a changed
option does not reuse another upstream’s), and with nothing saved, it
takes the upstream to run what it runs itself. Every request then goes to
the upstream and fails, until it answers: falling back to the origins
would hide an outage of the upstream, which the administrator chose.
Offline mode (§21) reads nothing: while it is on, the description is not
read, and the saved one, or the downstream’s own, is followed. With an
upstream, -auto-offline network follows the route to the upstream’s
address rather than the default route, so a downstream whose upstream is on
its link, with no way to the internet, stays online. A name is resolved
again at every check, and its last answer kept while it does not resolve;
until it has resolved once, the default route decides.
/pkgproxy/v1/info on the stats socket reports the upstream as
upstream_pkgproxy: its URL, where the description came from
(upstream, saved, baseline, newer, loop or unreachable), and the target of each
proxy engine.
An upstream that is the daemon itself, or that sends requests back to it, is §27.5’s matter.
27.3 Each proxy engine through the upstream
The proxy engines keep their origins. chain.Transport, inside
cmd.upstream and so inside offline mode’s transport, changes the URL of a
request to an origin the upstream serves into the same request to the
upstream, at that proxy engine’s path, with the upstream as Host:
https://proxy.golang.org/golang.org/x/mod/@v/list becomes
http://upstream:3142/go/golang.org/x/mod/@v/list. Everything that keys on
the origin sees the origin: the breakers of store-overloaded, the
statistics, the logs. A path with a . or .. element is refused rather
than sent, so nothing leaves the path of its proxy engine. Requests to the
upstream use no HTTP proxy: it is where the administrator said it is.
- snap: the store’s API goes to the upstream’s root. A file’s pending
record (§11.3) keeps the URL the store’s answer named, and how it is
fetched is decided when it is, by what the upstream runs then
(
snapproxy.Handler.DownloadContext, achain.Redirect): while the upstream runs snap, it is asked for the file by its hash,/v2/download/sha3-384/<hex>, whatever URL the answer named, as an answer the upstream serves from its cache names whatever host its first client knew it by; otherwise, the URL is fetched as it is. A record of the store’s own URL, kept from before the upstream was set, falls back to it when the upstream’s snap proxy engine answers 404. A record of a pkgproxy’s download URL, which an answer of the upstream’s names, is counted against the store’s downloads (§21’sdownloadclass), the origin it stands for. A refresh the upstream answered from its own records (X-Pkgproxy-Synthesised) is not recorded as the store’s. - Go: the module proxy’s requests go under the upstream’s path. The
downstream checks the modules it caches against the checksum database
itself, as
-go-sumdbsays,sum.golang.orgby default (§16.3): it asks the database directly, as a pkgproxy without an upstream does, so that a module changed on the way, or by the upstream, is caught. A lookup that fails, the database being unreachable from the downstream included, is a gap that is logged, and the module is served all the same. - Docker and LXD: under the upstream’s root and image server paths. An LXD image server the upstream does not serve by the same name is fetched from its origin. The downstream never uses the upstream’s HTTPS listener for LXD.
- apt: the forward proxy uses the upstream as its HTTP proxy for
http://URLs (dialguard.Guard.Proxy), so the upstream caches what it fetches.CONNECTtunnels, forhttps://sources, still go to their hosts: neither tier can cache them. - files:
/files/v1/fetch?url=X&sha256=His asked of the upstream’s own/files/v1/fetch, with the same query, by achain.Redirectof the request to X: offline mode counts it against X’s host. The downstream checks the checksum again, records the origin the upstream names inX-Pkgproxy-Origin-URL, and passes on the upstream’s 403. Both tiers apply their limits, so the stricter wins: before it asks the upstream, the downstream applies its own-files-allow-host,-files-max-sizeand-files-allow-network, the last to the addresses X’s host resolves to on the downstream (a name that does not resolve there is left to the upstream, which resolves it itself). The private address policy, which would refuse an upstream on the LAN, is not applied to the hop to the upstream. It is applied to every URL a client names, or a redirect leads to, the upstream’s own included, when files are fetched from their origins. - files-volatile:
/files/v1/volatile?url=Xis asked of the upstream’s own/files/v1/volatile, with the same query, by achain.Redirectof the request to X, as files by checksum are, within the same limits of both tiers (-files-volatile-max-sizein place of-files-max-size), and itsSTALEand fetch time are passed on, so that a cloud image’sSHA256SUMSis served while its origin is down; see §24.8. An upstream that does not list files-volatile is skipped for the origin.
What a client sends for its origin goes to the upstream with its request: snapd’s macaroons and device sessions, a Docker client’s registry token. The hop is plain HTTP, as is the one from the clients, and is to be on a network trusted as much (DEPLOYMENT.md, “Two tiers”).
A 404 or 410 that no proxy engine of the upstream sent says nothing of what was asked for: it means that the upstream changed since its description was read. It is not passed on, and so never cached as a negative answer (Go and Cargo cache those); the request fails, and the description is read again. Upstreams too old to stamp their answers, which answer as a pkgproxy does (§27.2), are taken as they come.
27.4 Refusals pass through, breakers stay closed
What offline mode refuses, the 503, or the snap proxy engine’s 424, is
marked X-Pkgproxy-Refused: <mode>. To a downstream, such an answer says
that one object is not in the upstream’s cache, not that the upstream is in
trouble: its breaker counts it as an answer, so that one object the
upstream lacks cannot cut the downstream off from everything else it has.
Nor is it asked again: a Download’s or a Go module’s retries (§24.6) stop at a
refusal, whatever Retry-After it carries, rather than hold the client for
something the upstream will not have until it is online again.
The snap proxy engine serves its stale copy in place of a refusal, as it
does in place of a 5xx, and otherwise passes the refusal on, a 424 that
snapd does not retry.
Freshness adds up across the tiers: an entry the upstream serves for up to
its TTL is then served by the downstream for up to its own. pkgproxyctl cache invalidate forgets what one tier has, not what the other has.
27.5 Loops
The daemon refuses to start when the upstream is itself by its address: one that resolves to one of the machine’s own, loopback included, at a port the daemon listens on. Another pkgproxy on the same machine, at another port, is a fine upstream.
What the address cannot tell, a name that leads back through a NAT or a
proxy, say, the description does, once the daemon serves its own: it reads
the upstream’s again as soon as its listeners are up, and every time after.
An upstream whose instance is its own is a loop: the daemon logs it,
fetches every proxy engine from its origin (source loop), and keeps
reading the description, as it would.
Two pkgproxies that are each other’s upstream, or a longer ring, each
describe themselves truly, and only the requests tell. Every request a
pkgproxy sends its upstream, by chain.Transport or as apt’s HTTP proxy,
carries
X-Pkgproxy-Via: <instance>[, <instance>...]
the instances of the pkgproxies it went through, the nearest to the client
first: those of the request the pkgproxy is fetching for, then its own.
A pkgproxy answers a request whose X-Pkgproxy-Via has its own instance,
or more than proxyengines.MaxHops (8) of them, with a 508 Loop Detected
marked X-Pkgproxy-Loop: <instance>, before any proxy engine sees it
(proxyengines.LoopGuard). The pkgproxy that sent the request, on such an
answer, logs it once, and fetches that proxy engine from its origin until
it next reads the description, the request first, when it has no body to
send again. In a ring, the last pkgproxy before the one that sees itself
again becomes the one that fetches from the origin, and the others keep
fetching through it. A shared download carries the X-Pkgproxy-Via of the
request that started it.
27.6 What is not done
- Cargo goes through the upstream once cargo’s proxy engine (§26) is part of a release: the redirection takes its origin like the others’.
- The mount paths of the downstream’s own proxy engines are flags, not snap options (gitlab.com/zygoon/pkgproxy#148).