Contents · An upstream pkgproxy

pkgproxy design: an upstream pkgproxy

Two tiers: a pkgproxy that fetches everything through another. Part of the pkgproxy design, whose index says which file holds each § number.

27. An upstream pkgproxy

-upstream-pkgproxy http://host:port (snap set pkgproxy upstream-pkgproxy=...) makes a pkgproxy, the downstream, send what its proxy engines fetch to another pkgproxy, the upstream, instead of the origins (gitlab.com/zygoon/pkgproxy#147). The case it is made for is CI: an infrastructure pkgproxy runs next to the runners, in offline-mode=store-overloaded, and the pkgproxy under test in the spread suite uses it as its upstream. What the upstream has cached is then a LAN away, and it is served while an origin is in trouble, as far as the upstream has it. Unset, a pkgproxy fetches from the origins, as it always did.

The option names one upstream for all the proxy engines. It replaces their own upstream options, so -snap-store-url, -go-upstream and -docker-upstream are refused with it. -go-sumdb is not an upstream to fetch from but what checks what is fetched, and it stays the downstream’s own (§27.3). A downstream serves the proxy engines its own settings turn on, at its own paths: what the upstream runs decides only where each is fetched from.

The terms: apt, snap, Go, Docker, LXD, files, files-volatile and cargo are the proxy engines of a pkgproxy.

27.1 What each proxy engine says: /pkgproxy/v1/proxy-engines

Every pkgproxy describes itself at GET /pkgproxy/v1/proxy-engines, on its main listener (internal/proxyengines). It is the only route of pkgproxy’s own API there; the others stay on the stats and admin sockets (§13.1). It says what is served where, and nothing of the daemon’s version or state, as anyone who can use pkgproxy can read it.

{"schema":1,"instance":"7b2f...","proxy-engines":{
  "apt":{"enabled":true,"kind":"forward"},
  "snap":{"enabled":true,"kind":"root"},
  "go":{"enabled":true,"kind":"path","path":"/go"},
  "docker":{"enabled":false,"kind":"root"},
  "lxd":{"enabled":true,"kind":"path","remotes":{"pkgproxy-ubuntu":"/lxd/pkgproxy-ubuntu"}},
  "files":{"enabled":true,"kind":"path","path":"/files/v1/fetch"},
  "files-volatile":{"enabled":true,"kind":"path","path":"/files/v1/volatile"}}}
  • schema is the format’s version; a downstream does not use another. A newer one is that of a newer upstream: the downstream says so in its log, follows what the upstream said before in a format it reads (saved or in memory), or else what it runs itself, as for an upstream that cannot be reached, and reads the description again every ten minutes, not sooner (source newer). Answers without X-Pkgproxy-Engine are no answers from it, as from any upstream that describes itself.
  • instance is random for each run of the daemon: a downstream whose upstream says its own instance is talking to itself (§27.5).
  • kind says how a downstream reaches a proxy engine: forward, as an HTTP proxy (apt); root, at the root (snap and Docker, whose clients need it there); path, under path, or, for LXD, each image server at its own path in remotes.

The downstream reads only absolute paths of the upstream’s own (proxyengines.ValidPath: no other host, no . or ..), and leaves out what it does not know, a kind or a proxy engine newer than itself: a newer upstream reads as an older one.

Every answer of a proxy engine on the main listener carries X-Pkgproxy-Engine: <name>, which the gateway sets before the proxy engine runs; the gateway’s own answers, such as its 404 for a path no proxy engine serves, carry none.

27.2 Following the description

internal/chain reads the description when the daemon starts, and every ten minutes after. A request to the upstream that fails, or that is answered without X-Pkgproxy-Engine, asks for it again, at most once a minute. Each reading replaces what is followed in one swap, which every request reads once, so a proxy engine never changes its target in the middle of a request. What the upstream says decides, for each proxy engine and each of LXD’s image servers:

  • one the upstream runs is fetched through it, at its path;
  • one the upstream has turned off is refused;
  • one the upstream does not list is fetched from its origin. That is how a downstream newer than its upstream works: an upstream from before cargo sends cargo to crates.io.

An upstream that answers /pkgproxy/v1/proxy-engines with a 404 of its own gateway is too old to describe itself, if it answers as a pkgproxy does where every pkgproxy from 0.12 on has a route: /files/v1/fetch, asked without a query, is a 400 of plain text (http.Error’s). It is then taken to run what 0.12 ran, at the default paths (chain.Baseline): apt, snap, Go, Docker, LXD with its default image servers, and files. A server that answers both with a 404, or anything but such a 400, is not a pkgproxy, as far as can be told: a wrong URL, say, to some web server. It is logged as such and followed as an upstream that cannot be reached (below), and its answers, which name no proxy engine, are not taken as answers.

An upstream older than the description differs from a newer one in two more ways, which nothing can tell from its answers. Its 404s are taken as they come, as its proxy engines’ answers, and Go caches them as negative answers for its TTL: a route it does not serve, as Baseline assumes it does, gives a negative answer that the origin would not have. And it does not mark what offline mode refuses (§27.4): its 503 counts against the downstream’s breaker of the origin, as an answer of an origin in trouble would, and enough of them leave that origin alone on the downstream too, for what the upstream has cached as well. Upgrade the upstream first.

An upstream that cannot be reached is a different matter. The downstream uses the description it saved in -cache-dir (upstream-pkgproxy.json, with the upstream’s URL, so that a changed option does not reuse another upstream’s), and with nothing saved, it takes the upstream to run what it runs itself. Every request then goes to the upstream and fails, until it answers: falling back to the origins would hide an outage of the upstream, which the administrator chose.

Offline mode (§21) reads nothing: while it is on, the description is not read, and the saved one, or the downstream’s own, is followed. With an upstream, -auto-offline network follows the route to the upstream’s address rather than the default route, so a downstream whose upstream is on its link, with no way to the internet, stays online. A name is resolved again at every check, and its last answer kept while it does not resolve; until it has resolved once, the default route decides.

/pkgproxy/v1/info on the stats socket reports the upstream as upstream_pkgproxy: its URL, where the description came from (upstream, saved, baseline, newer, loop or unreachable), and the target of each proxy engine.

An upstream that is the daemon itself, or that sends requests back to it, is §27.5’s matter.

27.3 Each proxy engine through the upstream

The proxy engines keep their origins. chain.Transport, inside cmd.upstream and so inside offline mode’s transport, changes the URL of a request to an origin the upstream serves into the same request to the upstream, at that proxy engine’s path, with the upstream as Host: https://proxy.golang.org/golang.org/x/mod/@v/list becomes http://upstream:3142/go/golang.org/x/mod/@v/list. Everything that keys on the origin sees the origin: the breakers of store-overloaded, the statistics, the logs. A path with a . or .. element is refused rather than sent, so nothing leaves the path of its proxy engine. Requests to the upstream use no HTTP proxy: it is where the administrator said it is.

  • snap: the store’s API goes to the upstream’s root. A file’s pending record (§11.3) keeps the URL the store’s answer named, and how it is fetched is decided when it is, by what the upstream runs then (snapproxy.Handler.DownloadContext, a chain.Redirect): while the upstream runs snap, it is asked for the file by its hash, /v2/download/sha3-384/<hex>, whatever URL the answer named, as an answer the upstream serves from its cache names whatever host its first client knew it by; otherwise, the URL is fetched as it is. A record of the store’s own URL, kept from before the upstream was set, falls back to it when the upstream’s snap proxy engine answers 404. A record of a pkgproxy’s download URL, which an answer of the upstream’s names, is counted against the store’s downloads (§21’s download class), the origin it stands for. A refresh the upstream answered from its own records (X-Pkgproxy-Synthesised) is not recorded as the store’s.
  • Go: the module proxy’s requests go under the upstream’s path. The downstream checks the modules it caches against the checksum database itself, as -go-sumdb says, sum.golang.org by default (§16.3): it asks the database directly, as a pkgproxy without an upstream does, so that a module changed on the way, or by the upstream, is caught. A lookup that fails, the database being unreachable from the downstream included, is a gap that is logged, and the module is served all the same.
  • Docker and LXD: under the upstream’s root and image server paths. An LXD image server the upstream does not serve by the same name is fetched from its origin. The downstream never uses the upstream’s HTTPS listener for LXD.
  • apt: the forward proxy uses the upstream as its HTTP proxy for http:// URLs (dialguard.Guard.Proxy), so the upstream caches what it fetches. CONNECT tunnels, for https:// sources, still go to their hosts: neither tier can cache them.
  • files: /files/v1/fetch?url=X&sha256=H is asked of the upstream’s own /files/v1/fetch, with the same query, by a chain.Redirect of the request to X: offline mode counts it against X’s host. The downstream checks the checksum again, records the origin the upstream names in X-Pkgproxy-Origin-URL, and passes on the upstream’s 403. Both tiers apply their limits, so the stricter wins: before it asks the upstream, the downstream applies its own -files-allow-host, -files-max-size and -files-allow-network, the last to the addresses X’s host resolves to on the downstream (a name that does not resolve there is left to the upstream, which resolves it itself). The private address policy, which would refuse an upstream on the LAN, is not applied to the hop to the upstream. It is applied to every URL a client names, or a redirect leads to, the upstream’s own included, when files are fetched from their origins.
  • files-volatile: /files/v1/volatile?url=X is asked of the upstream’s own /files/v1/volatile, with the same query, by a chain.Redirect of the request to X, as files by checksum are, within the same limits of both tiers (-files-volatile-max-size in place of -files-max-size), and its STALE and fetch time are passed on, so that a cloud image’s SHA256SUMS is served while its origin is down; see §24.8. An upstream that does not list files-volatile is skipped for the origin.

What a client sends for its origin goes to the upstream with its request: snapd’s macaroons and device sessions, a Docker client’s registry token. The hop is plain HTTP, as is the one from the clients, and is to be on a network trusted as much (DEPLOYMENT.md, “Two tiers”).

A 404 or 410 that no proxy engine of the upstream sent says nothing of what was asked for: it means that the upstream changed since its description was read. It is not passed on, and so never cached as a negative answer (Go and Cargo cache those); the request fails, and the description is read again. Upstreams too old to stamp their answers, which answer as a pkgproxy does (§27.2), are taken as they come.

27.4 Refusals pass through, breakers stay closed

What offline mode refuses, the 503, or the snap proxy engine’s 424, is marked X-Pkgproxy-Refused: <mode>. To a downstream, such an answer says that one object is not in the upstream’s cache, not that the upstream is in trouble: its breaker counts it as an answer, so that one object the upstream lacks cannot cut the downstream off from everything else it has. Nor is it asked again: a Download’s or a Go module’s retries (§24.6) stop at a refusal, whatever Retry-After it carries, rather than hold the client for something the upstream will not have until it is online again. The snap proxy engine serves its stale copy in place of a refusal, as it does in place of a 5xx, and otherwise passes the refusal on, a 424 that snapd does not retry.

Freshness adds up across the tiers: an entry the upstream serves for up to its TTL is then served by the downstream for up to its own. pkgproxyctl cache invalidate forgets what one tier has, not what the other has.

27.5 Loops

The daemon refuses to start when the upstream is itself by its address: one that resolves to one of the machine’s own, loopback included, at a port the daemon listens on. Another pkgproxy on the same machine, at another port, is a fine upstream.

What the address cannot tell, a name that leads back through a NAT or a proxy, say, the description does, once the daemon serves its own: it reads the upstream’s again as soon as its listeners are up, and every time after. An upstream whose instance is its own is a loop: the daemon logs it, fetches every proxy engine from its origin (source loop), and keeps reading the description, as it would.

Two pkgproxies that are each other’s upstream, or a longer ring, each describe themselves truly, and only the requests tell. Every request a pkgproxy sends its upstream, by chain.Transport or as apt’s HTTP proxy, carries

X-Pkgproxy-Via: <instance>[, <instance>...]

the instances of the pkgproxies it went through, the nearest to the client first: those of the request the pkgproxy is fetching for, then its own. A pkgproxy answers a request whose X-Pkgproxy-Via has its own instance, or more than proxyengines.MaxHops (8) of them, with a 508 Loop Detected marked X-Pkgproxy-Loop: <instance>, before any proxy engine sees it (proxyengines.LoopGuard). The pkgproxy that sent the request, on such an answer, logs it once, and fetches that proxy engine from its origin until it next reads the description, the request first, when it has no body to send again. In a ring, the last pkgproxy before the one that sees itself again becomes the one that fetches from the origin, and the others keep fetching through it. A shared download carries the X-Pkgproxy-Via of the request that started it.

27.6 What is not done

  • Cargo goes through the upstream once cargo’s proxy engine (§26) is part of a release: the redirection takes its origin like the others’.
  • The mount paths of the downstream’s own proxy engines are flags, not snap options (gitlab.com/zygoon/pkgproxy#148).