pkgproxy design: Go modules

The Go module proxy (the GOPROXY protocol) and checksum database verification. Part of the pkgproxy design, whose index says which file holds each § number.

• 16. Go module proxy (GOPROXY protocol)
  • 16.1 Why this is a good fit
  • 16.2 Shape: a third reverse-origin handler on the same listener
  • 16.3 Checksum database verification
  • 16.4 Client enablement

16. Go module proxy (GOPROXY protocol)

The Go side lives in internal/gomod (protocol parsing) and internal/gomodproxy (the handler); gitlab.com/zygoon/pkgproxy#13.

16.1 Why this is a good fit

CI systems that rebuild from a fresh disk image re-download every Go module from the internet on every build. The GOPROXY protocol (https://go.dev/ref/mod#goproxy-protocol) is a closed, five-endpoint GET-only API ($base/$module/@v/{list,$version.info,$version.mod, $version.zip} and $base/$module/@latest) with no query params and no required headers — even a static file server is a legal proxy. Crucially, .mod/.zip content for a given module@version is immutable by protocol: it must be byte-identical for all time, and clients pin it via go.sum plus the checksum database. So, like apt’s pool/** and snap’s downloads, versioned content is fetch-once-verify-cache-forever; only list/@latest/.info (which change as new versions get published) need a short TTL.

16.2 Shape: a third reverse-origin handler on the same listener

Like internal/snapproxy (§11.1) and unlike the apt forward proxy, the go client addresses the proxy URL as an origin — GOPROXY=http://host:3142/go — so internal/gomodproxy.Handler is a reverse-origin handler that chains to an upstream module proxy (-go-upstream, default https://proxy.golang.org) on a cache miss: a proxy-of-proxy, the same model corporate proxies use. Direct VCS fetching (git tags/commits, pseudo-versions, zip creation) is explicitly deferred — the chained upstream covers all public modules, and it’s a much larger job.

internal/gateway routes a request whose path starts with -go-base-path (default /go) here. It cannot collide with snap’s fixed /v2/... paths: a module path’s first element must contain a dot (it names a host).

Cache layout mirrors apt/snap: a third fslocal.Store rooted at <cache-dir>/go, sibling to apt and snap — see §11.4’s “each gets its own top-level directory” precedent.

A 404 from the upstream (the go command probing a module boundary, such as rsc.io before rsc.io/quote) is not cached, a known minor gap.

16.3 Checksum database verification

Fetched .mod/.zip are hash-checked and, by default, verified against the checksum database (sum.golang.org, via golang.org/x/mod/sumdb with the public key embedded exactly as the go command embeds it) before being admitted to the cache. A mismatch refuses to cache and returns 502 to the client — this matters more than apt’s InRelease trust-root gap (§4.2) because a poisoned cached zip would break every future build until manually deleted, not just one client’s one request. -go-sumdb=off disables verification entirely; a sumdb lookup failure (unreachable, or the version not yet recorded) is logged as a gap and cached anyway — the same stance apt’s InRelease trust root takes, and the client’s own go.sum/sumdb check remains the backstop either way.

The checksum database’s hash of a .mod file is not a plain sha256 of its bytes but dirhash.Hash1 over a single file named go.mod (golang.org/x/mod/sumdb/dirhash), as the go command’s own goModSum computes it.

16.4 Client enablement

pkgproxyctl setup go (and the daemon’s startup banner) prints copy-paste instructions. Unlike apt (an apt.conf.d drop-in) and snapd (a systemd drop-in), the go command has no system-wide config file; go env -w writes the invoking user’s own $GOENV. So the guidance offers go env -w GOPROXY=http://host:3142/go,direct for a user or export GOPROXY=... for one job. The trailing ,direct keeps modules pkgproxy can’t serve (private modules once GOPRIVATE excludes them) reachable from their source. GOSUMDB is left at its default: pkgproxy’s own verification (§16.3) adds to the client’s, it does not replace it. The piped setup go -undo runs go env -u GOPROXY only when the file go env GOENV names exists: without one, as for a user who never ran go env -w, go env -u fails although there is nothing to undo, so undoing before setting up, or twice, changes nothing and succeeds.