Go module proxy
go modules for users for administrators for CI operators
How GOPROXY points the go command at pkgproxy under /go, which module proxy requests are kept forever and which are rechecked, and how modules are checked against the checksum database before they are cached.
On the map: go modules
01 What this topic covers
The go command fetches modules from pkgproxy under /go. Module
files are checked against the checksum database and kept forever; version
lists are rechecked.
The Go module proxy is the part of pkgproxy that the go command fetches modules from, under the path /go. First, pkgproxyctl setup go points GOPROXY there. Then you see the module proxy's five kinds of request, and which of them are kept forever and which are rechecked. Last, you see how each module is checked against the checksum database before pkgproxy keeps it.
02 GOPROXY names /go
pkgproxyctl setup go | sh writes GOPROXY=http://host:3142/go,direct
where go env -w would, Go installed or not; export
GOPROXY=… does it for one CI job. ,direct is the
fallback on a 404 or 410.
The go command fetches modules from the proxy that GOPROXY names. pkgproxyctl setup go sets it for the current user to pkgproxy's address with the path /go, followed by ,direct. The go command then asks pkgproxy first, and fetches a module straight from its source only when pkgproxy answers that it does not have it. Modules that GOPRIVATE matches never go to a proxy at all. GOSUMDB is left as it is, so the go command still checks every module itself.
03 Five requests, two kinds
@v/list, @latest and .info are
revalidated after -go-revalidate-ttl (default: -revalidate-ttl, 5m);
.mod and .zip are immutable. The upstream is
-go-upstream, https://proxy.golang.org by default.
The module proxy protocol has five kinds of request, all plain GETs. The version list, the latest version and a version's info change as new versions are published, so pkgproxy serves them from the cache for five minutes, the go-revalidate-ttl flag, then asks again, and keeps serving the old answer if the upstream fails. A version's go.mod and zip never change, by the protocol's rules, so they are fetched once and kept forever. Misses go to proxy.golang.org, or to the go-upstream flag.
04 Checked against the checksum database
A .mod or .zip is committed only after its
h1: hash matches sum.golang.org. A mismatch is a
502 and nothing is cached; a failed lookup is logged and cached.
-go-sumdb off skips the check.
Before a go.mod or a zip is kept, pkgproxy hashes it the way the go command does and looks the module version up in the checksum database, sum.golang.org by default. If the database records a different hash, the file is not cached and the client gets a 502. If the database cannot be reached, or does not list the version yet, the file is cached anyway and the gap is logged; the client's own check still applies. The go-sumdb flag set to off turns this check off.
05 Where to go next for administrators in place of the general version
Next: Settings, for the go flags and their defaults; The store, for where the modules are kept.
Your go builds now fetch each module from the internet once, and every later build gets it from pkgproxy. To see the go flags, their defaults and how to set them, watch Settings. To see where the modules are kept on disk, and why nothing there is deleted on its own, watch The store.
06 Where to go next for CI operators in place of the general version
Next: pkgproxy for CI, for many machines behind one cache; Offline mode, for builds that see only what is cached.
Your go builds now fetch each module from the internet once, and every later build gets it from pkgproxy. To put fresh CI machines behind one cache, so that only the first job downloads a module, watch pkgproxy for CI. To run builds that see only what is cached, watch Offline mode.
07 Where to go next general version
Next: Offline mode, for builds from the cache with no network; pkgproxyctl setup, for the other clients.
Your go builds now fetch each module from the internet once, and every later build gets it from pkgproxy. To build from what is cached, with no network, watch Offline mode. To point apt, snapd, Docker and LXD at the same cache, watch pkgproxyctl setup.
Read more
Requests under /go
Every request is a plain GET (or HEAD) on the main port, under -go-base-path, /go by default. Module paths are case-encoded as the protocol says: each capital letter is written as ! and the small letter, so github.com/BurntSushi/toml is asked for as github.com/!burnt!sushi/toml.
| Path | Answer | Cache policy |
|---|---|---|
/go/MODULE/@v/list | the known versions, one per line | revalidated after -go-revalidate-ttl |
/go/MODULE/@latest | the latest version, as JSON | revalidated after -go-revalidate-ttl |
/go/MODULE/@v/VERSION.info | a version's time and name, as JSON | revalidated after -go-revalidate-ttl |
/go/MODULE/@v/VERSION.mod | the version's go.mod | kept forever, after the checksum database check |
/go/MODULE/@v/VERSION.zip | the version's source, as a zip | kept forever, after the checksum database check |
any other path under /go | 404 | not cached |
A .mod or .zip must name a canonical version, such as v1.2.3 rather than v1.2; anything else is a 404. The checksum database's own proxy paths, /go/sumdb/…, are not served: they get a 404, and the go command then asks sum.golang.org directly, as it does with no proxy.
Status codes
| Situation | Status the go command sees |
|---|---|
a method other than GET or HEAD | 405 |
| a path that is not one of the five requests | 404 |
.mod or .zip cached | 200 from the cache, also served to Range requests |
any path, the upstream answers 404 or 410 | the same 404 or 410, so ,direct takes over; remembered for -go-revalidate-ttl |
any path whose 404 or 410 is remembered | the same status, without asking the upstream; past the TTL, also while the upstream fails or cannot be reached |
.mod or .zip whose hash differs from the checksum database, or the database fails its own integrity check | 502, nothing cached |
.mod or .zip, upstream unreachable | 502 |
| list, latest or info, cached and within the TTL | 200 from the cache |
list, latest or info, past the TTL, upstream fails or answers an error other than 404 or 410 | 200 with the cached copy, and a warning in the log |
list, latest or info, past the TTL, upstream answers 404 or 410 | that status; the cached copy is dropped |
| list, latest or info, never cached, upstream answers an error | the upstream's status, or 502 when it cannot be reached |
Remembered "not found"
The go command looks for the module that holds a package by asking for several paths, such as golang.org/x/tools/gopls, golang.org/x/tools and golang.org/x, and expects 404 or 410 for those that are not modules. pkgproxy remembers each such answer for -go-revalidate-ttl, so a build asks the upstream only for what changed, and offline it gets the same answers, so go install example.com/tool@latest works for a tool installed through pkgproxy before.
A version asked for the moment it is tagged may be "not found" for up to -go-revalidate-ttl, as proxy.golang.org may not have it yet. GOPROXY=direct for that one command gets it.
The checksum database check
| Lookup result | What happens to the .mod or .zip |
|---|---|
the database lists the same h1: hash | cached |
| the database lists a different hash | not cached, 502 |
| the database's signature or tree does not verify | not cached, 502 |
| the database cannot be reached, or does not list the version yet | cached unchecked, a warning in the log |
-go-sumdb off | cached unchecked, no lookup |
Only sum.golang.org is supported, with the key the go command itself carries; any other -go-sumdb value stops pkgproxy at startup. The database is asked directly over HTTPS, and its tree head and lookups are kept under go/sumdb in the cache directory.
for administrators for CI operators
Settings
| Option | Environment variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-go-upstream | PKGPROXY_GO_UPSTREAM | flag or environment only | https://proxy.golang.org | the module proxy misses are fetched from |
-go-base-path | PKGPROXY_GO_BASE_PATH | flag or environment only | /go | the URL path of the module proxy, part of GOPROXY |
-go-sumdb | PKGPROXY_GO_SUMDB | flag or environment only | sum.golang.org | the checksum database, or off |
-go-revalidate-ttl | PKGPROXY_GO_REVALIDATE_TTL | go-revalidate-ttl | -revalidate-ttl, 5m | how long list, latest and info answers, and "not found" answers, are served before they are checked again |
All settings are in Settings.
On the client
| Variable | Set by pkgproxyctl setup go | Meaning |
|---|---|---|
GOPROXY | http://HOST:3142/go,direct, in the user's go env file, where go env -w would write it, whether go is installed yet or not | ask pkgproxy first; on a 404 or 410, fetch from the module's source |
GOSUMDB | left as it is | the go command still checks every module itself |
GOPRIVATE | left as it is | modules it matches never go to any proxy |
pkgproxyctl setup go | sh needs no go command, so Go can be set up first and installed after, and fetches its first module through pkgproxy. It fails only where go keeps no settings (GOENV=off, no HOME); then set GOPROXY in the environment. pkgproxyctl setup go -undo removes the GOPROXY line from that file, as go env -u GOPROXY does, whatever the user had set, not only pkgproxy's. For one CI job, export GOPROXY=http://HOST:3142/go,direct does the same without touching the user's settings. See Setting up clients.
In the documentation
- Set up a client on the same machine
- Settings: the Go upstream and checksum database
Roads from here
CORE · the path every side shares
- Immutable or revalidated — kept forever or rechecked
- Offline mode — building with what is cached
- The store — where the modules end up
SETUP AND THE COMMAND LINE
- Setting up clients — setup go and the other clients
OPERATIONS
- pkgproxy for CI — fresh CI machines behind one cache
- Settings — the go flags and their defaults