Go module proxy

go modules for users for administrators for CI operators

How GOPROXY points the go command at pkgproxy under /go, which module proxy requests are kept forever and which are rechecked, and how modules are checked against the checksum database before they are cached.

On the map: go modules

▶ Watch this deck — 7 scenes, ~4 min

01 What this topic covers

ABOUT THIS TOPIC Go module proxy where the go command fetches modules, under /go 1 GOPROXY names /go 2 Five requests, two kinds 3 Checked against the checksum database
The topic title, Go module proxy, and its three scenes in order: GOPROXY names /go, five requests of two kinds, the checksum database check

The go command fetches modules from pkgproxy under /go. Module files are checked against the checksum database and kept forever; version lists are rechecked.

The Go module proxy is the part of pkgproxy that the go command fetches modules from, under the path /go. First, pkgproxyctl setup go points GOPROXY there. Then you see the module proxy's five kinds of request, and which of them are kept forever and which are rechecked. Last, you see how each module is checked against the checksum database before pkgproxy keeps it.

02 GOPROXY names /go

GOPROXY $ pkgproxyctl setup go | sh # go env -w GOPROXY=http://pkgproxy.local:3142/go,direct go /go/… first pkgproxy proxy.golang.org direct: only after a 404 or 410 from pkgproxy module's source GOPRIVATE modules: never sent to a proxy GOSUMDB unchanged: go still checks every module
pkgproxyctl setup go sets GOPROXY to pkgproxy's /go path followed by direct; the go command asks pkgproxy, which fetches from proxy.golang.org, and goes to the module's source directly only on a 404 or 410

pkgproxyctl setup go | sh writes GOPROXY=http://host:3142/go,direct where go env -w would, Go installed or not; export GOPROXY=… does it for one CI job. ,direct is the fallback on a 404 or 410.

The go command fetches modules from the proxy that GOPROXY names. pkgproxyctl setup go sets it for the current user to pkgproxy's address with the path /go, followed by ,direct. The go command then asks pkgproxy first, and fetches a module straight from its source only when pkgproxy answers that it does not have it. Modules that GOPRIVATE matches never go to a proxy at all. GOSUMDB is left as it is, so the go command still checks every module itself.

03 Five requests, two kinds

FIVE REQUESTS GET /go/golang.org/x/mod/… …/@v/list …/@latest …/@v/v0.20.0.info rechecked served for 5 minutes, then asked again old answer kept if upstream fails …/@v/v0.20.0.mod …/@v/v0.20.0.zip kept forever fetched once, never rechecked misses go to proxy.golang.org flags: -go-revalidate-ttl 5m · -go-upstream
The five Go module proxy requests: list, latest and info are rechecked after five minutes; mod and zip are kept forever; misses go to proxy.golang.org

@v/list, @latest and .info are revalidated after -go-revalidate-ttl (default: -revalidate-ttl, 5m); .mod and .zip are immutable. The upstream is -go-upstream, https://proxy.golang.org by default.

The module proxy protocol has five kinds of request, all plain GETs. The version list, the latest version and a version's info change as new versions are published, so pkgproxy serves them from the cache for five minutes, the go-revalidate-ttl flag, then asks again, and keeps serving the old answer if the upstream fails. A version's go.mod and zip never change, by the protocol's rules, so they are fetched once and kept forever. Misses go to proxy.golang.org, or to the go-upstream flag.

04 Checked against the checksum database

CHECKSUM DATABASE mod@v0.20.0.zip h1:Qx7…= hashed as go does sum.golang.org the recorded hash same hash cached forever different hash not cached · the client gets 502 no answer, or not listed cached, and the gap logged · go still checks it turn the check off with -go-sumdb off
A fetched module zip is hashed and compared with the checksum database's record: a match is cached forever, a mismatch is refused with 502, and an unreachable database means it is cached and the gap logged

A .mod or .zip is committed only after its h1: hash matches sum.golang.org. A mismatch is a 502 and nothing is cached; a failed lookup is logged and cached. -go-sumdb off skips the check.

Before a go.mod or a zip is kept, pkgproxy hashes it the way the go command does and looks the module version up in the checksum database, sum.golang.org by default. If the database records a different hash, the file is not cached and the client gets a 502. If the database cannot be reached, or does not list the version yet, the file is cached anyway and the gap is logged; the client's own check still applies. The go-sumdb flag set to off turns this check off.

05 Where to go next for administrators in place of the general version

WHERE NEXT your go builds fetch each module from the internet once, and every later build gets it from pkgproxy you are here Go module proxy Settings the go flags, their defaults and how to set them The store where the modules are kept on disk
You are here, Go module proxy; next: Settings, The store

Next: Settings, for the go flags and their defaults; The store, for where the modules are kept.

Your go builds now fetch each module from the internet once, and every later build gets it from pkgproxy. To see the go flags, their defaults and how to set them, watch Settings. To see where the modules are kept on disk, and why nothing there is deleted on its own, watch The store.

06 Where to go next for CI operators in place of the general version

WHERE NEXT your go builds fetch each module from the internet once, and every later build gets it from pkgproxy you are here Go module proxy pkgproxy for CI fresh CI machines behind one cache Offline mode builds that see only what is cached
You are here, Go module proxy; next: pkgproxy for CI, Offline mode

Next: pkgproxy for CI, for many machines behind one cache; Offline mode, for builds that see only what is cached.

Your go builds now fetch each module from the internet once, and every later build gets it from pkgproxy. To put fresh CI machines behind one cache, so that only the first job downloads a module, watch pkgproxy for CI. To run builds that see only what is cached, watch Offline mode.

07 Where to go next general version

WHERE NEXT your go builds fetch each module from the internet once, and every later build gets it from pkgproxy you are here Go module proxy Offline mode builds answered from the cache with no network pkgproxyctl setup apt, snapd, Docker and LXD on the same cache
You are here, Go module proxy; next: Offline mode, pkgproxyctl setup

Next: Offline mode, for builds from the cache with no network; pkgproxyctl setup, for the other clients.

Your go builds now fetch each module from the internet once, and every later build gets it from pkgproxy. To build from what is cached, with no network, watch Offline mode. To point apt, snapd, Docker and LXD at the same cache, watch pkgproxyctl setup.

Read more

Requests under /go

Every request is a plain GET (or HEAD) on the main port, under -go-base-path, /go by default. Module paths are case-encoded as the protocol says: each capital letter is written as ! and the small letter, so github.com/BurntSushi/toml is asked for as github.com/!burnt!sushi/toml.

PathAnswerCache policy
/go/MODULE/@v/listthe known versions, one per linerevalidated after -go-revalidate-ttl
/go/MODULE/@latestthe latest version, as JSONrevalidated after -go-revalidate-ttl
/go/MODULE/@v/VERSION.infoa version's time and name, as JSONrevalidated after -go-revalidate-ttl
/go/MODULE/@v/VERSION.modthe version's go.modkept forever, after the checksum database check
/go/MODULE/@v/VERSION.zipthe version's source, as a zipkept forever, after the checksum database check
any other path under /go404not cached

A .mod or .zip must name a canonical version, such as v1.2.3 rather than v1.2; anything else is a 404. The checksum database's own proxy paths, /go/sumdb/…, are not served: they get a 404, and the go command then asks sum.golang.org directly, as it does with no proxy.

Status codes

SituationStatus the go command sees
a method other than GET or HEAD405
a path that is not one of the five requests404
.mod or .zip cached200 from the cache, also served to Range requests
any path, the upstream answers 404 or 410the same 404 or 410, so ,direct takes over; remembered for -go-revalidate-ttl
any path whose 404 or 410 is rememberedthe same status, without asking the upstream; past the TTL, also while the upstream fails or cannot be reached
.mod or .zip whose hash differs from the checksum database, or the database fails its own integrity check502, nothing cached
.mod or .zip, upstream unreachable502
list, latest or info, cached and within the TTL200 from the cache
list, latest or info, past the TTL, upstream fails or answers an error other than 404 or 410200 with the cached copy, and a warning in the log
list, latest or info, past the TTL, upstream answers 404 or 410that status; the cached copy is dropped
list, latest or info, never cached, upstream answers an errorthe upstream's status, or 502 when it cannot be reached

Remembered "not found"

The go command looks for the module that holds a package by asking for several paths, such as golang.org/x/tools/gopls, golang.org/x/tools and golang.org/x, and expects 404 or 410 for those that are not modules. pkgproxy remembers each such answer for -go-revalidate-ttl, so a build asks the upstream only for what changed, and offline it gets the same answers, so go install example.com/tool@latest works for a tool installed through pkgproxy before.

A version asked for the moment it is tagged may be "not found" for up to -go-revalidate-ttl, as proxy.golang.org may not have it yet. GOPROXY=direct for that one command gets it.

The checksum database check

Lookup resultWhat happens to the .mod or .zip
the database lists the same h1: hashcached
the database lists a different hashnot cached, 502
the database's signature or tree does not verifynot cached, 502
the database cannot be reached, or does not list the version yetcached unchecked, a warning in the log
-go-sumdb offcached unchecked, no lookup

Only sum.golang.org is supported, with the key the go command itself carries; any other -go-sumdb value stops pkgproxy at startup. The database is asked directly over HTTPS, and its tree head and lookups are kept under go/sumdb in the cache directory.

for administrators for CI operators

Settings

OptionEnvironment variableSnap optionDefaultMeaning
-go-upstreamPKGPROXY_GO_UPSTREAMflag or environment onlyhttps://proxy.golang.orgthe module proxy misses are fetched from
-go-base-pathPKGPROXY_GO_BASE_PATHflag or environment only/gothe URL path of the module proxy, part of GOPROXY
-go-sumdbPKGPROXY_GO_SUMDBflag or environment onlysum.golang.orgthe checksum database, or off
-go-revalidate-ttlPKGPROXY_GO_REVALIDATE_TTLgo-revalidate-ttl-revalidate-ttl, 5mhow long list, latest and info answers, and "not found" answers, are served before they are checked again

All settings are in Settings.

On the client

VariableSet by pkgproxyctl setup goMeaning
GOPROXYhttp://HOST:3142/go,direct, in the user's go env file, where go env -w would write it, whether go is installed yet or notask pkgproxy first; on a 404 or 410, fetch from the module's source
GOSUMDBleft as it isthe go command still checks every module itself
GOPRIVATEleft as it ismodules it matches never go to any proxy

pkgproxyctl setup go | sh needs no go command, so Go can be set up first and installed after, and fetches its first module through pkgproxy. It fails only where go keeps no settings (GOENV=off, no HOME); then set GOPROXY in the environment. pkgproxyctl setup go -undo removes the GOPROXY line from that file, as go env -u GOPROXY does, whatever the user had set, not only pkgproxy's. For one CI job, export GOPROXY=http://HOST:3142/go,direct does the same without touching the user's settings. See Setting up clients.

In the documentation

Roads from here

CORE · the path every side shares

SETUP AND THE COMMAND LINE

OPERATIONS