A request in code

contributors for contributors

How one request passes through the code, from reqlog.Wrap and the gateway's dispatch order to a proxy engine's handler, the store, a fill and the upstream transport, guarded for apt, and through an upstream pkgproxy.

On the map: inside pkgproxy

▶ Watch this deck — 8 scenes, ~6 min

01 What this topic covers

ABOUT THIS TOPIC A request in code the packages one request passes through, from listener to log line 1 One request through the packages 2 Why the proxy engines never collide 3 How a handler records its outcome 4 One transport stack, three bases 5 Through an upstream pkgproxy 6 What the upstream status hears
About this topic, A request in code: one request through the packages, the dispatch order, recording the outcome, the upstream transport, through an upstream pkgproxy, what the upstream status hears

Which packages one request passes through, from the listener to the access log line, the upstream transport, an upstream pkgproxy and the upstream status.

A request passes through a few packages in a fixed order, from the listener that accepts it to the line it leaves in the access log. If you have not seen The source tree, start there. First you follow one apt download through the packages. Then you see the order in which the gateway picks a proxy engine, and how a handler records its outcome. Then comes the transport every upstream fetch goes through, what changes with an upstream-pkgproxy, and how the upstream status hears of each request.

02 One request through the packages

ONE REQUEST, PACKAGE BY PACKAGE GET http://…/pool/… reqlog.Wrap offline.Middleware LoopGuard gateway → apt proxy.Handler Resolver.Resolve(r.URL) Store.Stat: ErrNotFound fills.Serve offline · idletimeout · retry · log · dialguard Origin.Fetch, once, through the apt proxy engine's transport Store.Create → hash → Commit client, tailing the fill msg="handled request" … handler=apt outcome=fetched status=200
One apt pool request on a miss: http.Server, reqlog.Wrap, offline.Middleware, proxyengines.LoopGuard and gateway.Handler lead to proxy.Handler, which resolves the key and finds nothing in the store; a fill fetches through the apt proxy engine's guarded transport, writes the store and the client at once, commits after the hash check; reqlog writes the line with outcome=fetched

reqlog.Wrap → offline.Middleware → proxyengines.LoopGuard → gateway.Handler → proxy.Handler: Resolve, Store.Stat, then fills.Serve, whose fetch commits through Store.Create after the hash check.

Follow one apt download from the pool, on a cache miss. The server hands it to reqlog.Wrap, which will write the access log line, then to the offline middleware, the loop guard and the gateway, which picks the apt handler. The proxy package resolves the URL to a cache key and asks the store, which has nothing, so the request starts a fill. The fill fetches from the origin once, through the apt proxy engine's own transport. The bytes go into a store writer and to the client at the same time, and the entry is committed once its hash matches. Then Wrap writes the line: outcome fetched.

03 Why the proxy engines never collide

ROUTE, IN ORDER 1 CONNECT or r.URL.Host != "" apt absolute-form and authority-form targets carry a host; snapd's origin-form ones do not 2 /pkgproxy/v1/proxy-engines description 3 /go/… -go-base-path · first element has a dot Go modules 4 /lxd/… -lxd-base-path LXD images 5 /files/… files 6 /v2/<name>/manifests|blobs|referrers|tags/list /v2/_catalog Docker matched by shape at the root · 404 when Docker is off 7 /v2/… /api/v1/… /v1/… snap relay 8 anything else 404, not relayed
The gateway's route function as an ordered list of checks, each leading to a proxy engine: CONNECT or a host in the URL to apt; /pkgproxy/v1/proxy-engines to the description; /go to Go modules; /lxd to LXD; /files to files; registry shapes under /v2, including tag lists and the catalog, to Docker, or to 404 with Docker off; /v2, /api/v1 and /v1 to the snap relay; anything else to 404

route checks CONNECT or a non-empty r.URL.Host first (apt), then proxyengines.Path, the base paths, registryproxy.Matches and snap's prefixes. A disabled proxy engine's traffic gets 404 from the gateway. Each proxy engine's answer carries X-Pkgproxy-Engine.

The gateway checks a request in a fixed order. A CONNECT, or a request line with an absolute URL, is parsed with a host, and only apt sends those, so apt and snapd cannot meet. Everything else has no host and goes by path: /pkgproxy/v1/proxy-engines, then /go, /lxd and /files. A Go module path starts with a host name, which contains a dot. Docker refuses a mirror address with a path, so its requests are matched by shape: manifests, blobs, referrers, tag lists and the catalog. Then the snap store's prefixes, and whatever is left gets a 404, not a relay.

04 How a handler records its outcome

RECORDING THE OUTCOME reqlog.Wrap(handler, logger, observers...) the line is written when the handler returns gateway reqlog.SetHandler(r, "apt") a proxyengine reqlog.SetOutcome(r, reqlog.OutcomeHit) on every path that serves an answer · the last call wins reqlog.SetAttr(r, logkey.AssertionsFetched, fetched) a snap refresh: the line appearing means the fetch ahead is over 4xx or 5xx, no outcome recorded outcome=error assumed by Wrap
reqlog.Wrap around the listener's handler; inside, the gateway calls SetHandler, a proxy engine calls SetOutcome and SetAttr; when the handler returns Wrap writes the line, defaulting the outcome to error for an unlabelled 4xx or 5xx

Of the requests that share one fetch, only those that waited are joined; the starter is fetched or revalidated, as internal/flight reports.

Each listener's handler is wrapped by reqlog.Wrap, which hands the request a place to record what happened and writes the line when the handler returns. The gateway records the handler name, and each proxy engine records the outcome with SetOutcome, the last call winning, on every path that serves an answer. A failure nothing labelled becomes error. SetAttr adds attributes of a proxy engine's own: a snap refresh adds how many assertions it fetched ahead, so the line appearing tells a test that the fetch ahead is over.

05 One transport stack, three bases

THE UPSTREAM TRANSPORT func (cmd *pkgproxyCmd) upstream(base http.RoundTripper, …) http.RoundTripper offline.Switch.Transport refuses everything while offline idletimeout.Transport{Timeout: 60s} upstreamretry.Transport # a 408, sent again upstreamlog.Transport # who answered an error http.DefaultTransport snap, go, docker, lxd dialguard…Transport() apt: never this machine its CONNECT dial too filesproxy.NewTransport files: no private addresses
Nested boxes: offline.Switch.Transport around idletimeout.Transport around upstreamretry.Transport around upstreamlog.Transport around one of three bases: http.DefaultTransport for snap, Go, Docker and LXD, dialguard's transport for apt, which its CONNECT tunnels also dial through, and filesproxy's transport for the files proxy engine

offline.Switch.Transport → idletimeout.Transport → upstreamretry.Transport → upstreamlog.Transport → a base: http.DefaultTransport, dialguard.Guard.Transport (apt; CONNECT dials offlineSwitch.Dial(guard.Dialer().DialContext)) or filesproxy.NewTransport.

In the code, the snap, Go, Docker and LXD proxy engines fetch through one round tripper, built by upstream Transport in the daemon command. The offline switch's Transport is the outer layer, so a request is refused there before anything else while offline. Inside it come the idle timeout, a retry of a request answered 408, and a log of who answered an error, around Go's default transport. Apt and the files proxy engine build the same layers around a transport of their own. Apt's dialguard refuses to connect to this machine for a client, and the files proxy engine refuses private addresses.

06 Through an upstream pkgproxy

INTERNAL/CHAIN -upstream-pkgproxy http://cache.example:3142 → chain.Chain GET /pkgproxy/v1/proxy-engines at start, then every 10m: one swap proxyengines.Document no request changes target halfway offline.Switch.Transport → … → upstreamlog.Transport chain.Transport(base) https://proxy.golang.org/golang.org/x/mod/@v/list → http://cache.example:3142/go/golang.org/x/mod/@v/list breakers, stats and logs still key on the origin apt: chain.ForwardTransport(guard.Transport()) X-Pkgproxy-Via: a, b, a → LoopGuard: 508 Loop Detected
With -upstream-pkgproxy, chain.Chain reads the upstream's proxy-engines document at start and every ten minutes and swaps it in at once; chain.Transport, innermost under upstreamlog and the offline switch, rewrites a request to proxy.golang.org into one to the upstream's /go path; apt uses ForwardTransport; LoopGuard in front of the gateway answers 508 to a request that went round

chain.Chain reads proxyengines.Document from /pkgproxy/v1/proxy-engines every DefaultRefreshInterval (10m). Transport wraps each base; apt's is ForwardTransport; files use Redirect. Requests carry X-Pkgproxy-Via; LoopGuard answers 508 past MaxHops (8) or its own instance.

With the -upstream-pkgproxy flag set, the daemon builds a chain from internal/chain. It reads the upstream's description of its proxy engines when the daemon starts and every ten minutes, and swaps what it follows in one step, so no request changes its target halfway. Its Transport is the innermost layer, under the log and the offline switch: it turns a request to an origin into the same request to the upstream, at that proxy engine's path, while breakers, statistics and logs still see the origin. Apt gets a forward transport instead. In front of the gateway, the loop guard answers a request that has been here before with 508.

07 What the upstream status hears

INTERNAL/UPSTREAMS offline.Switch *upstreams.Recorder transport.RoundTrip every upstream request Observe(Observation) once the body ends Record(Event) breaker, mode, daemon proxy engines: Switch.ServedStale(u) Switch.Synthesised(u) snap refresh stale · synthesised a few additions under the recorder's lock · no I/O on the request path
The offline switch holds an upstreams.Recorder; its transport calls Observe for each upstream request once the body ends, and Record for breaker and mode changes; proxy engines serving a stale copy call ServedStale, snap's answers in the store's place count as synthesised; all under the recorder's lock, no I/O

offline.Switch.Recorder is an *upstreams.Recorder: Observe(Observation) per request, Record(Event) per breaker, mode or daemon change. ServedStale comes from proxy, gomodproxy, registryproxy, snapproxy and revalidate.Doc; snap's writeSynthesised calls Synthesised, its placeholders Recorder.Observe directly.

The upstream status is kept by an upstreams.Recorder, which the offline switch holds. The switch's transport already sees every upstream request, so it reports each one to the recorder with Observe once its body ends, and each breaker or mode change with Record. A proxy engine that answers from a stale copy, because asking failed or was refused, calls ServedStale on the switch, and the snap proxy engine counts its answers in the store's place, recorded refreshes and placeholders, as synthesised. Each is a few additions under the recorder's own lock, with no input or output on the request path.

08 Where to go next general version

WHERE NEXT you can name the package behind each step of a request, from the listener to the access log and the upstream you are here A request in code internal/fill the fill a cache miss starts The proxy engines' packages what each proxy engine's handler does Cache page code the requests watched live
Where next from A request in code: internal/fill, The proxy engines' packages, Cache page code

Next: internal/fill, for the fill a cache miss starts; The proxy engines' packages, for what each handler does; Cache page code, for the requests watched live.

You can now name the package behind each step of a request, from the listener to the access log and out to the upstream. To see the fill that a cache miss starts, watch internal/fill. To see what each proxy engine's handler does inside, watch The proxy engines' packages. To see how the cache page shows these requests as they happen, watch Cache page code.

Read more

The handler chain of each listener

Outermost first. Each wrapper takes only its own paths and passes everything else, CONNECT and absolute-URL requests included, to the next.

ListenerChain
-addr (or -socket-path), and -content-socket-pathreqlog.Wrap → withSetupPage (with -setup-page, the default) → withLxdCertDownload (when the LXD HTTPS listener is on) → offline.Middleware → proxyengines.LoopGuard → gateway.Handler
-lxd-tls-addr (or -lxd-tls-socket-path), and -lxd-tls-content-socket-pathreqlog.Wrap → withLxdAPI → offline.Middleware → proxyengines.LoopGuard → gateway.Handler
-stats-socket-pathstatsapi.Handler: GET /pkgproxy/v1/stats, /pkgproxy/v1/info, /pkgproxy/v1/upstreams, /pkgproxy/v1/upstreams/history, /pkgproxy/v1/upstreams/events; no access log
-admin-socket-pathadminapi.Handler: DELETE /pkgproxy/v1/cache, GET and PUT /pkgproxy/v1/offline; no access log

Both traffic listeners pass the cache page's liveHub.observe to reqlog.Wrap as an observer. proxyengines.LoopGuard answers a request whose X-Pkgproxy-Via names this daemon's instance, or more than proxyengines.MaxHops (8) instances, with 508 Loop Detected and X-Pkgproxy-Loop, before any proxy engine sees it. The snap proxy engine is wrapped once more, inside the gateway, by offline.MiddlewareAnswer(…, snapproxy.RefusedAnswer), because snapd retries a 503 for half a minute. offline.Middleware turns a 502 into a 503 when the fetch was refused for being offline.

Paths taken before the gateway

WrapperMatchHandler name
withSetupPageGET or HEAD of /, /setup/<tool>, /setup/all, /pkgproxy/status; for a local or trusted visitor only (setupPageCache.mayView), /pkgproxy/cache, /pkgproxy/cache/activity, /pkgproxy/cache/events, /pkgproxy/cache/upstreams, /pkgproxy/upstreams, /pkgproxy/upstreams/<host>setup-page
withLxdCertDownloadGET or HEAD of /pkgproxy-lxd-cert.pemlxd-cert
withLxdAPIorigin-form / or /1.0 and below (lxdapi.Matches)lxd-api

The gateway's dispatch order

gateway.Handler.route, first match wins. The name is recorded with reqlog.SetHandler and is the internal/stats type, so an access log line and a stats row name the traffic alike. Before a named handler runs, ServeHTTP sets X-Pkgproxy-Engine (proxyengines.HeaderEngine) to its proxy engine (engineNames; files-volatile for /files/v1/volatile), so that a downstream pkgproxy tells a proxy engine's 404 from the gateway's, which carries none.

#MatchHandlerName
1CONNECT, or a non-empty r.URL.Host (an absolute-URL request line)Aptapt
2ProxyEngines set and the path is /pkgproxy/v1/proxy-engines (proxyengines.Path)proxyengines.Handlernone
3Go set and the path is -go-base-path (default /go) or below itGogomod
4Lxd set and the path is -lxd-base-path (default /lxd) or below itLxdlxd
5Files set and the path is /files or below itFilesfiles
6registryproxy.Matches: /v2/, /v2/<name>/manifests/<ref>, /blobs/<digest>, /referrers/<digest>, /tags/list, /v2/_catalogDocker, or 404 when it is nildocker
7/v2, /api/v1 or /v1, or below one of themSnapsnap
8anything elsehttp.NotFoundnone

"Below" means the base path followed by /. Docker and Lxd are left a nil interface when -docker-upstream off or -lxd-remote off turned the proxy engine off; the Go and files proxy engines are always set.

Inside the apt handler

proxy.Handler.ServeHTTP:

RequestWhat happens
CONNECThandleConnect: a blind tunnel, port 443 only, dialled through the dial guard
neither GET nor HEAD405
not an http:// absolute URL400
cachekey.PolicyImmutablereqlog.SetObject(r, "apt", key), then serveCached; a pool miss streams through serveFilled
cachekey.PolicyRevalidatereqlog.SetObject, then serveCached with revalidation
cachekey.PolicyPassthroughOnlypassthrough: relayed, not cached

Upstream transports

Proxy engineRound tripper, outermost first
snap, Go, Docker, LXD (upstreamTransport)offline.Switch.Transport → idletimeout.Transport → upstreamretry.Transport → upstreamlog.Transport → http.DefaultTransport, or chain.Transport(http.DefaultTransport) with -upstream-pkgproxy
apt (newAptHandler)the same four layers → dialguard.Guard.Transport(), or chain.ForwardTransport(guard.Transport()) with -upstream-pkgproxy (aptTransport, and guard.Proxy), inside origin.NewHTTPFetcher
apt CONNECTofflineSwitch.Dial(guard.Dialer().DialContext)
files (newFilesHandler)the same four layers → filesproxy.NewTransport(AddressPolicy{Allow: -files-allow-network}), wrapped by chain.Transport with -upstream-pkgproxy

The Go proxy engine's checksum database client uses the same upstreamTransport as its module fetches; with -upstream-pkgproxy it is not redirected, as -go-sumdb stays each pkgproxy's own. upstreamretry.Transport sends a request answered 408 again, up to DefaultAttempts (3) times; upstreamlog.Transport logs who answered an error. The offline switch's transport tells Switch.Recorder, an *upstreams.Recorder, of every request with Observe once its body ends, and of breaker and mode changes with Record (§28.6).

ErrorPackageMeaning
idletimeout.ErrStalledidletimeoutNo progress for -upstream-idle-timeout (default 60s; 0 waits forever)
offline.ErrOfflineofflineRefused: offline mode, or the host's breaker is open
chain.ErrRefusedchainThe upstream pkgproxy has this proxy engine turned off
chain.ErrLoopchainThe upstream pkgproxy is this one, or sends its requests back to it
dialguard.ErrRefuseddialguardLoopback, unspecified, or this machine's own address at -addr or -lxd-tls-addr's port
filesproxy.ErrPrivateAddressfilesproxyA private address outside -files-allow-network

internal/chain: through an upstream pkgproxy

Built by setUpChain when -upstream-pkgproxy is set, before the proxy engines' constructors (§27).

NameWhat it is
chain.ChainReads the upstream's proxyengines.Document at start, every DefaultRefreshInterval (10m), and at most every DefaultRetryInterval (1m) after a request to the upstream fails; each reading replaces what is followed in one swap
chain.Transport(base)Rewrites a request to an origin the upstream serves into the same request to the upstream, at that proxy engine's path; a path with . or .. is refused
chain.ForwardTransport(base)apt's: the upstream as its HTTP proxy for http:// URLs
chain.Redirect, chain.WithRedirectfiles and volatile files: the request to the origin becomes one to the upstream's own endpoint, with the same query
chain.BaselineWhat an upstream too old to describe itself is taken to run: 0.12's proxy engines at their default paths
chain.SourceWhere the description came from: upstream, saved, baseline, newer, loop, unreachable
chain.ViaHow a proxy engine is fetched: upstream, origin, refused
proxyengines.Documentschema, instance (random per run), proxy-engines: enabled, kind (forward, root, path), path, remotes
proxyengines.HeaderViaX-Pkgproxy-Via, the instances a request went through, nearest the client first

The saved description is <cache-dir>/upstream-pkgproxy.json. /pkgproxy/v1/info reports upstream_pkgproxy from Chain.Status.

reqlog: the access log line

reqlog.Wrap(next, logger, observers...) logs one record per request, message handled request, once next returns. Level is Info, Error for a 5xx.

KeyValue
methodr.Method
urlr.RequestURI, the target as the client sent it
remote_addrr.RemoteAddr
handlerwhat SetHandler recorded, last call wins; absent when nothing did
outcomewhat SetOutcome recorded, last call wins; error for a 4xx or 5xx nothing labelled
statusthe response status, 200 when none was written (a hijacked tunnel)
bytesbody bytes written, plus what AddBytes counted (a tunnel's copy)
durationhow long next took
proxy engine attributeswhat SetAttr added, such as assertions_fetched and chains_incomplete on a snap refresh
response5xx only: the first 512 bytes of a textual, uncompressed body
OutcomeMeaning
hitServed from the cache without contacting the origin
revalidatedThe origin was asked about a cached entry, which was then served
staleThe origin could not be reached; a cached entry due for revalidation was served
fetchedFetched, saved to the cache and served
joinedWaited for or followed another request's fetch
passthroughRelayed from the origin, not cached
synthesisedAnswered by pkgproxy without the origin: from what it recorded, or a placeholder device session
tunnelA blind CONNECT tunnel
errorThe request failed, or a response was cut short because its fill failed (status 200)

reqlog: events for observers

Only a request that named an object with SetObject or SetObjectFrom, or an open tunnel, has events. An Observer is called on the goroutine serving the request and must not block.

StepWhenFields set
StepBeginthe first SetObject(r, kind, key) or SetObjectFrom(r, kind, key, origin)ID, Time, RemoteAddr, Kind, Key, Origin
StepOutcomeeach SetOutcome after the beginOutcome
StepEndthe handler returns, or panics (outcome error, panic re-raised)final Outcome, Bytes, Status

Kind is the stats type (apt, snap, gomod, lxd, docker, files) or reqlog.KindTunnel (tunnel), whose Key is the host and port. The cache page turns these into its live views: see Cache page code.

Roads from here

START HERE

OPERATIONS

INSIDE PKGPROXY · for contributors