A request in code

contributors for contributors

How one request passes through the code, from reqlog.Wrap and the gateway's dispatch order to a side's handler, the store, a fill and the upstream transport, guarded for apt.

On the map: inside pkgproxy

▶ Watch this deck — 6 scenes, ~4 min

01 What this topic covers

ABOUT THIS TOPIC A request in code the packages one request passes through, from listener to log line 1 One request through the packages 2 Why the sides never collide 3 How a handler records its outcome 4 One transport stack, three bases
About this topic, A request in code: one request through the packages, the dispatch order, recording the outcome, the upstream transport

Which packages one request passes through, from the listener to the access log line and the upstream transport.

A request passes through a few packages in a fixed order, from the listener that accepts it to the line it leaves in the access log. If you have not seen The source tree, start there. First you follow one apt download through the packages. Then you see the order in which the gateway picks a side, so no two sides claim one request, and how a handler records its outcome. Last comes the transport every upstream fetch goes through.

02 One request through the packages

ONE REQUEST, PACKAGE BY PACKAGE GET http://…/pool/… reqlog.Wrap offline.Middleware gateway → apt proxy.Handler Resolver.Resolve(r.URL) Store.Stat: ErrNotFound fills.Serve offline · idletimeout · dialguard Origin.Fetch, once, through the apt side's transport Store.Create → hash → Commit client, tailing the fill msg="handled request" … handler=apt outcome=fetched status=200
One apt pool request on a miss: http.Server, reqlog.Wrap, offline.Middleware and gateway.Handler lead to proxy.Handler, which resolves the key and finds nothing in the store; a fill fetches through the apt side's guarded transport, writes the store and the client at once, commits after the hash check; reqlog writes the line with outcome=fetched

reqlog.Wrap → offline.Middleware → gateway.Handler → proxy.Handler: Resolve, Store.Stat, then fills.Serve, whose fetch commits through Store.Create after the hash check.

Follow one apt download from the pool, on a cache miss. The server hands it to reqlog.Wrap, which will write the access log line, then to the offline middleware and the gateway, which picks the apt handler. The proxy package resolves the URL to a cache key and asks the store, which has nothing, so the request starts a fill. The fill fetches from the origin once, through the apt side's own transport. The bytes go into a store writer and to the client at the same time, and the entry is committed once its hash matches. Then Wrap writes the line: outcome fetched.

03 Why the sides never collide

ROUTE, IN ORDER 1 CONNECT or r.URL.Host != "" apt absolute-form and authority-form targets carry a host; snapd's origin-form ones do not 2 /go/… -go-base-path · first element has a dot Go modules 3 /lxd/… -lxd-base-path LXD images 4 /files/… files 5 /v2/<name>/manifests|blobs|referrers|tags/list /v2/_catalog Docker matched by shape at the root · 404 when Docker is off 6 /v2/… /api/v1/… /v1/… snap relay 7 anything else 404, not relayed
The gateway's route function as an ordered list of checks, each leading to a side: CONNECT or a host in the URL to apt; /go to Go modules; /lxd to LXD; /files to files; registry shapes under /v2, including tag lists and the catalog, to Docker, or to 404 with Docker off; /v2, /api/v1 and /v1 to the snap relay; anything else to 404

route checks CONNECT or a non-empty r.URL.Host first (apt), then the base paths, then registryproxy.Matches, then snap's prefixes. A disabled side's traffic gets 404 from the gateway, a registry-shaped path included.

The gateway checks a request in a fixed order. A CONNECT, or a request line with an absolute URL, is parsed with a host, and only apt sends those, so apt and snapd cannot meet. Everything else has no host and goes by path: /go, then /lxd, then /files. A Go module path starts with a host name, which contains a dot. Docker refuses a mirror address with a path, so its requests are matched by shape: manifests, blobs, referrers, tag lists and the catalog. Such a path stays Docker's with Docker turned off, and gets a 404. Then the snap store's prefixes, and whatever is left gets a 404, not a relay.

04 How a handler records its outcome

RECORDING THE OUTCOME reqlog.Wrap(handler, logger, observers...) the line is written when the handler returns gateway reqlog.SetHandler(r, "apt") a side reqlog.SetOutcome(r, reqlog.OutcomeHit) on every path that serves an answer · the last call wins reqlog.SetAttr(r, logkey.AssertionsFetched, fetched) a snap refresh: the line appearing means the fetch ahead is over 4xx or 5xx, no outcome recorded outcome=error assumed by Wrap
reqlog.Wrap around the listener's handler; inside, the gateway calls SetHandler, a side calls SetOutcome and SetAttr; when the handler returns Wrap writes the line, defaulting the outcome to error for an unlabelled 4xx or 5xx

Of the requests that share one fetch, only those that waited are joined; the starter is fetched or revalidated, as internal/flight reports.

Each listener's handler is wrapped by reqlog.Wrap, which hands the request a place to record what happened and writes the line when the handler returns. The gateway records the handler name, and each side records the outcome with SetOutcome, the last call winning, on every path that serves an answer. A failure nothing labelled becomes error. SetAttr adds attributes of a side's own: a snap refresh adds how many assertions it fetched ahead, so the line appearing tells a test that the fetch ahead is over.

05 One transport stack, three bases

THE UPSTREAM TRANSPORT func (cmd *pkgproxyCmd) upstreamTransport() http.RoundTripper offline.Switch.Transport refuses everything while offline idletimeout.Transport{Timeout: 60s} http.DefaultTransport snap, go, docker, lxd dialguard.Guard.Transport() apt: never this machine its CONNECT dial too filesproxy.NewTransport(…) files: no private addresses cannot fetch from upstream: no progress for 1m0s
Nested boxes: offline.Switch.Transport around idletimeout.Transport around one of three bases: http.DefaultTransport for snap, Go, Docker and LXD, dialguard's transport for apt, which its CONNECT tunnels also dial through, and filesproxy's transport for the files side

offline.Switch.Transport around idletimeout.Transport around http.DefaultTransport; apt's base is dialguard.Guard.Transport (CONNECT dials offlineSwitch.Dial(guard.Dialer().DialContext)), files' is filesproxy.NewTransport. Stalls wrap idletimeout.ErrStalled.

In the code, the snap, Go, Docker and LXD sides fetch through one round tripper, built by upstream Transport in the daemon command. The offline switch's Transport is the outer layer, so a request is refused there before anything else while offline. Inside it, an idle timeout Transport wraps Go's default transport. Apt and the files side build the same two layers around a transport of their own. Apt's dialguard refuses to connect to this machine for a client, and its CONNECT tunnels dial through the same guard. The files side refuses private addresses. A stall fails with an error wrapping ErrStalled.

06 Where to go next general version

WHERE NEXT you can name the package behind each step of a request, from the listener to the access log and the upstream you are here A request in code internal/fill the fill a cache miss starts The sides' packages what each side's handler does Cache page code the requests watched live
Where next from A request in code: internal/fill, The sides' packages, Cache page code

Next: internal/fill, for the fill a cache miss starts; The sides' packages, for what each handler does; Cache page code, for the requests watched live.

You can now name the package behind each step of a request, from the listener to the access log and out to the upstream. To see the fill that a cache miss starts, watch internal/fill. To see what each side's handler does inside, watch The sides' packages. To see how the cache page shows these requests as they happen, watch Cache page code.

Read more

The handler chain of each listener

Outermost first. Each wrapper takes only its own paths and passes everything else, CONNECT and absolute-URL requests included, to the next.

ListenerChain
-addr (or -socket-path), and -content-socket-pathreqlog.Wrap → withSetupPage (with -setup-page, the default) → withLxdCertDownload (when the LXD HTTPS listener is on) → offline.Middleware → gateway.Handler
-lxd-tls-addr (or -lxd-tls-socket-path), and -lxd-tls-content-socket-pathreqlog.Wrap → withLxdAPI → offline.Middleware → gateway.Handler
-stats-socket-pathstatsapi.Handler: GET /pkgproxy/v1/stats, GET /pkgproxy/v1/info; no access log
-admin-socket-pathadminapi.Handler: DELETE /pkgproxy/v1/cache, GET and PUT /pkgproxy/v1/offline; no access log

Both traffic listeners pass the cache page's liveHub.observe to reqlog.Wrap as an observer. The snap side is wrapped once more, inside the gateway, by offline.MiddlewareAnswer(…, snapproxy.RefusedAnswer), because snapd retries a 503 for half a minute. offline.Middleware turns a 502 into a 503 when the fetch was refused for being offline.

Paths taken before the gateway

WrapperMatchHandler name
withSetupPageGET or HEAD of /, /setup/<tool>; for a local or trusted visitor only (setupPageCache.mayView), /pkgproxy/cache, /pkgproxy/cache/activity, /pkgproxy/cache/eventssetup-page
withLxdCertDownloadGET or HEAD of /pkgproxy-lxd-cert.pemlxd-cert
withLxdAPIorigin-form / or /1.0 and below (lxdapi.Matches)lxd-api

The gateway's dispatch order

gateway.Handler.route, first match wins. The name is recorded with reqlog.SetHandler and is the internal/stats type, so an access log line and a stats row name the traffic alike.

#MatchHandlerName
1CONNECT, or a non-empty r.URL.Host (an absolute-URL request line)Aptapt
2Go set and the path is -go-base-path (default /go) or below itGogomod
3Lxd set and the path is -lxd-base-path (default /lxd) or below itLxdlxd
4Files set and the path is /files or below itFilesfiles
5registryproxy.Matches: /v2/, /v2/<name>/manifests/<ref>, /blobs/<digest>, /referrers/<digest>, /tags/list, /v2/_catalogDocker, or 404 when it is nildocker
6/v2, /api/v1 or /v1, or below one of themSnapsnap
7anything elsehttp.NotFoundnone

"Below" means the base path followed by /. Docker and Lxd are left a nil interface when -docker-upstream off or -lxd-remote off turned the side off; the Go and files sides are always set.

Inside the apt handler

proxy.Handler.ServeHTTP:

RequestWhat happens
CONNECThandleConnect: a blind tunnel, port 443 only, dialled through the dial guard
neither GET nor HEAD405
not an http:// absolute URL400
cachekey.PolicyImmutablereqlog.SetObject(r, "apt", key), then serveCached; a pool miss streams through serveFilled
cachekey.PolicyRevalidatereqlog.SetObject, then serveCached with revalidation
cachekey.PolicyPassthroughOnlypassthrough: relayed, not cached

Upstream transports

SideRound tripper, outermost first
snap, Go, Docker, LXD (upstreamTransport)offline.Switch.Transport → idletimeout.Transport → http.DefaultTransport
apt (newAptHandler)offline.Switch.Transport → idletimeout.Transport → dialguard.Guard.Transport(), inside origin.NewHTTPFetcher
apt CONNECTofflineSwitch.Dial(guard.Dialer().DialContext)
files (newFilesHandler)offline.Switch.Transport → idletimeout.Transport → filesproxy.NewTransport(AddressPolicy{Allow: -files-allow-network})

The Go side's checksum database client uses the same upstreamTransport as its module fetches.

ErrorPackageMeaning
idletimeout.ErrStalledidletimeoutNo progress for -upstream-idle-timeout (default 60s; 0 waits forever)
offline.ErrOfflineofflineRefused: offline mode, or the host's breaker is open
dialguard.ErrRefuseddialguardLoopback, unspecified, or this machine's own address at -addr or -lxd-tls-addr's port
filesproxy.ErrPrivateAddressfilesproxyA private address outside -files-allow-network

reqlog: the access log line

reqlog.Wrap(next, logger, observers...) logs one record per request, message handled request, once next returns. Level is Info, Error for a 5xx.

KeyValue
methodr.Method
urlr.RequestURI, the target as the client sent it
remote_addrr.RemoteAddr
handlerwhat SetHandler recorded, last call wins; absent when nothing did
outcomewhat SetOutcome recorded, last call wins; error for a 4xx or 5xx nothing labelled
statusthe response status, 200 when none was written (a hijacked tunnel)
bytesbody bytes written, plus what AddBytes counted (a tunnel's copy)
durationhow long next took
side attributeswhat SetAttr added, such as assertions_fetched and chains_incomplete on a snap refresh
response5xx only: the first 512 bytes of a textual, uncompressed body
OutcomeMeaning
hitServed from the cache without contacting the origin
revalidatedThe origin was asked about a cached entry, which was then served
staleThe origin could not be reached; a cached entry due for revalidation was served
fetchedFetched, saved to the cache and served
joinedWaited for or followed another request's fetch
passthroughRelayed from the origin, not cached
synthesisedAnswered by pkgproxy from what it recorded, without the origin
tunnelA blind CONNECT tunnel
errorThe request failed

reqlog: events for observers

Only a request that named an object with SetObject or SetObjectFrom, or an open tunnel, has events. An Observer is called on the goroutine serving the request and must not block.

StepWhenFields set
StepBeginthe first SetObject(r, kind, key) or SetObjectFrom(r, kind, key, origin)ID, Time, RemoteAddr, Kind, Key, Origin
StepOutcomeeach SetOutcome after the beginOutcome
StepEndthe handler returns, or panics (outcome error, panic re-raised)final Outcome, Bytes, Status

Kind is the stats type (apt, snap, gomod, lxd, docker, files) or reqlog.KindTunnel (tunnel), whose Key is the host and port. The cache page turns these into its live views: see Cache page code.

Roads from here

START HERE

OPERATIONS

INSIDE PKGPROXY · for contributors