A request in code
contributors for contributors
How one request passes through the code, from reqlog.Wrap and the gateway's dispatch order to a proxy engine's handler, the store, a fill and the upstream transport, guarded for apt, and through an upstream pkgproxy.
On the map: inside pkgproxy
01 What this topic covers
Which packages one request passes through, from the listener to the access log line, the upstream transport, an upstream pkgproxy and the upstream status.
A request passes through a few packages in a fixed order, from the listener that accepts it to the line it leaves in the access log. If you have not seen The source tree, start there. First you follow one apt download through the packages. Then you see the order in which the gateway picks a proxy engine, and how a handler records its outcome. Then comes the transport every upstream fetch goes through, what changes with an upstream-pkgproxy, and how the upstream status hears of each request.
02 One request through the packages
reqlog.Wrap → offline.Middleware →
proxyengines.LoopGuard → gateway.Handler →
proxy.Handler: Resolve, Store.Stat,
then fills.Serve, whose fetch commits through
Store.Create after the hash check.
Follow one apt download from the pool, on a cache miss. The server hands it to reqlog.Wrap, which will write the access log line, then to the offline middleware, the loop guard and the gateway, which picks the apt handler. The proxy package resolves the URL to a cache key and asks the store, which has nothing, so the request starts a fill. The fill fetches from the origin once, through the apt proxy engine's own transport. The bytes go into a store writer and to the client at the same time, and the entry is committed once its hash matches. Then Wrap writes the line: outcome fetched.
03 Why the proxy engines never collide
route checks CONNECT or a non-empty
r.URL.Host first (apt), then proxyengines.Path,
the base paths, registryproxy.Matches and snap's prefixes.
A disabled proxy engine's traffic gets 404 from the gateway. Each proxy
engine's answer carries X-Pkgproxy-Engine.
The gateway checks a request in a fixed order. A CONNECT, or a request line with an absolute URL, is parsed with a host, and only apt sends those, so apt and snapd cannot meet. Everything else has no host and goes by path: /pkgproxy/v1/proxy-engines, then /go, /lxd and /files. A Go module path starts with a host name, which contains a dot. Docker refuses a mirror address with a path, so its requests are matched by shape: manifests, blobs, referrers, tag lists and the catalog. Then the snap store's prefixes, and whatever is left gets a 404, not a relay.
04 How a handler records its outcome
Of the requests that share one fetch, only those that waited are
joined; the starter is fetched or
revalidated, as internal/flight reports.
Each listener's handler is wrapped by reqlog.Wrap, which hands the request a place to record what happened and writes the line when the handler returns. The gateway records the handler name, and each proxy engine records the outcome with SetOutcome, the last call winning, on every path that serves an answer. A failure nothing labelled becomes error. SetAttr adds attributes of a proxy engine's own: a snap refresh adds how many assertions it fetched ahead, so the line appearing tells a test that the fetch ahead is over.
05 One transport stack, three bases
offline.Switch.Transport → idletimeout.Transport
→ upstreamretry.Transport → upstreamlog.Transport
→ a base: http.DefaultTransport,
dialguard.Guard.Transport (apt; CONNECT dials
offlineSwitch.Dial(guard.Dialer().DialContext)) or
filesproxy.NewTransport.
In the code, the snap, Go, Docker and LXD proxy engines fetch through one round tripper, built by upstream Transport in the daemon command. The offline switch's Transport is the outer layer, so a request is refused there before anything else while offline. Inside it come the idle timeout, a retry of a request answered 408, and a log of who answered an error, around Go's default transport. Apt and the files proxy engine build the same layers around a transport of their own. Apt's dialguard refuses to connect to this machine for a client, and the files proxy engine refuses private addresses.
06 Through an upstream pkgproxy
chain.Chain reads proxyengines.Document from
/pkgproxy/v1/proxy-engines every
DefaultRefreshInterval (10m). Transport wraps
each base; apt's is ForwardTransport; files use
Redirect. Requests carry X-Pkgproxy-Via;
LoopGuard answers 508 past MaxHops (8) or its own
instance.
With the -upstream-pkgproxy flag set, the daemon builds a chain from internal/chain. It reads the upstream's description of its proxy engines when the daemon starts and every ten minutes, and swaps what it follows in one step, so no request changes its target halfway. Its Transport is the innermost layer, under the log and the offline switch: it turns a request to an origin into the same request to the upstream, at that proxy engine's path, while breakers, statistics and logs still see the origin. Apt gets a forward transport instead. In front of the gateway, the loop guard answers a request that has been here before with 508.
07 What the upstream status hears
offline.Switch.Recorder is an
*upstreams.Recorder: Observe(Observation) per
request, Record(Event) per breaker, mode or daemon change.
ServedStale comes from proxy,
gomodproxy, registryproxy,
snapproxy and revalidate.Doc; snap's
writeSynthesised calls Synthesised, its
placeholders Recorder.Observe directly.
The upstream status is kept by an upstreams.Recorder, which the offline switch holds. The switch's transport already sees every upstream request, so it reports each one to the recorder with Observe once its body ends, and each breaker or mode change with Record. A proxy engine that answers from a stale copy, because asking failed or was refused, calls ServedStale on the switch, and the snap proxy engine counts its answers in the store's place, recorded refreshes and placeholders, as synthesised. Each is a few additions under the recorder's own lock, with no input or output on the request path.
08 Where to go next general version
Next: internal/fill, for the fill a cache miss starts; The proxy engines' packages, for what each handler does; Cache page code, for the requests watched live.
You can now name the package behind each step of a request, from the listener to the access log and out to the upstream. To see the fill that a cache miss starts, watch internal/fill. To see what each proxy engine's handler does inside, watch The proxy engines' packages. To see how the cache page shows these requests as they happen, watch Cache page code.
Read more
The handler chain of each listener
Outermost first. Each wrapper takes only its own paths and passes everything else, CONNECT and absolute-URL requests included, to the next.
| Listener | Chain |
|---|---|
-addr (or -socket-path), and -content-socket-path | reqlog.Wrap → withSetupPage (with -setup-page, the default) → withLxdCertDownload (when the LXD HTTPS listener is on) → offline.Middleware → proxyengines.LoopGuard → gateway.Handler |
-lxd-tls-addr (or -lxd-tls-socket-path), and -lxd-tls-content-socket-path | reqlog.Wrap → withLxdAPI → offline.Middleware → proxyengines.LoopGuard → gateway.Handler |
-stats-socket-path | statsapi.Handler: GET /pkgproxy/v1/stats, /pkgproxy/v1/info, /pkgproxy/v1/upstreams, /pkgproxy/v1/upstreams/history, /pkgproxy/v1/upstreams/events; no access log |
-admin-socket-path | adminapi.Handler: DELETE /pkgproxy/v1/cache, GET and PUT /pkgproxy/v1/offline; no access log |
Both traffic listeners pass the cache page's liveHub.observe to reqlog.Wrap as an observer. proxyengines.LoopGuard answers a request whose X-Pkgproxy-Via names this daemon's instance, or more than proxyengines.MaxHops (8) instances, with 508 Loop Detected and X-Pkgproxy-Loop, before any proxy engine sees it. The snap proxy engine is wrapped once more, inside the gateway, by offline.MiddlewareAnswer(…, snapproxy.RefusedAnswer), because snapd retries a 503 for half a minute. offline.Middleware turns a 502 into a 503 when the fetch was refused for being offline.
Paths taken before the gateway
| Wrapper | Match | Handler name |
|---|---|---|
withSetupPage | GET or HEAD of /, /setup/<tool>, /setup/all, /pkgproxy/status; for a local or trusted visitor only (setupPageCache.mayView), /pkgproxy/cache, /pkgproxy/cache/activity, /pkgproxy/cache/events, /pkgproxy/cache/upstreams, /pkgproxy/upstreams, /pkgproxy/upstreams/<host> | setup-page |
withLxdCertDownload | GET or HEAD of /pkgproxy-lxd-cert.pem | lxd-cert |
withLxdAPI | origin-form / or /1.0 and below (lxdapi.Matches) | lxd-api |
The gateway's dispatch order
gateway.Handler.route, first match wins. The name is recorded with reqlog.SetHandler and is the internal/stats type, so an access log line and a stats row name the traffic alike. Before a named handler runs, ServeHTTP sets X-Pkgproxy-Engine (proxyengines.HeaderEngine) to its proxy engine (engineNames; files-volatile for /files/v1/volatile), so that a downstream pkgproxy tells a proxy engine's 404 from the gateway's, which carries none.
| # | Match | Handler | Name |
|---|---|---|---|
| 1 | CONNECT, or a non-empty r.URL.Host (an absolute-URL request line) | Apt | apt |
| 2 | ProxyEngines set and the path is /pkgproxy/v1/proxy-engines (proxyengines.Path) | proxyengines.Handler | none |
| 3 | Go set and the path is -go-base-path (default /go) or below it | Go | gomod |
| 4 | Lxd set and the path is -lxd-base-path (default /lxd) or below it | Lxd | lxd |
| 5 | Files set and the path is /files or below it | Files | files |
| 6 | registryproxy.Matches: /v2/, /v2/<name>/manifests/<ref>, /blobs/<digest>, /referrers/<digest>, /tags/list, /v2/_catalog | Docker, or 404 when it is nil | docker |
| 7 | /v2, /api/v1 or /v1, or below one of them | Snap | snap |
| 8 | anything else | http.NotFound | none |
"Below" means the base path followed by /. Docker and Lxd are left a nil interface when -docker-upstream off or -lxd-remote off turned the proxy engine off; the Go and files proxy engines are always set.
Inside the apt handler
proxy.Handler.ServeHTTP:
| Request | What happens |
|---|---|
CONNECT | handleConnect: a blind tunnel, port 443 only, dialled through the dial guard |
| neither GET nor HEAD | 405 |
not an http:// absolute URL | 400 |
cachekey.PolicyImmutable | reqlog.SetObject(r, "apt", key), then serveCached; a pool miss streams through serveFilled |
cachekey.PolicyRevalidate | reqlog.SetObject, then serveCached with revalidation |
cachekey.PolicyPassthroughOnly | passthrough: relayed, not cached |
Upstream transports
| Proxy engine | Round tripper, outermost first |
|---|---|
snap, Go, Docker, LXD (upstreamTransport) | offline.Switch.Transport → idletimeout.Transport → upstreamretry.Transport → upstreamlog.Transport → http.DefaultTransport, or chain.Transport(http.DefaultTransport) with -upstream-pkgproxy |
apt (newAptHandler) | the same four layers → dialguard.Guard.Transport(), or chain.ForwardTransport(guard.Transport()) with -upstream-pkgproxy (aptTransport, and guard.Proxy), inside origin.NewHTTPFetcher |
| apt CONNECT | offlineSwitch.Dial(guard.Dialer().DialContext) |
files (newFilesHandler) | the same four layers → filesproxy.NewTransport(AddressPolicy{Allow: -files-allow-network}), wrapped by chain.Transport with -upstream-pkgproxy |
The Go proxy engine's checksum database client uses the same upstreamTransport as its module fetches; with -upstream-pkgproxy it is not redirected, as -go-sumdb stays each pkgproxy's own. upstreamretry.Transport sends a request answered 408 again, up to DefaultAttempts (3) times; upstreamlog.Transport logs who answered an error. The offline switch's transport tells Switch.Recorder, an *upstreams.Recorder, of every request with Observe once its body ends, and of breaker and mode changes with Record (§28.6).
| Error | Package | Meaning |
|---|---|---|
idletimeout.ErrStalled | idletimeout | No progress for -upstream-idle-timeout (default 60s; 0 waits forever) |
offline.ErrOffline | offline | Refused: offline mode, or the host's breaker is open |
chain.ErrRefused | chain | The upstream pkgproxy has this proxy engine turned off |
chain.ErrLoop | chain | The upstream pkgproxy is this one, or sends its requests back to it |
dialguard.ErrRefused | dialguard | Loopback, unspecified, or this machine's own address at -addr or -lxd-tls-addr's port |
filesproxy.ErrPrivateAddress | filesproxy | A private address outside -files-allow-network |
internal/chain: through an upstream pkgproxy
Built by setUpChain when -upstream-pkgproxy is set, before the proxy engines' constructors (§27).
| Name | What it is |
|---|---|
chain.Chain | Reads the upstream's proxyengines.Document at start, every DefaultRefreshInterval (10m), and at most every DefaultRetryInterval (1m) after a request to the upstream fails; each reading replaces what is followed in one swap |
chain.Transport(base) | Rewrites a request to an origin the upstream serves into the same request to the upstream, at that proxy engine's path; a path with . or .. is refused |
chain.ForwardTransport(base) | apt's: the upstream as its HTTP proxy for http:// URLs |
chain.Redirect, chain.WithRedirect | files and volatile files: the request to the origin becomes one to the upstream's own endpoint, with the same query |
chain.Baseline | What an upstream too old to describe itself is taken to run: 0.12's proxy engines at their default paths |
chain.Source | Where the description came from: upstream, saved, baseline, newer, loop, unreachable |
chain.Via | How a proxy engine is fetched: upstream, origin, refused |
proxyengines.Document | schema, instance (random per run), proxy-engines: enabled, kind (forward, root, path), path, remotes |
proxyengines.HeaderVia | X-Pkgproxy-Via, the instances a request went through, nearest the client first |
The saved description is <cache-dir>/upstream-pkgproxy.json. /pkgproxy/v1/info reports upstream_pkgproxy from Chain.Status.
reqlog: the access log line
reqlog.Wrap(next, logger, observers...) logs one record per request, message handled request, once next returns. Level is Info, Error for a 5xx.
| Key | Value |
|---|---|
method | r.Method |
url | r.RequestURI, the target as the client sent it |
remote_addr | r.RemoteAddr |
handler | what SetHandler recorded, last call wins; absent when nothing did |
outcome | what SetOutcome recorded, last call wins; error for a 4xx or 5xx nothing labelled |
status | the response status, 200 when none was written (a hijacked tunnel) |
bytes | body bytes written, plus what AddBytes counted (a tunnel's copy) |
duration | how long next took |
| proxy engine attributes | what SetAttr added, such as assertions_fetched and chains_incomplete on a snap refresh |
response | 5xx only: the first 512 bytes of a textual, uncompressed body |
| Outcome | Meaning |
|---|---|
hit | Served from the cache without contacting the origin |
revalidated | The origin was asked about a cached entry, which was then served |
stale | The origin could not be reached; a cached entry due for revalidation was served |
fetched | Fetched, saved to the cache and served |
joined | Waited for or followed another request's fetch |
passthrough | Relayed from the origin, not cached |
synthesised | Answered by pkgproxy without the origin: from what it recorded, or a placeholder device session |
tunnel | A blind CONNECT tunnel |
error | The request failed, or a response was cut short because its fill failed (status 200) |
reqlog: events for observers
Only a request that named an object with SetObject or SetObjectFrom, or an open tunnel, has events. An Observer is called on the goroutine serving the request and must not block.
| Step | When | Fields set |
|---|---|---|
StepBegin | the first SetObject(r, kind, key) or SetObjectFrom(r, kind, key, origin) | ID, Time, RemoteAddr, Kind, Key, Origin |
StepOutcome | each SetOutcome after the begin | Outcome |
StepEnd | the handler returns, or panics (outcome error, panic re-raised) | final Outcome, Bytes, Status |
Kind is the stats type (apt, snap, gomod, lxd, docker, files) or reqlog.KindTunnel (tunnel), whose Key is the host and port. The cache page turns these into its live views: see Cache page code.
Roads from here
START HERE
- A request, end to end — the same request without the code
OPERATIONS
- The access log — the line reqlog writes
INSIDE PKGPROXY · for contributors
- The source tree — where these packages live
- internal/fill — the fill a cache miss starts
- Store and keys — the store and the cache keys
- The proxy engines' packages — what each proxy engine's handler does
- Cache page code — the events reqlog tells the cache page