The access log

operations for administrators for CI operators

pkgproxy logs one line per request with the client, the side that served it and one of nine outcome words, at error level with the start of the answer when it failed.

On the map: operations

▶ Watch this deck — 6 scenes, ~4 min

01 What this topic covers

ABOUT THIS TOPIC The access log one line for every request pkgproxy handles 1 one line 2 nine outcomes 3 failures 4 where it goes
About this topic: The access log, for administrators and CI operators; its scenes: one line, nine outcomes, failures, where it goes

pkgproxy logs one line per request with who asked, which side answered and how, and more when it failed.

The access log is where pkgproxy writes one line for every request it handles, saying who asked and how the request was served. If you have not seen Fetch once, start there. You see the fields of one line, the nine words for how a request was served, what a failed request adds to its line, and where the log goes and how to search it.

02 One line per request

ONE LINE PER REQUEST level=INFO msg="handled request" method=GET url=http://deb.debian.org/debian/dists/bookworm/InRelease remote_addr=192.0.2.7:46382 handler=apt outcome=hit status=200 bytes=151472 duration=1.2ms logged when the handler returns: it covers work done after the response was sent method url as the client sent it: an absolute URL for apt, a path for the others remote_addr which machine asked handler apt snap gomod docker lxd files and setup-page … outcome how it was served: one of nine words status bytes duration the answer's status, its body bytes, the time taken
An access log line split into its fields: method and url as sent, remote_addr naming the client, handler naming the side, outcome, and status, bytes and duration

handler is apt (every request sent to pkgproxy as a proxy, tunnels included), snap, gomod, docker, lxd, lxd-api or files for the proxy sides, and setup-page or lxd-cert for pkgproxy's own pages. A line without one is a path nothing serves; a tunnel's bytes are what it copied to the client.

pkgproxy writes one line for each request it handles, when the handler returns, so the line covers work done after the response was sent. It names the method and the URL as the client sent it, which for apt is the mirror's full address; the remote address, so you can tell which machine asked; the handler, the side that served it; the outcome, how it was served; and the status, the number of body bytes and how long it took.

03 Nine outcomes

OUTCOME answered from what pkgproxy has straight from the upstream failed hit from the cache, upstream not asked joined waited for another request's fetch revalidated checked with the upstream, then served stale upstream unreachable: the copy served synthesised answered by pkgproxy from its records fetched fetched, saved to the cache, served passthrough relayed, not cached tunnel a blind CONNECT tunnel error the request failed assumed for a 4xx or 5xx that nothing labelled stable words, the same in every language: match on them
The nine outcomes in three columns: from pkgproxy's copy hit, joined, revalidated, stale and synthesised; from the upstream fetched, passthrough and tunnel; and error

hit, joined, revalidated, stale, synthesised, fetched, passthrough, tunnel, error. They are not translated.

The outcome is one of nine stable words, the same in every language. Hit was served from the cache without asking the upstream; joined waited for another request's fetch; revalidated checked the cached copy with the upstream first; stale served a copy due for a check because the upstream could not be reached; synthesised was answered by pkgproxy itself. Fetched came from the upstream and was kept, passthrough was relayed without keeping it, and tunnel is a blind CONNECT tunnel. Error means the request failed.

04 When a request fails

WHEN IT FAILS level=ERROR msg="cannot fetch from upstream" … error=… level=ERROR msg="handled request" method=GET url=http://deb.debian.org/… handler=apt outcome=error status=502 bytes=… duration=… response="cannot fetch from upstream" response: the start of the body · a failed cache lookup's error, or what the upstream answered text only not compressed, not binary at most 512 bytes valid UTF-8, trimmed 5xx only a relayed body may be sensitive $ snap logs pkgproxy -n all | grep 'status=5' failures, at ERROR $ snap logs pkgproxy -n all | grep 'status=424' snapd's errors, at INFO
A failed request: a line saying cannot fetch from upstream with the error, then the handled request line at ERROR level with status 502 and the response attribute; below, grep for status=5 and status=424

A 5xx line carries response, the first 512 bytes of a text body; compressed or non-text bodies are skipped. Look for status=5, and for snapd status=424.

A response with a server error status is logged at error level, with the start of its body as the response attribute: the error text of a failed cache lookup, or what the upstream answered. Only text is captured, at most 512 bytes, and only for server errors, because a relayed body can contain sensitive data. A line before it usually says what went wrong. To find failures, search the log for a status starting with five. The 424 answers snapd gets are logged at info level.

05 Where the log goes

WHERE THE LOG GOES pkgproxy stderr the systemd journal in the snap $ snap logs pkgproxy -n 100 $ snap logs pkgproxy -f $ journalctl -u snap.pkgproxy… without the snap: wherever your service manager puts stderr key=value pairs: grep is enough $ snap logs pkgproxy -n all | grep 'outcome=fetched' what went upstream $ snap logs pkgproxy -n all | grep 'handler=gomod' one side $ snap logs pkgproxy -f | grep 'remote_addr=192.0.2.7:' one machine
pkgproxy writes key=value lines to stderr; in the snap the systemd journal keeps them and snap logs or journalctl read them; without the snap the service manager decides; grep picks lines by outcome, handler or remote address

snap logs pkgproxy -n 100, snap logs pkgproxy -f, or journalctl -u snap.pkgproxy.pkgproxy.

The daemon writes its log to standard error. In the snap, systemd's journal keeps it: snap logs pkgproxy shows the last lines, and with the -f flag follows new ones, and journalctl for the snap's service reads the same. Without the snap, the log goes wherever your service manager puts standard error. Each line is a list of key=value pairs, so grep is enough: by outcome, by handler, or by remote address to follow one machine.

06 Where to go next general version

WHERE NEXT find any request: who asked, which side, how you are here The access log Statistics totals instead of lines The cache page the same requests, drawn as they happen Immutable or not what revalidated and stale mean
Where next from The access log: Statistics, totals instead of lines; The cache page, the same requests, drawn as they happen; Immutable or not, what revalidated and stale mean

Next: Statistics, for totals; The cache page, for the same requests drawn live; Immutable or not, for what revalidated and stale mean.

When a client is slow or fails, you can now find its request in the log and see which side answered it, and how. For totals instead of lines, watch Statistics. To see the same requests drawn as they happen, watch The cache page. For what revalidated and stale mean, watch Immutable or not.

Read more

A line, field by field

level=INFO msg="handled request" method=GET url=http://deb.debian.org/debian/dists/bookworm/InRelease remote_addr=192.0.2.7:46382 handler=apt outcome=hit status=200 bytes=151472 duration=1.2ms
FieldValue
levelINFO, or ERROR for a status of 500 or more
msghandled request, in the daemon's language; the keys and the words below are never translated
methodthe request method, CONNECT for a tunnel
urlthe request target as the client sent it: a full URL for apt, a path and query for the other sides
remote_addrthe client's address and port
handlerwhich part of pkgproxy answered; absent for a path nothing serves
outcomehow it was answered; absent for a successful request that recorded none
statusthe HTTP status sent, 200 for an open tunnel
bytesbody bytes sent to the client; for a tunnel, bytes copied from the upstream to the client
durationfrom the request's arrival until its handler returned, work after the response included
responseonly on a 5xx: the first 512 bytes of a text body; compressed or non-text bodies are left out
assertions_fetchedonly on a snap refresh answered online: assertions fetched ahead for offline installs
chains_incompletewith the above: how many recorded snaps still lack some assertions; above 0, a warning before names them

The line is written when the request's handler returns, so it comes after any warning or error line the same request logged.

Handler values

handlerRequests
apteverything sent to pkgproxy as a proxy: apt archives, any other http:// URL, and CONNECT tunnels
snapthe Snap Store API: /v2/… paths that are not registry paths, /api/v1/…, /v1/…
gomodthe Go module proxy, under /go
dockerthe Docker Hub mirror: registry-shaped /v2/… paths
lxdLXD image servers over simplestreams, under /lxd
lxd-apiLXD's image API on the HTTPS port, for LXD 6.10 and later
lxd-certthe download of pkgproxy's LXD certificate
filesfiles fetched by checksum, under /files
setup-pagethe setup page, its setup scripts and, for local visitors and those from trusted networks, the cache page and its live feeds

The side names match the entry types of pkgproxyctl stats, so a log line and a statistics row for the same traffic agree.

Outcome words

outcomeMeaningWhere it shows up besides the obvious
hitserved from the cache without asking the upstreama mutable entry inside its window
joinedwaited for or followed another request's fetchonly the requests that joined; the one that started the fetch is fetched or revalidated. On the Docker mirror every request sharing a manifest fetch is joined
revalidatedchecked with the upstream, then served (refreshed, or confirmed by a 304)apt indexes, Go lists, LXD catalogs, Docker tags
stalethe upstream could not be reached; a copy due for a check was servedalways preceded by a warning line with the reason
fetchedfetched from the upstream, kept and served
passthroughrelayed from the upstream without keeping ituncacheable paths, a hash mismatch, a HEAD of an uncached Docker layer, a document fetched but not stored
synthesisedanswered by pkgproxy from what it recorded, without asking the upstreamoffline snap refreshes, a HEAD of an uncached LXD image file or snap
tunnela blind CONNECT tunnelhttps:// apt sources and anything else tunnelled
errorthe request failedassumed for any 4xx or 5xx that recorded no other outcome

Finding things

To findSearch for
server errorsstatus=5
snapd's errors while offline or leaving the store alonestatus=424 (logged at INFO)
one machineremote_addr=192.0.2.7:
answers given from old copiesoutcome=stale
one sidehandler=docker

In the snap the log is in the journal: snap logs pkgproxy -n all, or journalctl -u snap.pkgproxy.pkgproxy. Elsewhere it is the daemon's standard error.

Who answered an error

When a mirror, store or registry answers with an error other than 404 and 410, a WARN line comes before the request's own line:

level=WARN msg="upstream answered with an error" method=GET url=https://api.snapcraft.io/v2/assertions/... status=408 proxy=http://egress.example:3128 server=squid/6.13 via="1.1 egress (squid/6.13)" squid_error="ERR_READ_TIMEOUT 0" content_type=text/html body="<html> ..."
FieldValue
urlthe upstream's address, without its query
proxythe proxy the request went through, or none
server, via, proxy_status, squid_errorthose headers of the answer, when it has them
bodythe start of the answer

A Via or X-Squid-Error header, or a proxy's error page in body, means a proxy on the way answered, not the upstream. A 408 is sent again, up to three times in all, each logged as repeating a request the upstream answered with 408; the snap store answers it now and then on a connection it is closing, and one followed by a success is nothing to worry about.

Roads from here

CORE · the path every side shares

APT

OPERATIONS