The access log
operations for administrators for CI operators
pkgproxy logs one line per request with the client, the side that served it and one of nine outcome words, at error level with the start of the answer when it failed.
On the map: operations
01 What this topic covers
pkgproxy logs one line per request with who asked, which side answered and how, and more when it failed.
The access log is where pkgproxy writes one line for every request it handles, saying who asked and how the request was served. If you have not seen Fetch once, start there. You see the fields of one line, the nine words for how a request was served, what a failed request adds to its line, and where the log goes and how to search it.
02 One line per request
handler is apt (every request sent to pkgproxy
as a proxy, tunnels included), snap, gomod,
docker, lxd, lxd-api or
files for the proxy sides, and setup-page or
lxd-cert for pkgproxy's own pages. A line without one is a path
nothing serves; a tunnel's bytes are what it copied to the client.
pkgproxy writes one line for each request it handles, when the handler returns, so the line covers work done after the response was sent. It names the method and the URL as the client sent it, which for apt is the mirror's full address; the remote address, so you can tell which machine asked; the handler, the side that served it; the outcome, how it was served; and the status, the number of body bytes and how long it took.
03 Nine outcomes
hit, joined, revalidated,
stale, synthesised, fetched,
passthrough, tunnel, error. They
are not translated.
The outcome is one of nine stable words, the same in every language. Hit was served from the cache without asking the upstream; joined waited for another request's fetch; revalidated checked the cached copy with the upstream first; stale served a copy due for a check because the upstream could not be reached; synthesised was answered by pkgproxy itself. Fetched came from the upstream and was kept, passthrough was relayed without keeping it, and tunnel is a blind CONNECT tunnel. Error means the request failed.
04 When a request fails
A 5xx line carries response, the first 512 bytes of a text
body; compressed or non-text bodies are skipped. Look for
status=5, and for snapd status=424.
A response with a server error status is logged at error level, with the start of its body as the response attribute: the error text of a failed cache lookup, or what the upstream answered. Only text is captured, at most 512 bytes, and only for server errors, because a relayed body can contain sensitive data. A line before it usually says what went wrong. To find failures, search the log for a status starting with five. The 424 answers snapd gets are logged at info level.
05 Where the log goes
snap logs pkgproxy -n 100, snap logs pkgproxy
-f, or journalctl -u snap.pkgproxy.pkgproxy.
The daemon writes its log to standard error. In the snap, systemd's journal keeps it: snap logs pkgproxy shows the last lines, and with the -f flag follows new ones, and journalctl for the snap's service reads the same. Without the snap, the log goes wherever your service manager puts standard error. Each line is a list of key=value pairs, so grep is enough: by outcome, by handler, or by remote address to follow one machine.
06 Where to go next general version
Next: Statistics, for totals; The cache page, for the same requests drawn live; Immutable or not, for what revalidated and stale mean.
When a client is slow or fails, you can now find its request in the log and see which side answered it, and how. For totals instead of lines, watch Statistics. To see the same requests drawn as they happen, watch The cache page. For what revalidated and stale mean, watch Immutable or not.
Read more
A line, field by field
level=INFO msg="handled request" method=GET url=http://deb.debian.org/debian/dists/bookworm/InRelease remote_addr=192.0.2.7:46382 handler=apt outcome=hit status=200 bytes=151472 duration=1.2ms
| Field | Value |
|---|---|
level | INFO, or ERROR for a status of 500 or more |
msg | handled request, in the daemon's language; the keys and the words below are never translated |
method | the request method, CONNECT for a tunnel |
url | the request target as the client sent it: a full URL for apt, a path and query for the other sides |
remote_addr | the client's address and port |
handler | which part of pkgproxy answered; absent for a path nothing serves |
outcome | how it was answered; absent for a successful request that recorded none |
status | the HTTP status sent, 200 for an open tunnel |
bytes | body bytes sent to the client; for a tunnel, bytes copied from the upstream to the client |
duration | from the request's arrival until its handler returned, work after the response included |
response | only on a 5xx: the first 512 bytes of a text body; compressed or non-text bodies are left out |
assertions_fetched | only on a snap refresh answered online: assertions fetched ahead for offline installs |
chains_incomplete | with the above: how many recorded snaps still lack some assertions; above 0, a warning before names them |
The line is written when the request's handler returns, so it comes after any warning or error line the same request logged.
Handler values
handler | Requests |
|---|---|
apt | everything sent to pkgproxy as a proxy: apt archives, any other http:// URL, and CONNECT tunnels |
snap | the Snap Store API: /v2/… paths that are not registry paths, /api/v1/…, /v1/… |
gomod | the Go module proxy, under /go |
docker | the Docker Hub mirror: registry-shaped /v2/… paths |
lxd | LXD image servers over simplestreams, under /lxd |
lxd-api | LXD's image API on the HTTPS port, for LXD 6.10 and later |
lxd-cert | the download of pkgproxy's LXD certificate |
files | files fetched by checksum, under /files |
setup-page | the setup page, its setup scripts and, for local visitors and those from trusted networks, the cache page and its live feeds |
The side names match the entry types of pkgproxyctl stats, so a log line and a statistics row for the same traffic agree.
Outcome words
outcome | Meaning | Where it shows up besides the obvious |
|---|---|---|
hit | served from the cache without asking the upstream | a mutable entry inside its window |
joined | waited for or followed another request's fetch | only the requests that joined; the one that started the fetch is fetched or revalidated. On the Docker mirror every request sharing a manifest fetch is joined |
revalidated | checked with the upstream, then served (refreshed, or confirmed by a 304) | apt indexes, Go lists, LXD catalogs, Docker tags |
stale | the upstream could not be reached; a copy due for a check was served | always preceded by a warning line with the reason |
fetched | fetched from the upstream, kept and served | |
passthrough | relayed from the upstream without keeping it | uncacheable paths, a hash mismatch, a HEAD of an uncached Docker layer, a document fetched but not stored |
synthesised | answered by pkgproxy from what it recorded, without asking the upstream | offline snap refreshes, a HEAD of an uncached LXD image file or snap |
tunnel | a blind CONNECT tunnel | https:// apt sources and anything else tunnelled |
error | the request failed | assumed for any 4xx or 5xx that recorded no other outcome |
Finding things
| To find | Search for |
|---|---|
| server errors | status=5 |
| snapd's errors while offline or leaving the store alone | status=424 (logged at INFO) |
| one machine | remote_addr=192.0.2.7: |
| answers given from old copies | outcome=stale |
| one side | handler=docker |
In the snap the log is in the journal: snap logs pkgproxy -n all, or journalctl -u snap.pkgproxy.pkgproxy. Elsewhere it is the daemon's standard error.
Who answered an error
When a mirror, store or registry answers with an error other than 404 and 410, a WARN line comes before the request's own line:
level=WARN msg="upstream answered with an error" method=GET url=https://api.snapcraft.io/v2/assertions/... status=408 proxy=http://egress.example:3128 server=squid/6.13 via="1.1 egress (squid/6.13)" squid_error="ERR_READ_TIMEOUT 0" content_type=text/html body="<html> ..."
| Field | Value |
|---|---|
url | the upstream's address, without its query |
proxy | the proxy the request went through, or none |
server, via, proxy_status, squid_error | those headers of the answer, when it has them |
body | the start of the answer |
A Via or X-Squid-Error header, or a proxy's error page in body, means a proxy on the way answered, not the upstream. A 408 is sent again, up to three times in all, each logged as repeating a request the upstream answered with 408; the snap store answers it now and then on a connection it is closing, and one followed by a success is nothing to worry about.
In the documentation
Roads from here
CORE · the path every side shares
- Fetch once — the shared fetch behind joined and fetched
- Immutable or revalidated — what revalidated and stale mean
- Offline mode — the answers given while offline
APT
- HTTPS apt sources — the tunnel outcome
OPERATIONS
- Statistics — totals instead of one line per request
- The cache page — the same requests drawn as they happen