Troubleshooting pkgproxy

What to look at when a tool fails to download through pkgproxy, or seems not to use it at all. Commands assume the snap; without it, use your own service’s name where this page says snap, and see DEPLOYMENT.md.

Which version is running

pkgproxyctl version

prints the version of pkgproxyctl, of the pkgproxy that is running and, in the snap, of the snap:

pkgproxyctl  v0.9.0
pkgproxy     v0.9.0
snap         0.9.0, revision 123

If the running pkgproxy is another build, it says so and how to restart it. The setup page at http://<host>:3142/ shows the version at the bottom. Read the documentation of the same version: open this file at that version’s tag.

Is pkgproxy running

snap services pkgproxy

If it is not active, start it with sudo snap start pkgproxy and look at the log for why it stopped. pkgproxy refuses to start when a setting is invalid, and says which one. pkgproxyctl stats, pkgproxyctl open and pkgproxyctl version say “pkgproxy is not running” when they cannot reach it.

Where the log is

snap logs pkgproxy -n 100
snap logs pkgproxy -f

or journalctl -u snap.pkgproxy.pkgproxy. Without the snap, pkgproxy logs to its standard error, so look wherever your service puts that, for example journalctl -u <your service>.

pkgproxy logs one line for each request it handles:

level=INFO msg="handled request" method=GET url=http://deb.debian.org/debian/dists/bookworm/InRelease remote_addr=192.0.2.7:46382 handler=apt outcome=hit status=200 bytes=151472 duration=1.2ms

• remote_addr is the client, so you can tell which machine sent it.
• handler is the kind of download: apt, snap, go, docker or lxd.
• outcome says how it was served: hit from the cache, fetched from the internet and kept, revalidated checked with the server first, stale an older copy because the server could not be reached, joined shared with another client’s download, passthrough passed on without keeping it, synthesised answered by pkgproxy itself without asking the server, tunnel an https:// connection pkgproxy cannot look into, and error.
• A snap refresh line ends with assertions_fetched and chains_incomplete: after answering snapd, pkgproxy fetches the assertions of the snaps it recorded that it does not have yet, so that they install offline. chains_incomplete above 0 means some snap still lacks assertions; a warning line before it names the snap and what is missing.

A request that failed with a 5xx status is logged at ERROR level, with the start of the answer as response. A line before it usually says what went wrong, such as cannot fetch from upstream with the error. To find failures:

snap logs pkgproxy -n all | grep 'status=5'

The errors for snapd are 424 (see below) and are logged at INFO; search for status=424 to find them.

pkgproxyctl offline shows the offline mode. While pkgproxy is not contacting some or all servers, pkgproxyctl stats lists them and the setup page says so to visitors.

What the errors mean

pkgproxy answers with these statuses on its own. Any other status comes from the mirror, store or registry, passed on.

pkgproxy checks each download against its published hash. When it does not match, pkgproxy keeps nothing and logs that it is not caching the download because it does not match its hash. apt, snapd and LXD then get the server’s answer as it is, while Go and Docker get a 502.

What each tool prints

apt shows only the status, not pkgproxy’s explanation, and the address in brackets is pkgproxy’s, not the mirror’s:

W: Failed to fetch http://deb.debian.org/debian/dists/bookworm/InRelease  503  Service Unavailable [IP: 192.0.2.1 3142]

In the store-overloaded mode apt 3 also prints TooManyRequests before the status, and waits as pkgproxy asks before it retries. In the offline mode pkgproxy does not ask it to wait, so apt update fails in seconds. apt older than 3.1.3, such as that of Ubuntu 24.04, can instead hang in apt update when a source is not cached; pkgproxyctl setup apt prevents that (see Working offline). When pkgproxy is down, apt says Could not connect to 192.0.2.1:3142. Look up the same URL in pkgproxy’s log.

snap shows pkgproxy’s explanation only when it fetches assertions:

error: cannot install "hello": ... assertion service error: "pkgproxy is offline and the content is not in its cache"

For installs and refreshes it shows only the status and the address:

error: cannot install "hello": cannot query the store for updates: got unexpected HTTP status code 424 via POST to "http://192.0.2.1:3142/v2/snaps/refresh"

A 424 from pkgproxy means it is offline, or is leaving the store alone for a while. snapd does not fall back to the real store.

go shows the explanation as server response:

go: example.com/m@v1.2.3: reading http://192.0.2.1:3142/go/example.com/m/@v/v1.2.3.info: 503 Service Unavailable
	server response: pkgproxy is offline: the content is not in the cache

With the GOPROXY=http://<host>:3142/go,direct that pkgproxyctl setup go sets, go downloads a module directly only when pkgproxy answers 404 or 410. Any other error, and pkgproxy being down, stop the build. Start pkgproxy, or use |direct in place of ,direct to download directly on any error.

lxc shows the status at the end of a chain of messages:

Error: Failed getting remote image info: Failed getting image: Failed parsing stream: Unable to fetch https://192.0.2.1:3143/lxd/pkgproxy-images/streams/v1/index.json: 503 Service Unavailable

LXD keeps image lists for an hour and uses its old copy when pkgproxy fails, so a problem can show up late. journalctl -u snap.lxd.daemon has the whole message.

Docker usually shows nothing; see the next section.

pkgproxy went offline by itself

On Linux, pkgproxy goes offline while the machine it runs on has no network, and comes back when it does. To see why it is offline, run:

pkgproxyctl offline status

which says, for example:

offline mode: "offline"
offline automatically, because this machine has no network; the mode "never" applies again when it is back

pkgproxyctl offline off goes online until the machine next goes offline. A network that only reaches a local mirror counts as no network, and pkgproxy in a container follows the container’s routes, not the host’s. To stop pkgproxy from going offline by itself, run:

sudo snap set pkgproxy auto-offline=off

or, without the snap, start it with -auto-offline off. See Going offline with the machine.

Docker downloads from Docker Hub instead

pkgproxyctl setup docker makes pkgproxy a registry mirror of Docker Hub. Docker tries the mirror first and, when it fails in almost any way (an error status, a missing image, pkgproxy being down), downloads from Docker Hub instead without saying so. The pull works, but nothing is cached. This also means that offline mode does not keep Docker offline.

To see it happen, look at Docker’s log, journalctl -u docker (or snap logs docker), for lines like:

level=info msg="Attempting next endpoint for pull after error: ..."

or, with the containerd image store, trying next host after status: 503 Service Unavailable. pkgproxy’s own log shows the request that failed, with handler=docker.

Docker uses mirrors only for images from Docker Hub, and only for pulls. Images from other registries, such as ghcr.io or quay.io, and private images never go through pkgproxy.

Is a tool really using pkgproxy

pkgproxyctl stats -r 10m

shows what was served from the cache in the last ten minutes. Run it right after the tool downloads something it has downloaded before; a first download is not a cache hit yet. pkgproxy’s log shows every request with the client’s address, whether it is a hit or not.

Each tool also shows its own setting:

Things that bypass pkgproxy even when it is set up:

• apt: https:// sources pass through without being cached (the log says outcome=tunnel). Use http:// sources; apt checks the signatures of what it downloads either way.
• Go: modules that match GOPRIVATE or GONOPROXY.
• LXD: images launched from images: or ubuntu: rather than pkgproxy-images: or pkgproxy-ubuntu:.
• snapd: snap get system proxy.store is about a different kind of store proxy, and is empty with pkgproxy.

Run a setup command without | sh to see exactly what it changes, and add -undo to see what removes it again.

pkgproxyctl setup asks the running pkgproxy which port it listens on. If it cannot reach pkgproxy, it uses the default ports without saying so, so start pkgproxy before you run it if you changed addr or lxd-tls-addr.

setup lxd says it cannot find the bridge LXD containers use

pkgproxyctl setup: cannot find the bridge LXD containers use: no nic of the default profile of project "default" is on a bridge LXD manages. Run "lxd init" to create one, or add a nic on one to that profile

The scripts of pkgproxyctl setup lxd and setup snapcraft ask LXD, with lxc, which bridge the containers of the project are on, so that they can reach pkgproxy at its address. They take the bridge of a nic device in the project’s default profile (in project default while the project, such as snapcraft before its first build, does not exist), and only one that LXD manages: never Docker’s docker0 or another bridge LXD does not know. See what the profile has, and whether LXD manages the network a nic names:

lxc profile show local:default
lxc network show local:lxdbr0

lxd init creates a managed bridge, lxdbr0, and puts the default profile on it. If the nics are on different managed bridges, the script says so: put them on one. If the bridge has no address (ipv4.address and ipv6.address both none), give it one with lxc network set.

None of this needs a snap interface: pkgproxyctl does not look for the bridge itself, the script does.

A broken or full cache

pkgproxy never deletes anything on its own, so its disk can fill up. Downloads then fail with errors in the log. See Disk space for how to see what takes the space and free it.

If apt keeps getting a broken package list, throw pkgproxy’s copy away with pkgproxyctl cache invalidate; see Removing a cached file early. For anything else that looks wrong in the cache, stop pkgproxy, remove the directory of that kind of download and start pkgproxy again, as shown in Disk space. Clients download what is gone again.

Reporting a problem

Open an issue with the output of pkgproxyctl version, what the tool printed, and the lines of pkgproxy’s log for the same requests.