How pkgproxy caches packages for apt, snapd, Go, Docker and LXD, one narrated deck per topic. Pick any place, follow the roads at the end of each deck, and switch perspective at the top to see what applies to you. The written documentation is at zygoon.pl/pkgproxy; to run pkgproxy in GitHub Actions jobs, see setup-pkgproxy.
Click a place to enter it. Inside a topic you see only its exits; m brings the whole map back. space plays, ← → step.
pkgproxy logs one line per request with who asked, which proxy engine answered and how, and more when it failed.
handler is apt (every request sent to pkgproxy
as a proxy, tunnels included), snap, gomod,
docker, lxd, lxd-api or
files for the proxy engines, and setup-page or
lxd-cert for pkgproxy's own pages. A line without one is a path
nothing serves; a tunnel's bytes are what it copied to the client.
hit, joined, revalidated,
stale, synthesised, fetched,
passthrough, tunnel, error. They
are not translated.
A 5xx line carries response, the first 512 bytes of a text
body; compressed or non-text bodies are skipped. Look for
status=5, and for snapd status=424.
snap logs pkgproxy -n 100, snap logs pkgproxy
-f, or journalctl -u snap.pkgproxy.pkgproxy.
Next: Statistics, for totals; The cache page, for the same requests drawn live; Immutable or not, for what revalidated and stale mean.
level=INFO msg="handled request" method=GET url=http://deb.debian.org/debian/dists/bookworm/InRelease remote_addr=192.0.2.7:46382 handler=apt outcome=hit status=200 bytes=151472 duration=1.2ms
| Field | Value |
|---|---|
level | INFO, or ERROR for a status of 500 or more |
msg | handled request, in the daemon's language; the keys and the words below are never translated |
method | the request method, CONNECT for a tunnel |
url | the request target as the client sent it: a full URL for apt, a path and query for the other proxy engines |
remote_addr | the client's address and port |
handler | which part of pkgproxy answered; absent for a path nothing serves |
outcome | how it was answered; absent for a successful request that recorded none |
status | the HTTP status sent, 200 for an open tunnel |
bytes | body bytes sent to the client; for a tunnel, bytes copied from the upstream to the client |
duration | from the request's arrival until its handler returned, work after the response included |
response | only on a 5xx: the first 512 bytes of a text body; compressed or non-text bodies are left out |
assertions_fetched | only on a snap refresh answered online: assertions fetched ahead for offline installs |
chains_incomplete | with the above: how many recorded snaps still lack some assertions; above 0, a warning before names them |
The line is written when the request's handler returns, so it comes after any warning or error line the same request logged. The health route, /pkgproxy/v1/health on the main port, writes no line, so a probe every few seconds does not bury the requests the log is for.
handler | Requests |
|---|---|
apt | everything sent to pkgproxy as a proxy: apt archives, any other http:// URL, and CONNECT tunnels |
snap | the Snap Store API: /v2/… paths that are not registry paths, /api/v1/…, /v1/… |
gomod | the Go module proxy, under /go |
docker | the Docker Hub mirror: registry-shaped /v2/… paths |
lxd | LXD image servers over simplestreams, under /lxd |
lxd-api | LXD's image API on the HTTPS port, for LXD 6.10 and later |
lxd-cert | the download of pkgproxy's LXD certificate |
files | files fetched by checksum, under /files |
setup-page | the setup page, its setup scripts and, for local visitors and those from trusted networks, the cache page and its live feeds |
The proxy engine names match the entry types of pkgproxyctl stats, so a log line and a statistics row for the same traffic agree.
outcome | Meaning | Where it shows up besides the obvious |
|---|---|---|
hit | served from the cache without asking the upstream | a mutable entry inside its window |
joined | waited for or followed another request's fetch | only the requests that joined; the one that started the fetch is fetched or revalidated. On the Docker mirror every request sharing a manifest fetch is joined |
revalidated | checked with the upstream, then served (refreshed, or confirmed by a 304) | apt indexes, Go lists, LXD catalogs, Docker tags, a volatile file the server confirmed |
stale | the upstream could not be reached; a copy due for a check was served | preceded by a warning line with the reason, except a snap entry served while the store's revalidation backs off; a volatile file's kept copy while its server cannot answer |
fetched | fetched from the upstream, kept and served | a volatile file the server sent |
passthrough | relayed from the upstream without keeping it | uncacheable paths, a hash mismatch, a HEAD of an uncached Docker layer, a document fetched but not stored, a 4xx of a volatile file's server |
synthesised | answered by pkgproxy itself, without asking the upstream | offline snap refreshes, a HEAD of an uncached LXD image file or snap, placeholder nonces and device sessions, and the 401 that sends snapd for a real session in place of a placeholder |
tunnel | a blind CONNECT tunnel | https:// apt sources and anything else tunnelled |
error | the request failed | assumed for any 4xx or 5xx that recorded no other outcome; also a 200 cut short because the download it followed failed, such as a file that failed its hash check |
| To find | Search for |
|---|---|
| server errors | status=5 |
| snapd's errors while offline or leaving the store alone | status=424 (logged at INFO) |
| one machine | remote_addr=192.0.2.7: |
| answers given from old copies | outcome=stale |
| one proxy engine | handler=docker |
In the snap the log is in the journal: snap logs pkgproxy -n all, or journalctl -u snap.pkgproxy.pkgproxy. Elsewhere it is the daemon's standard error.
When a mirror, store or registry answers with an error other than 404 and 410, a WARN line comes before the request's own line:
level=WARN msg="upstream answered with an error" method=GET url=https://api.snapcraft.io/v2/assertions/... status=408 proxy=http://egress.example:3128 server=squid/6.13 via="1.1 egress (squid/6.13)" squid_error="ERR_READ_TIMEOUT 0" content_type=text/html body="<html> ..."
| Field | Value |
|---|---|
url | the upstream's address, without its query |
proxy | the proxy the request went through, or none |
server, via, proxy_status, squid_error | those headers of the answer, when it has them |
view | the snap store's X-View-Name: the store's backend that answered; a store error without it came from the store's edge |
body | the start of the answer |
A Via or X-Squid-Error header, or a proxy's error page in body, means a proxy on the way answered, not the upstream. A 408 is sent again, up to three times in all, each logged as repeating a request the upstream answered with 408; the snap store answers it now and then on a connection it is closing, and one followed by a success is nothing to worry about.
Pool files are kept forever, dists files are rechecked after a short window, and a failing mirror is covered by the cached copy.
pool/** and dists/**/by-hash/** are immutable:
cached forever, never revalidated. Other dists/** files are
revalidated after a TTL. Anything else is passed through uncached.
-apt-revalidate-ttl (default: -revalidate-ttl, 5m) is how long a
dists/** entry is served without asking the mirror;
-min-forced-revalidate-interval (default 30s)
bounds what a client's Cache-Control: no-cache can force. Both
are also snap options.
A failed revalidation serves the stale copy (outcome=stale,
with a warning). The wait is bounded at 10 seconds and a failed origin is
not retried for a minute. 404/410 become
404; a known-superseded entry that cannot be refetched gets
502.
A dists/** miss, by-hash included, is fetched,
stored, parsed and then served from the store. A pool/** miss
is a fill that streams to its clients while it writes.
Next: Immutable or not, for the same rule on every proxy engine; Fetch once, for how a pool miss streams; Offline mode, for apt update without the internet.
The first path segment named pool or dists makes a URL an archive path, whatever comes before it; there is no list of archives to configure.
| Path | Kind | Cache policy | Checked against |
|---|---|---|---|
…/pool/… | packages and sources | kept forever, never rechecked | the SHA-256 from a Packages or Sources file already read; unchecked, with a warning, when none is known |
…/dists/…/by-hash/ALGO/HEX | an index named by its hash | kept forever, never rechecked | the hash in its own name |
…/dists/…/InRelease, Release, Release.gpg | the signed root of a suite | rechecked after -apt-revalidate-ttl | nothing: apt checks the signature |
any other …/dists/… file: Packages*, Sources*, Contents*, Translation-*, *.diff/Index | indexes | rechecked after -apt-revalidate-ttl | the hash Release lists for it, when known |
a path with no pool or dists segment | anything else | passed through, not cached | — |
a directory: a trailing slash, the bare dists or pool, or one the mirror redirects to a trailing slash or answers with HTML | a directory listing | relayed, not cached | — |
a . or .. segment anywhere, or an empty one (//) from dists/ or pool/ on | not the object's own name | relayed, not cached | — |
any segment ending in .meta.json | could clash with pkgproxy's metadata files | relayed, not cached | — |
https:// sources, through CONNECT | encrypted | tunnelled, not cached | — |
A listing that an older pkgproxy cached as a file where a directory belongs is removed once the mirror answers a file beneath it, never on the strength of a request alone; until then a request beneath it is relayed.
Compressed .xz indexes are cached and served but not read, so a pool file listed only in one is cached unchecked. HTTPS sources are covered in HTTPS apt sources, and how the cache key is built in apt cache keys and hash checks.
| Situation | What pkgproxy does |
|---|---|
cached and checked within -apt-revalidate-ttl | serves it, asks nothing |
| older than that | sends one If-Modified-Since request for every client waiting on it; a 304 keeps the copy and restarts the clock |
no Last-Modified was recorded | fetches it again in full |
a newer Release lists a different hash for the cached copy | fetches it again in full, even within the TTL |
the mirror fails, stalls, or answers 5xx or another unexpected status | serves the cached copy, warns in the log, and the access log says outcome=stale |
| the check takes more than 10 seconds | serves the cached copy while the check carries on |
| the mirror failed a check within the last minute | serves the cached copy without asking it |
the mirror answers 404 or 410 | 404 to apt |
| the copy is known to be superseded and cannot be fetched again | 502 to apt |
A miss on an index is fetched whole, stored, read for hashes and only then served. A miss on a pool file streams to every waiting client while it is written; see Fetch once.
apt sends these headers for Acquire::http::No-Cache and Acquire::http::Max-Age.
| Request header | Effect on a cached index within its TTL |
|---|---|
Cache-Control: no-cache | checked with the mirror first, unless checked within -min-forced-revalidate-interval |
Cache-Control: max-age=N | checked first if the last check is older than N seconds, with the same bound |
Pragma: no-cache, with no Cache-Control | the same as no-cache |
A forced check is a conditional request: a 304 keeps the copy, so it never repairs wrong content; pkgproxyctl cache invalidate does. Pool files and by-hash files are never rechecked, whatever the client asks. In offline mode no check is forced.
| Situation | Status apt sees |
|---|---|
a request line that is not an absolute http:// URL, as apt sends to a proxy | 400 |
a method other than GET, HEAD or CONNECT | 405 |
CONNECT to a port other than 443 | 403 |
a passed-through request or CONNECT to a loopback address, or to this machine at a port pkgproxy listens on | 403 |
the mirror answers 404 or 410 | 404 |
| a downloaded file fails its hash check | the mirror's bytes, unchanged and not cached |
| the mirror cannot be reached and nothing is cached | 502 |
| Option | Environment variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-apt-revalidate-ttl | PKGPROXY_APT_REVALIDATE_TTL | apt-revalidate-ttl | -revalidate-ttl, 5m | how long an index is served before it is checked again |
-revalidate-ttl | PKGPROXY_REVALIDATE_TTL | revalidate-ttl | 5m | the same for every proxy engine that has no setting of its own |
-min-forced-revalidate-interval | PKGPROXY_MIN_FORCED_REVALIDATE_INTERVAL | min-forced-revalidate-interval | 30s | how recently checked an index must be for a client's no-cache to be ignored |
-fill-drain-timeout | PKGPROXY_FILL_DRAIN_TIMEOUT | fill-drain-timeout | 10m | how long a pool file that failed its hash check keeps going to the clients already receiving it |
-upstream-idle-timeout | PKGPROXY_UPSTREAM_IDLE_TIMEOUT | upstream-idle-timeout | 1m | how long a mirror may send nothing before the request fails; 0 waits forever |
The 10-second wait and the one-minute pause after a failure are fixed. Cached files live under apt/objects/HOST/PATH in the cache directory, such as apt/objects/archive.ubuntu.com/ubuntu/pool/main/a/apt/apt_2.8.3_amd64.deb; see The store.
How apt's https sources pass through pkgproxy as an uncached tunnel, and how to get them cached instead.
apt sends CONNECT host:443 to whatever
Acquire::http::Proxy names for any https://
source; no Acquire::https::Proxy is needed to trigger it.
pkgproxy dials first, then takes over the client's connection and copies
bytes both ways. No TLS termination, no caching, no verification. A dial
that fails gets 502; one that reaches this machine's loopback
gets 403. Other ports get 403: pkgproxy only
tunnels CONNECT to port 443.
Refusing CONNECT (with 501) would make
apt update fail on any machine with one https-only source. The
port-443 tunnel keeps the non-goal: no TLS termination, no HTTPS caching.
Neither the tunnel nor plain http reaches this machine's loopback or
pkgproxy's own ports.
A tunnelled request is logged with outcome=tunnel. Switch the
source to http:// where the archive offers it; apt checks
signatures either way. https-only archives keep working, uncached.
Next: The access log, for finding tunnels; pool and dists, for what an http source gets; Files by checksum, for https that pkgproxy fetches itself.
Next: A forward proxy, for plain http sources; pool and dists, for what they get from the cache; Files by checksum, for https that pkgproxy fetches itself.
How an apt URL becomes a cache key, and how each download is checked against the hash its index lists.
The archive id is the host (with its port, unless it is 80) plus the path prefix before pool/
or dists/; the key is the archive id and the path from there
on. A URL with neither segment is passed through uncached.
deb.debian.org/debian and
deb.debian.org/debian-security are separate archives, as are KDE
Neon's /unstable and /release. Mirrors on
different hosts are cached separately: mirror aliasing is not built yet.
InRelease lists the hashes of the index files;
Packages lists Filename and SHA256
for each package. Both are parsed as they pass through. A
by-hash URL names its own hash.
A conflicting hash for a known pool path is logged and ignored. Content that fails its hash check is never committed; clients already receiving it get the origin's bytes unmodified, the rest a passthrough fetch.
InRelease, Release and Release.gpg
are cached unverified; apt verifies the signature. .xz indexes
are not parsed, so a pool file listed only there is cached without a hash
check, with a warning in the log.
Next: pool and dists, for what a key does on a hit; The store, for where downloads are written; Fetch once, for one fetch per key.
How apt sends every request to pkgproxy as a forward proxy, set up with one line and undone with one command.
pkgproxyctl setup apt | sudo sh writes
/etc/apt/apt.conf.d/99pkgproxy with one
Acquire::http::Proxy line, plus
Acquire::Retries::Delay "false"; when apt is older than 3.1.3.
sources.list stays as it is.
A forward-proxy request has an absolute URI: GET
http://deb.debian.org/…. pkgproxy fetches from the host it names,
with Via: 1.1 pkgproxy, so any number of mirrors go through one
proxy. This machine's loopback, and its own addresses at pkgproxy's ports,
get 403. A request with only a path is answered
400.
Mirror substitution rewrites sources.list to name the cache,
which fronts one upstream. A forward proxy takes requests for any host
through one cache, with one Acquire::http::Proxy line per
machine.
If-Modified-Since gets 304 from the cache.
Range is honoured only when If-Range matches the
entity being sent; otherwise 200 with the whole body. apt's
default Acquire::http::Pipeline-Depth is 10.
apt-config dump | grep -i proxy shows the setting. apt does
not fail over: while pkgproxy is down, its downloads fail.
pkgproxyctl setup apt -undo | sudo sh removes
/etc/apt/apt.conf.d/99pkgproxy. Without | sudo sh,
setup explains each step instead.
Next: Cache keys, for keys and hash checks; pool and dists, for what is kept and rechecked; HTTPS sources, for https archives.
Next: pool and dists, for what apt gets from the cache; HTTPS sources,
for https archives; pkgproxyctl setup, for the other tools.
setup lxd and the craft setups put apt, snapd and Go inside
LXD build containers behind pkgproxy.
setup lxd-remote serves LXD its images;
setup lxd points apt, snapd and go inside the containers at
pkgproxy. A craft build needs setup lxd-remote and
setup snapcraft (or rockcraft,
charmcraft).
environment.GOPROXY and
environment.SNAPPY_FORCE_API_URL, plus raw.lxc
pre-start and post-stop hooks that write and remove
99pkgproxy-lxd and pkgproxy-lxd.conf. Each hook
always exits 0 and never follows a symbolic link.
Without -project: a standalone pkgproxy profile
(lxc launch … --profile pkgproxy --profile default). With
-project P: a PATCH of P's default profile. setup
snapcraft, rockcraft and charmcraft fix P
to the tool's own project.
The host of -addr (or of the address the setup page was
reached at): an IP is used as is, a name is resolved with
getent, and none, a wildcard, localhost or
loopback means the managed LXD bridge's ipv4.address (else
ipv6.address).
The LXD calls stay in the printed script, run by the user in the
lxd group with lxc … local:. The admin socket,
-admin-socket-path, serves only DELETE
/pkgproxy/v1/cache and /pkgproxy/v1/offline.
-undo deletes the pkgproxy profile, or unsets
environment.GOPROXY, environment.SNAPPY_FORCE_API_URL
and raw.lxc on a project's default profile. Setup and undo can each be
run again safely, and a second snapcraft pack is served from
the cache.
Next: pkgproxy for CI, for many build machines behind one cache; Offline mode, for builds that see only what is cached.
Next: LXD images, for the images the containers start from; pool and dists, for what apt gets from the cache; Offline mode, for building with no network.
The cache page draws what fills the cache, what is used and what is being downloaded now, for the machine that runs pkgproxy and the networks it trusts.
http://localhost:3142/pkgproxy/cache, over TCP from this
machine, without Forwarded, Via,
X-Forwarded-* or X-Real-IP. pkgproxy's apt proxy
refuses to connect to this machine's loopback or its own ports, and what it
fetches carries Via, so it cannot relay the page.
trusted-networks (-trusted-networks): CIDR
prefixes separated by commas, empty by default. A refused request gets
the 404 of any unknown path. Give visitors the machine's address or
public-url: a short host name may be answered by any machine
on the link (mDNS, LLMNR).
Ground is what the item takes on disk, as du counts it. A
kind's smallest items fold into one "N more items" block. The layout is
fixed once the page loads, so nothing moves.
Hits in: All time, Last week, Last day or Last hour. Drag to turn and tilt, scroll to zoom, and click a block to zoom into its kind, then its group. Beside the map, a table of every item can be sorted and searched.
Machines are named by their host names, looked up from their addresses; virtual machines and snaps on a socket are "Socket clients". Lines are as wide as the bytes they carried and fade over six seconds. "Turntable" spins the map slowly. The upstreams' dots, also in the byte flow, are read every 10 seconds; pointing at one says which part is in trouble.
Views: Wall dashboard, Map, Byte flow and Details; #map,
#flow or #details opens one. At most four pages
watch live at once. Without JavaScript the page shows a flat treemap and
its tables.
Next: Statistics, for the same counts from a script; The access log, for one request after the fact; pkgproxy for CI, for a build farm's screen.
| Path, on the main port | What it answers |
|---|---|
/pkgproxy/cache | the cache page: HTML for a browser, the same figures as plain text otherwise |
/pkgproxy/cache/activity | what the page polls for its heights and hits |
/pkgproxy/cache/events | the stream of requests behind the wall dashboard and the byte flow |
/pkgproxy/cache/upstreams | what the page polls, every 10 seconds, for the status dots on the upstreams |
/pkgproxy/upstreams | the upstream status; see Upstream status |
/ | the setup page, with a line about the cache and one about the upstreams, each linking to its page |
All of them are served only with setup-page on. Every answer with the cache's figures is sent with Cache-Control: no-store.
| Setting | Snap option | Flag | Default | Meaning |
|---|---|---|---|---|
| Trusted networks | trusted-networks | -trusted-networks | (empty) | Networks, in CIDR notation and separated by commas (a single address is one machine), whose visitors see the cache page, its live views, the upstream status pages, the setup page's lines about them and the Cache and Upstreams links in every page's bar, as a visitor on the machine does |
sudo snap set pkgproxy trusted-networks=192.168.0.0/24,fd00::/8
A request sees the page only when all of these hold; anyone else gets the 404 of any unknown path, and the setup page shows them no line about the cache or the upstreams:
Forwarded, Via, X-Forwarded-*, X-Real-IP, True-Client-IP, CF-Connecting-IP;Host: an IP address, localhost, the machine's host name or the host of public-url. Another web site whose name someone points at the machine cannot read the page.0.0.0.0/0 and ::/0 are refused, so that opening the page to everyone is never a typo; a wide network such as 0.0.0.0/1 is still accepted. pkgproxy logs the networks it trusts when it starts.
The machine's names count as much as its addresses. Set public-url only to a name you control. Its short host name is trusted too, and on a network where any machine can answer for a short name (mDNS or LLMNR, as Windows and macOS use), a visitor reaching pkgproxy by that name trusts those answers: give visitors its address or public-url instead.
The pkgproxy-k8s charm: the rock as its workload, the cache
on Juju storage, options, Services and actions.
juju deploy pkgproxy-k8s: Charmhub releases the charm with the
pkgproxy-image resource of its commit, by digest. A local
charm needs --resource pkgproxy-image=IMAGE. One unit only.
Storage cache: a filesystem of at least 16G at
/var/cache/pkgproxy, owned by 584792, the rock's user. Size it
with --storage cache=SIZE. Without extended attributes the
unit is blocked.
juju config pkgproxy-k8s NAME=VALUE becomes
PKGPROXY_NAME when it is not the default.
pkgproxy -check-config runs first; a refusal blocks the unit.
The model's juju-http-proxy and the others become
HTTP_PROXY, HTTPS_PROXY and NO_PROXY.
service-type=cluster (default):
pkgproxy-k8s.MODEL.svc:3142.
service-type=loadbalancer adds APP-lb with
externalTrafficPolicy: Local. Refused:
juju trust APP --scope=cluster.
juju run pkgproxy-k8s/0 stats, invalidate url=URL
and setup-script client=CLIENT [undo=true], for apt, snap,
go, docker, lxd, lxd-remote, snapcraft, rockcraft and charmcraft. Offline
mode is the offline-mode option.
Next: In a container, for the rock; Juju machines, for the machine charm; The setup page, for pointing clients at it.
| Command | What it does |
|---|---|
juju deploy pkgproxy-k8s | the charm from Charmhub, with the rock released with it |
juju deploy pkgproxy-k8s --storage cache=100G | the same, with a cache larger than the 16G minimum |
juju deploy ./CHARM --resource pkgproxy-image=IMAGE | a charm you packed, with the rock you name |
pkgproxy-k8s, base Ubuntu 26.04, amd64. It needs Juju 3.6 or later.latest/edge follows the main branch, latest/stable the releases. Each charm revision is released with the rock of the same commit, by digest.Each option is the daemon setting of the same name, passed as its PKGPROXY_* variable only when it differs from the default. Settings has what each one does.
| Option | Default | Sets |
|---|---|---|
public-url | empty | the address the setup page tells visitors to use |
setup-page | true | the setup page at / and its scripts under /setup/ |
trusted-networks | empty | networks whose visitors may see the cache page |
offline-mode | store-overloaded | store-overloaded, never or offline |
revalidate-ttl | 5m0s | how long lists are served before they are checked |
upstream-idle-timeout | 1m0s | how long an upstream may send nothing |
upstream-retry-for | 2m0s | how long a broken-off download is asked for again |
upstream-check-interval | empty | how long a quiet upstream goes unasked before pkgproxy checks it; at least 5m (see Upstream status) |
upstream-pkgproxy | empty | another pkgproxy, as http://host:port, to fetch everything through (see Two tiers) |
docker-upstream | https://registry-1.docker.io | the Docker Hub registry, or off |
files-allow-host | empty | hosts files by checksum may come from |
files-allow-network | empty | private networks files may come from |
files-max-size | 8G | the largest file by checksum |
files-volatile-max-size | 1M | the largest volatile file fetched by URL |
service-type | cluster | cluster, or loadbalancer for a Service with an outside address (the charm's own) |
lxd-tls | true | false turns port 3143 off (lxd-tls-addr=off) |
pkgproxy -check-config in the workload with the new environment. A refused value blocks the unit with the daemon's message, and pkgproxy keeps the last good settings.juju-http-proxy, juju-https-proxy and juju-no-proxy reach pkgproxy as HTTP_PROXY, HTTPS_PROXY and NO_PROXY, in both spellings.Storage cache | a filesystem of at least 16G, mounted at /var/cache/pkgproxy; a persistent volume claim that outlives the pod |
| Owner | UID and GID 584792, the rock's user; the charm hands the volume to it before pkgproxy starts |
| Extended attributes | required; a volume without them blocks the unit |
| Ports | 3142, and 3143 unless lxd-tls is false |
| In the cluster | http://pkgproxy-k8s.MODEL.svc:3142 |
service-type=loadbalancer | a Service APP-lb of type LoadBalancer, with externalTrafficPolicy: Local, so pkgproxy sees each client's own address |
Where the cluster refuses to let the charm manage APP-lb, the unit is blocked with service-type=loadbalancer needs trust: juju trust APP --scope=cluster, and pkgproxy keeps serving the clients in the cluster. Back on cluster, and when the application is removed, APP-lb is deleted.
| Action | Parameters | Result |
|---|---|---|
stats | none | output: the report of pkgproxyctl stats |
invalidate | url (required): the http:// address of an apt file under dists/ or pool/ | output: what pkgproxyctl cache invalidate printed |
setup-script | client (required): apt, snap, go, docker, lxd, lxd-remote, snapcraft, rockcraft or charmcraft; undo (default false) | script: the script to run on the client |
juju run pkgproxy-k8s/0 ACTION [PARAM=VALUE ...]. A failure gives what pkgproxyctl printed.setup-script uses the host and port of public-url when it is set (port 80 when it names none), else the LoadBalancer Service's address, else APP.MODEL.svc:3142. For lxd-remote it adds port 3143 on the same host.setup-script refuses docker when docker-upstream is off, and lxd-remote when lxd-tls is false.apt, snap and docker as root, for go as yourself, and for lxd, lxd-remote and the craft tools as the user who runs lxc. Each craft tool takes its base images through lxd-remote, so set that up too.offline-mode option.The first that applies:
| Status | Message |
|---|---|
| blocked | scale to one unit: more are not supported yet (pkgproxy#111) |
| blocked | an unknown service-type, or the juju trust message above |
| blocked | the cache storage does not keep extended attributes: use another storage pool |
| blocked | the daemon's message about a refused option |
| waiting | waiting for the cache storage, waiting for Pebble, waiting for pkgproxy to answer |
| active | empty; offline: serving the cache only; or offline by itself: REASON |
Offline is active, not blocked: what the cache holds is still served.
The pkgproxy charm: the snap at a pinned revision, options
through snap set, optional storage and open ports, and the Kubernetes charm's actions.
The charm runs snap install pkgproxy --revision=N and
snap refresh --hold pkgproxy; juju refresh
moves it to the next pin. A non-empty pkgproxy-snap resource
is installed with --dangerous instead.
juju config pkgproxy NAME=VALUE runs
snap set pkgproxy NAME=VALUE, or NAME! for the
default. The model's proxies set http-proxy,
https-proxy and no-proxy.
juju deploy pkgproxy --storage cache=100G mounts it at
/var/snap/pkgproxy/common/cache. juju expose
pkgproxy opens 3142 and 3143. Offline shows as an active status
with a message.
juju run pkgproxy/0 stats, invalidate url=URL
and setup-script client=CLIENT [undo=true]. The address is
public-url's, else the unit's juju-info address.
Next: On Kubernetes, for the same options and actions; Settings, for what each option sets; The setup page, for pointing clients at it.
| Command | What it does |
|---|---|
juju deploy pkgproxy | the charm from Charmhub, which installs the snap revision it pins |
juju deploy pkgproxy --storage cache=100G | the same, with the cache on Juju storage |
juju expose pkgproxy | makes ports 3142 and 3143 reachable |
juju refresh pkgproxy | a new charm revision, and the snap revision it pins |
juju attach-resource pkgproxy pkgproxy-snap=./pkgproxy.snap | installs that snap with --dangerous instead of the pinned one |
pkgproxy, for Ubuntu 24.04 and 26.04, amd64 only for now.latest/edge follows the main branch, latest/stable the releases.pkgproxy from the store at the revision it pins for the machine's architecture, then holds it with snap refresh --hold pkgproxy. The snap does not refresh by itself.v0.12.1-6-g169a226, is matched to the snap's, v0.12.1+git6.169a226; the commit hashes need only agree in their common prefix.pkgproxy-snap resource. A non-empty one replaces the pinned snap, at install and whenever a new one is attached.The options of On Kubernetes, without service-type. Each is the snap option of the same name: the charm runs snap set pkgproxy NAME=VALUE when it differs from its default, and snap set pkgproxy NAME! otherwise.
| Option | Default | Snap option |
|---|---|---|
public-url | empty | public-url |
setup-page | true | setup-page |
trusted-networks | empty | trusted-networks |
offline-mode | store-overloaded | offline-mode |
revalidate-ttl | 5m0s | revalidate-ttl |
upstream-idle-timeout | 1m0s | upstream-idle-timeout |
upstream-retry-for | 2m0s | upstream-retry-for |
upstream-check-interval | empty | upstream-check-interval |
upstream-pkgproxy | empty | upstream-pkgproxy |
docker-upstream | https://registry-1.docker.io | docker-upstream |
files-allow-host | empty | files-allow-host |
files-allow-network | empty | files-allow-network |
files-max-size | 8G | files-max-size |
files-volatile-max-size | 1M | files-volatile-max-size |
lxd-tls | true | false sets lxd-tls-addr=off |
pkgproxy -check-config. When it refuses a value, snapd keeps the last good options and the unit is blocked with the hook's last line.juju-http-proxy, juju-https-proxy and juju-no-proxy set the snap's http-proxy, https-proxy and no-proxy. Without them those are unset, and the snap takes the proxies of /etc/environment.Storage cache | optional, at most one, at least 16G, mounted at /var/snap/pkgproxy/common/cache |
| Without it | the cache is on the machine's root disk |
| Extended attributes | required; storage without them blocks the unit |
| Ports | 3142, and 3143 unless lxd-tls is false, opened for juju expose |
| Status | Message |
|---|---|
| blocked | the cache storage does not keep extended attributes: use another storage pool |
| blocked | the configure hook's last line, for a refused option |
| waiting | waiting for pkgproxy to answer |
| active | empty; offline: serving the cache only; or offline by itself: REASON |
The same actions as the Kubernetes charm, each running the snap's pkgproxyctl on the machine:
| Action | Parameters | Result |
|---|---|---|
stats | none | output: the report of pkgproxyctl stats |
invalidate | url (required): the http:// address of an apt file under dists/ or pool/ | output: what pkgproxyctl cache invalidate printed |
setup-script | client (required): apt, snap, go, docker, lxd, lxd-remote, snapcraft, rockcraft or charmcraft; undo (default false) | script: the script to run on the client |
setup-script uses the host and port of public-url when it is set (port 80 when it names none), else the unit's address on port 3142. For lxd-remote it adds port 3143 on the same host.docker when docker-upstream is off, and lxd-remote when lxd-tls is false.apt, snap and docker as root, and for go, lxd, lxd-remote and the craft tools as yourself, the user who runs lxc. Each craft tool takes its base images through lxd-remote, so set that up too.juju run pkgproxy/0 setup-script client=apt --format=json | jq -r '.[].results.script' | sudo sh
| Machine charm | Kubernetes charm | |
|---|---|---|
| Runs | the snap, at a pinned revision | the rock, by digest |
| Options reach pkgproxy as | snap options | PKGPROXY_* variables |
| Cache storage | optional | required |
service-type | none: juju expose | cluster or loadbalancer |
Address setup-script gives without public-url | the unit's juju-info address, port 3142 | the LoadBalancer's, else APP.MODEL.svc:3142 |
CI runners, VMs and build containers share one pkgproxy, over the network or a UNIX socket, and can run against only what is cached.
curl -fsSL http://host:3142/setup/TOOL | sudo sh configures
a machine; ?undo=1 reverses it. public-url sets
the address the page names, for example behind NAT.
GitLab: extra_hosts = ["pkgproxy:host-gateway"] in
config.toml, plus a mounted apt.conf.d file and
GOPROXY. GitHub: options: --add-host=pkgproxy:host-gateway,
then curl -fsSL http://pkgproxy:3142/setup/apt | sh.
-socket-path replaces the TCP listener with a UNIX socket
(mode 0777), for QEMU's guestfwd; -addr still
names the host:port the instructions print.
-lxd-tls-socket-path does the same for -lxd-tls-addr.
The snap runs with -content-socket-path and
-lxd-tls-content-socket-path under
$SNAP_COMMON/sockets, shared by the sockets slot
(content pkgproxy-sockets). The TCP listeners stay as they are. snapd auto-connects a plug with the same content from the same publisher; sideloaded (--dangerous) snaps need snap connect.
setup docker edits registry-mirrors in
daemon.json (Docker Hub, anonymous pulls).
/files/v1/fetch?url=…&sha256=… caches by checksum, over
http or https; files-allow-host,
files-allow-network and files-max-size (8G)
limit it.
snap set pkgproxy offline-mode=offline or
pkgproxyctl offline on; neither restarts pkgproxy.
store-overloaded backs off from a troubled upstream.
auto-offline=off on an isolated network.
PKGPROXY_URL=http://HOST[:PORT], no path, plain HTTP; then
curl -fsSL $PKGPROXY_URL/setup/<tool> | sh,
/files/v1/fetch, or upstream-pkgproxy. The
daemon itself never reads it.
Next: The setup page, for the scripts every machine fetches; Hosted CI runners, for jobs on the forge's runners; Two tiers, for each job's pkgproxy behind the one next to the runners.
A job that runs on the machine itself, as a GitHub self-hosted runner does by default and GitLab's shell executor does, uses whatever the machine is set up to use. Set each tool up once:
pkgproxyctl setup apt | sudo sh pkgproxyctl setup snap | sudo sh pkgproxyctl setup docker | sudo sh sudo -u <runner user> sh -c 'pkgproxyctl setup go | sh'
setup go sets up the user who runs it, so run it as the user the runner runs its jobs as. On a machine that does not run pkgproxy itself, take the scripts from the one that does: curl -fsSL 'http://<host>:3142/setup/apt' | sudo sh.
A job container sees none of the runner machine's settings. Give it the host name pkgproxy for the machine, through Docker's host-gateway, and let the containers' networks reach port 3142 through the machine's firewall, for example sudo ufw allow from 172.16.0.0/12 to any port 3142 proto tcp.
GitLab's docker executor, in the runner's section of /etc/gitlab-runner/config.toml:
[[runners]]
environment = ["GOPROXY=http://pkgproxy:3142/go,direct"]
[runners.docker]
extra_hosts = ["pkgproxy:host-gateway"]
volumes = ["/cache", "/etc/gitlab-runner/pkgproxy-apt.conf:/etc/apt/apt.conf.d/99pkgproxy:ro"]
with the apt configuration every job container mounts:
echo 'Acquire::http::Proxy "http://pkgproxy:3142";' | sudo tee /etc/gitlab-runner/pkgproxy-apt.conf echo 'Acquire::Retries::Delay "false";' | sudo tee -a /etc/gitlab-runner/pkgproxy-apt.conf
The second line is for job images with apt older than 3.1.3, such as Ubuntu 24.04's. A GitHub job in a container:
jobs:
build:
runs-on: self-hosted
container:
image: ubuntu:24.04
options: --add-host=pkgproxy:host-gateway
steps:
- run: |
apt-get update && apt-get install -y curl
curl -fsSL http://pkgproxy:3142/setup/apt | sh
A tool that fetches large files by checksum gets http://pkgproxy:3142 in the job's environment; image-garden reads it from GARDEN_DL_PKGPROXY.
Software that can use a pkgproxy when there is one, such as image-garden, the snapcraft and rockcraft CI components and pkgproxy's own spread suite, reads PKGPROXY_URL:
PKGPROXY_URL=http://cache.example:3142
http://HOST[:PORT], with no path; a trailing slash is allowed.localhost and not a container runtime's name for the host.curl -fsSL $PKGPROXY_URL/setup/<tool> | sh, /files/v1/fetch, or as the upstream-pkgproxy of a pkgproxy of its own (see Two tiers).The daemon itself never reads PKGPROXY_URL.
A job that runs its own pkgproxy, as the snap in a fresh VM, starts with an empty cache. Set its upstream-pkgproxy to the pkgproxy next to the runners, which runs with offline-mode=store-overloaded, and what that one has cached is a LAN away for every job, also while an origin is down. See Two tiers.
Run the same job twice, then run pkgproxyctl stats: every download of the second run counts as a hit. The access log says the same per request (outcome=hit), and so does the cache page. A job that does not show up there does not use pkgproxy.
On runners the forge hosts, pkgproxy runs inside the job and its cache lives in the CI cache between jobs; setup-pkgproxy does that on GitHub.
uses: zyga/setup-pkgproxy@v2. save-package-cache
is default-branch, always or never;
a job that added nothing saves nothing. Set up other tools with
pkgproxyctl setup TOOL -addr "${{ steps.pkgproxy.outputs.addr }}".
v1 workflows end with a step that stops pkgproxy.service
and a step with actions/cache/save. Caches saved by v1.0
and v1.1 were a plain directory, so the first run after moving from them
starts cold.
The job starts pkgproxy on 127.0.0.1:3142 and runs
pkgproxyctl setup against it. The CI cache restores the
whole cache at the start of each job and saves it when it changed.
Archive with tar --xattrs --xattrs-include='user.*' after
pkgproxy has stopped, and unpack the same way before it starts. A file
that lost its attribute reads its .meta.json file, kept for
going back to 0.11; one that lost both is a miss, and pkgproxy says so
once in its log.
PKGPROXY_CACHE_DIR under $CI_PROJECT_DIR,
cache: when: always, and after_script to save
even a failed job's downloads. Keep pkgproxy.log as an
artifact.
A public pkgproxy is an open proxy: no client authentication, plain
http:// to any host but its own and CONNECT to
port 443 of any of them, and files by checksum up to files-max-size each,
in a cache that deletes nothing. Run pkgproxy in the job instead.
Next: pkgproxy for CI, for runners you host; The store, for the cache on disk; Offline mode, for leaving a troubled mirror or store alone.
setup-pkgproxy is a GitHub Action that runs pkgproxy in the job. Both major versions:
go install on a miss, and install it as /usr/local/bin/pkgproxy with a pkgproxyctl link;/var/cache/pkgproxy, and start pkgproxy.service;configure-host is false;tar --xattrs (from v1.2.0 on), which the unit writes to /var/cache/pkgproxy-archive/cache.tar when pkgproxy stops.v2 (zyga/setup-pkgproxy@v2) | v1 (zyga/setup-pkgproxy@v1) | |
|---|---|---|
| saving the cache | the action's post step, at the end of the job, passed, failed or cancelled | the workflow's own steps: stop pkgproxy.service, hand the archive to the runner user, actions/cache/save |
| which runs save | save-package-cache: default-branch (default), always or never | wherever the workflow's save step runs, pull requests included |
| cache keys | a new key per saving job, <package-cache-key>-<arch>-<run>-…; the next job restores the newest | one key per CPU arch, <package-cache-key>-<arch>-success or -partial |
| a job that added nothing | saves nothing | the save step skips an exact hit (package-cache-hit) |
| pkgproxy | 0.12.0 | 0.12.0 |
on:
push:
branches: [main] # only the default branch saves the cache
pull_request:
# ...
steps:
- uses: zyga/setup-pkgproxy@v2
id: pkgproxy
- run: pkgproxyctl setup go -addr "${{ steps.pkgproxy.outputs.addr }}" | sh
| Input | Default | |
|---|---|---|
version | v0.12.0 | the pkgproxy version to build |
go-version | 1.27.1 | the Go that builds it on a binary-cache miss |
addr | :3142 | the address pkgproxy listens on |
binary-cache-key | pkgproxy-bin | key prefix of the built binary |
package-cache-key | pkgproxy-cache-v4 | key prefix of the package cache |
save-package-cache | default-branch | which runs save the package cache |
configure-host | true | run pkgproxyctl setup apt and setup snap on the runner |
| Output | |
|---|---|
addr | the address to pass to pkgproxyctl setup … -addr |
cache-dir | /var/cache/pkgproxy |
package-cache-restored-key | the key the cache was restored from, empty on a cold start |
package-cache-key | the key the post step saves under, empty when this run may not save |
Every run can restore the caches of its own ref, of its pull request's base branch and of the default branch. A cache a pull request saved helps only that pull request, and takes room the default branch's needs, so v2 saves on the default branch only by default. A workflow that runs only for pull requests then never saves the cache, so run it on pushes to the default branch too, as in the example above. In a matrix with several jobs on one CPU arch, give all but one save-package-cache: never.
- uses: zyga/setup-pkgproxy@v1
id: pkgproxy
# ... the job's own steps
- if: always()
run: |
sudo systemctl stop pkgproxy.service || true
sudo chown -R "$USER:$USER" ${{ steps.pkgproxy.outputs.cache-dir }}
- if: steps.pkgproxy.outputs.package-cache-hit != 'true'
uses: actions/cache/save@v6
with:
path: ${{ steps.pkgproxy.outputs.cache-dir }}
key: ${{ steps.pkgproxy.outputs.package-cache-key-success }}
In v1, cache-dir is /var/cache/pkgproxy-archive, and the outputs package-cache-hit, package-cache-key-success and package-cache-key-partial drive the save. To move to v2, change @v1 to @v2, delete the two save steps, and read the outputs v2 has instead. v1 stays maintained, on the action's releases/v1 branch. v1.2 saves the same archive under the same pkgproxy-cache-v4 prefix, so its cache warms v2; caches saved by v1.0 and v1.1 were the plain directory, and the first run after them starts cold.
pkgproxy keeps each cached file's metadata in the user.pkgproxy.meta extended attribute of the file, which is replaced together with the file, and a copy in a .meta.json file beside it, for going back to 0.11. A CI cache that archives the directory itself may not keep extended attributes: a file that lost its attribute then reads its .meta.json file, and one that lost both is a miss, downloaded again, and pkgproxy logs once that it found such a file. Archive the cache yourself, keeping the attributes, and let the CI cache keep that one file:
# stop pkgproxy first, so that nothing is half written tar --xattrs --xattrs-include='user.*' -cf pkgproxy-cache.tar .pkgproxy # before the next start tar --xattrs --xattrs-include='user.*' -xf pkgproxy-cache.tar
A cache directory on a file system without extended attributes keeps pkgproxy from starting, and its log says why.
A GitLab job runs in a container without a service manager: start pkgproxy in the background, with its cache in the project directory, and keep its archive in GitLab's cache. Its sockets stay in /run/pkgproxy, which keeps their paths short.
variables:
PKGPROXY_CACHE_DIR: $CI_PROJECT_DIR/.pkgproxy
PKGPROXY_ADDR: 127.0.0.1:3142
PKGPROXY_LXD_TLS_ADDR: "off"
build:
image: golang:1.25
cache:
key: pkgproxy
paths: [pkgproxy-cache.tar]
when: always
before_script:
- go install gitlab.com/zygoon/pkgproxy/cmd/pkgproxy@latest
- ln -s pkgproxy "$(go env GOPATH)/bin/pkgproxyctl"
- if [ -f pkgproxy-cache.tar ]; then tar --xattrs --xattrs-include='user.*' -xf pkgproxy-cache.tar; fi
- pkgproxy > pkgproxy.log 2>&1 & echo $! > pkgproxy.pid
- until curl -fs -o /dev/null http://127.0.0.1:3142/; do sleep 1; done
- pkgproxyctl setup apt -addr 127.0.0.1:3142 | sh
- pkgproxyctl setup go -addr 127.0.0.1:3142 | sh
script:
- ...
after_script:
- pid=$(cat pkgproxy.pid); kill "$pid"; while kill -0 "$pid" 2>/dev/null; do sleep 1; done
- tar --xattrs --xattrs-include='user.*' -cf pkgproxy-cache.tar .pkgproxy
Every setting is read from its PKGPROXY_ variable. From 0.12.0 on, a job can download a released pkgproxy and check it instead of building it with go install. Keep pkgproxy.log as an artifact to see what each request was served from.
On runners you host, a pkgproxy in each job can fetch through one next to the runners: set the job's upstream-pkgproxy to it (see Two tiers). The hop carries snapd's and Docker's credentials in plain HTTP, so only on a network you trust; a public pkgproxy is not one (below). Software in the job that reads PKGPROXY_URL (see pkgproxy for CI) needs an address its VMs and containers can reach too: 127.0.0.1 works only for what runs on the job's machine itself.
| clients | no authentication: every client that reaches pkgproxy may use it |
| apt's proxy engine | passes plain http:// requests to any host but pkgproxy's own machine, and tunnels CONNECT to port 443 of any of them |
| files by checksum | downloads any public URL anyone names, up to files-max-size each |
| the cache | never deletes anything, so anyone can fill its disk |
Run pkgproxy in the job instead.
Every daemon setting is one name, set by a flag, an environment variable or a snap option, checked before it applies.
-revalidate-ttl 10m,
PKGPROXY_REVALIDATE_TTL=10m and snap set pkgproxy
revalidate-ttl=10m set one setting. Precedence: flag, then
environment, then default.
pkgproxy-env runs snapctl get for each key it
lists and exports PKGPROXY_*, then execs the daemon.
snap unset pkgproxy KEY restores the default.
The configure hook runs pkgproxy-env pkgproxy -check-config.
A bad value fails snap set; the daemon is not touched.
$SNAP_DATA/configured.env holds the last
PKGPROXY_* set minus PKGPROXY_OFFLINE_MODE. Same:
pkgproxyctl offline mode -keep-automatic. Different:
snapctl restart.
Not options: cache-dir, the socket paths,
check-config, offline, the upstream URLs, the
base paths, lxd-remote, go-sumdb.
snap set accepts any other key and ignores it; the Settings
table in DEPLOYMENT.md lists the options.
http-proxy, https-proxy, no-proxy:
over /etc/environment, none for off, a restart
to apply. Then snap unset system proxy.http, so snapd reaches
pkgproxy directly.
Next: Immutable or not, for the TTL settings; Offline mode, for the offline-mode setting; Statistics, for what the daemon reports.
Every setting is one name, such as revalidate-ttl. It is set by:
| Route | Form | Example |
|---|---|---|
| Command line | -name value | pkgproxy -revalidate-ttl 10m |
| Environment | PKGPROXY_ and the name in capitals, - as _ | PKGPROXY_REVALIDATE_TTL=10m |
| Snap option | the name, for the options listed below | sudo snap set pkgproxy revalidate-ttl=10m |
sudo snap unset pkgproxy NAME goes back to it, and snap get pkgproxy shows what you set.snap set accepts any key. A key that is not a snap option below is stored and has no effect.90s, 10m or 1h30m. Sizes like 8G or 500M.pkgproxy -check-config reads every flag and PKGPROXY_* variable, checks them and exits without starting the daemon. The snap's configure hook runs it on every snap set, so a bad value fails snap set and the running daemon is not touched. It refuses:
revalidate-ttl=ten, an unknown offline-mode or auto-offline word, or a bad size;addr or lxd-tls-addr that is not host:port or :port, a port outside 0 to 65535, or a host that is neither an IP address nor a host name (whether the name resolves is found only when the daemon starts);revalidate-ttl, or one proxy engine's TTL, of zero or less;trusted-networks entry that is neither an address nor a network, or a network of every address (0.0.0.0/0, ::/0);-offline together with an -offline-mode that names a different mode;pkgproxy -migrate-cache copies the .meta.json file of each file pkgproxy 0.11 cached into the file's extended attribute, keeps the .meta.json files, prints how many it migrated, and exits without starting the daemon. It stops after -migrate-cache-timeout (2m); a file it did not reach still works. The snap runs it after every refresh and keeps its output in /var/snap/pkgproxy/current/migrate-cache.log. Without the snap, nothing requires it. See The store.
The configure hook compares the settings the daemon would start with, the proxy options included and offline-mode left out, with the copy it saved last time in /var/snap/pkgproxy/current/configured.env.
| Case | What happens |
|---|---|
| The daemon is not running | nothing; the new values apply when it starts |
Only offline-mode changed, or nothing | pkgproxyctl offline mode -keep-automatic MODE switches the running daemon; no restart |
| Anything else changed, the copy is missing, or the switch fails | the service restarts |
pkgproxyctl offline also writes offline-mode back as a snap option, without running the hook, so the mode survives a restart.
| Setting | Variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-addr | PKGPROXY_ADDR | addr | :3142 | address of the main HTTP port, for every client and the setup page |
-lxd-tls-addr | PKGPROXY_LXD_TLS_ADDR | lxd-tls-addr | :3143 | address of the HTTPS port lxc needs for image servers; off turns it off |
-socket-path | PKGPROXY_SOCKET_PATH | flag or environment only | (none) | listen on this UNIX socket instead of -addr; setup still names -addr |
-lxd-tls-socket-path | PKGPROXY_LXD_TLS_SOCKET_PATH | flag or environment only | (none) | serve HTTPS on this UNIX socket instead of -lxd-tls-addr |
-content-socket-path | PKGPROXY_CONTENT_SOCKET_PATH | pinned by the snap | (none); snap: $SNAP_COMMON/sockets/pkgproxy.sock | also serve the main port on this UNIX socket, to share the cache with another snap |
-lxd-tls-content-socket-path | PKGPROXY_LXD_TLS_CONTENT_SOCKET_PATH | pinned by the snap | (none); snap: $SNAP_COMMON/sockets/pkgproxy-lxd-tls.sock | the same for the HTTPS port |
-stats-socket-path | PKGPROXY_STATS_SOCKET_PATH | flag or environment only | /run/pkgproxy/stats.sock; snap: $SNAP_COMMON/stats.sock | UNIX socket that pkgproxyctl stats, open and setup read |
-admin-socket-path | PKGPROXY_ADMIN_SOCKET_PATH | flag or environment only | /run/pkgproxy/admin.sock; snap: $SNAP_COMMON/admin.sock | UNIX socket that pkgproxyctl cache and offline change pkgproxy through |
-setup-page | PKGPROXY_SETUP_PAGE | setup-page | true | serve the setup page at / and its scripts under /setup/; false also hides the applications menu entry |
-public-url | PKGPROXY_PUBLIC_URL | public-url | (the address the visitor used) | http://host[:port] the setup page tells visitors to use |
-trusted-networks | PKGPROXY_TRUSTED_NETWORKS | trusted-networks | (none: this machine only) | networks in CIDR notation, separated by commas, such as 192.168.0.0/24,fd00::/8, whose visitors see the cache page and what the cache holds as visitors on this machine do; see The cache page |
-read-header-timeout | PKGPROXY_READ_HEADER_TIMEOUT | read-header-timeout | 30s | how long a client may take to send its request headers |
-shutdown-grace | PKGPROXY_SHUTDOWN_GRACE | shutdown-grace | 10s | how long requests in progress get to finish when pkgproxy stops |
See One listener for what each port and socket answers.
| Setting | Variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-cache-dir | PKGPROXY_CACHE_DIR | pinned by the snap | /var/cache/pkgproxy; snap: $SNAP_COMMON/cache | directory that holds the cache and the LXD certificate; it must be on a file system that keeps extended attributes (ext4, XFS, btrfs, ZFS, tmpfs from Linux 6.6), or pkgproxy refuses to start; see The store |
-fill-drain-timeout | PKGPROXY_FILL_DRAIN_TIMEOUT | fill-drain-timeout | 10m | how long a failed-hash download of an apt package, snap or LXD image keeps going to clients already receiving it; Docker layers and files are cut short |
-cache-metadata | PKGPROXY_CACHE_METADATA | cache-metadata | xattr | where pkgproxy writes what it knows of each cached file: xattr to the file's extended attribute, and to a .meta.json file beside it so that going back to 0.11 keeps the cache; meta-json to the .meta.json files alone, as 0.11 did, only to test the migration of such a cache |
-upstream-idle-timeout | PKGPROXY_UPSTREAM_IDLE_TIMEOUT | upstream-idle-timeout | 1m | how long an upstream may send nothing before its request fails; 0 waits forever |
-upstream-retry-for | PKGPROXY_UPSTREAM_RETRY_FOR | upstream-retry-for | 2m | how long after an upstream last sent a byte of a snap, LXD image or file by checksum that broke off pkgproxy still asks it for the rest, waiting 1, 2, 4, 8 and 16 s between tries; a Go module download that fails is asked again from the start, the same way, for this long after it was first asked for; 0 never asks again |
-stats-checkpoint-interval | PKGPROXY_STATS_CHECKPOINT_INTERVAL | stats-checkpoint-interval | 30s | how often statistics are saved; a crash loses what changed since |
-lxd-checkpoint-interval | PKGPROXY_LXD_CHECKPOINT_INTERVAL | lxd-checkpoint-interval | 30s | how often LXD image file hashes are saved |
-upstream-check-interval | PKGPROXY_UPSTREAM_CHECK_INTERVAL | upstream-check-interval | 0 (off) | how long an upstream nobody asked goes unasked before pkgproxy sends it a small check of its own: the snap store's search, details, assertions and catalogue, the Go proxy and the Docker registry; at least 5m; see Upstream status |
See The store and Stalled upstreams.
| Setting | Variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-revalidate-ttl | PKGPROXY_REVALIDATE_TTL | revalidate-ttl | 5m | how long what changes is served before it is checked again, for every proxy engine without its own value |
-apt-revalidate-ttl | PKGPROXY_APT_REVALIDATE_TTL | apt-revalidate-ttl | revalidate-ttl | the same, apt indexes alone |
-snap-revalidate-ttl | PKGPROXY_SNAP_REVALIDATE_TTL | snap-revalidate-ttl | revalidate-ttl | the same, snap details and assertions alone |
-go-revalidate-ttl | PKGPROXY_GO_REVALIDATE_TTL | go-revalidate-ttl | revalidate-ttl | the same, Go module version lists alone |
-docker-revalidate-ttl | PKGPROXY_DOCKER_REVALIDATE_TTL | docker-revalidate-ttl | revalidate-ttl | the same, Docker image tags alone |
-lxd-revalidate-ttl | PKGPROXY_LXD_REVALIDATE_TTL | lxd-revalidate-ttl | revalidate-ttl | the same, LXD image lists alone |
-min-forced-revalidate-interval | PKGPROXY_MIN_FORCED_REVALIDATE_INTERVAL | min-forced-revalidate-interval | 30s | how soon after a check an apt index is still served from the cache when a client asks for a fresh copy |
Every TTL must be longer than zero. See Immutable or revalidated.
| Setting | Variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-offline-mode | PKGPROXY_OFFLINE_MODE | offline-mode | store-overloaded | store-overloaded, never or offline; switch at run time with pkgproxyctl offline. The default was never up to 0.12.0 |
-offline | PKGPROXY_OFFLINE | flag or environment only | (unset) | short for -offline-mode offline; -offline=false is -offline-mode never |
-auto-offline | PKGPROXY_AUTO_OFFLINE | auto-offline | network on Linux, off elsewhere | off, network (no network, airplane mode included) or airplane-mode (all radios off) |
See Offline mode.
| Setting | Variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-snap-store-url | PKGPROXY_SNAP_STORE_URL | flag or environment only | https://api.snapcraft.io | Snap Store API that snaps and assertions come from |
-snap-external-url | PKGPROXY_SNAP_EXTERNAL_URL | flag or environment only | (the address the client used) | http://host[:port] put in the download links handed to snapd |
-go-upstream | PKGPROXY_GO_UPSTREAM | flag or environment only | https://proxy.golang.org | Go module proxy that modules come from |
-go-sumdb | PKGPROXY_GO_SUMDB | flag or environment only | sum.golang.org | checksum database modules are checked against before they are cached; off caches them unchecked |
-go-base-path | PKGPROXY_GO_BASE_PATH | flag or environment only | /go | URL path of the Go module proxy |
-docker-upstream | PKGPROXY_DOCKER_UPSTREAM | docker-upstream | https://registry-1.docker.io | registry Docker Hub images come from; off turns the mirror off |
-lxd-remote | PKGPROXY_LXD_REMOTE | flag or environment only | five remotes, see LXD image servers | name=url of an image server to cache; repeatable; name= drops a default; off caches none |
-lxd-base-path | PKGPROXY_LXD_BASE_PATH | flag or environment only | /lxd | URL path image servers are served under, each at /lxd/NAME |
-upstream-pkgproxy | PKGPROXY_UPSTREAM_PKGPROXY | upstream-pkgproxy | (unset) | another pkgproxy, as http://host:port, to fetch everything through; refused together with -snap-store-url, -go-upstream or -docker-upstream; see Two tiers |
apt has no upstream setting: apt names its mirror in each request.
| Setting | Variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-files-allow-host | PKGPROXY_FILES_ALLOW_HOST | files-allow-host | (empty: every host) | comma-separated hosts files may come from; .example.org allows every host under it; * allows all |
-files-allow-network | PKGPROXY_FILES_ALLOW_NETWORK | files-allow-network | (empty) | comma-separated private networks in CIDR form that files may come from; other private addresses are refused |
-files-max-size | PKGPROXY_FILES_MAX_SIZE | files-max-size | 8G | largest file fetched; 0 means no limit |
-files-volatile-max-size | PKGPROXY_FILES_VOLATILE_MAX_SIZE | files-volatile-max-size | 1M | largest volatile file fetched by URL, such as a SHA256SUMS; a larger one is refused; 0 means no limit |
See Files by checksum.
These are snap options without flags. The snap exports them as the standard variables, upper and lower case, for pkgproxy alone; without the snap, set those variables in pkgproxy's environment.
| Snap option | Variables | Unset | Meaning |
|---|---|---|---|
http-proxy | HTTP_PROXY, http_proxy | http_proxy of /etc/environment | proxy for http:// mirrors and registries; none for no proxy |
https-proxy | HTTPS_PROXY, https_proxy | https_proxy of /etc/environment | the same, for https:// |
no-proxy | NO_PROXY, no_proxy | no_proxy of /etc/environment | hosts and networks reached without the proxy, such as .example.com,10.0.0.0/8; none for no exceptions |
snap set refuses a proxy that is not an http:// or https:// URL, and a change restarts pkgproxy.proxy.http and proxy.https into /etc/environment; on classic Ubuntu it does not, so pkgproxy has no proxy there unless one of these is set.https:// sources connect directly and do not use the proxy.How the cache page is built from a scan of the cache, the recorder's activity and the access log's live events, and who may see it.
withSetupPage serves /pkgproxy/cache,
/pkgproxy/cache/activity,
/pkgproxy/cache/events and /pkgproxy/cache/upstreams
to local and trusted visitors from setupPageCache: a
cacheusage.Monitor, stats.Recorder.Activity, the
liveHub and mapUpstreams.
cacheusage.Scan sums allocated blocks, as du
does; Monitor.Status walks when the last report is older than
DefaultMaxAge (1 min) and waits DefaultWait (1 s).
Report.Blocks is cut to BlockLimits, five times
the SVG treemap's, which is what a browser without JavaScript gets.
stats.Recorder.Activity returns one Activity per
store key; newCacheActivity sums them by block, by the same
group and leaf IDs the layout gives, falling back to the group's rest,
then the kind's. activityTTL is 5 s.
reqlog.Wrap(next, logger, observers...) calls each
Observer with StepBegin,
StepOutcome and StepEnd, from the goroutine
serving the request, so an observer must not block.
reqlog.KindTunnel names a tunnel, its key the host and port.
liveHub.send never waits: a full queue
(liveBuffer) counts the event as lost. Lookups take at most
2 s, 8 at a time, and are kept 10 minutes; past 1,024 clients the old
ones are forgotten. An idle stream says it is alive every 20 s.
setupPageCache.mayView checks the listener,
forwardingHeaders, the unmapped remote address against
loopback, ownAddrs and -trusted-networks, and
the Host (namesMachine) against DNS rebinding.
The flag refuses 0.0.0.0/0 and ::/0; anyone
else is passed on to the gateway.
Next: The cache page, for what an administrator sees; A request in code, for where the events start; Store and keys, for the keys the blocks are sorted by.
All on the -addr listener, served by withSetupPage (only with -setup-page), for GET and HEAD, to a local visitor or one from -trusted-networks only. Anyone else's request goes on to the gateway and gets its 404. The handler name in the access log is setup-page.
| Path | Constant | Answer | Headers |
|---|---|---|---|
/pkgproxy/cache | cachePagePath | The cache page: HTML (cachepage.html.tmpl, with the map's JSON and cachepage.js) when Accept prefers it, otherwise the same figures as plain text through guide | Cache-Control: no-store, Vary: Accept, Accept-Language, Cookie, Host, Content-Language |
/pkgproxy/cache/activity | cacheActivityPath | application/json: a cacheActivity, or null when the recorder has no activity | Cache-Control: no-store |
/pkgproxy/cache/events | cacheEventsPath | text/event-stream: one liveEvent per data: line; a HEAD gets the headers only; passed to the gateway when the hub is nil | Cache-Control: no-store |
/pkgproxy/cache/upstreams | cacheUpstreamsPath | application/json: mapUpstreams, by host a mapDot (c: ok, warn, bad or none; s: the state's icon; t: the tooltip's lines); cachepage.js pollUpstreams reads it every 10 s, 60 s while hidden, and drawDot puts a dot on each upstream's mark | Cache-Control: no-store |
/ | setupPagePath | The setup page; for a local or trusted visitor it also carries a cacheSummary and an upstream line, with links to the cache page and /pkgproxy/upstreams | Cache-Control: no-cache, no-store for a local or trusted visitor |
The cache page opens with the bar every page shares (pageShell, §20.10), its banner and, in its footer, the language switcher: cachePage has Shell, Banner and Langs, and no longer a back link.
setupPageCache.mayView(r) is true only when all of these hold, for the page, its activity and its events alike:
Forwarded, Via, X-Forwarded-For, X-Forwarded-Host, X-Forwarded-Proto, X-Forwarded-Port, X-Forwarded-Server, X-Real-Ip, True-Client-Ip, Cf-Connecting-Ip;isOwnAddr) or in a trusted network (isTrustedAddr);Host is an IP address, localhost, this machine's name or the host of -public-url (namesMachine), against DNS rebinding.-trusted-networks (trustedNetworksFlag; snap option trusted-networks; empty by default) takes CIDR prefixes separated by commas, a single address as a host prefix, through parseNetworks, which also takes an IPv4-mapped network as the IPv4 network it maps. A prefix of length 0 (0.0.0.0/0, ::/0) is refused. When any is set, the daemon logs showing the cache page to visitors from trusted networks with networks once at start. newSetupPageCache gets the prefixes and -public-url's host.
{"now": <Unix seconds>, "leaves": [<leafActivity>…]}. The series end with the minute, hour and UTC day now is in.
| JSON key | Field | Meaning |
|---|---|---|
g | Group | The group's ID, or the kind alone for what of a kind is not drawn as a group |
id | Leaf | The leaf's ID; absent for a group's or a kind's rest |
c | Content | Content rather than indexes or metadata |
l | Last | Last served, Unix seconds |
h | Hits | Hits in the last hour, day, week, all time |
s | Saved | Bytes saved in the same four windows |
mi, ho, d | Minutes, Hours, Days | Sparse series: pairs of an index and a count, over 60 minutes, 24 hours and 60 days |
IDs join a kind, then a label and a name per level, with \x1f (groupID, leafID): the same IDs the map's blocks carry, so the page finds a leaf's block however it was laid out.
cacheMapData, embedded in the page: kinds (kind, name, size, r, rest), groups (k, id, name, full, size, r, rest), blocks (k, g, id, name, size, r, meta), text (translated strings), activity (the first cacheActivity), live (cacheLiveData: events, upstreams, logos, text; absent without the hub), size, free. r is a rectangle's corners in a ground of cacheMapWorld (1000) units, to a hundredth.
t | Sent when | Other keys |
|---|---|---|
hi | the stream starts | at |
busy | already liveMaxWatchers pages watch; preceded by retry: 30000, and the stream ends | none |
b | reqlog.StepBegin | id, at, c (client address, or socket), n (name, once known), k (kind), g and l (block IDs) or h (a tunnel's host), u (upstream host) |
o | reqlog.StepOutcome | id, at, o |
e | reqlog.StepEnd | id, at, o (final), b (bytes) |
n | a client's reverse lookup found a name | c, n |
lost | the queue had been full and has just drained | lost: how many events were dropped |
An idle stream gets : keepalive every liveKeepalive. Nothing is encoded while no page watches.
| Name | Value | What it bounds |
|---|---|---|
liveBuffer | 512 | Events queued per watching page before they are lost |
liveMaxWatchers | 4 | Pages watching at once (a browser opens at most 6 connections to one host) |
liveBusyRetry | 30s | When a page told busy tries again |
liveKeepalive | 20s | Keepalive comment on an idle stream |
liveLookupTimeout | 2s | One reverse lookup of a client |
liveMaxLookups | 8 | Lookups at once |
liveNameTTL | 10m | How long a client's name is kept |
liveMaxClients | 1024 | Clients' names kept before the old ones are forgotten |
liveNamesTTL | 5s | How often the names of snaps, registry blobs and files are read again from the recorder |
activityTTL | 5s | How long one encoded activity answer is shared |
poll() in cachepage.js | 10s, 60s while the tab is hidden | How often the page fetches the activity |
cacheusage.DefaultMaxAge | 1m | Age of the last scan after which a page view starts another |
cacheusage.DefaultWait | 1s | How long a page view waits for a scan in progress |
MapBudget, MinKindBudget, MaxGroups, MinShareOf | 300, 32, 48, 8 | MapLimits, the SVG treemap a page without JavaScript shows: at most 800 rectangles |
BlockLimits | 5 × MapLimits | Report.Blocks, what the 3D map and the activity use |
| Name | What it is |
|---|---|
Scan(ctx, dir, names) (*Report, error) | Walks the cache directory, no symlinks, no other file systems, summing allocated blocks as du does; each object is counted with its .meta.json file, which pkgproxy writes beside it for 0.11 |
Tally, NewTally, Tally.Add, Tally.Report | The summing behind Scan |
Report | Total; Kinds within MapLimits, Blocks within BlockLimits, largest first |
Kind, Group, Leaf, Rest | A kind of download, an archive, snap, module, image, repository or file name, its members, and what a level leaves out (Count, Size) |
Limits | Budget, MinKindBudget, MaxGroups |
Monitor, NewMonitor, Monitor.Status, Monitor.Last | The last report, rescanned in the background under the daemon's context; Status has Report, Err, Age, Measuring |
Classify(kind, key, names) (group, leaf Ident) | A store key to its group and leaf |
Ident, Label | A name, or a label standing in for one: LabelIndexes, LabelUnknownSnap, LabelSnapMetadata, LabelUnknownRepo, LabelRegistryMetadata, LabelOther, LabelUnknownFile |
Names | What the recorder knows of objects kept by hash: Snaps, DockerRepos, Files |
DiskSpace(dir) (Space, error) | Free (for unprivileged users) and total space of the file system |
AptDir … FilesDir, KindOther | The store subdirectories, and the kind of everything else under the cache directory |
| Kind | Group | Leaf |
|---|---|---|
| apt | archive, <host>/<prefix> | source package of a pool file; dists/** under LabelIndexes |
| snap | snap name | revision; everything but blobs under LabelSnapMetadata |
| gomod | module path | version (its .info, .mod and .zip) |
| lxd | <remote>/<image path> | serial |
| docker | repository the blob was served through | blob; manifests and tags under LabelRegistryMetadata |
| files | name it was served under | <algo>:<short hex> |
How internal/fill lets one download serve every client that asks for the same file while it downloads, and what happens when the file fails its hash check.
A handler calls Group.Serve with a fill.Spec:
Fetch fills the Store (often a
fill.Download), and Cached,
Streaming and Failed are the proxy engine's own answers.
Small objects go through internal/flight instead.
fill.Group maps keys to *Fill under one mutex.
join joins a fill, or finds the object in the
Store, or starts run under
context.WithoutCancel. Commit happens before the fill leaves
the table.
An ObjectWriter that is also an io.ReaderAt can
be tailed. progressWriter advances written and
closes the changed channel; readers wait on it, then switch to
OpenRange on the committed object.
Serve hands httprange.Serve an
Entity whose Open(offset, length) returns a
tailing reader; its FlushHeaders sends the
status at once. With Spec.Size there are no validators yet,
so an If-Range fails closed and the whole body is sent.
Without a tail, serveAfterwards calls Cached.
join starts go g.run(context.WithoutCancel(ctx), …);
each reader selects on its own r.Context().
idletimeout.Transport bounds the fetch by inactivity
(-upstream-idle-timeout, default 60s), not total time.
Fill.Abandon(timeout) forgets the fill, wakes its readers and
drains. Apt, snap and LXD pass -fill-drain-timeout
(fill.DefaultDrainTimeout, 10m, when unset). Spec.HoldLastByte reads the last byte
only from the committed object. A response cut short is logged
outcome=error, status 200.
Next: Store and keys, for the store a fill writes; How it is tested, for joins and timeouts in a bubble; Fetch once, for the behaviour from outside.
fill.Group maps keys to *Fill under one mutex; join holds it while it looks up the map and, on a miss, calls Store.Stat.
| State | How it got there | What a new request gets |
|---|---|---|
| Not in the map, not in the Store | nothing fetched it, or a fetch failed | join starts go g.run(context.WithoutCancel(ctx), …); this request is the starter (started) |
| Not in the map, committed in the Store | a fill committed and left the map | Cached(w, r, meta, true) |
| In the map, headers not in yet | the fetch has not called Begin | waits on ready; with Spec.Size > 0 it answers at once instead |
| In the map, tailable | Begin got a known size and a writer that is an io.ReaderAt | Streaming(w, r, meta, !started), then httprange.Serve reading from a tailing reader |
| In the map, not tailable | unknown size, or a writer that cannot be read back | waits for done, then Cached(…, !started) or Failed (serveAfterwards) |
| Abandoned | the fetch called Abandon: forgotten from the map, closing set | starts a new fill; the readers already following keep reading until the drain timeout |
| Done | run forgets the key, then finish closes done | as the first two rows: committed, or missing again |
A fill commits to the Store before it leaves the map, so a key found in neither is really missing.
| Name | Kind | What it does |
|---|---|---|
Group | type | The fills in progress for one Store, by key; zero value ready |
Group.Serve(w, r, Spec) | method | Answers from the Store, or joins or starts the fill and answers once the upstream's headers are in (200, 206 for a Range resume, 304). A response cut short because the fill failed is logged with outcome error, its status already sent (cutReader) |
Spec | type | What Serve needs from a proxy engine (below) |
Fetch | func type | func(ctx, *Fill) error: download into the Store, Begin once the headers are in, write through Writer, commit only if it verifies; ctx is detached from every client |
Fill | type | One fill in progress |
Fill.Begin(meta, w) | method | Publishes the metadata (Size -1 if unknown) and, if w can be read back and the size is known, the pending bytes |
Fill.Writer(w) | method | Wraps the Store's writer so each write advances written and wakes the readers |
Fill.Err() | method | The fetch's error, once it is done |
Fill.Abandon(timeout) | method | Content not cacheable: forget the fill, let the readers already following get the bytes, wait for them up to timeout, then cut off stragglers |
Download | type | A Fetch for content-addressed objects at a plain URL |
Download.Fetch(ctx, f) | method | GET, Create, Begin, hash while copying, commit only if the hash (and size) match |
Passthrough(w, r, client, url) | func | A fresh, uncached, streamed GET for a client whose fill failed verification before anything was streamed |
BodyLength(r, meta) | func | How many bytes Serve streams for r from a fill in progress |
DefaultDrainTimeout | const | 10 minutes |
Retry(ctx, logger, url, retryFor, attempt) | func | Runs attempt again from the start until it succeeds or fails in a way not worth another try: up to DefaultRetries more attempts after the backoff, none starting more than retryFor after the first; for a fetch that is not a Download (gomodproxy's .mod and .zip) |
UpstreamError(err) | func | Marks the error of a request that got no response as the upstream's, so Retry may retry it |
UpstreamBody(r) | func | Wraps an upstream's body so its read errors are the upstream's: a body that breaks off is always retried |
UpstreamStatus | type | The non-200 status an upstream answered, wrapped by the error; 408, 429, 500, 502, 503 and 504 are retried |
DefaultRetries | const | 5 |
DefaultRetryFor | const | 2 minutes |
ErrHashMismatch | var | Content did not hash to what it was requested by; nothing is cached |
ErrSizeMismatch | var | Content not of Download.Size; nothing is cached |
ErrTooLarge | var | Content larger than Download.MaxSize; nothing is cached |
TooLargeError(n, limit), MoreThanError(reason, limit) | func | The errors of an object announced, or found, larger than a limit; filesproxy's volatile files use them too |
ErrUpstreamRefused | var | The upstream is a pkgproxy that refused the object (X-Pkgproxy-Refused): it does not have it, and it is not asked again (§27.4) |
| Field | Meaning |
|---|---|
Store, Key | Where the object is, committed or pending |
Fetch | Starts the download when the object is neither stored nor being filled |
HoldLastByte | No follower gets the last byte before the commit: a bad object ends short |
Size | Known in advance: a miss is answered without waiting for the upstream's headers |
Cached(w, r, meta, hit) | Answers from the committed object; hit is false for the request that started the fill |
Streaming(w, r, meta, hit) | Sets the proxy engine's headers and records the serve before a response streams from the fill; Serve sets the status, Content-Length and Content-Range |
Failed(w, r, err) | Answers a fill that failed before anything was streamed, or a Store lookup that failed |
| Field | Meaning |
|---|---|
Client | Nil means http.DefaultClient |
URL, Store, Key | What to fetch and where to keep it |
Algo, NewHash, Hex | The hash's name for ContentHash, its constructor, and the digest the content must have |
Now | Stamps StoredAt; nil means time.Now |
DrainTimeout | Positive: on a mismatch, Abandon(DrainTimeout); zero leaves it to Spec.HoldLastByte |
MaxSize | Positive: a larger announced size is not read, and more bytes than that fail the fetch |
Size | Positive: the content must have exactly this size; also lets followers tail an upstream that announces none |
RecordOrigin | Store the URL after redirects as OriginURL |
OriginHeader | The header of the upstream's answer that names the URL the content came from, which RecordOrigin records instead: X-Pkgproxy-Origin-URL of an upstream pkgproxy |
Retries | Failures in a row that are asked again, for the rest with a Range request once some is kept; zero means DefaultRetries, less than zero none |
RetryFor | No retry starts more than this after the upstream last sent a byte; zero means DefaultRetryFor, less than zero none (-upstream-retry-for 0) |
Logger | Logs each retry, download interrupted, asking the upstream again |
| Proxy engine | Key | Fetch | Size | On a bad hash |
|---|---|---|---|---|
apt (proxy.serveFilled, pool files only) | cachekey key | fetchStoreAndIndex, checked against the index | none | Abandon(-fill-drain-timeout) in storeResponseBody; a client that got nothing yet is answered by a fresh passthrough |
snap (snapproxy) | sha3-384/<hex> | fill.Download, sha3-384, DrainTimeout from -fill-drain-timeout | from the store's answer | Abandon |
LXD (lxdproxy/blob.go) | <remote>/<path> | fill.Download, sha256 from the catalog, DrainTimeout from -fill-drain-timeout | from the catalog | Abandon |
Docker (registryproxy.serveFilled) | blobs/sha256/<hex> | fetchBlob, checked against the digest | none | HoldLastByte |
files (filesproxy.serveFill) | sha256/<hex> or sha512/<hex> | fill.Download with MaxSize (-files-max-size), Size (size=), RecordOrigin | none | HoldLastByte |
Go's .mod and .zip, Docker manifests and apt's dists/** files are not streamed: one shared fetch through flight.Group, served once it is stored.
Both use retryable and backoff (1, 2, 4, 8, 16 s, capped at 30 s). Retried: a body that breaks off, whatever the error; a request with no response for a broken or quiet connection (EOF, ECONNRESET, ECONNABORTED, EPIPE, idletimeout.ErrStalled, a timeout, an HTTP/2 stream or connection error, GOAWAY); an HTTP proxy's CONNECT refused with a retried status; and the retried statuses. Not retried: no such host, connection refused, the dial guard's or the offline switch's refusal, an upstream pkgproxy's refusal (ErrUpstreamRefused), a cancelled context, and any error of the Store's.
Download.Fetch | Retry | |
|---|---|---|
| Used by | snapproxy, lxdproxy, filesproxy | gomodproxy, for .mod and .zip |
| A retry asks for | the rest, Range: bytes=<kept>- | the whole object again |
| The time bound counts from | the last byte the upstream sent | the first attempt |
Retry-After | honoured within the bound | not read |
Breaker open in store-overloaded | waits for it within the bound | refused, not retried |
Every attempt after the first carries offline.WithRetry, so a host's breaker counts the first failure only. gomodproxy passes retryFor -1 when a stale "not found" is at hand, which it serves instead; .info, @v/list and @latest are not retried, as they fall back on a stale copy.
Where pkgproxy's code and design live, and the conventions every change follows.
cmd/pkgproxy is the multi-call binary;
internal/ holds the proxy engines (proxy is apt), the
shared path, the protocol parsers and the supporting packages. Nothing
under internal/ is importable from outside the module.
main.go routes on argv[0] with go-cmdr's
router.Cmd; ctl.go builds
pkgproxyctl's tree. Every leaf prints its help with
printUsage; tests use invoke(t, "pkgproxyctl", …).
docs/design.md holds §1, §12 and §18 and the index; each
area's file holds its own §, cited as docs/design/lxd.md §17.11;
go test ./docs guards the published site.
Issues and milestones live in GitLab (glab issue list), not
in a TODO file. git config core.hooksPath githooks, then
git commit -s; ./check.sh before pushing.
Next: A request in code, for one request through the packages; How it is tested, for what a change needs; Translations, for errors and messages that translate.
| Path | What it holds |
|---|---|
cmd/pkgproxy/ | The one multi-call binary: the daemon pkgproxy and the command line pkgproxyctl, a symlink to the same file |
internal/ | Every other package, one per concern; not importable from outside module gitlab.com/zygoon/pkgproxy |
internal/integration/ | Tests only: podman runs of real apt and go (skipped under -short or without podman), and snapd/, a nested Go module that drives snapd's own store client |
data/ | Assets embedded with //go:embed: the icon (also the snap's), the upstream logos of the cache page's live views, the Ubuntu mark |
docs/ | The design (design.md, design/*.md), statsapi.md, statsapi-internals.md, testing.md, troubleshooting.md, and docs_test.go, which guards the published site |
snap/ | snapcraft.yaml, the configure hook, local/bin/pkgproxy-env (snap options to PKGPROXY_* variables), the desktop file |
rockcraft.yaml | The rock: an OCI image of the same binary under Pebble (§29) |
charms/pkgproxy-k8s/ | The Kubernetes charm, in Python with ops, which runs the rock (§30): src/charm.py, src/pkgproxy.py (the Pebble layer), src/loadbalancer.py, src/common.py, unit and integration tests, its own pyproject.toml and uv.lock |
charms/pkgproxy/ | The machine charm, which installs the snap (§31): src/charm.py, the same src/common.py, snap.toml (the snap revisions it pins) and pin-snap, which rewrites them at publish time |
tests/ | The spread suite: tests/<client>/<task>/task.yaml, helpers in tests/lib/, fixtures for the snapcraft, rockcraft and charmcraft packs, tests/rock/ and the manual tests/charm/ |
testdata/apt/ | Release, InRelease, Packages and Sources excerpts of Debian and Ubuntu, for the parsers' unit tests |
githooks/ | pre-commit, enabled with git config core.hooksPath githooks |
.dagger/ | The Dagger module whose checks ./check.sh and GitLab CI run |
spread.yaml, .image-garden.mk | The spread project and the VM images image-garden makes for it |
check.sh | exec dagger check |
DEPLOYMENT.md, README.md, NEWS | Published at https://zygoon.pl/pkgproxy/ with docs/ |
| Command | File |
|---|---|
pkgproxy (the daemon) | cmd_pkgproxy.go, with filesflags.go and envflags.go (PKGPROXY_* defaults) |
pkgproxyctl (the router) | ctl.go |
pkgproxyctl stats | cmd_stats.go |
pkgproxyctl upstreams | cmd_upstreams.go (§28.9) |
pkgproxyctl setup | cmd_setup.go; one child each: cmd_setup_apt.go, _snap, _go, _docker, _lxd, _lxd_remote, _snapcraft, _rockcraft, _charmcraft, _status |
pkgproxyctl cache invalidate | cmd_cache.go |
pkgproxyctl offline | cmd_offline.go |
pkgproxyctl open | cmd_open.go |
pkgproxyctl version | cmd_version.go, with version.go |
Files without the cmd_ prefix hold what several commands share:
| File | What it holds |
|---|---|
main.go | The router.Cmd on argv[0], the locale and the translator |
usage.go | printUsage, the help layout of every leaf command (docs/design/cli.md §14.3) |
guide.go | The guide builder: translated paragraphs and verbatim code blocks |
instructions.go | Client configuration paths and the startup banner |
host.go | host, what commands take from outside, which tests replace |
setuppage.go, setuppage_cache.go, setuppage.html.tmpl, setuptool.html.tmpl | The setup page, a page per tool, its scripts under /setup/, and the cache page |
pageshell.go | The bar every page opens with, its status element, /pkgproxy/status and the /setup/all script (§20.10) |
banner.go | The degraded or offline banner of every page and of pkgproxyctl stats and upstreams (§28.4) |
kindhint.go, lang.go | What an error kind such as http-429 means, in the visitor's language; the page language from ?lang=, the pkgproxy-lang cookie or Accept-Language, and the footer's switcher (§20.3) |
upstreams.go | The upstream status recorder in -cache-dir/upstreams, and the snap session placeholders' prefix file |
upstreamcheck.go | -upstream-check-interval and the checks of quiet upstreams (§28.8) |
upstreamspage.go, upstreamdetail.go, upstreamhow.go, upstreamspage.html.tmpl, upstreamdetail.html.tmpl | /pkgproxy/upstreams, a page per host, the setup page's 30-day summary, the cache page's status dots, and what each page says of how it knows (§28.3) |
upstreampkgproxy.go | -upstream-pkgproxy: the chain, upstream-pkgproxy.json, the self check (§27) |
cachemap.go, cachelive.go, cachepage.js | The cache page's map, its activity and its live events |
lxdtls.go, lxdremote.go, lxdbridge.go, lxdhook.go | The LXD HTTPS listener and certificate, -lxd-remote, the bridge and profile of setup lxd |
statssocket.go, snapoption.go | Default socket and cache paths; reading snap options |
signals_*.go, umask_*.go | Platform parts: shutdown signals, the umask around socket creation |
| Package | Purpose |
|---|---|
proxy | apt: the forward proxy, cache key, store, origin and index wired into one handler (§7.5, §8) |
snapproxy | snap: a reverse relay snapd talks to through SNAPPY_FORCE_API_URL, caching snaps and assertions (§11) |
gomodproxy | Go: a GOPROXY origin chained to an upstream module proxy, with checksum database checks (§16) |
registryproxy | Docker: the read-only OCI distribution API for anonymous pulls (§19) |
lxdproxy | LXD: simplestreams image servers under <base>/<name> (§17) |
lxdapi | The part of LXD's REST API that LXD 6.10 and later download images through (§17.14) |
filesproxy | Files by checksum, GET /files/v1/fetch?url=…&sha256=… (§24), and volatile files, GET /files/v1/volatile?url=… (§24.8) |
| Package | Purpose |
|---|---|
gateway | Dispatches each request to one proxy engine on the one listener, and stamps its answers X-Pkgproxy-Engine |
reqlog | Wraps a listener's handler: one access log line per request, and events for observers (§23) |
offline | The offline switch: modes, per-host breakers, the transport, dial and middleware wrappers; tells the upstream status recorder of every upstream request (§21) |
fill | Serves a large blob while it downloads into the store (§8.1) |
flight | singleflight with typed results and a joined bit |
revalidate | One detached revalidation per key, bounded wait, recent-failure memory |
store, store/fslocal, store/memstore | The store interface, its disk implementation, its in-memory one for tests (§7.1) |
cachekey | apt request URL to cache key and policy (§4, §7.3) |
index | Expected hashes of apt pool and by-hash paths from parsed indexes (§4.2, §7.4) |
lxdindex | Expected SHA256 of LXD blob paths from parsed catalogs, checkpointed to lxd-index.json |
httprange | Range, If-Range and conditional requests for apt, snap and LXD answers |
idletimeout | Fails an upstream request that makes no progress |
upstreamretry | Sends a request an upstream answered 408 again |
upstreamlog | Logs who answered when an upstream answers with an error |
chain | Sends what the proxy engines fetch through an upstream pkgproxy (§27) |
proxyengines | /pkgproxy/v1/proxy-engines, the description a downstream reads, and the loop guard (§27.1, §27.5) |
dialguard | Keeps apt's forward proxy from connecting to this machine for a client |
origin | Fetches from apt archives over HTTP |
| Package | Purpose |
|---|---|
apt | Release, InRelease, Packages and Sources: names, sizes and hashes only |
snap | The Snap Store API v2 knowledge the snap proxy engine acts on, and the assertion chain check |
gomod | The GOPROXY protocol's five endpoints: parsing, validation, cache keys |
simplestreams | LXD's stream index and product catalogs |
stats, statsapi | The counters and their read-only JSON API on the stats socket (§13) |
upstreams | The upstream status: buckets, states and the files under -cache-dir/upstreams (§28) |
adminapi | The mutating API on the admin socket: cache invalidation, offline mode |
cacheusage | Measures the cache directory for the cache page (§20.7) |
treemap | Lays out the cache page's treemap |
hostnet | Follows whether the host is offline, for -auto-offline |
l10n | The translation catalog and the log handler that translates (§22) |
logkey, logmsg, httpheader, enumutil, relpath | Shared spellings: log attribute keys, log messages, header names, enum fallbacks, relative path validation |
docs/design.md holds §1, §12 and §18 and the index; each area has its own file under docs/design/, and the § numbers are shared by all of them. Cite a section as docs/design/lxd.md §17.11.
| § | File | Area |
|---|---|---|
| §1 | design.md | Goal and non-goals |
| §2–§8 | design/apt.md | What apt asks for, the forward proxy, normalization, policy, data model, Go interfaces (§7.1–§7.5), the fetch path (§8, §8.1) |
| §11 | design/snap.md | Snap packages and assertions (§11–§11.7) |
| §12 | design.md | Listener: TCP or UNIX socket |
| §13, §15 | design/stats.md | The stats API (§13.1–§13.6) and the default socket path |
| §14 | design/cli.md | pkgproxyctl as a multi-call binary, piped scripts, help layout (§14–§14.3) |
| §16–§16.4 | design/gomod.md | The Go module proxy and checksum database |
| §16.6–§16.10 | design/build-containers.md | LXD build containers and the admin API socket |
| §17 | design/lxd.md | LXD simplestreams image mirrors (§17–§17.14) |
| §18 | design.md | Tuning knobs: flags, environment, snap set |
| §19 | design/docker.md | The Docker Hub registry mirror (§19–§19.6.1) |
| §20 | design/setup-page.md | The setup page, the cache page and the bar every page shares (§20–§20.10) |
| §21 | design/offline.md | Offline mode |
| §22 | design/l10n.md | Localization |
| §23 | design/access-log.md | The access log and its outcomes |
| §24 | design/files.md | Files by checksum and volatile files (§24–§24.8) |
| §25 | design/upgrades.md | Upgrades, reverts and snap epochs: the .meta.json files kept for 0.11, the post-refresh migration (§25–§25.6) |
| §27 | design/upstream.md | An upstream pkgproxy: two tiers (§27–§27.6) |
| §28 | design/upstreams.md | Upstream status (§28–§28.13) |
| §29 | design/rock.md | The rock |
| §30 | design/charm.md | The Kubernetes charm |
| §31 | design/machine-charm.md | The machine charm |
A new area gets its own file, the next free § number and a line in the index in docs/design.md; a change to the design updates its document in the same commit. go test ./docs checks links and anchors, raw tags outside code, one title per file and the tables of contents.
| Rule | Where it is written |
|---|---|
Work starts from a GitLab issue in a milestone (glab issue list, glab issue create) | AGENTS.md |
Errors with l10n.D.Errorf, worded "cannot …"; log messages wrapped in i18n.MarkForTranslation | AGENTS.md, §22; see Translations |
Every commit signed off (git commit -s); CI runs dco-check | AGENTS.md, .gitlab-ci.yml |
Linters are tool dependencies in go.mod (go get -tool), run with go tool | AGENTS.md |
./check.sh before pushing; the pre-commit hook before each commit | AGENTS.md; see How it is tested |
Which packages one request passes through, from the listener to the access log line, the upstream transport, an upstream pkgproxy and the upstream status.
reqlog.Wrap → offline.Middleware →
proxyengines.LoopGuard → gateway.Handler →
proxy.Handler: Resolve, Store.Stat,
then fills.Serve, whose fetch commits through
Store.Create after the hash check.
route checks CONNECT or a non-empty
r.URL.Host first (apt), then proxyengines.Path,
the base paths, registryproxy.Matches and snap's prefixes.
A disabled proxy engine's traffic gets 404 from the gateway. Each proxy
engine's answer carries X-Pkgproxy-Engine.
Of the requests that share one fetch, only those that waited are
joined; the starter is fetched or
revalidated, as internal/flight reports.
offline.Switch.Transport → idletimeout.Transport
→ upstreamretry.Transport → upstreamlog.Transport
→ a base: http.DefaultTransport,
dialguard.Guard.Transport (apt; CONNECT dials
offlineSwitch.Dial(guard.Dialer().DialContext)) or
filesproxy.NewTransport.
chain.Chain reads proxyengines.Document from
/pkgproxy/v1/proxy-engines every
DefaultRefreshInterval (10m). Transport wraps
each base; apt's is ForwardTransport; files use
Redirect. Requests carry X-Pkgproxy-Via;
LoopGuard answers 508 past MaxHops (8) or its own
instance.
offline.Switch.Recorder is an
*upstreams.Recorder: Observe(Observation) per
request, Record(Event) per breaker, mode or daemon change.
ServedStale comes from proxy,
gomodproxy, registryproxy,
snapproxy and revalidate.Doc; snap's
writeSynthesised calls Synthesised, its
placeholders Recorder.Observe directly.
Next: internal/fill, for the fill a cache miss starts; The proxy engines' packages, for what each handler does; Cache page code, for the requests watched live.
Outermost first. Each wrapper takes only its own paths and passes everything else, CONNECT and absolute-URL requests included, to the next.
| Listener | Chain |
|---|---|
-addr (or -socket-path), and -content-socket-path | reqlog.Wrap → withSetupPage (with -setup-page, the default) → withLxdCertDownload (when the LXD HTTPS listener is on) → offline.Middleware → proxyengines.LoopGuard → gateway.Handler |
-lxd-tls-addr (or -lxd-tls-socket-path), and -lxd-tls-content-socket-path | reqlog.Wrap → withLxdAPI → offline.Middleware → proxyengines.LoopGuard → gateway.Handler |
-stats-socket-path | statsapi.Handler: GET /pkgproxy/v1/stats, /pkgproxy/v1/info, /pkgproxy/v1/upstreams, /pkgproxy/v1/upstreams/history, /pkgproxy/v1/upstreams/events; no access log |
-admin-socket-path | adminapi.Handler: DELETE /pkgproxy/v1/cache, GET and PUT /pkgproxy/v1/offline; no access log |
Both traffic listeners pass the cache page's liveHub.observe to reqlog.Wrap as an observer. proxyengines.LoopGuard answers a request whose X-Pkgproxy-Via names this daemon's instance, or more than proxyengines.MaxHops (8) instances, with 508 Loop Detected and X-Pkgproxy-Loop, before any proxy engine sees it. The snap proxy engine is wrapped once more, inside the gateway, by offline.MiddlewareAnswer(…, snapproxy.RefusedAnswer), because snapd retries a 503 for half a minute. offline.Middleware turns a 502 into a 503 when the fetch was refused for being offline.
| Wrapper | Match | Handler name |
|---|---|---|
withSetupPage | GET or HEAD of /, /setup/<tool>, /setup/all, /pkgproxy/status; for a local or trusted visitor only (setupPageCache.mayView), /pkgproxy/cache, /pkgproxy/cache/activity, /pkgproxy/cache/events, /pkgproxy/cache/upstreams, /pkgproxy/upstreams, /pkgproxy/upstreams/<host> | setup-page |
withLxdCertDownload | GET or HEAD of /pkgproxy-lxd-cert.pem | lxd-cert |
withLxdAPI | origin-form / or /1.0 and below (lxdapi.Matches) | lxd-api |
gateway.Handler.route, first match wins. The name is recorded with reqlog.SetHandler and is the internal/stats type, so an access log line and a stats row name the traffic alike. Before a named handler runs, ServeHTTP sets X-Pkgproxy-Engine (proxyengines.HeaderEngine) to its proxy engine (engineNames; files-volatile for /files/v1/volatile), so that a downstream pkgproxy tells a proxy engine's 404 from the gateway's, which carries none.
| # | Match | Handler | Name |
|---|---|---|---|
| 1 | CONNECT, or a non-empty r.URL.Host (an absolute-URL request line) | Apt | apt |
| 2 | ProxyEngines set and the path is /pkgproxy/v1/proxy-engines (proxyengines.Path) | proxyengines.Handler | none |
| 3 | Go set and the path is -go-base-path (default /go) or below it | Go | gomod |
| 4 | Lxd set and the path is -lxd-base-path (default /lxd) or below it | Lxd | lxd |
| 5 | Files set and the path is /files or below it | Files | files |
| 6 | registryproxy.Matches: /v2/, /v2/<name>/manifests/<ref>, /blobs/<digest>, /referrers/<digest>, /tags/list, /v2/_catalog | Docker, or 404 when it is nil | docker |
| 7 | /v2, /api/v1 or /v1, or below one of them | Snap | snap |
| 8 | anything else | http.NotFound | none |
"Below" means the base path followed by /. Docker and Lxd are left a nil interface when -docker-upstream off or -lxd-remote off turned the proxy engine off; the Go and files proxy engines are always set.
proxy.Handler.ServeHTTP:
| Request | What happens |
|---|---|
CONNECT | handleConnect: a blind tunnel, port 443 only, dialled through the dial guard |
| neither GET nor HEAD | 405 |
not an http:// absolute URL | 400 |
cachekey.PolicyImmutable | reqlog.SetObject(r, "apt", key), then serveCached; a pool miss streams through serveFilled |
cachekey.PolicyRevalidate | reqlog.SetObject, then serveCached with revalidation |
cachekey.PolicyPassthroughOnly | passthrough: relayed, not cached |
| Proxy engine | Round tripper, outermost first |
|---|---|
snap, Go, Docker, LXD (upstreamTransport) | offline.Switch.Transport → idletimeout.Transport → upstreamretry.Transport → upstreamlog.Transport → http.DefaultTransport, or chain.Transport(http.DefaultTransport) with -upstream-pkgproxy |
apt (newAptHandler) | the same four layers → dialguard.Guard.Transport(), or chain.ForwardTransport(guard.Transport()) with -upstream-pkgproxy (aptTransport, and guard.Proxy), inside origin.NewHTTPFetcher |
| apt CONNECT | offlineSwitch.Dial(guard.Dialer().DialContext) |
files (newFilesHandler) | the same four layers → filesproxy.NewTransport(AddressPolicy{Allow: -files-allow-network}), wrapped by chain.Transport with -upstream-pkgproxy |
The Go proxy engine's checksum database client uses the same upstreamTransport as its module fetches; with -upstream-pkgproxy it is not redirected, as -go-sumdb stays each pkgproxy's own. upstreamretry.Transport sends a request answered 408 again, up to DefaultAttempts (3) times; upstreamlog.Transport logs who answered an error. The offline switch's transport tells Switch.Recorder, an *upstreams.Recorder, of every request with Observe once its body ends, and of breaker and mode changes with Record (§28.6).
| Error | Package | Meaning |
|---|---|---|
idletimeout.ErrStalled | idletimeout | No progress for -upstream-idle-timeout (default 60s; 0 waits forever) |
offline.ErrOffline | offline | Refused: offline mode, or the host's breaker is open |
chain.ErrRefused | chain | The upstream pkgproxy has this proxy engine turned off |
chain.ErrLoop | chain | The upstream pkgproxy is this one, or sends its requests back to it |
dialguard.ErrRefused | dialguard | Loopback, unspecified, or this machine's own address at -addr or -lxd-tls-addr's port |
filesproxy.ErrPrivateAddress | filesproxy | A private address outside -files-allow-network |
Built by setUpChain when -upstream-pkgproxy is set, before the proxy engines' constructors (§27).
| Name | What it is |
|---|---|
chain.Chain | Reads the upstream's proxyengines.Document at start, every DefaultRefreshInterval (10m), and at most every DefaultRetryInterval (1m) after a request to the upstream fails; each reading replaces what is followed in one swap |
chain.Transport(base) | Rewrites a request to an origin the upstream serves into the same request to the upstream, at that proxy engine's path; a path with . or .. is refused |
chain.ForwardTransport(base) | apt's: the upstream as its HTTP proxy for http:// URLs |
chain.Redirect, chain.WithRedirect | files and volatile files: the request to the origin becomes one to the upstream's own endpoint, with the same query |
chain.Baseline | What an upstream too old to describe itself is taken to run: 0.12's proxy engines at their default paths |
chain.Source | Where the description came from: upstream, saved, baseline, newer, loop, unreachable |
chain.Via | How a proxy engine is fetched: upstream, origin, refused |
proxyengines.Document | schema, instance (random per run), proxy-engines: enabled, kind (forward, root, path), path, remotes |
proxyengines.HeaderVia | X-Pkgproxy-Via, the instances a request went through, nearest the client first |
The saved description is <cache-dir>/upstream-pkgproxy.json. /pkgproxy/v1/info reports upstream_pkgproxy from Chain.Status.
reqlog.Wrap(next, logger, observers...) logs one record per request, message handled request, once next returns. Level is Info, Error for a 5xx.
| Key | Value |
|---|---|
method | r.Method |
url | r.RequestURI, the target as the client sent it |
remote_addr | r.RemoteAddr |
handler | what SetHandler recorded, last call wins; absent when nothing did |
outcome | what SetOutcome recorded, last call wins; error for a 4xx or 5xx nothing labelled |
status | the response status, 200 when none was written (a hijacked tunnel) |
bytes | body bytes written, plus what AddBytes counted (a tunnel's copy) |
duration | how long next took |
| proxy engine attributes | what SetAttr added, such as assertions_fetched and chains_incomplete on a snap refresh |
response | 5xx only: the first 512 bytes of a textual, uncompressed body |
| Outcome | Meaning |
|---|---|
hit | Served from the cache without contacting the origin |
revalidated | The origin was asked about a cached entry, which was then served |
stale | The origin could not be reached; a cached entry due for revalidation was served |
fetched | Fetched, saved to the cache and served |
joined | Waited for or followed another request's fetch |
passthrough | Relayed from the origin, not cached |
synthesised | Answered by pkgproxy without the origin: from what it recorded, or a placeholder device session |
tunnel | A blind CONNECT tunnel |
error | The request failed, or a response was cut short because its fill failed (status 200) |
Only a request that named an object with SetObject or SetObjectFrom, or an open tunnel, has events. An Observer is called on the goroutine serving the request and must not block.
| Step | When | Fields set |
|---|---|---|
StepBegin | the first SetObject(r, kind, key) or SetObjectFrom(r, kind, key, origin) | ID, Time, RemoteAddr, Kind, Key, Origin |
StepOutcome | each SetOutcome after the begin | Outcome |
StepEnd | the handler returns, or panics (outcome error, panic re-raised) | final Outcome, Bytes, Status |
Kind is the stats type (apt, snap, gomod, lxd, docker, files) or reqlog.KindTunnel (tunnel), whose Key is the host and port. The cache page turns these into its live views: see Cache page code.
How each proxy engine is built from a store, a transport and shared helpers, what the Go and Docker proxy engines do inside, and the snap proxy engine's code for a store in trouble.
newAptHandler … newFilesHandler in
cmd/pkgproxy, each with fslocal.New(cacheDir/<engine>),
after setUpChain. A disabled proxy engine's constructor returns nil, and the gateway field stays a
nil interface. Each handler names the object a request is for with
reqlog.SetObject, for the cache page's live views.
revalidate.Group.Do shares one detached fetch per key, bounded
by Policy.Wait (10 s) and Policy.RetryInterval
(1 min). Doc.ServedStale tells the upstream status of a stale
copy served. The snap and Docker proxy engines serve stale entries with
their own code.
gomod.Request names the endpoint; Endpoint.Immutable
picks the path. fetchAndStoreImmutable runs under
flight, hashes with dirhash and commits after
verifyAgainstSumdb; serveTTL uses
revalidate.ServeDoc. See design §16.
tags/<name>/_/<tag>/<id> holds a digest;
manifests/sha256/… and blobs/sha256/… hold
immutable content. revalidateTag uses headTag;
a stale pointer is served when the registry cannot be asked.
Placeholders are pkgproxy-<16 hex>-n-… or
-s-…, the prefix kept in
<cache-dir>/snap-session-prefix.
revalidationBackoff: Retry-After, or the TTL
doubled per failure, up to maxRevalidationBackoff;
markRepeatable; StoreChecks.
Next: internal/fill, for the fills blob proxy engines stream through; Store and keys, for the store every proxy engine writes; Go module proxy, for the Go proxy engine from outside.
RunFlag calls setUpChain (§27), then the constructors in this order, after opening the stats recorder, and before it listens. Each opens fslocal.New(-cache-dir/<dir>) and passes the logger, stats.Recorder and the offline switch.
| Proxy engine | Package and type | Constructor | Store dir | Gateway field |
|---|---|---|---|---|
| apt | proxy.Handler | newAptHandler | apt/ | Apt |
| snap | snapproxy.Handler | newSnapHandler | snap/ | Snap, inside offline.MiddlewareAnswer |
| Go | gomodproxy.Handler | newGoHandler | go/ | Go, at -go-base-path |
| LXD | lxdproxy.Handler | newLxdHandler | lxd/ | Lxd, at -lxd-base-path; nil when off |
| Docker | registryproxy.Handler | newDockerHandler | docker/ | Docker; nil when off |
| files | filesproxy.Handler | newFilesHandler | files/ | Files, at /files |
The gateway's ProxyEngines field is &proxyengines.Handler{Document: own}, the description this daemon serves at /pkgproxy/v1/proxy-engines. The files Handler also has Upstream and VolatileUpstream (set with -upstream-pkgproxy), Policy, Refusing and VolatileMaxSize.
| Proxy engine | Upstream | Flag and default | Other settings |
|---|---|---|---|
| apt | whatever host the client's URL names | none | -apt-revalidate-ttl, -min-forced-revalidate-interval, -fill-drain-timeout |
| snap | the Snap Store API, and the CDN URLs it hands out | -snap-store-url https://api.snapcraft.io | -snap-external-url, -snap-revalidate-ttl, -fill-drain-timeout |
| Go | a module proxy | -go-upstream https://proxy.golang.org | -go-sumdb sum.golang.org (or off), -go-base-path /go, -go-revalidate-ttl |
| Docker | one registry | -docker-upstream https://registry-1.docker.io (off turns the proxy engine off) | -docker-revalidate-ttl |
| LXD | named simplestreams servers | -lxd-remote name=url, repeatable (off turns the proxy engine off) | -lxd-base-path /lxd, -lxd-revalidate-ttl, -lxd-checkpoint-interval, -lxd-tls-addr, -fill-drain-timeout |
| files | whatever URL the client names | none | -files-allow-host, -files-allow-network, -files-max-size (8G), -files-volatile-max-size (filesproxy.DefaultVolatileMaxSize, 1 MiB) |
-upstream-pkgproxy sends every proxy engine's fetches through another pkgproxy (§27); it is refused together with -snap-store-url, -go-upstream or -docker-upstream (flagsReplacedByUpstreamPkgproxy). -upstream-check-interval (0, off; at least 5m) has the store, Go and Docker upstreams checked when quiet (§28.8): snapproxy.StoreChecks and checkTargets.
Every -<engine>-revalidate-ttl defaults to -revalidate-ttl, 5 minutes. The default LXD remotes: pkgproxy-images (images.lxd.canonical.com), pkgproxy-ubuntu and pkgproxy-ubuntu-daily (cloud-images.ubuntu.com /releases, /daily), and craft-providers' craft-com.ubuntu.cloud-buildd and craft-com.ubuntu.cloud-buildd-daily (/buildd/releases, /buildd/daily).
| Proxy engine | Kept forever (immutable) | Revalidated after its TTL | Not cached |
|---|---|---|---|
| apt | pool/**, dists/**/by-hash/**, checked against the hashes the parsed indexes list | other dists/**: Release, InRelease, Packages, Sources | URLs without dists or pool; CONNECT tunnels |
| snap | .snap files and their dm-verity data by sha3-384; the store's download paths an anonymous answer names next to a hash (store-path/<hash>) | info, find, catalogue answers and assertions | answers to a user's macaroon that show private snaps; anything on a brand store's device session |
| Go | .mod and .zip of a version, checked against the checksum database before commit (unchecked with -go-sumdb off) | @v/list, @latest, .info | a path that is not one of the protocol's five endpoints |
| Docker | blobs and manifests by sha256 digest, checked against it | tags: a pointer to a digest per set of accepted media types, checked with a HEAD | methods other than GET and HEAD (405); tag lists and the catalog (registry 404) |
| LXD | image files by the sha256 their product catalog lists | streams/v1/index.json and product catalogs | a path no parsed catalog listed: 404, never fetched |
| files | the file by sha256 or sha512, whatever its URL | volatile files, volatile/<hex>: fetched from the origin at every request, the last copy kept | a private address outside -files-allow-network; a host outside -files-allow-host; a file over -files-max-size, or a volatile one over -files-volatile-max-size |
| Proxy engine | fill.Group (large blobs, streamed) | flight.Group (one shared fetch) | revalidate.Group (bounded revalidation) | Stale entries on upstream failure |
|---|---|---|---|---|
| apt | pool files | dists/**, and the passthrough fallback | yes, RevalidateWait, OriginRetryInterval | yes |
| snap | .snap files | relayed metadata fetches, and assertions fetched ahead | no | its own code, with a back-off: revalidationBackoff (backoff.go) serves an expired entry without asking for Retry-After, or its TTL doubled per failure, up to maxRevalidationBackoff (1h); at most 10 000 keys |
| Go | no | .mod and .zip | yes, through revalidate.ServeDoc | yes |
| Docker | blobs | manifests | no | its own code: a stale tag pointer |
| LXD | image files | no | yes, through revalidate.ServeDoc | yes |
| files | files | no | no | volatile files: the kept copy, X-Pkgproxy-Cache: STALE, while the origin cannot answer; read into memory up to -files-volatile-max-size, not through a fill |
revalidate.Policy: Wait defaults to revalidate.DefaultWait (10s), after which Do returns ErrSlow; RetryInterval to DefaultRetryInterval (1m), within which a host that failed gets ErrFailing without a fetch; Epoch is the offline switch's, so a change of mode forgets failures. revalidate.Doc.ServedStale, when set, is told of every stale copy served, for the upstream status (§28.13); the proxy engines that serve stale with their own code call offline.Switch.ServedStale themselves.
| File | What it does |
|---|---|
session.go | While the store's auth is not contacted (offline, or its auth breaker open), or in store-overloaded when the store fails a nonce or session request, answers it with a placeholder, pkgproxy-<16 hex>-n-<32 hex> or -s-, marked X-Pkgproxy-Synthesised. Once auth answers again, a request carrying this pkgproxy's placeholder gets 401 with WWW-Authenticate: Macaroon refresh_device_session=1; while auth is still refused but the request's class is not, the placeholder is dropped. The prefix is kept in <cache-dir>/snap-session-prefix (cmd/pkgproxy/upstreams.go). |
backoff.go | revalidationBackoff, above. An answer, or a change of offline mode, forgets a key. |
repeatable.go | markRepeatable: a nonce and a refresh are marked idempotent, with a nil X-Idempotency-Key, which is not sent, and keep a body of up to 1 MiB, so net/http sends them again on a new connection when the store's edge closed the pooled one, as it does after each 429 and 408. A session request, which spends its nonce, is sent once. |
check.go | Check, StoreChecks: the idle checks of the store's classes (§28.8). |
newDockerHandler returns nil for -docker-upstream off, newLxdHandler nil for -lxd-remote off. RunFlag assigns them to the gateway only when non-nil: a nil *lxdproxy.Handler stored in an http.Handler field would be a non-nil interface, and the gateway's h.Lxd != nil check would call through it. With Docker off, a registry-shaped path still matches and gets 404; with LXD off, /lxd falls through to the snap prefixes and then 404.
The store interface, its metadata and commit, and how apt's cache keys and policies reach it.
internal/store defines Store and
ObjectMeta; fslocal.New is called once per proxy
engine on -cache-dir/<engine>; memstore serves the
tests. Keys: cachekey.PathResolver for apt, simple prefixes
such as sha3-384/ or blobs/sha256/ elsewhere.
store.Store: Stat, Open,
OpenRange, Create (an ObjectWriter
with Commit/Discard, optionally
io.ReaderAt), UpdateMeta, Delete.
fslocal on disk, memstore in tests.
store.ObjectMeta, marshalled with encoding/json
into the user.pkgproxy.meta xattr of
objects/<key>, gzipped above 3 KiB (64 KiB of JSON at
most; OriginURL is dropped if it would not fit). ContentHash is
<algo>:<digest> in the proxy engine's own terms.
<key>.meta.json holds it too, for pkgproxy 0.11
(§25.3).
fslocal: tmp/obj-<hex>,
setAttr, Sync, Close,
Rename to objects/<key>, with
<key>.meta.json removed before and rewritten after. A
reader that has the old file open keeps reading it. defer w.Discard() is
always safe. memstore copies its buffer on
Commit and copies out on read.
cachekey.PathResolver returns Resolved;
internal/apt parses Release and stanzas;
index.Store implements index.Lookup. The proxy
handler in internal/proxy wires them together.
PolicyImmutable commits Immutable: true;
PolicyRevalidate is checked against StoredAt and
-apt-revalidate-ttl (by default -revalidate-ttl), and touch refreshes it with
UpdateMeta on a 304. PolicyPassthroughOnly goes
to passthrough. cachekey.KeyPolicy serves
stats.IsContent.
Next: internal/fill, for clients reading while the store is written; The proxy engines' packages, for each proxy engine's keys; The store, for the cache on disk.
| Method | Contract |
|---|---|
Open(ctx, key) (io.ReadCloser, ObjectMeta, error) | The body and metadata of a committed object; ErrNotFound otherwise |
OpenRange(ctx, key, offset, length) (io.ReadCloser, ObjectMeta, error) | Bytes [offset, offset+length); length < 0 reads to the end |
Stat(ctx, key) (ObjectMeta, error) | Metadata only; use it instead of opening and discarding the body |
Create(ctx, key) (ObjectWriter, error) | A writer whose object is visible to Open and Stat only after Commit; a Commit over an existing object switches body and metadata together, and a reader that has the old body keeps reading it |
Delete(ctx, key) error | Removes the object and its metadata; a reader that has it open keeps reading it |
UpdateMeta(ctx, key, meta) error | Replaces a committed object's metadata, body untouched (a 304); an error wrapping ErrNotFound if there is none |
| Other API | Contract |
|---|---|
ObjectWriter | io.Writer, Commit(ctx, meta) error, Discard() error; Discard after Commit is a no-op, so defer w.Discard() is always safe |
ObjectWriter that is also io.ReaderAt | Reads back the bytes written so far, concurrently with writes, for a fill in progress; may fail once Commit or Discard returned |
store.Put(ctx, st, key, data, meta) | Create, Write, Commit of a small in-memory object, discarding on failure; sets meta.Size |
store.ErrNotFound | Returned by Open, OpenRange and Stat for a key with no committed object |
| Field | Type | Set by | Meaning |
|---|---|---|---|
Size | int64 | every proxy engine | Body length |
ETag | string | proxy engines that relay it | The origin's ETag |
LastModified | time.Time | proxy engines that relay it | The origin's Last-Modified, sent back when revalidating |
StoredAt | time.Time | every proxy engine | When it was stored or last revalidated; TTLs count from it |
ContentHash | string | apt on every entry; the other proxy engines when stored | <algorithm>:<digest> in the proxy engine's own terms: SHA256:<hex> (apt's index names), sha3-384:<hex> (snaps), sha256:<hex> (LXD, registry digests, files, Go's mutable endpoints), h1:<base64> (Go's dirhash) |
Immutable | bool | content-addressed entries | Never revalidated |
ContentType | string, omitempty | Docker manifests | The origin's media type, where the key cannot tell it |
MaxAge | time.Duration, omitempty | when the origin announces one | Cache-Control: max-age that differs from the proxy engine's TTL |
OriginURL | string, omitempty | files by checksum | The URL fetched, after redirects; dropped by fslocal when the metadata would not fit otherwise |
fslocal.New(dir) creates dir, objects/ and tmp/ (mode 0755), opens an os.Root on dir, so no key and no symlink can reach outside it, and probes for extended attributes by setting user.pkgproxy.meta on a tmp/probe-<hex> file; it fails on a file system without them (tmpfs before Linux 6.6, NFSv3, FAT, and every directory on a system other than Linux), and the daemon then refuses to start. The daemon calls it once per side, on -cache-dir/<side>.
Store.MissingMeta, when set, is called with the key of an object file that has neither the attribute nor a .meta.json: what a copy of the cache that dropped extended attributes and the .meta.json files leaves. cmd/pkgproxy sets it on every side's store to warnMissingMeta (cachestore.go), which logs once per store, so a restored cache that looks empty says why.
<-cache-dir>/ default /var/cache/pkgproxy
apt/ snap/ go/ docker/ lxd/ files/
objects/<key> the body; ObjectMeta in its user.pkgproxy.meta xattr
objects/<key>.meta.json the same ObjectMeta as plain JSON (the sidecar), for pkgproxy 0.11
tmp/obj-<32 hex> a body being written
go/sumdb/ the checksum database client's own cache
lxd/tls/ the LXD HTTPS listener's certificate
stats.json the stats recorder's checkpoint
lxd-index.json the LXD blob hashes' checkpoint
upstreams/ the upstream status history (§28.7)
upstream-pkgproxy.json the upstream pkgproxy's last description (§27.2)
snap-session-prefix the prefix of the snap device session placeholders, mode 0600
| Operation | On disk |
|---|---|
Create | tmp/obj-<hex>, opened O_EXCL, mode 0644; ten tries at a fresh random name |
Commit | encodeMeta, setAttr on the temporary file, Sync, Close, MkdirAll of the key's directory, removes the old sidecar, Rename to objects/<key>, then writes the new sidecar; a crash in between leaves an object without one, and failing to write it fails nothing |
Discard | Closes and removes the temporary file; nothing after Commit |
Stat, Open, OpenRange | Open objects/<key> and read the metadata from that open file: the attribute, or the sidecar when it has none; a file with neither, or not a regular file, is ErrNotFound, and a file with neither is also passed to MissingMeta |
UpdateMeta | Replaces the attribute in place on the open file (fsetxattr, atomic), body untouched, then replaces the sidecar through a temporary file and a rename; lost if a Commit replaced the file meanwhile, as the new object has its own |
Delete | Removes the object, then its sidecar; a missing file is not an error |
MigrateSidecars | Walks objects/ and gives the attribute to each file that has only a sidecar, with XATTR_CREATE on the file it opened, so an attribute a Commit or UpdateMeta sets meanwhile stays; keeps the sidecars; returns how many it migrated |
Store.SidecarOnly, set by -cache-metadata=meta-json (the snap option cache-metadata), writes the sidecar alone, as 0.11 did, and reads it before the attribute: it makes a 0.11 cache for the spread test tests/install/migrate-meta, and gives up the atomic switch of body and metadata, so it is for tests only.
| epoch 0 | 0.11 and earlier: the metadata in <key>.meta.json alone |
| epoch 1 | the metadata in user.pkgproxy.meta |
epoch: 1* in snap/snapcraft.yaml | reads 0 and 1, writes 1; with the sidecars it still writes 0 too, because snap revert (snapstate.RevertToRevision) checks no epoch |
snap/hooks/post-refresh | runs pkgproxy -cache-dir "$SNAP_COMMON/cache" -migrate-cache into $SNAP_DATA/migrate-cache.log, and never fails, since snapd undoes a refresh whose hook fails |
-migrate-cache | migrateCache (migratecache.go): MigrateSidecars on each proxy engine's store, for at most -migrate-cache-timeout (2m); runs before the other settings are checked, then exits |
| retiring epoch 0 | a later release, most likely 0.13, declares plain epoch: 1, stops writing and reading sidecars and removes them; snapd offers it only to 1* revisions, so a revert from it lands on one (§25.6) |
The attribute holds JSON as it is up to maxMetaSize, 3072 bytes (ext4 keeps all of a file's attributes in one 4 KiB block); larger JSON, up to maxMetaJSON, 64 KiB, is compressed with gzip, whose magic bytes no JSON starts with. Metadata that still does not fit loses OriginURL, and is refused if it is still too large. getfattr --only-values -n user.pkgproxy.meta FILE | zcat shows a compressed value.
A key is a slash-separated relative path. cleanKey rejects one that relpath.Clean cleans to ., .., ../… or an absolute path, before os.Root enforces containment. memstore keeps committed objects in a map under a mutex, buffers a writer until Commit, and copies on the way in and out.
| Proxy engine | Store directory | Keys |
|---|---|---|
| apt | apt/ | <archive id>/dists/… and <archive id>/pool/…, from cachekey |
| snap | snap/ | sha3-384/<hex> (snaps), pending/sha3-384/<hex> (hash to CDN URL and size), store-path/<sha256 hex of the path> (a store download path to its blob's hash), info/<name>…, find/<hash>, assertions/<type>/<key>…, catalog/<hash>, refresh/… |
| Go | go/ | the GOPROXY URL space: <escaped module>/@v/<escaped version>.info, .mod, .zip, <module>/@v/list, <module>/@latest |
| Docker | docker/ | blobs/sha256/<hex>, manifests/sha256/<hex>, tags/<name>/_/<tag>/<id> (a digest per set of accepted media types) |
| LXD | lxd/ | <remote>/<path>, such as <remote>/images/<os>/<release>/<arch>/<variant>/<serial>/<file> |
| files | files/ | sha256/<hex> or sha512/<hex>; volatile files volatile/<hex>, the SHA-256 of the normalized URL (VolatileKey) |
cachekey.PathResolver{}.Resolve(u) finds the first dists or pool path segment. The archive id is the host (with its port unless it is 80) plus every non-empty segment before that one; the key is <archive id>/<archive path>.
| Path class | Policy | Store contract |
|---|---|---|
…/pool/** | PolicyImmutable | Committed with Immutable: true, only if it matches the hash the parsed indexes list (a path they do not list is cached with a warning); streamed through a fill on a miss |
…/dists/**/by-hash/<ALGO>/<hex> | PolicyImmutable | As above; the hash comes from the URL (KnownHash), no index lookup |
other …/dists/** (Release, InRelease, Packages, Sources, …) | PolicyRevalidate | Committed with the origin's LastModified; once StoredAt is older than -apt-revalidate-ttl (default -revalidate-ttl, 5m) a conditional GET goes out, and a 304 calls UpdateMeta |
no dists or pool segment | PolicyPassthroughOnly | No key; relayed, never stored |
| Example URL | Key |
|---|---|
http://archive.ubuntu.com/ubuntu/pool/main/a/apt/apt.deb | archive.ubuntu.com/ubuntu/pool/main/a/apt/apt.deb |
http://repo.example:8080/debian/dists/stable/Release | repo.example:8080/debian/dists/stable/Release |
http://host:80//dists/x/Release | host/dists/x/Release |
cachekey.Resolved field | Meaning |
|---|---|
Key | <ArchiveID>/<ArchivePath>; empty for passthrough |
ArchivePath | From dists/ or pool/ on; what index.Lookup and Release entries key on |
ArchiveID | Host (and port) plus path prefix, such as archive.ubuntu.com/ubuntu |
Policy | PolicyImmutable, PolicyRevalidate, PolicyPassthroughOnly (String(): immutable, revalidate, passthrough-only) |
OriginURL | The URL to fetch upstream |
KnownHash, HasKnownHash | The apt.Hash a by-hash path names |
cachekey.KeyPolicy(key) reads the policy back from a stored key; stats.IsContent uses it to tell content from indexes.
pkgproxy mirrors Docker Hub for anonymous pulls: it holds the token, rechecks tags, and keeps manifests and layers by digest.
pkgproxyctl setup docker | sudo sh edits
/etc/docker/daemon.json (or the Docker snap's) and restarts
the daemon. registry-mirrors takes no path, so the API is at
/v2/. Only docker.io images use it.
GET /v2/ is answered 200 by pkgproxy. On a miss, the
registry's 401 Bearer challenge is met with an anonymous
token, cached per repository. Pushes and authenticated pulls are not
supported.
A tag pointer (tags/<name>/_/<tag>/…) is trusted
for -docker-revalidate-ttl (default: -revalidate-ttl, 5m), then resolved
with HEAD. Manifests and blobs are keyed
manifests/sha256/<hex> and
blobs/sha256/<hex>, verified and immutable.
A blob miss is a fill shared by every pull. Its last byte is held back until the sha256 digest matches and the blob is committed; a mismatch is not cached and its responses end short.
-docker-upstream (default
https://registry-1.docker.io, or off, after which registry paths get a 404) and
-docker-revalidate-ttl (default: -revalidate-ttl, 5m); both
are snap options, docker-upstream and
docker-revalidate-ttl. Only pulls are served; the cache is
<cache-dir>/docker.
Next: pkgproxy for CI, for build machines pulling the same images; Statistics, for layers counted per image.
Next: Immutable or not, for what is kept and what is rechecked; Snap Store relay, the other proxy engine under /v2; pkgproxyctl setup, for the other clients.
The mirror answers at the root of port 3142. These shapes are the registry's; any other path under /v2 belongs to the Snap Store relay. Only GET and HEAD are served; any other method is a 405.
| Path | What pkgproxy does | Cache policy |
|---|---|---|
/v2/ | Answers 200 itself, with an empty JSON body: no login needed | not cached; never asks the registry |
/v2/<name>/manifests/<tag> | Resolves the tag to a manifest digest with a HEAD to the registry, then serves that manifest | the tag's digest is revalidated after docker-revalidate-ttl; the manifest is kept forever |
/v2/<name>/manifests/sha256:<hex> | Serves the manifest, fetched once and checked against its digest | kept forever |
/v2/<name>/blobs/sha256:<hex> | Serves the layer or configuration blob; a miss is one download shared by every pull, Range included | kept forever, checked while it streams in; a HEAD on a miss asks the registry and caches nothing |
/v2/<name>/referrers/sha256:<hex> | Relays the request to the registry | not cached |
/v2/<name>/tags/list, /v2/_catalog | 404 | not served |
Manifests and blobs are shared by every repository and tag that names their digest. A cached digest is served under any repository name without asking the registry. With docker-upstream set to off, every path above is a 404.
| Registry's answer while resolving a tag | Client gets |
|---|---|
| Cannot be reached | The stale manifest, if the tag resolved before, with a warning in the log |
| 404 | 404: the tag is gone, and the client falls back to Docker Hub |
| 401 or 403, the tag resolved before | The stale manifest, asked again on every request |
| 401 or 403, nothing cached | 404 |
| A failed fetch of a manifest or blob not cached | 502 |
| Setting | Flag | Environment | Snap option | Default |
|---|---|---|---|---|
The registry to fetch from, or off to turn the mirror off | -docker-upstream | PKGPROXY_DOCKER_UPSTREAM | docker-upstream | https://registry-1.docker.io |
| How long a tag's digest is trusted | -docker-revalidate-ttl | PKGPROXY_DOCKER_REVALIDATE_TTL | docker-revalidate-ttl | revalidate-ttl, 5m |
Under the docker directory of the cache (/var/snap/pkgproxy/common/cache/docker in the snap, /var/cache/pkgproxy/docker otherwise), each object with its metadata in an extended attribute of its file:
docker/objects/ blobs/sha256/<hex> layers and configuration blobs manifests/sha256/<hex> manifests and image indexes, with their media type tags/<name>/_/<tag>/<id> the digest a tag last resolved to, per set of accepted media types
Nothing there is deleted on its own. To drop it all, stop pkgproxy (sudo snap stop pkgproxy), remove the directory and start it again. See The store.
A client asks pkgproxy for a large file by URL and checksum. The checksum is the cache key, and only bytes that match it are kept; a checksum list that changes under its URL is kept for when its server is down.
GET /files/v1/fetch?url=…&sha256=… (or
sha512, optional size). pkgproxy is the HTTPS
client and verifies the origin's certificate. X-Pkgproxy-Origin-URL
and X-Pkgproxy-Cache describe the answer.
The key is sha256/<hex> or sha512/<hex>
in <cache-dir>/files, whatever the URL. Entries are
immutable; a new file has a new checksum and misses.
A miss is a fill shared by every request, Range included. The last byte is held until the checksum, and the size if given, match; otherwise nothing is cached and every response ends short.
GET /files/v1/volatile?url=…: X-Pkgproxy-Cache is
FRESH, REVALIDATED or, from the copy kept while the
server is down, STALE, with X-Pkgproxy-Fetched.
Files over files-volatile-max-size (1M) are refused.
files-allow-host (empty: any host),
files-allow-network (empty: no private networks) and
files-max-size (8G, 0 for no limit),
as snap options or flags. The private-address check runs on every
connection, redirects included.
Next: Settings, for the files settings; The cache page, where files by checksum are a kind of their own, grouped by name.
Next: pkgproxy for CI, for many machines behind one cache; Offline mode, for files served with no network.
Next: Fetch once, for one download shared by every client; Offline mode, for files served with no network; The store, for where they are kept.
GET /files/v1/fetch?url=<URL>&sha256=<hex>[&size=<bytes>] GET /files/v1/fetch?url=<URL>&sha512=<hex>[&size=<bytes>] HEAD the same
Any other path under /files is a 404, and any method but GET and HEAD a 405.
| Parameter | Required | Value |
|---|---|---|
url | yes | An absolute http:// or https:// URL with a host, percent-encoded. No user name or password. |
sha256 | one of the two | 64 hexadecimal digits. |
sha512 | one of the two | 128 hexadecimal digits. Give one checksum, not both. |
size | no | The file's size in bytes, a positive number. Checked like the checksum, and lets clients follow a download whose server sends no length. |
| Header | Value |
|---|---|
Content-Type | application/octet-stream |
Content-Length | The file's size |
Accept-Ranges | bytes: Range requests and resumed downloads work, on a hit and while the file downloads |
X-Pkgproxy-Cache | hit from a complete cached file; miss while the file downloads, whether this request started the download or joined it |
X-Pkgproxy-Origin-URL | The URL the bytes were downloaded from, after redirects. On a hit it is the URL of the request that fetched the file. |
| Status | When |
|---|---|
| 200 or 206 | The file, from the cache or as it downloads. |
| 400 | A missing or malformed parameter; or, on a hit, a size other than the cached file's. |
| 403 | The host is not allowed by files-allow-host; size is larger than files-max-size; or a connection would go to a private address files-allow-network does not allow, after any redirect. |
| 404 | A HEAD for a file that is not cached (HEAD never contacts the origin). |
| 405 | A method other than GET or HEAD. |
| 502 | The download failed, after asking the origin again for up to upstream-retry-for (see Fetch once): the origin could not be fetched, more than 10 redirects, a file larger than files-max-size, or bytes that do not match the checksum or size. |
| 503 | Offline mode, for a file that is not cached; cached files are still served. |
A download that fails after its headers were sent ends short instead, so the client rejects it. On any error, the client downloads the file directly.
GET /files/v1/volatile?url=<URL> HEAD the same
For small files that change under their URL, such as a cloud image's SHA256SUMS. The URL follows the rules of fetch. Every request goes to the server, conditionally when a copy is kept; pkgproxy keeps the last copy.
| Answer | When |
|---|---|
200, X-Pkgproxy-Cache: FRESH | The server sent the file; it replaces the copy kept. |
200, X-Pkgproxy-Cache: REVALIDATED | The server said the copy kept is current (its 304). |
200, X-Pkgproxy-Cache: STALE | The copy kept, because the server cannot answer: it is down, times out, answers 5xx or 429, its breaker is open in store-overloaded, or pkgproxy is offline. |
| 404, 410 or another 4xx | The server's own answer, passed on; the copy kept stays. |
| 403 | The host or address is not allowed, or the file is larger than files-volatile-max-size. |
| 502 | Nothing kept, and the server cannot answer, or answered something else, such as a redirect. |
| 503 | Offline mode, with nothing kept. |
Every answer with the file carries the server's Content-Type, ETag and Last-Modified, X-Pkgproxy-Origin-URL, and X-Pkgproxy-Fetched: when the copy was fetched, or last confirmed by the server. A client's own conditional and Range requests work against the server's validators. The access log records fetched, revalidated, stale, or passthrough for a 4xx passed on. A kept copy is never deleted; pkgproxyctl cache invalidate <URL> throws one away.
| Setting | Flag | Environment | Default | Meaning |
|---|---|---|---|---|
files-allow-host | -files-allow-host | PKGPROXY_FILES_ALLOW_HOST | empty: any host | Hosts clients may name, separated by commas: a host name, or a domain starting with . for every host under it. Empty or * allows every host. Checked on the URL the client names, not on redirects. |
files-allow-network | -files-allow-network | PKGPROXY_FILES_ALLOW_NETWORK | empty: none | Private networks, in CIDR notation and separated by commas, that files may be fetched from. |
files-max-size | -files-max-size | PKGPROXY_FILES_MAX_SIZE | 8G | Largest file fetched, such as 8G or 500M; 0 means no limit. |
files-volatile-max-size | -files-volatile-max-size | PKGPROXY_FILES_VOLATILE_MAX_SIZE | 1M | Largest volatile file fetched, such as 1M or 64K; a larger one is refused with 403 and the copy kept stays. 0 means no limit. |
All four are also snap options: sudo snap set pkgproxy files-max-size=20G. The private addresses refused unless allowed are loopback, private and unique local, link-local, unspecified, multicast, 0.0.0.0/8 and 100.64.0.0/10. A proxy pkgproxy fetches through (see Settings) is used wherever it is, on loopback or the local network too, as for apt; a client that names the proxy's own address as a file's URL is fetched from directly, and checked like any other.
Files are kept in the files directory of the cache, as files/objects/sha256/<hex> or files/objects/sha512/<hex>, each with its metadata, the origin URL among it, in an extended attribute of the file. Volatile files are kept beside them, as files/objects/volatile/<hex>, the SHA-256 of the URL. The cache is /var/snap/pkgproxy/common/cache in the snap and /var/cache/pkgproxy otherwise. Files are never revalidated or deleted; see The store.
How one upstream download is shared by every request for a file, and what happens when it goes wrong.
Concurrent requests for one file share one upstream download. A fill
streams to every waiting client while it is written (apt's
pool/**, snaps, LXD image files, Docker blobs, files by
checksum); indexes, Go modules and manifests are served once complete.
A plain request gets 200 once the upstream's headers are in;
a Range or If-Range resume gets 206
from the same fill. With the size known in advance (snaps, LXD image
files) the headers go out at once.
A fill that gives up ends every follower's response short and caches
nothing; the next request starts a new fill. -upstream-idle-timeout
(60s) bounds upstream inactivity, not total time.
A Range: bytes=<kept>- retry, 5 in a row after 1, 2,
4, 8 and 16 s, while followers wait. -upstream-retry-for
(2m after the last byte; 0 never asks again).
Also retried before the body: a dropped connection and 408, 429, 500,
502, 503, 504, a proxy's included.
Drain (apt, snap, LXD): followers get the origin's bytes, bounded by
-fill-drain-timeout (10m), for apt, snaps and LXD
images alike. Hold the last byte (Docker blobs, files): a
mismatch ends the response short.
Next: The store, for where a download is written; Stalled upstreams, for when an upstream goes silent; pool and dists, for apt's files.
Next: Stalled upstreams, for when an upstream goes silent; pool and dists, for apt's files; Immutable or not, for what is kept and what is rechecked.
| What happens | Snaps, LXD images, files by checksum | Go module .mod and .zip | Apt packages, Docker blobs |
|---|---|---|---|
the upstream breaks off mid-body, or fails before it starts with a dropped connection, 408, 429, 500, 502, 503 or 504 | asked again for the rest, up to 5 times in a row after 1, 2, 4, 8 and 16 s, within upstream-retry-for (2m) of the last byte; followers keep waiting | asked again from the start, up to 5 more times after 1, 2, 4, 8 and 16 s, while the next try starts within upstream-retry-for (2m) of the first; the go command waits | the fill fails, followers are cut short |
a Retry-After longer than what is left of upstream-retry-for | fails at once | not read | — |
in store-overloaded, the host's breaker is open when a retry is due | waits for the breaker to let it through, within upstream-retry-for | the retry fails at once, as the breaker refuses it | — |
| an error answer instead of the file | at most 64 KiB of it is read | the upstream's status is passed on once the tries run out | — |
| more bytes than the store or image server promised (snaps, LXD images) | the fill fails, followers are cut short | — | — |
| the bytes do not match the hash, whichever tries they came from | not cached | not cached, 502 | not cached |
A cached file that is replaced while it is sent, such as an InRelease that another machine's apt update refreshed just then, is sent with the size, ETag and Last-Modified of the copy actually sent.
Which entries are kept forever and which are rechecked after a short window, in every proxy engine.
Content addressed by hash or by version is immutable: fetched once, verified and cached forever. Indexes, version lists and tags are mutable and revalidated after a TTL. Files by checksum have no mutable part.
Inside the TTL a mutable entry is a hit. After it, apt sends
If-Modified-Since: 304 renews the entry,
200 replaces it. The other proxy engines fetch again; Docker resolves
a tag with HEAD.
-revalidate-ttl (5m) applies to every proxy engine;
-apt-revalidate-ttl, -snap-revalidate-ttl,
-go-revalidate-ttl, -lxd-revalidate-ttl and
-docker-revalidate-ttl override it for one proxy engine each, with
snap options of the same names. Zero is refused. A snap store
max-age takes precedence; offline mode makes every window
endless.
Cache-Control: no-cache, max-age=N or
Pragma: no-cache force a conditional GET, except within
-min-forced-revalidate-interval (30s) of the last
check. A 304 keeps the entry; pkgproxyctl cache
invalidate removes one.
A failed revalidation serves the stale entry with a warning. apt, Go and
LXD wait at most 10 s for it and skip a failing host for 1 min. For apt, an origin
404/410 is relayed as 404; a hash
mismatch is passed through uncached.
Next: Offline mode, for freshness frozen; pool and dists, for apt in detail; Stalled upstreams, for when an upstream counts as stalled.
Kept forever means fetched once, checked against its hash where one is known, and never asked about again. Rechecked means served from the cache for the window in the last column, then checked with the upstream. Relayed means passed between client and upstream without being kept.
| Entry | Policy | Window |
|---|---|---|
anything under pool/, and dists/…/by-hash/… | kept forever, hash checked | none |
Release, InRelease, Release.gpg | rechecked with If-Modified-Since | apt-revalidate-ttl |
every other file under dists/ (Packages, Sources, Contents, Translation, diff indexes) | rechecked with If-Modified-Since | apt-revalidate-ttl |
a URL with no pool or dists path segment | relayed | none |
an https:// source, through a CONNECT tunnel | relayed, unseen | none |
An index with no Last-Modified from the mirror, or one a newer index already shows to be superseded, is fetched again in full instead of conditionally.
| Entry | Policy | Window |
|---|---|---|
| snap files, by their SHA3-384 | kept forever | none |
snap-revision assertions, and account, snap-declaration, store or validation-set assertions pinned to a sequence number | kept forever | none |
every other assertion, account-key included | rechecked | snap-revalidate-ttl |
| snap info, find, categories, sections, names, repairs, and the old v1 search and details | rechecked | the store's max-age, else snap-revalidate-ttl |
| refresh and install requests | relayed; each result recorded for answering offline | none |
| a snap's or its dm-verity data's download path, named next to its hash in an answer to an anonymous request | kept forever, as the snap it names | none |
| other download redirects without a hash, such as resources | followed and streamed, not kept | none |
| logins, device sessions and serials, cohorts, metadata | relayed; nonces and sessions answered with placeholders while the store's auth is not contacted | none |
| any other store path | relayed | none |
The store's max-age wins whether it is shorter or longer than the window; a max-age of zero counts as none.
| Entry | Policy | Window |
|---|---|---|
.mod and .zip of a version | kept forever, after the checksum database check | none |
list, @latest and .info | fetched again in full | go-revalidate-ttl |
| a 404 from the upstream | not kept | none |
| Entry | Policy | Window |
|---|---|---|
| layers and configs, by digest | kept forever, digest checked | none |
| manifests and indexes, by digest | kept forever, digest checked | none |
| the digest a tag resolves to, per set of accepted media types | rechecked with a HEAD | docker-revalidate-ttl |
a HEAD of a layer not yet cached | answered from the registry's HEAD, not kept | none |
| referrers | relayed | none |
| Entry | Policy | Window |
|---|---|---|
streams/v1/index.json and every other .json catalog | fetched again in full | lxd-revalidate-ttl |
| an image file whose path a catalog listed | kept forever, hash checked | none |
| a path no catalog listed | answered 404, never fetched | none |
LXD's image API, on the HTTPS port, reads the same catalogs and files, so it follows the same policy.
Every file is kept forever under its checksum and never rechecked; there is nothing to revalidate.
| Setting | Default | Window for |
|---|---|---|
revalidate-ttl | 5m | every proxy engine whose own setting is unset |
apt-revalidate-ttl | revalidate-ttl | apt indexes |
snap-revalidate-ttl | revalidate-ttl | snap details and assertions without a max-age |
go-revalidate-ttl | revalidate-ttl | Go version lists, @latest, .info |
docker-revalidate-ttl | revalidate-ttl | Docker tags |
lxd-revalidate-ttl | revalidate-ttl | LXD catalogs |
min-forced-revalidate-interval | 30s | apt only: how long after a check a client's request for a fresh copy is still served from the cache |
Each is a -flag, a PKGPROXY_ variable and a snap option of the same name; see Settings. A window must be longer than zero: pkgproxy refuses 0 at start and in -check-config. In offline mode every window is endless; see Offline mode.
| Proxy engine | Cache-Control: no-cache, max-age=N, Pragma: no-cache |
|---|---|
| apt | a conditional request upstream before serving, unless the index was checked within min-forced-revalidate-interval; requests forcing one index share one check |
| snap, Go, Docker, LXD, files | ignored: the window decides |
A Pragma: no-cache counts only when there is no Cache-Control. A check confirms or replaces an index; it never deletes one. To throw a cached apt file away, use pkgproxyctl cache invalidate.
| Proxy engine | Upstream unreachable, stalled or answering 5xx | Upstream says the entry is gone |
|---|---|---|
| apt | cached copy served with a warning; waits at most 10 s, host skipped for 1 min | 404 or 410 relayed as 404 |
| Go, LXD | cached copy served with a warning; waits at most 10 s, host skipped for 1 min | cached copy served with a warning, as for any answer other than 200 |
| snap | cached copy served with a warning, also on a 429 from the store; the store is not asked for that entry again until a back-off ends (its Retry-After, else the time to live doubled per failure, up to an hour), and meanwhile the copy is served stale without a warning | upstream's answer relayed |
| Docker | stale tag served with a warning; a 401 or 403 also keeps a tag that resolved before | 404, and the client falls back to Docker Hub |
An outcome of stale in the access log marks each such answer.
The go command fetches modules from pkgproxy under /go. Module
files are checked against the checksum database and kept forever; version
lists are rechecked, and a module download that breaks off is asked for
again.
pkgproxyctl setup go | sh writes GOPROXY=http://host:3142/go,direct
where go env -w would, Go installed or not; export
GOPROXY=… does it for one CI job. ,direct is the
fallback on a 404 or 410.
@v/list, @latest and .info are
revalidated after -go-revalidate-ttl (default: -revalidate-ttl, 5m);
.mod and .zip are immutable. The upstream is
-go-upstream, https://proxy.golang.org by default.
A .mod or .zip is committed only after its
h1: hash matches sum.golang.org. A mismatch is a
502 and nothing is cached; a failed lookup is logged and cached.
-go-sumdb off skips the check.
A .mod or .zip whose body breaks off, or that
gets no answer on a dropped connection, or a 408, 429, 500, 502, 503 or
504, is fetched again from the start: up to 5 retries after 1, 2, 4, 8
and 16 s, within -upstream-retry-for (2m) of the
first request. A 404 or 410 is not retried.
Next: Settings, for the go flags and their defaults; The store, for where the modules are kept.
Next: pkgproxy for CI, for many machines behind one cache; Offline mode, for builds that see only what is cached.
Next: Offline mode, for builds from the cache with no network; pkgproxyctl setup, for the other clients.
Every request is a plain GET (or HEAD) on the main port, under -go-base-path, /go by default. Module paths are case-encoded as the protocol says: each capital letter is written as ! and the small letter, so github.com/BurntSushi/toml is asked for as github.com/!burnt!sushi/toml.
| Path | Answer | Cache policy |
|---|---|---|
/go/MODULE/@v/list | the known versions, one per line | revalidated after -go-revalidate-ttl |
/go/MODULE/@latest | the latest version, as JSON | revalidated after -go-revalidate-ttl |
/go/MODULE/@v/VERSION.info | a version's time and name, as JSON | revalidated after -go-revalidate-ttl |
/go/MODULE/@v/VERSION.mod | the version's go.mod | kept forever, after the checksum database check |
/go/MODULE/@v/VERSION.zip | the version's source, as a zip | kept forever, after the checksum database check |
any other path under /go | 404 | not cached |
A .mod or .zip must name a canonical version, such as v1.2.3 rather than v1.2; anything else is a 404. The checksum database's own proxy paths, /go/sumdb/…, are not served: they get a 404, and the go command then asks sum.golang.org directly, as it does with no proxy.
| Situation | Status the go command sees |
|---|---|
a method other than GET or HEAD | 405 |
| a path that is not one of the five requests | 404 |
.mod or .zip cached | 200 from the cache, also served to Range requests |
any path, the upstream answers 404 or 410 | the same 404 or 410, so ,direct takes over; remembered for -go-revalidate-ttl |
any path whose 404 or 410 is remembered | the same status, without asking the upstream; past the TTL, also while the upstream fails or cannot be reached |
.mod or .zip whose hash differs from the checksum database, or the database fails its own integrity check | 502, nothing cached |
.mod or .zip, the body breaks off, the connection drops or times out, or the upstream answers 408, 429, 500, 502, 503 or 504 | fetched again from the start, up to 5 more times after 1, 2, 4, 8 and 16 s, while the go command waits; then 200 with the whole module |
.mod or .zip, still failing after the last try, or when the next try would start more than -upstream-retry-for (2m) after the first | the upstream's own status, or 502 when it could not be reached or its body broke off |
.mod or .zip, the host does not resolve or refuses connections, or pkgproxy is offline | 502 at once, without asking again |
.mod or .zip that an upstream pkgproxy refuses as not in its cache while it is offline (see Two tiers) | fails at once, without asking again |
.mod or .zip whose 404 or 410 is remembered, past the TTL, the upstream fails | the remembered status, without asking again |
| list, latest or info, cached and within the TTL | 200 from the cache |
list, latest or info, past the TTL, upstream fails or answers an error other than 404 or 410 | 200 with the cached copy, and a warning in the log |
list, latest or info, past the TTL, upstream answers 404 or 410 | that status; the cached copy is dropped |
| list, latest or info, never cached, upstream answers an error | the upstream's status, or 502 when it cannot be reached |
The go command looks for the module that holds a package by asking for several paths, such as golang.org/x/tools/gopls, golang.org/x/tools and golang.org/x, and expects 404 or 410 for those that are not modules. pkgproxy remembers each such answer for -go-revalidate-ttl, so a build asks the upstream only for what changed, and offline it gets the same answers, so go install example.com/tool@latest works for a tool installed through pkgproxy before.
A version asked for the moment it is tagged may be "not found" for up to -go-revalidate-ttl, as proxy.golang.org may not have it yet. GOPROXY=direct for that one command gets it.
| Lookup result | What happens to the .mod or .zip |
|---|---|
the database lists the same h1: hash | cached |
| the database lists a different hash | not cached, 502 |
| the database's signature or tree does not verify | not cached, 502 |
| the database cannot be reached, or does not list the version yet | cached unchecked, a warning in the log |
-go-sumdb off | cached unchecked, no lookup |
Only sum.golang.org is supported, with the key the go command itself carries; any other -go-sumdb value stops pkgproxy at startup. The database is asked directly over HTTPS, and its tree head and lookups are kept under go/sumdb in the cache directory.
| Option | Environment variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-go-upstream | PKGPROXY_GO_UPSTREAM | flag or environment only | https://proxy.golang.org | the module proxy misses are fetched from |
-go-base-path | PKGPROXY_GO_BASE_PATH | flag or environment only | /go | the URL path of the module proxy, part of GOPROXY |
-go-sumdb | PKGPROXY_GO_SUMDB | flag or environment only | sum.golang.org | the checksum database, or off |
-go-revalidate-ttl | PKGPROXY_GO_REVALIDATE_TTL | go-revalidate-ttl | -revalidate-ttl, 5m | how long list, latest and info answers, and "not found" answers, are served before they are checked again |
-upstream-retry-for | PKGPROXY_UPSTREAM_RETRY_FOR | upstream-retry-for | 2m | how long after the first request a .mod or .zip fetch may still be tried again; 0 never asks again |
All settings are in Settings.
| Variable | Set by pkgproxyctl setup go | Meaning |
|---|---|---|
GOPROXY | http://HOST:3142/go,direct, in the user's go env file, where go env -w would write it, whether go is installed yet or not | ask pkgproxy first; on a 404 or 410, fetch from the module's source |
GOSUMDB | left as it is | the go command still checks every module itself |
GOPRIVATE | left as it is | modules it matches never go to any proxy |
pkgproxyctl setup go | sh needs no go command, so Go can be set up first and installed after, and fetches its first module through pkgproxy. It fails only where go keeps no settings (GOENV=off, no HOME); then set GOPROXY in the environment. pkgproxyctl setup go -undo removes the GOPROXY line from that file, as go env -u GOPROXY does, whatever the user had set, not only pkgproxy's. For one CI job, export GOPROXY=http://HOST:3142/go,direct does the same without touching the user's settings. See Setting up clients.
How the upstream idle timeout fails a download that stops, without cutting off a slow one.
-upstream-idle-timeout (default 60s) fails a
request only after that long without progress. A 2 GB layer at 1 MB/s
takes its 33 minutes. 0 turns the check off.
The timer runs while the headers are awaited and while each read of the body waits for bytes, and is stopped between reads. Time package proxy spends passing the bytes on to a client is not the upstream's.
A shared download whose upstream stalls fails like on any upstream error and is removed from the downloads in progress; a snap, LXD image or file by checksum asks once more first, about 2 minutes in all. The next request for the file starts a new download rather than waiting on the old connection.
-read-header-timeout (30s),
-upstream-idle-timeout (60s) and
-shutdown-grace (10s), also as snap options of
the same names. Connecting upstream is bounded at 30s, the
TLS handshake at 10s.
Next: Fetch once, for the shared download; Immutable or not, for a stalled recheck; Offline mode, for leaving a failing upstream alone.
| What failed | Asked again | Up to |
|---|---|---|
any request answered 408 Request Timeout | the same request, when its body can be sent again | 3 attempts in all |
a snap, LXD image or file by checksum that broke off or never started: a dropped or silent connection, or 408, 429, 500, 502, 503, 504, a proxy's included | the rest of it, with Range from the bytes kept | 5 tries in a row after 1, 2, 4, 8 and 16 s, within upstream-retry-for of the last byte |
An upstream that never answers a snap, LXD image or file by checksum is given up on after two attempts of upstream-idle-timeout, about 2 minutes. Every byte that arrives resets the count and the clock. A download that cannot be made at all (no such host, refused, offline mode) is not asked again. Apt and Docker downloads fail at the first error.
| Setting | Snap option | Default | Bounds |
|---|---|---|---|
-upstream-idle-timeout | upstream-idle-timeout | 60s | how long one upstream request may receive nothing; 0 waits forever |
-upstream-retry-for | upstream-retry-for | 2m | how long after the last byte a snap, LXD image or file by checksum is still asked for again; 0 never asks again |
How errors, help, guides, pages and logs are translated through one catalog, with commands and paths kept out of it.
Prose is translated and reflowed; commands, paths and configuration stay
verbatim. TestCtlSetupApt_GuidePolish pins this output.
A page takes ?lang=, then the pkgproxy-lang cookie,
then Accept-Language (pageLocale).
l10n.D.Errorf keeps errors.Is identity for
sentinels. Error() is English; go-cmdr prints the
translation. Write "cannot …", not "failed to …" or "could not …".
g.para(…) takes prose from l10n.D.Tr;
g.code(…) takes lines that are never translated or reflowed.
The msgid has %s where the path goes.
l10n.NewLogHandler wraps the daemon's slog text
handler. A message from a library at run time passes through untranslated.
go tool i18n scan, merge -no-obsolete, translate
pl.po, then check -auto -S, all with -C
internal/l10n.
Next: How it is tested, for translated output in tests; The setup page, for the visitor's language; The source tree, for the other conventions.
const: the scanner follows constants passed to Tr, TrN and Errorf.TRANSLATORS: right above the call; other comments stay out of the catalog.go tool i18n scan -C internal/l10n go tool i18n merge -C internal/l10n -no-obsolete # translate the new and fuzzy entries in internal/l10n/po/pl.po go tool i18n check -C internal/l10n -auto -S
go tool i18n is gitlab.com/zygoon/go-i18n/cmd/i18n, a tool dependency in go.mod. Scan and merge run at development time only; the binary embeds the result.
| Text | Call | Language |
|---|---|---|
| An error, sentinels included | l10n.D.Errorf("cannot …", …), never fmt.Errorf or errors.New | Error() English; go-cmdr's printer shows LocalizedError in the process's locale, wrapped causes included |
| Help, command output, page text | l10n.D.Tr(ctx, …), l10n.D.TrN(ctx, …, n) for plurals | The locale on ctx |
| A log message | logger.Info(i18n.MarkForTranslation("…"), …) | Translated by l10n.NewLogHandler as it is written |
| Setup guidance | g.para(l10n.D.Tr(ctx, …)) for prose, g.code(…) for commands, paths and configuration | Prose translated and wrapped to 74 columns (guideWidth); code verbatim, indented four spaces |
| A message a piped script prints | shellFail(l10n.D.Tr(ctx, …)) | Translated and shell-quoted; the rest of the script is not translated |
Wording: a failed action reads cannot <verb> …, not "failed to …", "could not …" or a gerund wrapper such as "creating X: %w". An error that describes a state ("malformed field line") stays as it is, and a package prefix such as apt: is kept. Daemon log messages and HTTP error bodies use the same wording. A msgid is one unwrapped paragraph with %s or %q where a path or name goes.
| Translated | Not translated |
|---|---|
pkgproxyctl and pkgproxy help, through go-cmdr's own catalog too | Error() of every error: logs and tests read the same in every locale |
Errors as pkgproxyctl and pkgproxy print them | HTTP error bodies, including snapd error results, the store-style error list of a refused snapd request, and offline.PlainAnswer's body |
| Setup guides on a terminal, and the daemon's startup banner | Scripts piped to sh, apart from what shellFail prints |
| Every page: the setup page, a tool's page, the cache page, the upstream status pages, the bar and the banner, in the visitor's language | Commands, paths, configuration and YAML: never part of a msgid |
What an error kind means, on hover (kindHint: http-429 is "Too Many Requests: the server asked pkgproxy to slow down") | The error kinds themselves, such as http-429 or timeout, in plain text, the API and the strips' cell tooltips |
| Daemon log messages, and error-valued log attributes | Log attribute keys and other values |
pkgproxyctl stats, offline, version and the other commands' output | A log message that comes from a library at run time, such as the checksum database client's |
The snap's daemon runs under systemd, normally in the C locale, so its log is in English unless the service is given a locale.
| Where | Locale |
|---|---|
main | i18n.DetectLocale(), the process's locale, on the base context; i18n.BindAll(ctx, l10n.D, cmdrl10n.Domain) binds pkgproxy's and go-cmdr's catalogs |
Every page, /pkgproxy/status, setup scripts | pageLocale(w, r): ?lang=<tag> when l10n.Lookup knows it, remembered in the cookie pkgproxy-lang (a year, HttpOnly, SameSite=Lax); ?lang= with no known tag forgets the cookie; else the cookie; else l10n.AcceptLanguage(r.Header.Get("Accept-Language")), the best match among English and the catalogs, English for none or a malformed header. Set on a context used only to render, never to log; the answers say Vary: …, Cookie, … |
l10n.NewLogHandler(ctx, next) | The record's context when it carries a locale, otherwise the daemon's |
| File | What it is |
|---|---|
internal/l10n/po/pkgproxy.pot | The template scan writes: every msgid, with its source locations |
internal/l10n/po/pl.po | Polish, with three plural forms |
internal/l10n/l10n.go | //go:embed po; registers domain D = gitlab.com/zygoon/pkgproxy; English is the source language; every <lang>.po in po is a language |
internal/l10n/log.go | NewLogHandler: translates each record's message and renders error attributes through D.LocalizedError, groups included |
A new language is a new po/<tag>.po: AcceptLanguage, Lookup and Languages pick it up from the embedded directory. Add its name in itself to selfNames in internal/l10n/l10n.go too, or the switcher in every page's footer (langSwitcherFor, "Language: English · polski") shows its bare tag.
| Test | What it checks |
|---|---|
cmd/pkgproxy TestCtlSetupApt_GuidePolish | setup apt in Polish: prose translated and rewrapped, configuration identical to English |
TestCtl_HelpPolish, TestCtlVersion_Polish, TestSetupPage_Polish | Help, version and the setup page in Polish |
TestSetupPage_LanguageSwitcher | ?lang=, the cookie and the footer's links |
TestUpstreamsPage_Polish, TestUpstreamsPage_HowItKnows_Polish, TestKindHint_Polish | The upstream status pages, what they say of how they know, and the error kinds' hints, in Polish |
internal/l10n TestAcceptLanguage | Header to locale |
internal/l10n TestLogHandler_* | Messages and error attributes translated, groups, the record's locale winning, unknown messages passing through |
internal/registryproxy TestErrors_Localize | The Docker proxy engine's errors localize |
Command tests run in English through invoke; invokeIn(t, locale, argv...) runs them in another locale.
The port or socket every client connects to, the other sockets the daemon opens, what answers a request before the proxy engines, and the health route.
-addr (default :3142) is the TCP listener.
-socket-path replaces it with a UNIX socket, for example for
QEMU's guestfwd. Setup instructions always name
-addr.
-lxd-tls-addr (:3143) is HTTPS for lxc.
stats.sock and admin.sock live in
/var/snap/pkgproxy/common in the snap, /run/pkgproxy
otherwise. The snap adds content sockets under
$SNAP_COMMON/sockets.
bind(2) applies the umask; connect(2) needs
write permission. The umask is cleared around every bind, so each socket
is 0777. A stale socket file is removed first; this is not a
check for a second daemon.
-addr and -lxd-tls-addr must be
host:port with a numeric port. pkgproxy
-check-config validates flags and PKGPROXY_*
variables and exits; the snap's configure hook runs it.
Main port: access log, setup page (/, /setup/,
/pkgproxy/status, /pkgproxy/cache,
/pkgproxy/upstreams), LXD certificate, offline mode, the loop
check (508), then the proxy engines. HTTPS port: access log,
LXD image API, offline mode, the same proxy engines.
GET or HEAD /pkgproxy/v1/health on
-addr (or -socket-path) and the content socket:
200 and {"status":"ok","mode":"store-overloaded"},
with automatic while offline by itself. Served with
-setup-page=false; not in the access log.
Next: Settings, for the address and socket path; LXD images, for the HTTPS port; The cache page, for what the main port shows a local visitor.
Plain HTTP on -addr, or on the UNIX socket -socket-path names instead. A request is offered to each row in turn; the first that claims it answers. The name in the last column is what the access log records as the handler.
| Order | Request | Answered by | Local or trusted only | Log name |
|---|---|---|---|---|
| 0 | GET or HEAD /pkgproxy/v1/health | the health route: 200 while pkgproxy serves | no | not logged |
| 1 | GET or HEAD / | the setup page | no | setup-page |
| 1 | GET or HEAD /setup/TOOL, ?undo=1 for the undo script | a setup script for sh | no | setup-page |
| 1 | GET or HEAD /pkgproxy/cache | the cache page | yes | setup-page |
| 1 | GET or HEAD /pkgproxy/cache/activity | the cache page's figures, as JSON | yes | setup-page |
| 1 | GET or HEAD /pkgproxy/cache/events | the cache page's live stream | yes | setup-page |
| 1 | GET or HEAD /pkgproxy/cache/upstreams | the status dots of the cache page's map, as JSON | yes | setup-page |
| 1 | GET or HEAD /pkgproxy/upstreams, /pkgproxy/upstreams/HOST | the upstream status pages | yes | setup-page |
| 1 | GET or HEAD /pkgproxy/status | the status element of every page's bar, never cached | no | setup-page |
| 2 | GET or HEAD /pkgproxy-lxd-cert.pem | the HTTPS port's certificate, in PEM | no | lxd-cert |
| 3 | CONNECT host:port, or an absolute URL such as GET http://archive.ubuntu.com/… | the apt forward proxy | no | apt |
| 3 | /pkgproxy/v1/proxy-engines | which proxy engines this pkgproxy runs, and where, for a pkgproxy using it as its upstream | no | (none) |
| 4 | /go and below | the Go module proxy | no | gomod |
| 5 | /lxd/NAME/… | an LXD image server, by simplestreams | no | lxd |
| 6 | /files/v1/fetch?…, /files/v1/volatile?… | files by checksum, and files that change under their URL | no | files |
| 7 | /v2/, /v2/REPO/manifests/REF, /v2/REPO/blobs/DIGEST, /v2/REPO/referrers/DIGEST | the Docker Hub mirror | no | docker |
| 7 | /v2/REPO/tags/list, /v2/_catalog | the Docker Hub mirror, which answers 404 | no | docker |
| 8 | any other path under /v2, /api/v1 or /v1 | the Snap Store relay | no | snap |
| 9 | anything else | 404, sent nowhere | no | (none) |
setup-page says. A forward-proxy request for the same path goes upstream as any other.GET and HEAD; other methods fall through. Row 1 goes away with setup-page set to false, row 2 when the HTTPS port is off.trusted-networks (-trusted-networks, empty by default); without a reverse proxy's header such as Via or X-Forwarded-For; and with a Host that is an IP address, localhost, the machine's name or the host of public-url.-go-base-path and -lxd-base-path. A proxy engine that is turned off claims nothing: its paths reach row 9, and Docker-shaped paths get a 404 rather than going to the Snap Store.X-Pkgproxy-Via, or more than 8, is a loop between pkgproxies, answered 508 Loop Detected; see Two tiers. Every answer of a proxy engine carries X-Pkgproxy-Engine with its name: apt, snap, go, docker, lxd, files or files-volatile.GET /pkgproxy/v1/health is for service managers, load balancers, Pebble checks and Kubernetes probes. It is on the main port (or the socket -socket-path names in its place) and the main content socket, not on the HTTPS port.
$ curl http://localhost:3142/pkgproxy/v1/health {"status":"ok","mode":"store-overloaded"}
| Field | Meaning |
|---|---|
status | always ok: an answer at all means pkgproxy serves; otherwise the connection is refused |
mode | the offline mode in effect; offline is still 200, as the cache serves what it holds |
automatic | only while pkgproxy is offline by itself: no-network or airplane-mode |
It names no upstreams, as anyone who reaches the port may ask it; the stats socket's /pkgproxy/v1/info lists them.
HTTPS on -lxd-tls-addr, with the self-signed certificate from the cache directory, or on the UNIX socket -lxd-tls-socket-path names. It exists for lxc, which talks to image servers only over HTTPS. It is off when -lxd-tls-addr is off or no LXD remote is configured.
| Order | Request | Answered by | Log name |
|---|---|---|---|
| 1 | GET / | LXD's API root, as LXD asks when it adds a remote | lxd-api |
| 1 | GET /1.0 | the server description | lxd-api |
| 1 | GET /1.0/images?project=NAME | the images of the remote NAME | lxd-api |
| 1 | GET /1.0/images/aliases/ALIAS?project=NAME | an alias of that remote | lxd-api |
| 1 | GET /1.0/images/FINGERPRINT?project=NAME | one image | lxd-api |
| 1 | GET /1.0/images/FINGERPRINT/export?project=NAME | the image's files, as LXD 6.10 and later download them | lxd-api |
| 2 | anything else | the same proxy engines as the main port, from row 3 on | as on the main port |
Under /1.0, other paths get LXD's 404 and other methods a 405. The setup page, the cache page and the certificate download are on the main port only. The lxc remote add commands that pkgproxyctl setup lxd-remote prints use https://HOST:3143/lxd/NAME with simplestreams.
| Socket | Snap | Without the snap | Serves | Used by |
|---|---|---|---|---|
| stats | /var/snap/pkgproxy/common/stats.sock | /run/pkgproxy/stats.sock | GET /pkgproxy/v1/stats, GET /pkgproxy/v1/info, GET /pkgproxy/v1/upstreams and its /history and /events | pkgproxyctl stats, upstreams, open and setup |
| admin | /var/snap/pkgproxy/common/admin.sock | /run/pkgproxy/admin.sock | DELETE /pkgproxy/v1/cache with the apt URL in a JSON body, GET and PUT /pkgproxy/v1/offline | pkgproxyctl cache invalidate, pkgproxyctl offline, the snap's configure hook |
| content, main | /var/snap/pkgproxy/common/sockets/pkgproxy.sock | where -content-socket-path points | the main port's table, without the cache page's rows, the health route included | another snap, through the sockets content interface |
| content, HTTPS | /var/snap/pkgproxy/common/sockets/pkgproxy-lxd-tls.sock | where -lxd-tls-content-socket-path points | the HTTPS port's table | the same |
-socket-path and -lxd-tls-socket-path replace it.0777: anyone on the machine may connect, so anyone may switch offline mode or drop an apt file. The stats socket only reads.See Statistics for the stats API, pkgproxyctl for the commands and Settings for the flags.
pkgproxy serves LXD's simplestreams image servers over its own HTTPS port. It rechecks the catalogs and keeps every image file it has checked.
lxc remote add NAME https://host:3143/lxd/NAME --protocol=simplestreams.
lxc refuses a simplestreams remote that is not https://, so
pkgproxy has a TLS listener, -lxd-tls-addr, with a
self-signed certificate kept under lxd/tls in the cache
directory.
--accept-certificate is not read for simplestreams remotes.
pkgproxyctl setup lxd-remote | sh adds each remote and
fetches /pkgproxy-lxd-cert.pem into lxc's
servercerts/NAME.crt (under ~/snap/lxd/common/config for the LXD snap, else ~/.config/lxc). It refuses sudo, which
would configure root's lxc instead.
The defaults: pkgproxy-images, pkgproxy-ubuntu,
pkgproxy-ubuntu-daily, and craft-providers'
craft-com.ubuntu.cloud-buildd and
craft-com.ubuntu.cloud-buildd-daily, which setup converges
onto pkgproxy and -undo restores.
streams/v1/index.json and the catalogs revalidate after
-lxd-revalidate-ttl (default: -revalidate-ttl, 5m). Image files are allowlisted by the
catalogs, SHA-256 verified and cached forever; the allowlist is saved in
lxd-index.json every -lxd-checkpoint-interval
(30s) and on shutdown.
On LXD 6.10 setup lxd-remote adds an lxd image
registry per remote over one public cluster link, which pins pkgproxy's
certificate. pkgproxy serves LXD's image API, read-only, on the HTTPS
port. Registries belong to the LXD server, so they serve every user of it.
-lxd-remote name=url (repeatable; name= drops a
default; off), -lxd-tls-addr (:3143),
-lxd-base-path (/lxd),
-lxd-revalidate-ttl (default: -revalidate-ttl, 5m), -lxd-checkpoint-interval
(30s).
Next: Build containers, for what runs inside them; pkgproxy for CI, for many machines behind one cache.
Next: Build containers, for what runs inside them; Fetch once, for one shared image download; Offline mode, for launching with no network.
With no -lxd-remote at all, these five are served. The address lxc uses is https://HOST:3143/lxd/NAME.
| Remote | Real image server | Launch with |
|---|---|---|
pkgproxy-images | https://images.lxd.canonical.com | lxc launch pkgproxy-images:debian/12 |
pkgproxy-ubuntu | https://cloud-images.ubuntu.com/releases | lxc launch pkgproxy-ubuntu:24.04 |
pkgproxy-ubuntu-daily | https://cloud-images.ubuntu.com/daily | lxc launch pkgproxy-ubuntu-daily:24.04 |
craft-com.ubuntu.cloud-buildd | https://cloud-images.ubuntu.com/buildd/releases | used by snapcraft, rockcraft and charmcraft |
craft-com.ubuntu.cloud-buildd-daily | https://cloud-images.ubuntu.com/buildd/daily | used by snapcraft, rockcraft and charmcraft |
-lxd-remote NAME=URL adds a remote or replaces a default of that name, -lxd-remote NAME= drops a default, and -lxd-remote off turns the LXD proxy engine off. The option repeats, once per remote. A name may not be empty, . or .., or hold /, :, ,, ?, #, %, white space or a control character.
Paths are relative to a remote, so /lxd/pkgproxy-ubuntu/streams/v1/index.json is fetched from https://cloud-images.ubuntu.com/releases/streams/v1/index.json. Only GET and HEAD are answered; anything else is a 405.
Path under /lxd/NAME/ | What it is | Cache policy |
|---|---|---|
streams/v1/index.json | the index of the remote's catalogs | revalidated after -lxd-revalidate-ttl; not verified, as nothing lists its hash |
any other path ending in .json | a product catalog, such as streams/v1/images.json | revalidated after -lxd-revalidate-ttl; read for the SHA-256 of every file it lists |
| a file a catalog listed | an image file: metadata tarball, squashfs root, VM disk | fetched once, checked against the catalog's SHA-256, kept forever |
| a file no catalog listed | — | 404, never fetched |
| a remote that is not configured, or no path after the name | — | 404 |
a path that climbs out with .. | — | 400 |
A failed recheck of a catalog serves the cached copy, as in the other proxy engines. A HEAD for an image file that is not cached yet is answered from the catalog, with its size, without fetching it. An image file that fails its hash check is not cached, but the clients already receiving it get the upstream's bytes unchanged, for at most -fill-drain-timeout; lxc checks the image itself. A client's Authorization header is never sent upstream, so private image servers are not supported.
| Path | Main port, 3142 | HTTPS port, 3143 |
|---|---|---|
/lxd/NAME/… | yes, over plain HTTP, for curl and scripts | yes; the only form lxc accepts |
/pkgproxy-lxd-cert.pem | yes: the certificate, to pin | no |
/, /1.0, /1.0/images/… | no | yes: LXD's image API, for LXD 6.10 |
LXD 6.10 downloads images through an image registry, which asks the HTTPS port this read-only part of LXD's own API. ?project=NAME names the remote; the answers come from the same cached catalogs and image files.
| Request | Answer |
|---|---|
GET /, GET /1.0 | a public server with no API extensions |
GET /1.0/images, with or without ?recursion=1 | the remote's images for the requester's architecture |
GET /1.0/images/aliases/ALIAS | the image the alias names: the container image, or the virtual machine image when there is no container image |
GET /1.0/images/FINGERPRINT | the image, by its fingerprint or a unique prefix of it |
GET /1.0/images/FINGERPRINT/export | its files: one for a unified image, or metadata and then rootfs or rootfs.img as a multipart answer |
The requester's architecture comes from LXD's User-Agent. These requests appear in the access log with handler=lxd-api.
Under the cache directory, /var/snap/pkgproxy/common/cache in the snap and /var/cache/pkgproxy otherwise:
| Path | What it holds |
|---|---|
lxd/objects/NAME/… | the cached catalogs and image files, one directory per remote, each file with its metadata in an extended attribute |
lxd/tls/cert.pem | the self-signed certificate of the HTTPS port: ECDSA P-256, valid for ten years |
lxd/tls/key.pem | its private key, readable by its owner only |
lxd-index.json | the SHA-256 of every image file the catalogs listed, saved every -lxd-checkpoint-interval and on shutdown |
Keep lxd/tls when moving or clearing the cache: a new certificate breaks every remote pinned to the old one until pkgproxyctl setup lxd-remote runs again. Without lxd-index.json, a request for an image file whose catalog has not been read since the restart gets a 404.
| Option | Environment variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-lxd-remote | PKGPROXY_LXD_REMOTE | flag or environment only | the five above | remotes to serve, as NAME=URL; off serves none |
-lxd-base-path | PKGPROXY_LXD_BASE_PATH | flag or environment only | /lxd | the URL path the remotes are served under |
-lxd-tls-addr | PKGPROXY_LXD_TLS_ADDR | lxd-tls-addr | :3143 | address of the HTTPS port, or off |
-lxd-tls-socket-path | PKGPROXY_LXD_TLS_SOCKET_PATH | flag or environment only | none | serve the HTTPS port on this UNIX socket instead; setup scripts still name -lxd-tls-addr |
-lxd-tls-content-socket-path | PKGPROXY_LXD_TLS_CONTENT_SOCKET_PATH | flag or environment only | none; the snap sets /var/snap/pkgproxy/common/sockets/pkgproxy-lxd-tls.sock | also serve the HTTPS port on this UNIX socket, to share it with another snap |
-lxd-revalidate-ttl | PKGPROXY_LXD_REVALIDATE_TTL | lxd-revalidate-ttl | -revalidate-ttl, 5m | how long index.json and catalogs are served before they are checked again |
-lxd-checkpoint-interval | PKGPROXY_LXD_CHECKPOINT_INTERVAL | lxd-checkpoint-interval | 30s | how often the learned image hashes are saved |
With -lxd-tls-addr off, pkgproxyctl setup lxd-remote prints no script, since lxc has no HTTPS address to use. All settings are in Settings; what the setup script changes is in Setting up clients.
The three offline modes, what works and what fails without the internet, the breakers that leave a troubled upstream alone, and how the mode is switched.
-offline-mode is store-overloaded (default),
never or offline. -offline
is short for -offline-mode offline; given both, they must
agree.
Hits are served with every TTL frozen. Upstream requests and
CONNECT dials are refused before they start. A miss is a
503 (pkgproxy is offline, no
Retry-After); snapd gets a 424.
Never-cached content fails offline: HTTPS apt sources, new modules and
images, unseen snap searches. A dists/** entry the index knows
is superseded is refused online or off. Snaps seen online install and
refresh; a new device session is answered with placeholders (see Snap
Store relay).
One breaker per host: open after 3 consecutive failures, or at once on
429/503 with Retry-After; 30 s,
doubled up to 10 min; then one probe. A 500 or
501 is not a failure.
The store's API host has a breaker per endpoint class: find,
info, refresh, assertions,
catalog, auth, download, and the
host's own for the rest. Other hosts, the CDN included, keep one. The
status reads upstream api.snapcraft.io find: left alone after
trouble, ….
pkgproxyctl offline (status), on,
off and mode MODE, or snap set pkgproxy
offline-mode=…: no restart either way. In the snap the switch is
saved as the option.
-auto-offline is network (default on Linux),
airplane-mode or off (the only value elsewhere).
Changes settle for 5 s; offline-mode is not changed.
Next: pkgproxy for CI, for builds that see only the cache; Immutable or
not, for what offline freezes; pkgproxyctl, for the offline
command.
Next: Immutable or not, for what offline freezes;
pkgproxyctl, for the offline command; Snap Store relay, for
snaps without a network and what still needs the store.
offline-mode | Upstreams | Windows | What the cache cannot answer |
|---|---|---|---|
store-overloaded (default) | contacted, except a host whose breaker is open | as configured | from a host left alone: 503 with Retry-After, the time until its next try; snapd gets 424 |
never | contacted as needed; one that cannot be reached is waited on until the connection gives up | as configured | fetched |
offline | never contacted, no DNS lookup or connection | endless: every cached entry is a hit | 503 without Retry-After; snapd gets 424 |
The breaker of store-overloaded, one per upstream host, and one per endpoint class on the snap store's API host (below): it opens after 3 failures in a row (a transport error, a timeout, 429, 502, 503, 504), or at once on a 429 or 503 with Retry-After (honoured up to 10 min). It stays open 30 s, doubling each time it opens again, up to 10 min; then one probe request decides. A 500 or 501 is not a failure. The default was never up to pkgproxy 0.12.0; a mode chosen with snap set or pkgproxyctl offline is kept across the change.
The store's API host (api.snapcraft.io, or the host of the store URL) has a breaker for each class of its endpoints, whatever the method, so the part that fails is left alone and the rest still answers. Every other host, the store's download servers included, has one breaker.
| Class | Paths below the store URL |
|---|---|
find | /v2/snaps/find, /api/v1/snaps/search |
info | /v2/snaps/info/…, /api/v1/snaps/details/… |
refresh | /v2/snaps/refresh |
assertions | /v2/assertions/…, /api/v1/snaps/assertions/… |
catalog | /api/v1/snaps/names, /api/v1/snaps/sections, /v2/snaps/categories, /v2/snaps/category/… |
auth | /api/v1/snaps/auth/nonces, /api/v1/snaps/auth/sessions |
download | /api/v1/snaps/download/…, /api/v1/snaps/dm-verity/download/…, /api/v1/snaps/resources/…, /api/v1/resources/download/…, /api/v1/media/download/… |
| (none: the host's own) | everything else, and every tunnel to the host; a tunnel is refused while this breaker is open, but a tunnel that fails does not count towards opening it |
A class is named after its host, as in api.snapcraft.io find, in pkgproxyctl offline status, on the setup page and in the log line of each opening and recovery. Before pkgproxy 0.12.2 the host had one breaker, and a refused search or catalogue left the whole store alone, device sessions included.
| Proxy engine | Still served | Refused |
|---|---|---|
| apt | cached pool/ files and dists/ indexes, as they were | anything not cached; an index a newer Release shows to be superseded; https:// sources, whose tunnels are refused |
| Snap Store | cached snap info, assertions and snap files; install, download and refresh of snaps that were answered online; searches made online | searches never made online and actions never answered online, as 424. A device with a serial but no store session gets placeholder nonces and sessions from pkgproxy (see Snap Store relay) |
| Go modules | cached .mod, .zip, list, @latest and .info; the "not found" answers the go command gets while it looks for the module that holds a package, so go install example.com/tool@latest works for a tool installed through pkgproxy before | modules and versions never fetched |
| Docker Hub | tags resolved before, cached manifests and layers | images never pulled |
| LXD images | cached catalogs and image files, through lxc and through LXD's image API | images never fetched |
| Files by checksum | files already fetched | files never fetched |
| Client | Status | Body |
|---|---|---|
| apt, go, Docker, LXD, curl | 503 | pkgproxy is offline: the content is not in the cache |
the same, in store-overloaded | 503, Retry-After | pkgproxy is not contacting the upstream for a while, as it looked overloaded: the content is not in the cache; for a part of the snap store, pkgproxy is not contacting api.snapcraft.io (find) for a while, … |
| snapd | 424 | a store error list: pkgproxy is offline and the content is not in its cache |
snapd, in store-overloaded | 424 | pkgproxy is giving the store a rest, as it looked overloaded, and the content is not in its cache; for one part of it, pkgproxy is giving the store's find endpoints a rest, as they looked overloaded, and the content is not in its cache |
The bodies are in English whatever the daemon's language. snapd does not retry a 424, where it would retry a 5xx for half a minute.
auto-offline | pkgproxy is offline while | Reported reason |
|---|---|---|
network (default on Linux) | the machine has no usable default route | no-network |
airplane-mode | the machine has at least one radio and every radio is off | airplane-mode |
off (the only value outside Linux) | never by itself | none |
A change counts once it has held for 5 s. The configured offline-mode is kept and applies again once the machine is back. A switch by hand wins until the machine next goes offline. With an upstream-pkgproxy (see Two tiers), network follows the route to the upstream's address instead of the default route, so a pkgproxy whose upstream is on its local network, with no way to the internet, stays online.
| Command | Does |
|---|---|
pkgproxyctl offline or pkgproxyctl offline status | shows the mode in effect, why if automatic, and each upstream being left alone with its next try |
pkgproxyctl offline on | mode offline |
pkgproxyctl offline off | mode never (not the mode before, nor the default store-overloaded) |
pkgproxyctl offline mode MODE | mode never, store-overloaded or offline |
pkgproxyctl offline mode -keep-automatic MODE | sets the mode for when the machine is back, staying offline while it is not; the snap's configure hook uses it |
sudo snap set pkgproxy offline-mode=MODE | the same setting, without a restart |
sudo snap set pkgproxy auto-offline=off | stop going offline with the machine |
Each command takes -socket-path, the admin socket. In the snap a switch is saved as the offline-mode option and survives a restart; outside it a switch lasts until the daemon restarts, and pkgproxyctl says so. Any change of mode forgets the breakers, the hosts whose recheck failed, and the snap store entries whose revalidation is backing off.
| Setting | Flag | Variable | Snap option | Default |
|---|---|---|---|---|
| start mode | -offline-mode | PKGPROXY_OFFLINE_MODE | offline-mode | store-overloaded |
short for offline | -offline | PKGPROXY_OFFLINE | flag or environment only | off |
| going offline with the machine | -auto-offline | PKGPROXY_AUTO_OFFLINE | auto-offline | network on Linux, else off |
Given both -offline and -offline-mode, they must name the same mode, or pkgproxy refuses to start.
GET and PUT of /pkgproxy/v1/offline on the admin socket read and set the mode; pkgproxyctl offline is its front end. The stats socket's /pkgproxy/v1/info reports the same, read-only.
| Field | Meaning |
|---|---|
mode | the mode in effect: never, store-overloaded or offline |
configured_mode | while offline automatically: the mode that applies again |
automatic | while offline automatically: no-network or airplane-mode |
auto_offline | the auto-offline setting |
keep_automatic | in a PUT: keep an automatic override |
upstreams | breakers that are not closed, sorted by host, then class: host, class (left out for a host's own breaker), state (open or half-open), retry_in_seconds |
restart_mode | the mode after a restart |
persistent | whether a switch survives a restart |
persist_error | from a PUT: why the mode could not be saved |
One request through pkgproxy, from the client to the cache on disk and back.
Five kinds of client share one daemon and one port, 3142; LXD
image servers also need the HTTPS port 3143. Each download is
fetched upstream once and served from disk after that.
apt is recognized by its absolute request line. The others are told apart
by path: /go, /lxd, /files, Docker's
/v2/…/manifests|blobs|referrers|tags/list,
/v2/_catalog and /v2/, and the rest of
/v2 for snapd. Anything else is answered with 404.
The key names the archive and the path inside it. The store has one directory per kind of download, and a write becomes visible only on commit: a temporary file renamed into place.
Concurrent requests for one key share one upstream fetch, a fill. It runs detached from the clients, streams to them while it writes, and is committed only after the content checks out.
Immutable content is cached forever without revalidation. Mutable indexes are served for a short TTL and then revalidated with a conditional GET. Nothing is ever evicted.
Next: One listener, for the ports and sockets; The store, for the cache on disk; Settings, for the flags and snap options.
Next: pkgproxy for CI, for many machines behind one cache; Fetch once, for one download shared by many jobs; Offline mode, for builds that see only what was cached.
Next: pkgproxyctl setup, for pointing your machines at it;
Fetch once, for one download shared by every client; Immutable or not,
for what is kept and what is rechecked.
pkgproxyctl is the pkgproxy binary under a second name, with a few commands that reach the daemon over two sockets.
pkgproxyctl is a symlink to pkgproxy, which
dispatches on the name it was invoked as. In the snap, if the shell cannot
find pkgproxyctl, use pkgproxy.pkgproxyctl.
Without the snap, every release from 0.12.0 has the binary for Linux on amd64
and arm64 on the releases page.
pkgproxyctl -h lists the seven commands; pkgproxyctl
setup -h and pkgproxyctl offline -h list theirs.
stats.sock serves stats, upstreams, version,
open and setup; admin.sock serves
cache and offline. Each command's
-socket-path points it elsewhere.
Usage: … [OPTIONS] MODE, a description, then
Available options. A stray argument is an error, not
something to guess around.
Next: pkgproxyctl setup, to set up clients; Statistics, for what pkgproxyctl stats reports; Offline mode, for what pkgproxyctl offline switches.
| Command | Arguments | Options | Socket | What it does |
|---|---|---|---|---|
stats | none | -json, -limit N (default 20), -r WHEN, -socket-path | stats | shows what is cached and how much downloading it saved |
upstreams | none | -json, -socket-path | stats | shows how each upstream server, and each part of the snap store, answers now; see Upstream status |
setup TOOL | none | see below | stats, to read -addr | prints how to point a tool at pkgproxy: a guide on a terminal, a script for sh when piped |
cache invalidate | URL | -socket-path | admin | throws away the cached copy of one apt file, or a kept volatile file |
offline | none | -socket-path | admin | the same as offline status |
offline status | none | -socket-path | admin | shows the mode and each upstream left alone for now |
offline on | none | -socket-path | admin | serves only from the cache: the mode offline |
offline off | none | -socket-path | admin | contacts upstreams as needed: the mode never |
offline mode | never, store-overloaded or offline | -keep-automatic, -socket-path | admin | sets the mode |
open | none | -socket-path | stats | opens the setup page in the desktop's browser |
version | none | -socket-path | stats | shows the version of pkgproxyctl and of the running daemon, and in the snap the snap's version and revision |
-h at any level lists what is below it; pkgproxyctl setup -h lists the tools.stats and upstreams start with the status banner's line while pkgproxy is degraded or offline.-r takes a Unix timestamp, an RFC 3339 time or how long ago, such as 30m or 2h.-keep-automatic sets the mode that applies once the machine is online again, without ending an automatic offline; the snap's configure hook uses it.-socket-path defaults to /var/snap/pkgproxy/common/stats.sock or admin.sock in the snap, and to /run/pkgproxy/stats.sock or admin.sock otherwise.cache invalidate takes the http:// URL apt fetches the file from, with dists/ or pool/ in its path, such as http://deb.debian.org/debian/dists/bookworm/InRelease, or the URL of a volatile file as clients name it. Other kinds of download cannot be thrown away one by one.version says so when the running daemon is another build than pkgproxyctl, and how to restart it.pkgproxy.open and the applications menu entry run pkgproxyctl open. If the shell cannot find pkgproxyctl, use pkgproxy.pkgproxyctl.| Tool | Sets up | Run the script as | Options besides -addr and -undo |
|---|---|---|---|
apt | apt's proxy | root | none |
snap | snapd's store, through a systemd drop-in; restarts snapd | root | none |
go | GOPROXY for the user | the user | -go-base-path (also -base-path), default /go |
docker | Docker Hub mirror; restarts Docker | root | none |
lxd | a pkgproxy profile for LXD containers, or a project's default profile | the user, in the lxd group | -project NAME |
lxd-remote | lxc image remotes through the HTTPS port, and their certificate | the user who runs lxc, not with sudo | -lxd-tls-addr, -lxd-base-path (also -base-path), -lxd-remote |
snapcraft | the snapcraft LXD project's build containers | the user, in the lxd group | none |
rockcraft | the rockcraft LXD project's build containers | the user, in the lxd group | none |
charmcraft | the charmcraft LXD project's build containers | the user, in the lxd group | none |
status | nothing: prints a script that reports what is set up | the user | -lxd-tls-addr, -go-base-path, -lxd-base-path, -lxd-remote; no -undo |
-addr and -lxd-tls-addr are read from the running daemon when not given; -go-base-path, -lxd-base-path and -lxd-remote are not, and default to the daemon's defaults.pkgproxyctl setup apt | sudo sh, or | sh for the tools run as the user. -undo prints the script that removes the setup.exit 1 to the pipe, so the pipe fails.setup status reports, for each other tool: set up, not set up, set up for another address, needs setup again, not installed, or cannot check.See Setting up clients for what each script writes.
cannot … error, with exit status 1. Asking for help exits with status 64.The pkgproxy image: what is in it, running it with a cache volume, its settings, and its health check and stop.
A base: bare rock for amd64: /usr/bin/pkgproxy,
pkgproxyctl as a symlink, CA certificates and Pebble, no
shell. It runs as UID and GID 584792;
/var/cache/pkgproxy is owned by it.
docker volume create pkgproxy-cache, then docker run
-v pkgproxy-cache:/var/cache/pkgproxy -p 3142:3142 -p 3143:3143.
A bind mount needs chown 584792:584792, or podman's
:U.
-e PKGPROXY_PUBLIC_URL=…, or --args pkgproxy
-addr=:8080; a flag wins. docker logs pkgproxy shows
the log; docker exec pkgproxy pkgproxyctl stats needs no
flags.
The health check asks
http://localhost:3142/pkgproxy/v1/health. pkgproxy exiting
ends the container. Pebble's kill-delay is 15s; use
docker stop -t 20 pkgproxy.
Next: pkgproxyctl setup, to point your tools at it; Settings, for every option; On Kubernetes, for the image under Juju.
| Name | registry.gitlab.com/zygoon/pkgproxy |
| Tags | edge follows the main branch; each release has its own tag, such as v0.13.0 |
| Architecture | amd64 (the published image) |
| Base | bare: the static binary, CA certificates and Pebble, no shell |
| Entrypoint | Pebble, which starts the pkgproxy service |
| User | UID and GID 584792, for Pebble and pkgproxy |
| Version | git describe with - for +: v0.12.1, or v0.12.1-git5.abc1234 after a tag |
| Path | What it holds |
|---|---|
/usr/bin/pkgproxy | the daemon |
/usr/bin/pkgproxyctl | a symlink to the same binary |
/var/cache/pkgproxy | the cache; mount a volume here. Owned by 584792 |
/run/pkgproxy | the stats and admin sockets, made again at every start; needs no volume |
The image declares no volume and no port. A new named docker or podman volume is seeded from the image, owner included. A host directory must be owned by 584792 (chown 584792:584792), or mounted with podman's :U. The cache needs extended attributes; named volumes and most host directories have them.
| Port | For |
|---|---|
3142 | every client but LXD, the setup page and the health route |
3143 | LXD's image server, over TLS |
An option comes from, strongest first:
1. A flag after --args pkgproxy: docker run IMAGE --args pkgproxy -addr=:8080. 2. The container's environment: docker run -e PKGPROXY_NAME=value, or a pod's env. 3. The daemon's default.
The image sets no PKGPROXY_* variable; its only variable is PEBBLE_VERBOSE=1, so that docker logs shows pkgproxy's log. See Settings for every option.
| Health check | health, level ready: GET http://localhost:3142/pkgproxy/v1/health. It fails when -addr names another port |
| When pkgproxy exits | Pebble shuts down and the container ends; the runtime's restart policy decides what comes next |
| Kill delay | 15s, above pkgproxy's 10s shutdown grace |
| Stopping | docker stop -t 20 pkgproxy; docker waits only 10s by default. Kubernetes gives a pod 30s |
docker volume create pkgproxy-cache docker run -d --name pkgproxy --restart unless-stopped \ -v pkgproxy-cache:/var/cache/pkgproxy \ -p 3142:3142 -p 3143:3143 \ registry.gitlab.com/zygoon/pkgproxy:edge docker logs pkgproxy docker exec pkgproxy pkgproxyctl stats docker exec pkgproxy pkgproxyctl setup apt -addr cache.example:3142 docker stop -t 20 pkgproxy
The setup scripts name the address given with -addr: the one your machines reach the container on.
One pkgproxyctl setup command per client, printed as a guide or piped as a script, safe to run again and to undo.
On a terminal pkgproxyctl setup apt prints a guide; piped, it
prints a script. Either way apt ends up with an
Acquire::http::Proxy line in
/etc/apt/apt.conf.d/99pkgproxy.
Each piped script is one { … } block. It runs only where its
tool is, warns unless it runs as root, and chains its steps with
&& so that a failed step fails the script.
apt, snap and docker change system
files and need root. go, lxd,
lxd-remote, snapcraft, rockcraft and
charmcraft change your own settings and run as you.
-addr defaults to what /pkgproxy/v1/info reports,
else :3142. A wildcard host becomes the host name. A bad
address prints the error on stderr and exit 1 to the pipe.
pkgproxyctl setup X -undo | sh reverses setup X
(with sudo sh for apt, snap and docker). Both can be run
again safely.
pkgproxyctl setup status | sh reports configuration only, one
line per setup command. It runs as you, without sudo.
Next: Build containers, for what setup lxd and the craft tools set up; LXD images, for the remotes setup lxd-remote adds; Statistics, to check the builds use the cache.
Next: The setup page, for machines without pkgproxy; Statistics, to check that a client uses the cache; pkgproxyctl, for the other commands.
Every script but Go's runs only where its tool is; elsewhere it says there is nothing to do and succeeds. Go's needs no go command and sets up Go before it is installed. <host> is the address the setup names, as in pkgproxy:3142.
| Command | Runs as | What it changes |
|---|---|---|
setup apt | root (warns otherwise) | Writes /etc/apt/apt.conf.d/99pkgproxy with Acquire::http::Proxy "http://<host>";. Where apt is older than 3.1.3 it adds Acquire::Retries::Delay "false";, so apt does not hang while pkgproxy is offline. |
setup snap | root (warns otherwise) | Writes /etc/systemd/system/snapd.service.d/pkgproxy.conf setting SNAPPY_FORCE_API_URL=http://<host>, then systemctl daemon-reload and, where snapd is installed, restarts it. Needs systemd. |
setup docker | root (fails otherwise) | Puts http://<host> first in registry-mirrors in /var/snap/docker/current/config/daemon.json (Docker snap) or /etc/docker/daemon.json, creating the file if needed and keeping other settings and mirrors, then restarts Docker. A file it cannot understand is left alone, with an error. |
setup go | you | Writes GOPROXY=http://<host>/go,direct to your own Go environment file, where go env -w would, so it works before Go is installed. |
setup lxd | you, in the lxd group | Creates or replaces the LXD profile pkgproxy: environment.GOPROXY, environment.SNAPPY_FORCE_API_URL and two container hooks (below). Add the profile to containers yourself. |
setup lxd -project P | you, in the lxd group | The same settings in the default profile of LXD project P, as the craft setups do. |
setup snapcraft, setup rockcraft, setup charmcraft | you, in the lxd group | setup lxd for the LXD project of the same name: creates the project if missing (copying the default profile of the default project), deletes its base-instance containers so the next build starts from a fresh base, and sets the three settings on the project's default profile. |
setup lxd-remote | you; refuses to run under sudo | For each image remote, lxc remote add <name> https://<host-name>:3143/lxd/<name> --protocol=simplestreams, with pkgproxy's certificate from http://<host>/pkgproxy-lxd-cert.pem saved as servercerts/<name>.crt in your lxc configuration directory (~/snap/lxd/common/config or ~/.config/lxc). A remote of the same name at another address is replaced. On an LXD with image registries it also adds a cluster link named pkgproxy-… and an image registry per remote. |
setup status | you | Changes nothing: prints a report (see below). |
The profile's hooks run on the LXD host, around each container that uses it, so nothing needs installing in the image.
| Hook | What it does in the container's root file system |
|---|---|
| before start | Writes /etc/apt/apt.conf.d/99pkgproxy-lxd (Acquire::http::Proxy and Acquire::Retries::Delay "false") and /etc/systemd/system/snapd.service.d/pkgproxy-lxd.conf (SNAPPY_FORCE_API_URL). Rewrites https:// to http:// for the archive, security, ports and old-releases hosts of ubuntu.com in /etc/apt/sources.list and /etc/apt/sources.list.d/*.sources. |
| after stop | Removes the two files it wrote. |
Containers reach pkgproxy at the address the setup names, or, when that address is this machine itself (a loopback address), at the address of the LXD bridge the project's default profile uses.
| Command | -undo |
|---|---|
setup apt | Removes /etc/apt/apt.conf.d/99pkgproxy. |
setup snap | Removes the snapd drop-in, reloads systemd and restarts snapd. |
setup docker | Removes only pkgproxy's entry from registry-mirrors, and the key when no mirror is left, then restarts Docker. |
setup go | Removes the GOPROXY line from your Go environment file, as go env -u GOPROXY does, if it has one. |
setup lxd | Deletes the pkgproxy profile. |
setup lxd -project P and the craft setups | Unsets environment.GOPROXY, environment.SNAPPY_FORCE_API_URL and raw.lxc on the project's default profile. The project is kept. |
setup lxd-remote | Removes each remote that points at pkgproxy and its certificate, and puts craft-com.ubuntu.cloud-buildd and craft-com.ubuntu.cloud-buildd-daily back at https://cloud-images.ubuntu.com/buildd/releases and …/buildd/daily. Removes pkgproxy's image registries, the images LXD downloaded through them, and its cluster links. |
| Option | Commands | Meaning |
|---|---|---|
-addr | all | pkgproxy's listen address. Taken from the running daemon when not given, else :3142. A wildcard host becomes this machine's host name. |
-undo | all but status | Print the commands that reverse the setup. |
-go-base-path, -base-path | go, status (-go-base-path only) | pkgproxy's Go base path, /go by default; not taken from the daemon. |
-project | lxd | LXD project whose default profile to change, instead of a pkgproxy profile. |
-lxd-base-path, -base-path | lxd-remote, status (-lxd-base-path only) | pkgproxy's LXD base path, /lxd by default; not taken from the daemon. |
-lxd-remote | lxd-remote, status | The image remotes, as given to the daemon; not taken from it. By default craft-com.ubuntu.cloud-buildd, craft-com.ubuntu.cloud-buildd-daily, pkgproxy-images, pkgproxy-ubuntu and pkgproxy-ubuntu-daily. |
-lxd-tls-addr | lxd-remote, status | pkgproxy's HTTPS address for LXD images. Taken from the daemon when not given, else :3143. |
pkgproxyctl setup status | sh prints one line per setup command with one of these states, and notes under it.
| State | Meaning |
|---|---|
| set up | Set up for the pkgproxy at -addr. |
| not set up | Nothing of this setup is there. |
| set up for another address | Set up, but for another pkgproxy address. |
| needs setup again | There, but it cannot work as it is, such as an LXD remote at an old address; run that setup again. |
| not installed | The tool is not on this machine. |
| cannot check | The script could not tell, with the reason in a note. |
pkgproxyctl is the pkgproxy binary under another name. The snap installs both. Without the snap, every release from 0.12.0 on publishes the binary for Linux on amd64 and arm64, with a SHA256SUMS file to check it against, on the releases page. To install the newest release, with arch set to the machine's architecture:
arch=amd64 url=https://gitlab.com/zygoon/pkgproxy/-/releases/permalink/latest/downloads curl -fL -O "$url/pkgproxy-linux-$arch" -O "$url/SHA256SUMS" sha256sum --check --ignore-missing SHA256SUMS sudo install -m 755 "pkgproxy-linux-$arch" /usr/local/bin/pkgproxy sudo ln -sf pkgproxy /usr/local/bin/pkgproxyctl
Or build it with Go: go install gitlab.com/zygoon/pkgproxy/cmd/pkgproxy@latest, then ln -s pkgproxy "$(go env GOPATH)/bin/pkgproxyctl".
A machine without pkgproxyctl gets the same scripts from the daemon's setup page (The setup page):
| URL | What it serves |
|---|---|
http://<host>:3142/setup/<tool> | #!/bin/sh and the lines pkgproxyctl setup <tool> prints when piped, as text/plain |
http://<host>:3142/setup/<tool>?undo=1 | The -undo script |
http://<host>:3142/setup/all | A script that runs each tool's script for the tools found on the machine, sudo only where root is needed; run it as yourself |
http://<host>:3142/setup/all?undo=1 | The same with ?undo=1, in the reverse order |
curl gets the script; a browser at /setup/<tool> gets that tool's page, with the command, the commands it runs and the undo.
The tools are apt, snap, go, docker (unless the Docker mirror is off), lxd, lxd-remote (only with image remotes and the HTTPS listener on), snapcraft, rockcraft and charmcraft; status is not offered. The scripts name the address the visitor reached pkgproxy at, or the public-url setting. Pipe those for apt, snap and docker to sudo sh, and the others to sh.
The page at the proxy's own address that sets up any machine, with a page per tool, the desktop entry, the status bar, and the cache and upstream lines.
curl -fsSL http://host:3142/setup/all | sh, run as yourself;
its undo is ?undo=1. Each tool's page is at
/setup/<tool>. GET / is HTML for a browser,
plain text for curl. -setup-page=false turns it off.
/setup/apt is #!/bin/sh plus what
pkgproxyctl setup apt | sh would run, as text/plain
for curl; a browser gets the apt page. Minimal Debian and Ubuntu images
have no curl: paste the commands instead.
-public-url http://host[:port] (snap option
public-url) beats the request's Host, which
beats -addr.
pkgproxy.open runs pkgproxyctl open: it reads
/pkgproxy/v1/info, opens http://localhost:3142/
with xdg-open, and on failure says why on stderr.
The status element is re-read from /pkgproxy/status every
30 seconds while the page is visible; offline shows a hollow dot, so it
reads without colour. In plain text, the setup page shows a notice
instead. The footer has a language switcher.
"How the upstream servers have answered, last 30 days"; "Other servers"
is shown only once three or more were asked. The lines link to
/pkgproxy/cache and /pkgproxy/upstreams; a request
from an untrusted network, or carrying Forwarded,
Via or X-Forwarded-*, gets a 404.
Next: The cache page, for what it shows; Offline mode, for what fails while offline; pkgproxyctl setup, for the same scripts from the command line.
Next: pkgproxyctl setup, for the same scripts on the proxy's machine; Offline mode, for what fails while it is offline.
| Address | A browser gets | curl gets |
|---|---|---|
/ | The setup page: one command for every tool, a tile per tool, the 30-day summary | Every tool's commands, as plain text |
/setup/all | The script | A script that runs, for each tool found on the machine, that tool's script; sudo only where root is needed. It goes on past a tool that fails and exits non-zero if any did. |
/setup/all?undo=1 | The script | The same, undoing each tool, in the reverse order |
/setup/<tool> | The tool's page: the command, the commands it runs, and the undo | The tool's script |
/setup/<tool>?undo=1 | The tool's page | The undo script |
/pkgproxy/status | The status element alone, never cached | The same |
The tools are those whose proxy engine is turned on: apt, snap, go, docker, lxd, lxd-remote, snapcraft, rockcraft and charmcraft. /setup/all checks for each with command -v (for docker, also /var/snap/docker/current), and prints pkgproxy: <tool> before each.
Every page has a language switcher in its footer. The page's language is, in order: ?lang=, the pkgproxy-lang cookie the switcher sets for a year, then the browser's Accept-Language.
snapd uses pkgproxy as its store over plain HTTP. The store's answers are kept, rechecked or only relayed, and the device-session handshake is answered with placeholders while the store's auth is not contacted.
pkgproxyctl setup snap | sudo sh writes a systemd drop-in
that sets SNAPPY_FORCE_API_URL and restarts snapd. snapd then
uses pkgproxy as its store over HTTP; pkgproxy relays to the real store over
HTTPS. No TLS is intercepted.
Every object with a url and a sha3-384 gets its
URL rewritten to /v2/download/sha3-384/<hex>; the real
URL is recorded under pending/. The host is each request's own,
cached answers included, or -snap-external-url.
A miss is one shared fill from the store's URL. The snap is committed as
sha3-384/<hex> only when it matches the hash, and is
never revalidated. A mismatch is not cached; snapd rejects it too.
snap-revision and sequence-pinned assertions, other than
account-key, are permanent.
Other assertions, info and find are revalidated
after the store's max-age, else after
-snap-revalidate-ttl (default: -revalidate-ttl, 5m).
POST /v2/snaps/refresh is relayed, and each action's result
recorded; offline, snapd's install is answered from a record whose snap
and assertion chain are cached. Auth, device sessions (while the store's auth is
contacted) and unknown paths are relayed uncached; private answers are never kept.
/api/v1/snaps/auth/nonces and …/sessions are
answered with placeholders, marked X-Pkgproxy-Synthesised. A
placeholder later gets 401 with
WWW-Authenticate: Macaroon refresh_device_session=1. In mode
never the store's failure is passed on.
Next: Immutable or not, for what is kept forever and what is
rechecked; Offline mode, for installs with no network; pkgproxyctl
setup, for the other clients.
snapd sends every store request to the address in SNAPPY_FORCE_API_URL, so the relay answers paths under /v2, /api/v1 and /v1 on port 3142, after the Docker registry's shapes (Docker Hub registry mirror). pkgproxy relays to https://api.snapcraft.io.
| Path | What pkgproxy does | Cache policy |
|---|---|---|
GET /v2/download/sha3-384/<hex> | Serves the snap (or delta) a rewritten link names, fetched once from the real URL recorded for the hash; Range included | kept forever, checked against the hash; a HEAD on a miss is answered from the record without fetching |
GET /v2/assertions/<type>/<key> | Serves an assertion | see the next table |
GET /v2/snaps/info/<name> | Relays, rewriting download links | revalidated |
GET /v2/snaps/find | Relays a search | revalidated; a search of private snaps is relayed uncached |
POST /v2/snaps/refresh | Relays installs, refreshes and downloads, rewriting download links | the answer is not cached; each action's result is recorded |
GET /v2/snaps/categories, /v2/snaps/category/<name>, /v2/repairs/… | Relays | revalidated |
GET /api/v1/snaps, /api/v1/snaps/sections, /api/v1/snaps/names, /api/v1/snaps/search, /api/v1/snaps/details/<name> | The older search API, relayed with download links rewritten | revalidated |
/api/v1/snaps/download/…, /api/v1/snaps/dm-verity/download/…, /api/v1/snaps/resources/…, /api/v1/resources/download/…, /api/v1/media/download/… | A snap's or its dm-verity data's download path that an answer to an anonymous request named next to its sha3-384 is served like the blob it names; any other is followed through the store's redirect to its CDN and streamed | the named ones kept forever with their blob; the rest not cached: there is no hash to key it by |
/api/v1/snaps/auth/…, /api/v1/snaps/metadata, /v1/…, POST /v2/cohorts | Relays untouched: logins, device sessions and serials; while the store's auth is not contacted, nonces and sessions are answered with placeholders (below) | not cached |
any other path under /v2 | Relays, rewriting download links | not cached |
| Assertion type | Without ?sequence=N | With ?sequence=N |
|---|---|---|
snap-revision | kept forever | kept forever |
account, snap-declaration, store, validation-set | revalidated | kept forever |
account-key | revalidated | revalidated |
| any other type | revalidated | revalidated |
Revalidated means served from the cache for the max-age the store announced for the entry, else for snap-revalidate-ttl, then asked for again. When the store cannot answer a revalidation (no connection, a 429 or a 5xx), the expired entry is served, with a warning in the log, and the store is not asked for it again for a while: the failure's Retry-After, or else the entry's time to live doubled for each failure in a row, up to an hour. Meanwhile the entry is served as stale with no warning. An answer, or a change of the offline mode, ends the wait.
In every relayed JSON answer, an object with both a url and a sha3-384 gets its url replaced by http://<address>/v2/download/sha3-384/<hex>, and assertion-stream-urls are re-based to the same address. The older API's download_url and anon_download_url are rewritten the same way. <address> is the one snapd reached pkgproxy at, or snap-external-url. The cache keeps the store's answer as it came, and every answer from it is rewritten for the machine asking, so a container on a bridge and the host on localhost each get their own address.
| Request | Served from the cache | Kept |
|---|---|---|
| Anonymous, or a device session on the public store | yes | yes |
| A logged-in user | yes | only answers whose snaps all say "private": false; the catalogue is relayed uncached |
| A device session on a brand store | no | nothing, and no refresh results are recorded |
While pkgproxy is not contacting the store (Offline mode), a POST /v2/snaps/refresh is answered from the recorded results, with an X-Pkgproxy-Synthesised header. An action is answered only when its snap file and every assertion that vouches for it are cached; otherwise it gets an error of its own, and the other actions still succeed. Actions with a cohort, ignore-validation or validation sets are never recorded or answered. An install is recorded under the channel asked and under the channel snapd then tracks, so an offline snap refresh of a snap installed from stable (tracked as latest/stable) is answered. Installs recorded before 0.12.2 are found once they are installed again online.
A refresh and a nonce request are sent again on a new connection when the store had closed the one they went out on, as its edge does after each 429; snapd no longer fails with cannot get nonce from store for that.
Before most store requests, snapd makes sure the device has a session. A device that has its serial but no session yet first sends POST /api/v1/snaps/auth/nonces, then POST /api/v1/snaps/auth/sessions. pkgproxy relays both and caches neither, as a nonce is used once and a session belongs to one device, except:
| When | pkgproxy |
|---|---|
Offline, or the store's auth part left alone in store-overloaded | Answers both itself with placeholders, pkgproxy-<random>-n-… and -s-…, marked X-Pkgproxy-Synthesised and logged synthesised |
store-overloaded, and the store fails the request (no connection, 429 or 5xx) | The same |
never | Passes the store's failure on |
| A request carries this pkgproxy's placeholder, and the store's auth answers again | Answers 401 with WWW-Authenticate: Macaroon refresh_device_session=1 without sending it, so snapd gets a real session |
| A request carries this pkgproxy's placeholder, the auth is still refused but the request's own part is not | Sends it without device authorization |
Cache hits are served whatever session they carry. A placeholder of another pkgproxy, or a real session, is passed on.
In store-overloaded, each part of the store has a breaker of its own (see Offline mode): a snap find the store answers with 429 leaves snap info, install, refresh and the device sessions working. Before pkgproxy 0.12.2 the store was one breaker, and a refused search closed off all of it. When pkgproxy does not contact the store, snapd's error names the part that is left alone: pkgproxy is giving the store's find endpoints a rest, as they looked overloaded, and the content is not in its cache. A relayed request pkgproxy does not send is logged at info level as not contacting the upstream: the offline mode refused the request, with mode, host and class, not as a failure of the store. The log line of an error the store answers names the store's backend that sent it as view=; an error without it came from in front of the store.
| Setting | Flag | Environment | Snap option | Default |
|---|---|---|---|---|
| Snap Store API to relay to | -snap-store-url | PKGPROXY_SNAP_STORE_URL | flag or environment only | https://api.snapcraft.io |
Address put in rewritten links, as http://host[:port] | -snap-external-url | PKGPROXY_SNAP_EXTERNAL_URL | flag or environment only | the address the client used |
How long details and assertions are trusted when the store announces no max-age | -snap-revalidate-ttl | PKGPROXY_SNAP_REVALIDATE_TTL | snap-revalidate-ttl | revalidate-ttl, 5m |
| How long clients already following a snap download that failed its hash check keep receiving it | -fill-drain-timeout | PKGPROXY_FILL_DRAIN_TIMEOUT | fill-drain-timeout | 10m |
Under the snap directory of the cache (/var/snap/pkgproxy/common/cache/snap in the snap, /var/cache/pkgproxy/snap otherwise), each object with its metadata in an extended attribute of its file:
snap/objects/ sha3-384/<hex> snaps and deltas pending/sha3-384/<hex> the real URL and size a hash stands for, kept after the snap is cached assertions/<type>/<key> assertions info/<name>… snap details, per query and selecting headers find/<hash> searches catalog/<hash> categories, repairs and the older search API refresh/… recorded install and refresh results store-path/<hash> the hash a store download path stands for
The random prefix of device-session placeholders is kept in snap-session-prefix at the top of the cache directory.
See The store.
pkgproxy counts what every cached file saved and reports it through pkgproxyctl stats and a read-only socket.
Per cache key: hits, fetches, the size, and
bytes-saved, what the hits sent. A joined request is a hit;
the one that started the fetch is a fetch.
Each entry has content: true or false, decided by its key
when the report is made. The totals add only content.
pkgproxyctl stats, -r 1h for the last hour,
-limit N (20 by default), -json for the
entries as JSON.
-stats-socket-path, by default
$SNAP_COMMON/stats.sock in the snap and
/run/pkgproxy/stats.sock otherwise. The reference is
docs/statsapi.md.
-stats-checkpoint-interval (snap option
stats-checkpoint-interval, default 30s) writes
<cache-dir>/stats.json. A crash loses at most one
interval.
Next: The cache page, for the counts as a live map; The access log, for one line per request; Immutable or not, for why metadata is rechecked.
Next: pkgproxy for CI, to measure one build; The cache page, to see a build's downloads live; The access log, for one line per request.
Next: Fetch once, for why a joined request is a hit; Immutable or not, for why metadata is rechecked; pkgproxyctl, for the other commands.
| Option | Default | Does |
|---|---|---|
-limit N | 20 | show at most N entries, those that saved the most |
-r WHEN | all time | only changes since WHEN: a Unix timestamp, an RFC 3339 time, or how long ago, such as 30m or 2h |
-json | off | print the entries as JSON instead of a report |
-socket-path PATH | the stats socket below | where to ask |
| How pkgproxy runs | Socket |
|---|---|
| the pkgproxy snap | /var/snap/pkgproxy/common/stats.sock |
| bundled in another snap | /var/snap/<snap>/common/stats.sock |
| on its own | /run/pkgproxy/stats.sock |
Moved with -stats-socket-path or PKGPROXY_STATS_SOCKET_PATH; it is not a snap option. Every local user can read it; nothing can write through it. Ask it with any HTTP client that speaks to a UNIX socket; the host name in the URL is ignored:
curl --unix-socket /run/pkgproxy/stats.sock 'http://localhost/pkgproxy/v1/stats?limit=10' curl --unix-socket /run/pkgproxy/stats.sock "http://localhost/pkgproxy/v1/stats?since=$(date -d '1 hour ago' +%s)" curl --unix-socket /run/pkgproxy/stats.sock http://localhost/pkgproxy/v1/info
| Path | Method | Answers |
|---|---|---|
/pkgproxy/v1/stats | GET, HEAD | the entries that saved the most, biggest first |
/pkgproxy/v1/info | GET, HEAD | the listen addresses, the version and the offline state |
| Status | When |
|---|---|
200 | a JSON answer |
400 | since is not a whole number |
404 | any other path, a trailing / included |
405 | any method but GET or HEAD |
Errors are plain text.
| Parameter | Default | Meaning |
|---|---|---|
limit=N | 200 | at most N entries; anything but a positive whole number means the default |
since=T | all time | count only from T, in Unix seconds; empty is the same as absent |
Without since the answer is an array of entries; with it, an object {"since": …, "entries": […]} whose since is the start really used, in the daemon's time zone. An empty answer is [] or "entries":[], never null. Only entries hit at least once appear.
| Field | Meaning |
|---|---|
type | apt, snap, gomod, lxd, docker or files |
key | what was downloaded; see below |
content | true for content, false for metadata |
size-bytes | its size |
hits | times served without going upstream, joined requests included |
fetches | times pkgproxy went upstream for it, to download or to recheck |
bytes-saved | what the hits sent: the whole file for a GET, the range asked for, nothing for a HEAD or a 304 |
type | key | Content | Metadata |
|---|---|---|---|
apt | archive and path, such as deb.debian.org/debian/pool/main/h/hello/hello_2.10-3_amd64.deb | under pool/, and dists/…/by-hash/… | every other file under dists/ |
gomod | module file, such as golang.org/x/text/@v/v0.3.0.zip | .zip, .mod | list, @latest, .info |
lxd | <remote>/<path>, such as pkgproxy-ubuntu/streams/v1/index.json | image files | anything ending in .json |
docker | <repository>@<digest>; a layer pulled through two repositories shows twice | every entry | none counted |
files | <file name>@<algorithm>:<hex>, the name being the URL's last part | every entry | none counted |
snap | the snap's name, one entry for all its revisions; "" for downloads not matched to a name | every entry | none counted |
For a snap, size-bytes is the revision served last, fetches counts every download of the snap, cached or not, and bytes-saved adds up all revisions. What the cache saved is the sum of bytes-saved over the entries whose content is true; the ranking puts both kinds together.
since within | Counts start at the beginning of its |
|---|---|
| the last hour | minute |
| the last 2 days | hour |
| the last 60 days | day |
| further back | moved forward to 60 days ago |
All-time counts are never trimmed.
| Field | Meaning |
|---|---|
addr | the client address, as given to -addr (:3142) |
lxd_tls_addr | the HTTPS address for LXD, as given to -lxd-tls-addr (:3143), or off |
setup_page | whether / serves the setup page; absent before 0.9.0, read as true |
version | the daemon's version, as pkgproxyctl version shows it |
offline.mode | never, store-overloaded or offline, read at the time of asking |
offline.automatic | while offline with the machine: no-network or airplane-mode |
offline.configured_mode | while offline with the machine: the mode that applies again |
offline.upstreams | upstreams being left alone, sorted by host, then class, each with host, class for one part of the snap store (find, info, refresh, assertions, catalog, auth or download; absent for a host's own breaker), state (open or half-open) and retry_in_seconds |
offline.since | when the mode in effect began (since 0.12.2) |
offline.degraded_since | while some upstream is being left alone, when that began (since 0.12.2) |
upstream_pkgproxy | with upstream-pkgproxy set: url, source (upstream, saved, baseline, newer, loop or unreachable) and proxy_engines, where each is fetched from: via is upstream, origin or refused; see Two tiers |
pkgproxyctl setup, open, stats and version read this endpoint to find the daemon's ports, its setup page, its offline mode and its build. See Offline mode.
The main port answers a shorter question for probes: GET /pkgproxy/v1/health gives status (always ok), mode and, while offline by itself, automatic, and names no upstreams; see One listener.
The stats socket also serves GET /pkgproxy/v1/upstreams, each upstream server and part of the snap store now, /pkgproxy/v1/upstreams/history and /pkgproxy/v1/upstreams/events. pkgproxyctl upstreams prints the first; see Upstream status. pkgproxyctl stats starts with the status banner's line while pkgproxy is degraded or offline.
| Setting | Flag | Snap option | Default |
|---|---|---|---|
| how often counts are saved | -stats-checkpoint-interval | stats-checkpoint-interval | 30s |
| where the socket is | -stats-socket-path | flag or environment only | see above |
The counts and their history are saved to stats.json in the cache directory, /var/snap/pkgproxy/common/cache/stats.json in the snap, at every interval and on a clean shutdown. A crash loses what changed since the last save.
Where the cache lives on disk, how a download is committed, and how to free space.
-cache-dir: /var/snap/pkgproxy/common/cache in
the snap, /var/cache/pkgproxy otherwise. Each kind has
objects/ and tmp/; stats.json,
lxd-index.json and lxd/tls sit beside them.
The body is objects/<key>; its metadata is JSON in
the file's user.pkgproxy.meta extended attribute, and in
<key>.meta.json beside it, for pkgproxy 0.11. pkgproxy
refuses to start on a file system without extended attributes (tmpfs
before Linux 6.6, NFSv3, FAT); copy the cache with cp -a,
rsync -aX or tar --xattrs.
sudo snap revert pkgproxy goes back to 0.11 with the cache
intact. After each refresh, the snap copies the .meta.json
files of what 0.11 cached into the attributes, and logs how many in
/var/snap/pkgproxy/current/migrate-cache.log; outside the
snap, pkgproxy -migrate-cache does the same, though nothing
needs it.
The metadata is set on the file in tmp/, which is flushed
and renamed into objects/: body and metadata switch
together. A 304 replaces only the
user.pkgproxy.meta attribute.
Nothing is evicted. Stop pkgproxy, remove one kind's directory, start it
again. After removing lxd, run pkgproxyctl setup
lxd-remote | sh; keep stats.json to keep the
statistics.
pkgproxyctl cache invalidate <url> sends
DELETE /pkgproxy/v1/cache over admin.sock. It
works for apt only; other kinds are verified before they are stored.
Next: Statistics, for what the cache holds and saved; The cache page, for what is cached and served now; Settings, for the cache directory.
| What | In the snap | Without the snap |
|---|---|---|
Cache directory (-cache-dir) | /var/snap/pkgproxy/common/cache | /var/cache/pkgproxy |
| Statistics socket | /var/snap/pkgproxy/common/stats.sock | /run/pkgproxy/stats.sock |
| Admin socket | /var/snap/pkgproxy/common/admin.sock | /run/pkgproxy/admin.sock |
| Content sockets, shared with other snaps | /var/snap/pkgproxy/common/sockets/ | only where -content-socket-path and -lxd-tls-content-socket-path point |
| Settings the configure hook last applied | /var/snap/pkgproxy/current/configured.env | (none) |
| What the last refresh migrated | /var/snap/pkgproxy/current/migrate-cache.log | (none) |
The snap's common directory survives snap updates and refreshes.
/var/snap/pkgproxy/common/
├── cache/
│ ├── apt/
│ │ ├── objects/
│ │ │ └── archive.ubuntu.com/ubuntu/
│ │ │ ├── dists/noble/InRelease
│ │ │ ├── dists/noble/InRelease.meta.json
│ │ │ ├── pool/main/h/hello/hello_2.10-3_amd64.deb
│ │ │ └── pool/main/h/hello/hello_2.10-3_amd64.deb.meta.json
│ │ └── tmp/ downloads in progress
│ ├── snap/
│ │ ├── objects/
│ │ │ ├── sha3-384/HEX snap files, by hash
│ │ │ ├── info/NAME snap details
│ │ │ ├── find/… refresh/… catalog/…
│ │ │ ├── assertions/…
│ │ │ └── pending/sha3-384/HEX where a rewritten download really points
│ │ └── tmp/
│ ├── go/
│ │ ├── objects/
│ │ │ └── MODULE/@v/ list, VERSION.info, VERSION.mod, VERSION.zip
│ │ ├── sumdb/ checksum database tiles and notes
│ │ └── tmp/
│ ├── docker/
│ │ ├── objects/
│ │ │ ├── blobs/sha256/HEX layers and configs
│ │ │ ├── manifests/sha256/HEX
│ │ │ └── tags/REPOSITORY/_/TAG/ACCEPT
│ │ └── tmp/
│ ├── lxd/
│ │ ├── objects/
│ │ │ └── REMOTE/… the image server's own paths
│ │ ├── tmp/
│ │ └── tls/
│ │ ├── cert.pem the HTTPS port's self-signed certificate
│ │ └── key.pem and its private key
│ ├── files/
│ │ ├── objects/
│ │ │ ├── sha256/HEX
│ │ │ └── sha512/HEX
│ │ └── tmp/
│ ├── stats.json saved statistics
│ └── lxd-index.json saved hashes of LXD image files
├── stats.sock
├── admin.sock
└── sockets/
├── pkgproxy.sock the main port, for another snap
└── pkgproxy-lxd-tls.sock the HTTPS port, for another snap
Without the snap, what the snap keeps in cache/ is in /var/cache/pkgproxy, and stats.sock and admin.sock are in /run/pkgproxy. Every cached object is one file, with its metadata in an extended attribute of that file and in a .meta.json file beside it. A kind's directory is made when its side starts, so a side turned off, such as Docker with docker-upstream set to off, has none; lxd/tls exists only while the HTTPS port is on.
| Kind | Key under objects/ | Example |
|---|---|---|
| apt | mirror host (with a port other than 80), the path before dists or pool, then the archive path | archive.ubuntu.com/ubuntu/pool/main/h/hello/hello_2.10-3_amd64.deb |
| snap | sha3-384/ and the snap's hash; details, searches, refreshes and assertions under their own prefixes | sha3-384/1f2e… |
| go | the module path and version, case-escaped as the Go module proxy protocol names them | golang.org/x/mod/@v/v0.20.0.zip |
| docker | blobs/sha256/, manifests/sha256/, or tags/ with the repository, tag and a short hash of the client's accepted types | tags/library/ubuntu/_/24.04/any |
| lxd | the remote's name, then the image server's path | pkgproxy-ubuntu/server/releases/noble/… |
| files | the checksum's algorithm and its hex digest, whatever URL the file came from | sha256/9f86… |
Each object's metadata is JSON in the user.pkgproxy.meta extended attribute of its file, so it is set and replaced together with the file. getfattr -n user.pkgproxy.meta FILE shows it. Metadata of more than 3 KiB is kept compressed with gzip, up to 64 KiB of JSON: getfattr --only-values -n user.pkgproxy.meta FILE | zcat shows it.
| Field | Meaning |
|---|---|
Size | size of the object in bytes |
ETag, LastModified | the upstream's validators, sent back when the entry is rechecked |
StoredAt | when the object was stored or last rechecked |
ContentHash | the hash of the object as stored, as ALGORITHM:DIGEST in the kind's own terms: SHA256:HEX (or the index's other algorithm) for apt, sha3-384:HEX for snaps, sha256:HEX for LXD images, Docker digests and files, h1:BASE64 for Go module zips and go.mod files, and sha256:HEX of what was stored for Go's version lists and .info files; empty for the rest. Where the kind has a published hash to check against, the object matched it before it was stored; an apt file no index lists and Go's version lists are stored with the hash unchecked |
Immutable | the entry is kept forever and never rechecked |
ContentType | the media type to serve it with, for proxy engines that keep one |
MaxAge | a lifetime the upstream announced, for proxy engines that honour one |
OriginURL | the URL the object was downloaded from, for proxy engines that record it; left out if the metadata would not fit otherwise |
pkgproxy also writes the same JSON, uncompressed, to <file>.meta.json beside each object, which is where pkgproxy 0.11 kept it, so that going back to 0.11 keeps the cache (below). A file without the attribute takes its metadata from that file; one without either is a miss and is fetched again, and pkgproxy logs the first one it finds in each kind's directory. Anything under tmp/ is never served and can be removed while pkgproxy is stopped.
| the cache from 0.11 | every object has only its .meta.json file; 0.12 reads it as it is |
| after each snap refresh | the post-refresh hook runs pkgproxy -migrate-cache, which copies each .meta.json file of what 0.11 cached into the file's extended attribute, keeps the .meta.json files, and stops after two minutes; it writes how many files it migrated to /var/snap/pkgproxy/current/migrate-cache.log. A file it did not reach still works |
| without the snap | pkgproxy -migrate-cache does the same, then exits; nothing requires it |
| going back | sudo snap revert pkgproxy: 0.11 finds every object it would have written itself |
| going back any other way | refused: the snap declares epoch 1*, which reads the cache of 0.11 (epoch 0) and its own (epoch 1), and snapd refuses to refresh or install a revision of epoch 0, such as 0.11, over it |
| after a revert, on 0.12 again | what 0.11 stored reads right; a file 0.11 revalidated is revalidated again sooner than it need be |
| later | a release, most likely 0.13, will declare epoch 1: it stops writing and reading the .meta.json files, and removes them. snapd moves every machine through an epoch 1* revision first, so a revert from it lands there, not on 0.11 |
| file systems that keep extended attributes | ext4, XFS, btrfs, ZFS; tmpfs from Linux 6.6 |
| file systems that do not | tmpfs before Linux 6.6, NFSv3, FAT: pkgproxy refuses to start with its cache there |
| copying or backing up the cache | cp -a, rsync -aX or tar --xattrs, which keep extended attributes |
| a copy that dropped them | every file reads its metadata from its .meta.json file; a file without that too is a miss and is downloaded again the next time a client needs it, and pkgproxy logs the first such file it finds in each kind's directory |
| To | Do |
|---|---|
| see what each kind takes | du -sh /var/snap/pkgproxy/common/cache/* |
| free one kind | sudo snap stop pkgproxy, sudo rm -rf /var/snap/pkgproxy/common/cache/KIND, sudo snap start pkgproxy |
| free the LXD kind | as above, then pkgproxyctl setup lxd-remote again on every client, because the certificate is new |
| throw away one apt file | pkgproxyctl cache invalidate URL, with the http:// URL apt fetches it from |
| keep the statistics | leave stats.json in place |
See pkgproxyctl for cache invalidate and Statistics for what stats.json holds.
Unit tests, command tests, synctest bubbles, exploratory runs, the spread suite, outages on purpose, the rock and the charms, and the checks before a commit.
Index parsing, cache keys, hash checks and the Store are
plain unit tests. ./check.sh is dagger check:
fmt, vet, golint, build, staticcheck, goconst, errcheck, wsl, ineffassign,
go test -race -short on two Go versions, and reuse lint.
invoke(t, "pkgproxyctl", "setup", …) wraps go-cmdr's
cmdtest; assert the whole stdout, with
<IGNORE> where it varies. noDaemon(t)
points SNAP_COMMON at an empty directory;
unixServer fakes a daemon.
synctest.Test runs the test in a bubble;
synctest.Wait() returns when everything else is blocked.
Timeouts and TTLs pass on a fake clock. Use
httptest.NewTestServer and its Client(), not a
real socket.
Exploratory apt runs in a podman container against the
pkgproxy on the host; no CI job snapshots a mirror. A bug found this way
becomes a narrow unit test of the logic that failed.
tests/<client>/: setup,
setup-setup, undo, undo-undo,
undo-first, smoke.
tests/lib/invariants.sh runs after every task. CI runs the
LXD 6 system; LXD 5.21 is left to local runs.
pp_guard_on [all] [reject|drop];
pp_assert_served_locally accepts hit,
stale and synthesised.
tests/offline-build/pack is manual, about an hour, on
ubuntu-cloud-26.04-at-bare, with pkgproxy as
pkgproxy-offline.service.
tests/rock/: apt, options.
tests/charm/ (manual): integration on MicroK8s,
machine on LXD. The spread-charm job runs it
with PKGPROXY_JUJU_CHANNEL 3.6/stable and
4/stable. ./check.sh runs
charm-lint and charm-test with uv.
git config core.hooksPath githooks enables
githooks/pre-commit, which also runs
go mod tidy -diff in every module. Tests, golint and reuse lint are left to
./check.sh and CI. Commit with git commit -s
for the DCO check.
Next: internal/fill, for the code synctest tests drive; The source tree, for the conventions; pkgproxy for CI, for the spread suite as a farm.
| Kind | Where | How to run |
|---|---|---|
| Unit tests of pure logic | *_test.go beside the code under internal/; fixtures in testdata/apt/ | go test ./internal/..., or one package |
| Command tests | cmd/pkgproxy/*_test.go, one beside each cmd_*.go | go test ./cmd/pkgproxy |
| Synctest tests (shared fetches, timeouts, TTLs) | in the packages they test: fill, flight, idletimeout, proxy, snapproxy, lxdproxy, reqlog, cmd/pkgproxy and others | as their package |
| Documentation checks | docs/docs_test.go: TestTablesOfContents, TestLinks, TestNoRawTags, TestHeader | go test ./docs |
| Podman runs of real apt and go | internal/integration/ | go test ./internal/integration; skipped under -short, without podman, or when a container cannot reach the host |
| snapd's own store client against the snap proxy engine | internal/integration/snapd/, a nested module | cd internal/integration/snapd && go test ./... |
| Exploratory apt runs | by hand: a Debian or Ubuntu container under podman with its apt pointed at the host's pkgproxy | not automated; a bug found becomes a narrow unit test |
| Spread suite | spread.yaml, tests/<suite>/<task>/task.yaml, tests/lib/ | spread garden: (below) |
| Charm unit tests | charms/<name>/tests/unit/, with ops.testing, for pkgproxy-k8s and pkgproxy | uv run pytest -q in charms/<name>/, or dagger check charm-test |
| Charm integration tests | charms/<name>/tests/integration/, with jubilant | the manual spread suite tests/charm/ (below), or uv run pytest tests/integration against a Juju model of your own |
| Charms kept in step with the daemon | cmd/pkgproxy/charm_test.go | go test ./cmd/pkgproxy |
| A benchmark | internal/upstreams/history_test.go: BenchmarkHistory30Days, a series' 30 days of hours, the same cost whatever the files hold (§28.7) | go test -run '^$' -bench History30Days ./internal/upstreams; neither ./check.sh nor CI runs it |
| Everything CI runs but spread | .dagger/main.go | ./check.sh |
| Helper | File | What it does |
|---|---|---|
TestMain | main_test.go | Started as pkgproxy or pkgproxyctl (a symlink), the test binary runs main, so tests can run the snap's shell scripts against the real commands |
invoke(t, argv...) | main_test.go | Runs the real command tree in English with go-cmdr's cmdtest; its buffers make every setup command print its piped script |
invokeIn(t, locale, argv...) | main_test.go | The same in another locale |
invokeOn(t, h, argv...), invokeWith | main_test.go | With a host of test doubles, or another command tree |
check(t, errs...) | main_test.go | Reports every failed expectation of one run |
noDaemon(t) | main_test.go | Points SNAP_COMMON, and with it the default stats and admin sockets, at an empty directory |
unixServer(t, handler) | cmd_stats_test.go | Serves a fake daemon on a UNIX socket |
charm_test.go keeps the charms in step with what they wrap:
| Test | Checks |
|---|---|
TestCharmOptionsMatchFlags | in both charms, every option is a daemon flag of the same name and default, or one of the charm's own; every flag is an option, or listed with a reason as not one |
TestMachineCharmOptionsAreSnapOptions | every machine charm option is a snap option pkgproxy-env maps (lxd-tls as lxd-tls-addr) |
TestCharmServiceMatchesRock | the Kubernetes charm's Pebble service runs the rock's command (without its empty [ ] arguments), with the rock's kill-delay |
TestCharmSetupScriptMatchesSetup | the setup-script action offers every tool pkgproxyctl setup sets up, in its order |
TestCharmsShareCommon | src/common.py is the same file in every charm |
Expectations compare the whole output: inv.ExpectStdout, ExpectStderr, ExpectStderrTemplate, with <IGNORE> or <VARIABLE name> where output varies (host names, temporary paths, time).
synctest.Test; synctest.Wait() returns once every other goroutine in the bubble is blocked.time.Sleep or poll: timeouts, TTLs and sleeps pass on the bubble's fake clock.httptest.NewTestServer and its Client(): a goroutine blocked on a real socket keeps the clock from moving../check.sh is exec dagger check; GitLab CI's dagger-check job runs the same module and then dagger call coverage export --path=coverage. It needs the dagger CLI and docker or podman. The container is golang:1.27 with shared module and build caches.
| Check | Runs |
|---|---|
go-fmt | go fmt ./... prints nothing |
go-vet | go vet ./... |
go-lint | go tool golang.org/x/lint/golint ./... |
go-build | go build ./... |
static-check | go tool honnef.co/go/tools/cmd/staticcheck ./... |
go-const | goconst, ignoring immutable, revalidate, tcp, unix, json, limit |
err-check | go tool github.com/kisielk/errcheck ./... |
wsl | go tool github.com/bombsimon/wsl/v5/cmd/wsl ./... |
ineff-assign | go tool github.com/gordonklaus/ineffassign ./... |
test | go test -race -short -coverprofile=coverage.txt -covermode=atomic ./..., then go tool cover -func |
go-test-latest | go test -race -short ./... with GOTOOLCHAIN set to the latest stable Go from go.dev |
snapd-test | go vet and go test -race -count=1 ./... in internal/integration/snapd |
snapd-lint | gofmt, staticcheck, errcheck and ineffassign in internal/integration/snapd, built with -modfile=/src/go.mod |
reuse-lint | reuse lint in fsfe/reuse:latest |
charm-lint | uv sync --locked, then uv run mypy (--strict, from the charm's pyproject.toml) in charms/pkgproxy-k8s and charms/pkgproxy |
charm-test | uv sync --locked, then uv run pytest -q (tests/unit) in each charm |
The charm checks run in ghcr.io/astral-sh/uv:0.9.30-python3.14-trixie-slim, with a shared uv cache. --locked fails when uv.lock no longer matches pyproject.toml.
| Command | Does |
|---|---|
dagger check -l | List the checks |
dagger check go-vet static-check | Run some by name |
dagger check --failfast | Stop at the first failure |
dagger call coverage export --path=coverage | coverage.txt, coverage-func.txt, Cobertura coverage.xml |
Enable once per clone (all its worktrees share it): git config core.hooksPath githooks. It needs no container and runs, with the tools go.mod pins:
| Check | Command |
|---|---|
| gofmt | gofmt -l on every tracked .go file outside .dagger/ |
| go-vet | go vet ./... |
| staticcheck | go tool honnef.co/go/tools/cmd/staticcheck ./... |
| goconst | on a copy of the files CI sees, with CI's ignore list |
| errcheck | go tool github.com/kisielk/errcheck ./... |
| wsl | go tool github.com/bombsimon/wsl/v5/cmd/wsl ./... |
| ineffassign | go tool github.com/gordonklaus/ineffassign ./... |
| go mod tidy | go mod tidy -diff in every module but .dagger, the nested ones included |
It also reminds you to regenerate .dagger/dagger.gen.go when a staged .dagger Go file changed without it. Tests, golint and REUSE lint are left to ./check.sh and CI.
Every commit needs Signed-off-by:: commit with git commit -s, repair with git commit --amend -s or git rebase --signoff. CI's dco job runs dco-check on merge requests and the default branch.
| System | LXD | Workers | Suites |
|---|---|---|---|
ubuntu-cloud-26.04-at-lxd6 | 6 | 5 | every suite but lxd-upgrade, with tests/rock/ and the manual tests/charm/; the only system CI runs |
ubuntu-cloud-26.04-at-lxd5 | 5.21 | 3 | lxd, lxd-remote, snapcraft, rockcraft, charmcraft, lxd-upgrade |
ubuntu-cloud-26.04-at-bare | none | 1 | the manual tests/offline-build/ alone; the project's prepare, prepare-each and restore-each do nothing on it |
| Suite | Tasks |
|---|---|
tests/apt/ | the six setup tasks, offline, store-overloaded, upstream-silent (a silent archive: apt works from the cache, served stale) |
tests/snap/ | the six setup tasks, offline, offline-refresh, offline-session (a snapd with a serial but no session installs from the cache), store-overloaded, store-down (a store that cannot be reached at all), store-silent (a store that hangs) |
tests/docker/ | the six setup tasks, offline |
tests/go/ | the six setup tasks, setup-before-go, offline, offline-build (go get, build and install offline after the module cache is wiped) |
tests/lxd-remote/ | the six setup tasks, offline, refuses-sudo |
tests/lxd/ | the six setup tasks |
tests/files/ | smoke, offline, volatile (SHA256SUMS online twice, then the kept copy offline, marked STALE; refused once invalidated) |
tests/install/ | daemon, snap-option (one variant per snap option), offline-switch, auto-offline, migrate-meta, proxy-options |
tests/snapcraft/, tests/rockcraft/, tests/charmcraft/ | pack, twice, the second from the cache |
tests/lxd-upgrade/ | setup-again, on -at-lxd5: LXD 5.21 to 6 |
tests/offline-build/ | manual, about an hour: pack, a fresh machine with no snapd that packs tests/snapcraft/fixture once online and again offline with the leak guard and no default routes, from pkgproxy alone; pkgproxy runs outside snapd as pkgproxy-offline.service, state in /var/lib/pkgproxy-offline; what was refused is in the artifact refused.log |
tests/rock/ | apt: downloads through the rock in docker from a named volume, and finds the cache again in a new container; options: a variable, a flag, an invalid option, and stopping it |
tests/charm/ | manual: integration (the Kubernetes charm, on MicroK8s) and machine (the machine charm, on the VM's LXD) |
| Task | Asserts |
|---|---|
setup | The setup command, piped to sh, configures the client |
setup-setup | Twice leaves the machine as once |
undo | Setup then -undo leaves the machine as it was |
undo-undo | A second -undo succeeds and changes nothing |
undo-first | -undo on a machine never set up succeeds and changes nothing |
smoke | The client downloads through pkgproxy; the second download is a cache hit |
tests/lib/invariants.sh runs after every task: no proxy setting left behind, offline mode never, auto-offline=off, the default routes. tests/lib/pkgproxy.sh has the pp_* helpers (pp_not for a negative, pp_user to run as ubuntu, pp_hits_delta, pp_content_re, pp_assert_no_upstream, which accepts a hit, a synthesised answer, a stale copy or a refusal, pp_assert_served_locally T [ERE], which requires hit, stale or synthesised of every access log line since T, pp_volatile_drop, pp_tool_snaps, …); pp_guard_on [all] [reject|drop] loads the leak guard, an nft rule on the daemon's cgroup: reject refuses each new connection, as an upstream that is down, drop discards the packets, as one that hangs, and all stops already open connections too. PP_UNIT, PP_COMMON, PP_STATS_SOCK and PP_ADMIN_SOCK point the helpers at a pkgproxy outside the snap; tests/lib/workload/<client>.sh has the wl_* functions the setup tasks share.
rm -f pkgproxy_*.snap && snapcraft pack # the one snap the suite installs, in the main checkout rm -f pkgproxy_*.rock && rockcraft pack # the one rock tests/rock/ and tests/charm/ run spread -list spread garden: spread garden:ubuntu-cloud-26.04-at-lxd6:tests/apt/ spread garden:ubuntu-cloud-26.04-at-lxd6:tests/apt/smoke spread -debug ... # a shell in the VM when a task fails spread -reuse ... # keep the VMs for the next run
It needs image-garden and spread (the image-garden snap has both), KVM and about 6 GiB per VM, and network access. Optionally, an infrastructure pkgproxy as PKGPROXY_UPSTREAM_URL (by default PKGPROXY_URL, which CI runners set), an address the VMs reach, never localhost: the pkgproxy under test then fetches everything through it (upstream-pkgproxy, §27), and the snapcraft, rockcraft and charmcraft suites install their tools from it with pp_tool_snaps, so that they do not warm the cache of the pkgproxy under test. tests/rock/ publishes the rock's port 3142 on 4142, beside the snap.
tests/charm/ is manual: spread runs it only when it is named, as the cluster and the controllers take minutes to set up.
spread garden:ubuntu-cloud-26.04-at-lxd6:tests/charm/ PKGPROXY_JUJU_CHANNEL=4/stable spread garden:ubuntu-cloud-26.04-at-lxd6:tests/charm/
Its prepare, once per VM, lets pod and container traffic through Docker's FORWARD policy (DOCKER-USER rules for cali* and lxdbr0), makes the network namespace pkgproxy-client (192.168.250.2) as a client off the cluster, and runs concierge with tests/charm/concierge.yaml: Juju from PKGPROXY_JUJU_CHANNEL (default 3.6/stable), a strictly confined MicroK8s with hostpath-storage, dns, registry and metallb (10.64.140.43 to 49), and the controllers concierge-microk8s and concierge-lxd, whose models run update-status every minute, as Juju 4 runs config-changed only after the next one. It copies the rock into localhost:32000, and takes the charms CI packed from the root of the tree, or packs them with charmcraft pack --destructive-mode.
| Task | Runs | Asserts |
|---|---|---|
integration | charms/pkgproxy-k8s/tests/integration | active with the cache on storage; apt twice through the Service, the second a hit; still a hit after the pod is deleted; an option reaches the service; a refused option blocks the unit while the old configuration serves; offline mode; the stats, invalidate and setup-script actions; the LoadBalancer keeps the client's address |
machine | charms/pkgproxy/tests/integration | deployed with the snap as pkgproxy-snap; apt twice, the second a hit; an option reaches snap get; a refused option blocks the unit and keeps the old value; a refresh installs the snap again and the package is still a hit |
A run takes 9 to 13 minutes, most of it concierge. pytest.xml is each task's artifact (spread -artifacts). The Kubernetes tests also run against a cluster of your own: PKGPROXY_CHARM, PKGPROXY_IMAGE, JUJU_MODEL, and for the LoadBalancer test PKGPROXY_CLIENT_NETNS and PKGPROXY_CLIENT_ADDR, then uv run pytest tests/integration in charms/pkgproxy-k8s/.
| Job | Does |
|---|---|
snap-ci | packs the snap under test with snapcraft-gitlab-ci's build component, snapcraft pack --destructive-mode in a throwaway 26.04 VM, from a copy of the working tree with .git; no needs, so nothing untracked stamps the binaries +dirty; keeps it in snaps/ |
snap-ci-check | the binaries are not +dirty |
snap-ci-arm64, snap-ci-arm64-check | the same, natively on an aarch64 runner; spread tests the amd64 snap only |
rock-ci, rock-ci-check | packs the rock with rockcraft-gitlab-ci into rocks/, and checks its binary |
charm-ci, charm-ci-check | packs charms/pkgproxy-k8s/ with charmcraft-gitlab-ci into pkgproxy-k8s_amd64.charm, and checks its metadata (the container's user, the cache storage) and that its virtualenv has no test packages |
charm-machine-24.04, charm-machine-26.04 | pack charms/pkgproxy/ once for each base |
spread | moves the snap and the rock into the root of the tree and runs the -at-lxd6 tasks in five VMs, with spread-gitlab-ci's spread@0.4 component, through spread-gitlab-report; the runner's PKGPROXY_URL is the upstream of the pkgproxy under test |
spread-charm | runs tests/charm/ in one VM, as a matrix of PKGPROXY_JUJU_CHANNEL 3.6/stable and 4/stable |
spread-offline-build | runs tests/offline-build/pack with the snap snap-ci packed, in a VM of its own: on a schedule of main that sets PKGPROXY_OFFLINE_BUILD=1, which the publishing jobs skip, and by hand in any other pipeline; retries: 1 |
rock | after spread, pushes the rock, edge from main and the version from a tag, and names it by digest as ROCK_IMAGE |
snap-edge, snap-stable, charm-edge, charm-stable | publish to the Snap Store and Charmhub |
A retry of a spread job, GitLab's or by hand, runs only the tasks that did not pass: the component keeps them in .spread-rerun/, cached under spread-rerun-$CI_PIPELINE_ID-$CI_JOB_NAME_SLUG.
spread-charm takes 9 to 13 minutes of a large KVM slot, so it runs only on a merge request labelled spread-charm (add the label, then push or run the pipeline again), on a pipeline with PKGPROXY_SPREAD_CHARM=1 (by hand or on a schedule, which then runs snap-ci, rock-ci, charm-ci and spread-charm alone), and always on main and version tags. Ask for it on every merge request that changes the charms, the rock or tests/charm/.
upstream-pkgproxy sends what one pkgproxy fetches to another,
which caches it for every pkgproxy behind it.
snap set pkgproxy upstream-pkgproxy=http://cache.example:3142
(flag -upstream-pkgproxy). Refused together with
-snap-store-url, -go-upstream or
-docker-upstream. Unset, pkgproxy fetches from the origins.
GET /pkgproxy/v1/proxy-engines on the main port, readable by
anyone who can use pkgproxy. An upstream too old to describe itself, but
answering as a pkgproxy does, is taken to run what 0.12 ran. Saved as
<cache-dir>/upstream-pkgproxy.json.
apt: http:// through the upstream, https://
tunnels direct. Go: sum.golang.org asked by each tier; a
lookup that fails is logged and the module served. Files: each tier's
files-allow-host, files-allow-network and size
limits apply.
Offline mode's refusals carry X-Pkgproxy-Refused; they do
not open the downstream's breakers and are not retried. With
auto-offline=network, the route to the upstream decides.
The hop carries snapd's macaroons and device sessions and Docker registry tokens, in plain HTTP. Use it only on a network trusted as much as the clients' own.
Each request to an upstream carries X-Pkgproxy-Via; a loop
is answered 508 Loop Detected with X-Pkgproxy-Loop,
and logged. The description's random instance also tells a
pkgproxy that it is its own upstream.
Next: pkgproxy for CI, for the pkgproxy next to the runners; Upstream status, for how the upstream has answered; Offline mode, for serving while an origin is down.
| Setting | Flag | Environment | Default | Meaning |
|---|---|---|---|---|
upstream-pkgproxy | -upstream-pkgproxy | PKGPROXY_UPSTREAM_PKGPROXY | unset | Another pkgproxy, as http://host:port, to fetch everything through. |
It cannot be combined with -snap-store-url, -go-upstream or -docker-upstream. -go-sumdb stays each pkgproxy's own. pkgproxy refuses to start when the upstream is itself: an address of this machine, loopback included, at a port it listens on.
$ curl http://cache.example:3142/pkgproxy/v1/proxy-engines
{"schema":1,"instance":"7b2f...","proxy-engines":{
"apt":{"enabled":true,"kind":"forward"},
"snap":{"enabled":true,"kind":"root"},
"go":{"enabled":true,"kind":"path","path":"/go"},
"docker":{"enabled":false,"kind":"root"},
"lxd":{"enabled":true,"kind":"path","remotes":{"pkgproxy-ubuntu":"/lxd/pkgproxy-ubuntu"}},
"files":{"enabled":true,"kind":"path","path":"/files/v1/fetch"},
"files-volatile":{"enabled":true,"kind":"path","path":"/files/v1/volatile"}}}
Every pkgproxy serves this on its main port, to anyone who can use it. It names nothing of the daemon's version or state. Every answer of a proxy engine carries X-Pkgproxy-Engine: <name>.
| The upstream | The proxy engine is |
|---|---|
| runs it | fetched through the upstream, at its path |
| has it turned off | refused |
| does not list it | fetched from its origin |
| answers in a newer format | followed as last read, or as this pkgproxy runs |
| is 0.12 or 0.12.1, which do not describe themselves | taken to run apt, snap, Go, Docker, LXD and files at the default paths |
| cannot be reached | followed as saved in <cache-dir>/upstream-pkgproxy.json, or as this pkgproxy runs; requests fail until it answers |
The description is read at start and every ten minutes, and again, at most once a minute, after a request to the upstream fails. Nothing is read while pkgproxy is offline. /pkgproxy/v1/info on the stats socket reports upstream_pkgproxy: the URL, where the description came from (upstream, saved, baseline, newer, loop or unreachable), and the target of each proxy engine.
| Proxy engine | Through the upstream |
|---|---|
| apt | The upstream is the HTTP proxy for http:// sources. https:// sources still go to their servers through tunnels. |
| snap | The store's API at the upstream's root; snaps by their hash, /v2/download/sha3-384/<hex>. |
| Go | Under the upstream's /go. Each pkgproxy checks modules against sum.golang.org itself; a lookup that fails is logged and the module served. |
| Docker | At the upstream's root. |
| LXD | At the upstream's path for each image server it serves by the same name; others from their origin. |
| files | The upstream's /files/v1/fetch. Each pkgproxy applies its own files-allow-host, files-allow-network and files-max-size; the checksum is checked again. |
| volatile files | The upstream's /files/v1/volatile, within both pkgproxies' limits; its STALE copy is passed on. |
Breakers, statistics and logs key on the origin, not the upstream. Freshness adds up: an entry the upstream serves for up to its TTL is then served for up to this pkgproxy's own. pkgproxyctl cache invalidate forgets what one tier has, not the other's.
X-Pkgproxy-Refused: <mode>. It counts as an answer for this pkgproxy's breakers and is not retried.auto-offline=network, no network means no route to the upstream's address.Every request to an upstream carries X-Pkgproxy-Via, the instances of the pkgproxies it went through. A pkgproxy that finds its own instance there, or more than 8, answers 508 Loop Detected with X-Pkgproxy-Loop: <instance>. The pkgproxy that sent it logs the loop once and fetches that proxy engine from its origin until it next reads the description.
The hop between the two is plain HTTP and carries what clients send for the origins: snapd's macaroons and device sessions, and Docker registry tokens. Use it on a network you trust as much as the clients' own.
pkgproxy records how each upstream server answers, from its clients'
requests and, when turned on, its own checks, and shows it on every page,
on the status pages and in pkgproxyctl upstreams.
Recorded from clients' requests, per server and, for the snap store, per
part. Kept under <cache-dir>/upstreams/ and never
deleted. States: answering, answering slowly, failing, left alone, being
tried again, pkgproxy offline, no recent requests, never asked.
Degraded (amber) names up to three servers or parts left alone and the
last failure's kind; offline (grey) says why. Nothing is shown while
pkgproxy is online and every breaker is closed. Public visitors see no
host names. pkgproxyctl stats opens with the same line.
/pkgproxy/upstreams and /pkgproxy/upstreams/<host>,
for this machine and trusted-networks, linked from the bar
of every page. The setup page shows anyone a 30-day summary; "Other
servers" appears only once three or more were asked something.
All answered: full height, green; up to 5 % failed: 3/4; up to 50 %: 1/2; over 50 %: 1/4, red. Patterns mark cells with no requests. Figures: "99.6% answered · data for 31% of the time". Hover or tap a cell for its record.
upstream-check-interval (0, off; at least
5m). A check is counted apart from clients' requests, and
also tries again a part left alone once its time is up. Never while
pkgproxy is offline.
"Last 30 days: left alone or offline for 4h12m; pkgproxy answered 1284
requests itself." pkgproxyctl upstreams (-json
for the stats API's answer) has the columns LEFT ALONE/OFFLINE 24H,
ANSWERED BY PKGPROXY 24H and CHECKED.
Next: Offline mode, for the breakers that leave a server alone; The cache page, for the status on the live map; Two tiers, for a pkgproxy behind another.
| Where | Who | What |
|---|---|---|
| The banner at the top of every page | everyone; host names only for this machine and trusted networks | Degraded (amber) or offline (grey), for how long and why. Nothing while online with every breaker closed. |
| The status element in every page's bar | everyone, at their level of detail | Online, Degraded or Offline, re-read every 30 seconds while the page is open. |
/ (the setup page) | everyone | How the upstream servers have answered over the last 30 days: Snap Store, Go modules, Docker Hub and, once three or more were asked, Other servers. No other host is named. |
/pkgproxy/upstreams | this machine and trusted-networks | Each server now and over 24 hours, 30 days or a year. |
/pkgproxy/upstreams/<host> | this machine and trusted-networks | One server; for the snap store, each part, who answered (its edge or a backend), and the times left alone in the last 30 days. |
| The cache page's live views | this machine and trusted-networks | A dot on each upstream's mark, in the shape and colour of its worst part's state. |
pkgproxyctl upstreams | the stats socket | The same as a table, for the last 24 hours. -json prints the stats API's answer. |
pkgproxyctl stats | the stats socket | Starts with the banner's line. |
The pages answer text/plain too, with the same tables.
| Word | Meaning |
|---|---|
| answering | The last request was answered, within the last 15 minutes. |
| answering slowly | Answering, but the median time to the response headers over 15 minutes is over the part's threshold: search 3 s, refresh and catalogue 2 s, the rest 1 s, a whole host 2 s. |
| failing | The last requests failed; the breaker is still closed. |
| left alone | The breaker is open: requests are not sent ("next try in 4m10s"). |
| being tried again | The breaker is half open. |
| pkgproxy offline | Offline mode is offline. The server may well be fine. |
| no recent requests | Nothing was asked in the last 15 minutes. Not an outage. |
| never asked | Nothing was ever asked. |
Breakers open only in offline-mode=store-overloaded; see Offline mode. Failures are named by kind: http-408, http-429, http-500, http-502, http-503, http-504, timeout, reset, eof, dns, connect, tls or other. On the pages each is underlined and says what it means on hover or keyboard focus.
| Cell | Drawn as |
|---|---|
| All requests answered | green, full height |
| Up to 5 % failed | light amber, 3/4 height |
| Up to 50 % failed | amber, 1/2 height |
| Over 50 % failed | red, 1/4 height |
| Only pkgproxy's checks answered | green horizontal stripes |
| Left alone | a red diagonal hatch |
| pkgproxy offline | a grey cross-hatch |
| pkgproxy not running | grey dots |
| Nothing asked | a faint grey block |
A cell is 15 minutes over 24 hours, 6 hours over 30 days (a calendar of weeks), and a day over a year (a calendar of months). The figures beside a strip are the share of clients' requests answered, of those answered or failed, and the share of the time with data. Under each server, and in a host's table of parts: how long it was left alone or pkgproxy offline, and how many requests pkgproxy answered itself, from a stale copy in its cache or with a placeholder or recorded answer in the snap store's place.
| Setting | Flag | Default | Meaning |
|---|---|---|---|
upstream-check-interval | -upstream-check-interval | 0: off | How long a server nobody asked goes unasked before pkgproxy sends it a check of its own. At least 5m. |
| Server | Check | At most every |
|---|---|---|
| snap store details | GET /v2/snaps/info/hello | 30 min, or the interval if longer |
| snap store assertions | GET /v2/assertions/account/canonical | 30 min, or the interval if longer |
| snap store search | GET /v2/snaps/find?name=hello | 1 h, or the interval if longer |
| snap store catalogue | GET /v2/snaps/categories | 1 day |
| the Go proxy | GET / | the interval |
| the Docker registry | GET /v2/ (a 401 counts as an answer) | the interval |
| store sign-in, refreshes, downloads; every other host | none |
A check is counted apart from clients' requests. A part left alone is not checked; one being tried again is. Nothing is checked while pkgproxy is offline. Each status page says under its heading what is checked, and how often.
$ pkgproxyctl upstreams Offline mode: store-overloaded. All times are UTC. Checks: off; status comes from clients' requests only (upstream-check-interval). SERVER STATE SINCE RECENT LEFT ALONE/OFFLINE 24H ANSWERED BY PKGPROXY 24H CHECKED LAST FAILURE api.snapcraft.io catalog left alone, next try in 4m10s 09:14 0 of 1 answered 4h48m 1284 off 14:01 http-429 from the store's edge api.snapcraft.io find answering slowly 09:12 12 of 14 answered - - off -
RECENT counts the last 15 to 30 minutes. CHECKED is the check interval, off, or never for a part that is never checked. Options: -socket-path (the stats socket), -json.
On the stats socket:
| Path | Answer |
|---|---|
GET /pkgproxy/v1/upstreams | Every server and part now: state, breaker, last answer, request and failure, the last 15 minutes' counts and latencies, the last 24 hours' time left alone or offline and answers by pkgproxy, and, with checks on, check_interval_seconds and check_every_seconds. |
GET /pkgproxy/v1/upstreams/history?host=&class=&step=&since=&until= | The buckets of one series in which something happened. |
GET /pkgproxy/v1/upstreams/events?since=&host= | Breaker and mode changes, starts and stops. |
GET /pkgproxy/v1/info | Its offline object has since and, while something is left alone, degraded_since. |
<cache-dir>/upstreams/
live.json what is open now, and the last 48 hours' 15-minute
buckets; rewritten at most every 5 minutes
<year>/hours-<MM>.jsonl the hours and events of a UTC month,
gzipped (.jsonl.gz) once the year is over
<year>/days.jsonl the days and events of a year
Closed buckets are appended and synced; nothing is deleted. A crash loses at most five minutes of the open buckets' counts. A live.json that cannot be read is moved aside, to live.json.bad; pkgproxy starts all the same.
pkgproxy caches what apt, snapd, Go, Docker and LXD download, for every machine on your network. This topic covers the problem, the solution, the setup and the limits.
Every machine, VM, container and CI job downloads the same packages and images from the internet on its own, and an empty build environment does it again on every run.
One pkgproxy for the whole network: the first request for a file downloads it once, and every later one is served from its disk. Without the internet, it serves what is cached.
sudo snap install pkgproxy starts the service on
3142 (and 3143 for LXD).
pkgproxyctl setup <tool> prints a script; other machines
fetch it from http://<host>:3142/setup/<tool>.
-undo prints the way back.
No TLS termination: https apt sources go through an uncached
CONNECT tunnel to port 443. Files by checksum are the
exception. No signature checks (apt does them) and no garbage collection.
Next: pkgproxy for CI, for many runners behind one cache; A request, for one request end to end; The store, for where the cache lives.
Next: A request, for one request end to end; pkgproxyctl
setup, for pointing your tools at it; Offline mode, for working
without the internet.
| Channel | What it has |
|---|---|
latest/stable | each release, from its tag |
latest/edge | the main branch, as CI builds it, released from time to time |
sudo snap install pkgproxy
The snap is built for amd64 and arm64. Releases before 0.12.2 were also built for ppc64el, s390x and riscv64; their last revisions stay in the store.