How pkgproxy caches packages for apt, snapd, Go, Docker and LXD, one narrated deck per topic. Pick any place, follow the roads at the end of each deck, and switch perspective at the top to see what applies to you. The written documentation is at zygoon.pl/pkgproxy; to run pkgproxy in GitHub Actions jobs, see setup-pkgproxy.
Click a place to enter it. Inside a topic you see only its exits; m brings the whole map back. space plays, ← → step.
pkgproxy logs one line per request with who asked, which side answered and how, and more when it failed.
handler is apt (every request sent to pkgproxy
as a proxy, tunnels included), snap, gomod,
docker, lxd, lxd-api or
files for the proxy sides, and setup-page or
lxd-cert for pkgproxy's own pages. A line without one is a path
nothing serves; a tunnel's bytes are what it copied to the client.
hit, joined, revalidated,
stale, synthesised, fetched,
passthrough, tunnel, error. They
are not translated.
A 5xx line carries response, the first 512 bytes of a text
body; compressed or non-text bodies are skipped. Look for
status=5, and for snapd status=424.
snap logs pkgproxy -n 100, snap logs pkgproxy
-f, or journalctl -u snap.pkgproxy.pkgproxy.
Next: Statistics, for totals; The cache page, for the same requests drawn live; Immutable or not, for what revalidated and stale mean.
level=INFO msg="handled request" method=GET url=http://deb.debian.org/debian/dists/bookworm/InRelease remote_addr=192.0.2.7:46382 handler=apt outcome=hit status=200 bytes=151472 duration=1.2ms
| Field | Value |
|---|---|
level | INFO, or ERROR for a status of 500 or more |
msg | handled request, in the daemon's language; the keys and the words below are never translated |
method | the request method, CONNECT for a tunnel |
url | the request target as the client sent it: a full URL for apt, a path and query for the other sides |
remote_addr | the client's address and port |
handler | which part of pkgproxy answered; absent for a path nothing serves |
outcome | how it was answered; absent for a successful request that recorded none |
status | the HTTP status sent, 200 for an open tunnel |
bytes | body bytes sent to the client; for a tunnel, bytes copied from the upstream to the client |
duration | from the request's arrival until its handler returned, work after the response included |
response | only on a 5xx: the first 512 bytes of a text body; compressed or non-text bodies are left out |
assertions_fetched | only on a snap refresh answered online: assertions fetched ahead for offline installs |
chains_incomplete | with the above: how many recorded snaps still lack some assertions; above 0, a warning before names them |
The line is written when the request's handler returns, so it comes after any warning or error line the same request logged.
handler | Requests |
|---|---|
apt | everything sent to pkgproxy as a proxy: apt archives, any other http:// URL, and CONNECT tunnels |
snap | the Snap Store API: /v2/… paths that are not registry paths, /api/v1/…, /v1/… |
gomod | the Go module proxy, under /go |
docker | the Docker Hub mirror: registry-shaped /v2/… paths |
lxd | LXD image servers over simplestreams, under /lxd |
lxd-api | LXD's image API on the HTTPS port, for LXD 6.10 and later |
lxd-cert | the download of pkgproxy's LXD certificate |
files | files fetched by checksum, under /files |
setup-page | the setup page, its setup scripts and, for local visitors and those from trusted networks, the cache page and its live feeds |
The side names match the entry types of pkgproxyctl stats, so a log line and a statistics row for the same traffic agree.
outcome | Meaning | Where it shows up besides the obvious |
|---|---|---|
hit | served from the cache without asking the upstream | a mutable entry inside its window |
joined | waited for or followed another request's fetch | only the requests that joined; the one that started the fetch is fetched or revalidated. On the Docker mirror every request sharing a manifest fetch is joined |
revalidated | checked with the upstream, then served (refreshed, or confirmed by a 304) | apt indexes, Go lists, LXD catalogs, Docker tags |
stale | the upstream could not be reached; a copy due for a check was served | always preceded by a warning line with the reason |
fetched | fetched from the upstream, kept and served | |
passthrough | relayed from the upstream without keeping it | uncacheable paths, a hash mismatch, a HEAD of an uncached Docker layer, a document fetched but not stored |
synthesised | answered by pkgproxy from what it recorded, without asking the upstream | offline snap refreshes, a HEAD of an uncached LXD image file or snap |
tunnel | a blind CONNECT tunnel | https:// apt sources and anything else tunnelled |
error | the request failed | assumed for any 4xx or 5xx that recorded no other outcome |
| To find | Search for |
|---|---|
| server errors | status=5 |
| snapd's errors while offline or leaving the store alone | status=424 (logged at INFO) |
| one machine | remote_addr=192.0.2.7: |
| answers given from old copies | outcome=stale |
| one side | handler=docker |
In the snap the log is in the journal: snap logs pkgproxy -n all, or journalctl -u snap.pkgproxy.pkgproxy. Elsewhere it is the daemon's standard error.
Pool files are kept forever, dists files are rechecked after a short window, and a failing mirror is covered by the cached copy.
pool/** and dists/**/by-hash/** are immutable:
cached forever, never revalidated. Other dists/** files are
revalidated after a TTL. Anything else is passed through uncached.
-apt-revalidate-ttl (default: -revalidate-ttl, 5m) is how long a
dists/** entry is served without asking the mirror;
-min-forced-revalidate-interval (default 30s)
bounds what a client's Cache-Control: no-cache can force. Both
are also snap options.
A failed revalidation serves the stale copy (outcome=stale,
with a warning). The wait is bounded at 10 seconds and a failed origin is
not retried for a minute. 404/410 become
404; a known-superseded entry that cannot be refetched gets
502.
A dists/** miss, by-hash included, is fetched,
stored, parsed and then served from the store. A pool/** miss
is a fill that streams to its clients while it writes.
Next: Immutable or not, for the same rule on every side; Fetch once, for how a pool miss streams; Offline mode, for apt update without the internet.
The first path segment named pool or dists makes a URL an archive path, whatever comes before it; there is no list of archives to configure.
| Path | Kind | Cache policy | Checked against |
|---|---|---|---|
…/pool/… | packages and sources | kept forever, never rechecked | the SHA-256 from a Packages or Sources file already read; unchecked, with a warning, when none is known |
…/dists/…/by-hash/ALGO/HEX | an index named by its hash | kept forever, never rechecked | the hash in its own name |
…/dists/…/InRelease, Release, Release.gpg | the signed root of a suite | rechecked after -apt-revalidate-ttl | nothing: apt checks the signature |
any other …/dists/… file: Packages*, Sources*, Contents*, Translation-*, *.diff/Index | indexes | rechecked after -apt-revalidate-ttl | the hash Release lists for it, when known |
a path with no pool or dists segment | anything else | passed through, not cached | — |
https:// sources, through CONNECT | encrypted | tunnelled, not cached | — |
Compressed .xz indexes are cached and served but not read, so a pool file listed only in one is cached unchecked. HTTPS sources are covered in HTTPS apt sources, and how the cache key is built in apt cache keys and hash checks.
| Situation | What pkgproxy does |
|---|---|
cached and checked within -apt-revalidate-ttl | serves it, asks nothing |
| older than that | sends one If-Modified-Since request for every client waiting on it; a 304 keeps the copy and restarts the clock |
no Last-Modified was recorded | fetches it again in full |
a newer Release lists a different hash for the cached copy | fetches it again in full, even within the TTL |
the mirror fails, stalls, or answers 5xx or another unexpected status | serves the cached copy, warns in the log, and the access log says outcome=stale |
| the check takes more than 10 seconds | serves the cached copy while the check carries on |
| the mirror failed a check within the last minute | serves the cached copy without asking it |
the mirror answers 404 or 410 | 404 to apt |
| the copy is known to be superseded and cannot be fetched again | 502 to apt |
A miss on an index is fetched whole, stored, read for hashes and only then served. A miss on a pool file streams to every waiting client while it is written; see Fetch once.
apt sends these headers for Acquire::http::No-Cache and Acquire::http::Max-Age.
| Request header | Effect on a cached index within its TTL |
|---|---|
Cache-Control: no-cache | checked with the mirror first, unless checked within -min-forced-revalidate-interval |
Cache-Control: max-age=N | checked first if the last check is older than N seconds, with the same bound |
Pragma: no-cache, with no Cache-Control | the same as no-cache |
A forced check is a conditional request: a 304 keeps the copy, so it never repairs wrong content; pkgproxyctl cache invalidate does. Pool files and by-hash files are never rechecked, whatever the client asks. In offline mode no check is forced.
| Situation | Status apt sees |
|---|---|
a request line that is not an absolute http:// URL, as apt sends to a proxy | 400 |
a method other than GET, HEAD or CONNECT | 405 |
CONNECT to a port other than 443 | 403 |
a passed-through request or CONNECT to a loopback address, or to this machine at a port pkgproxy listens on | 403 |
the mirror answers 404 or 410 | 404 |
| a downloaded file fails its hash check | the mirror's bytes, unchanged and not cached |
| the mirror cannot be reached and nothing is cached | 502 |
| Option | Environment variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-apt-revalidate-ttl | PKGPROXY_APT_REVALIDATE_TTL | apt-revalidate-ttl | -revalidate-ttl, 5m | how long an index is served before it is checked again |
-revalidate-ttl | PKGPROXY_REVALIDATE_TTL | revalidate-ttl | 5m | the same for every side that has no setting of its own |
-min-forced-revalidate-interval | PKGPROXY_MIN_FORCED_REVALIDATE_INTERVAL | min-forced-revalidate-interval | 30s | how recently checked an index must be for a client's no-cache to be ignored |
-fill-drain-timeout | PKGPROXY_FILL_DRAIN_TIMEOUT | fill-drain-timeout | 10m | how long a pool file that failed its hash check keeps going to the clients already receiving it |
-upstream-idle-timeout | PKGPROXY_UPSTREAM_IDLE_TIMEOUT | upstream-idle-timeout | 1m | how long a mirror may send nothing before the request fails; 0 waits forever |
The 10-second wait and the one-minute pause after a failure are fixed. Cached files live under apt/objects/HOST/PATH in the cache directory, such as apt/objects/archive.ubuntu.com/ubuntu/pool/main/a/apt/apt_2.8.3_amd64.deb; see The store.
How apt's https sources pass through pkgproxy as an uncached tunnel, and how to get them cached instead.
apt sends CONNECT host:443 to whatever
Acquire::http::Proxy names for any https://
source; no Acquire::https::Proxy is needed to trigger it.
pkgproxy dials first, then takes over the client's connection and copies
bytes both ways. No TLS termination, no caching, no verification. A dial
that fails gets 502; one that reaches this machine's loopback
gets 403. Other ports get 403: pkgproxy only
tunnels CONNECT to port 443.
Refusing CONNECT (with 501) would make
apt update fail on any machine with one https-only source. The
port-443 tunnel keeps the non-goal: no TLS termination, no HTTPS caching.
Neither the tunnel nor plain http reaches this machine's loopback or
pkgproxy's own ports.
A tunnelled request is logged with outcome=tunnel. Switch the
source to http:// where the archive offers it; apt checks
signatures either way. https-only archives keep working, uncached.
Next: The access log, for finding tunnels; pool and dists, for what an http source gets; Files by checksum, for https that pkgproxy fetches itself.
Next: A forward proxy, for plain http sources; pool and dists, for what they get from the cache; Files by checksum, for https that pkgproxy fetches itself.
How an apt URL becomes a cache key, and how each download is checked against the hash its index lists.
The archive id is the host (with its port, unless it is 80) plus the path prefix before pool/
or dists/; the key is the archive id and the path from there
on. A URL with neither segment is passed through uncached.
deb.debian.org/debian and
deb.debian.org/debian-security are separate archives, as are KDE
Neon's /unstable and /release. Mirrors on
different hosts are cached separately: mirror aliasing is not built yet.
InRelease lists the hashes of the index files;
Packages lists Filename and SHA256
for each package. Both are parsed as they pass through. A
by-hash URL names its own hash.
A conflicting hash for a known pool path is logged and ignored. Content that fails its hash check is never committed; clients already receiving it get the origin's bytes unmodified, the rest a passthrough fetch.
InRelease, Release and Release.gpg
are cached unverified; apt verifies the signature. .xz indexes
are not parsed, so a pool file listed only there is cached without a hash
check, with a warning in the log.
Next: pool and dists, for what a key does on a hit; The store, for where downloads are written; Fetch once, for one fetch per key.
How apt sends every request to pkgproxy as a forward proxy, set up with one line and undone with one command.
pkgproxyctl setup apt | sudo sh writes
/etc/apt/apt.conf.d/99pkgproxy with one
Acquire::http::Proxy line, plus
Acquire::Retries::Delay "false"; when apt is older than 3.1.3.
sources.list stays as it is.
A forward-proxy request has an absolute URI: GET
http://deb.debian.org/…. pkgproxy fetches from the host it names,
with Via: 1.1 pkgproxy, so any number of mirrors go through one
proxy. This machine's loopback, and its own addresses at pkgproxy's ports,
get 403. A request with only a path is answered
400.
Mirror substitution rewrites sources.list to name the cache,
which fronts one upstream. A forward proxy takes requests for any host
through one cache, with one Acquire::http::Proxy line per
machine.
If-Modified-Since gets 304 from the cache.
Range is honoured only when If-Range matches the
entity being sent; otherwise 200 with the whole body. apt's
default Acquire::http::Pipeline-Depth is 10.
apt-config dump | grep -i proxy shows the setting. apt does
not fail over: while pkgproxy is down, its downloads fail.
pkgproxyctl setup apt -undo | sudo sh removes
/etc/apt/apt.conf.d/99pkgproxy. Without | sudo sh,
setup explains each step instead.
Next: Cache keys, for keys and hash checks; pool and dists, for what is kept and rechecked; HTTPS sources, for https archives.
Next: pool and dists, for what apt gets from the cache; HTTPS sources,
for https archives; pkgproxyctl setup, for the other tools.
setup lxd and the craft setups put apt, snapd and Go inside
LXD build containers behind pkgproxy.
setup lxd-remote serves LXD its images;
setup lxd points apt, snapd and go inside the containers at
pkgproxy. A craft build needs setup lxd-remote and
setup snapcraft (or rockcraft,
charmcraft).
environment.GOPROXY and
environment.SNAPPY_FORCE_API_URL, plus raw.lxc
pre-start and post-stop hooks that write and remove
99pkgproxy-lxd and pkgproxy-lxd.conf. Each hook
always exits 0 and never follows a symbolic link.
Without -project: a standalone pkgproxy profile
(lxc launch … --profile pkgproxy --profile default). With
-project P: a PATCH of P's default profile. setup
snapcraft, rockcraft and charmcraft fix P
to the tool's own project.
The host of -addr (or of the address the setup page was
reached at): an IP is used as is, a name is resolved with
getent, and none, a wildcard, localhost or
loopback means the managed LXD bridge's ipv4.address (else
ipv6.address).
The LXD calls stay in the printed script, run by the user in the
lxd group with lxc … local:. The admin socket,
-admin-socket-path, serves only DELETE
/pkgproxy/v1/cache and /pkgproxy/v1/offline.
-undo deletes the pkgproxy profile, or unsets
environment.GOPROXY, environment.SNAPPY_FORCE_API_URL
and raw.lxc on a project's default profile. Setup and undo can each be
run again safely, and a second snapcraft pack is served from
the cache.
Next: pkgproxy for CI, for many build machines behind one cache; Offline mode, for builds that see only what is cached.
Next: LXD images, for the images the containers start from; pool and dists, for what apt gets from the cache; Offline mode, for building with no network.
The cache page draws what fills the cache, what is used and what is being downloaded now, for the machine that runs pkgproxy and the networks it trusts.
http://localhost:3142/pkgproxy/cache, over TCP from this
machine, without Forwarded, Via,
X-Forwarded-* or X-Real-IP. pkgproxy's apt proxy
refuses to connect to this machine's loopback or its own ports, and what it
fetches carries Via, so it cannot relay the page.
trusted-networks (-trusted-networks): CIDR
prefixes separated by commas, empty by default. A refused request gets
the 404 of any unknown path. Give visitors the machine's address or
public-url: a short host name may be answered by any machine
on the link (mDNS, LLMNR).
Ground is what the item takes on disk, as du counts it. A
kind's smallest items fold into one "N more items" block. The layout is
fixed once the page loads, so nothing moves.
Hits in: All time, Last week, Last day or Last hour. Drag to turn and tilt, scroll to zoom, and click a block to zoom into its kind, then its group. Beside the map, a table of every item can be sorted and searched.
Machines are named by their host names, looked up from their addresses; virtual machines and snaps on a socket are "Socket clients". Lines are as wide as the bytes they carried and fade over six seconds. "Turntable" spins the map slowly.
Views: Wall dashboard, Map, Byte flow and Details; #map,
#flow or #details opens one. At most four pages
watch live at once. Without JavaScript the page shows a flat treemap and
its tables.
Next: Statistics, for the same counts from a script; The access log, for one request after the fact; pkgproxy for CI, for a build farm's screen.
| Path, on the main port | What it answers |
|---|---|
/pkgproxy/cache | the cache page: HTML for a browser, the same figures as plain text otherwise |
/pkgproxy/cache/activity | what the page polls for its heights and hits |
/pkgproxy/cache/events | the stream of requests behind the wall dashboard and the byte flow |
/ | the setup page, with a line about the cache and a link to the page |
All of them are served only with setup-page on. Every answer with the cache's figures is sent with Cache-Control: no-store.
| Setting | Snap option | Flag | Default | Meaning |
|---|---|---|---|---|
| Trusted networks | trusted-networks | -trusted-networks | (empty) | Networks, in CIDR notation and separated by commas (a single address is one machine), whose visitors see the cache page, its live views and the setup page's cache line as a visitor on the machine does |
sudo snap set pkgproxy trusted-networks=192.168.0.0/24,fd00::/8
A request sees the page only when all of these hold; anyone else gets the 404 of any unknown path, and the setup page shows them no line about the cache:
Forwarded, Via, X-Forwarded-*, X-Real-IP, True-Client-IP, CF-Connecting-IP;Host: an IP address, localhost, the machine's host name or the host of public-url. Another web site whose name someone points at the machine cannot read the page.0.0.0.0/0 and ::/0 are refused, so that opening the page to everyone is never a typo; a wide network such as 0.0.0.0/1 is still accepted. pkgproxy logs the networks it trusts when it starts.
The machine's names count as much as its addresses. Set public-url only to a name you control. Its short host name is trusted too, and on a network where any machine can answer for a short name (mDNS or LLMNR, as Windows and macOS use), a visitor reaching pkgproxy by that name trusts those answers: give visitors its address or public-url instead.
CI runners, VMs and build containers share one pkgproxy, over the network or a UNIX socket, and can run against only what is cached.
curl -fsSL http://host:3142/setup/TOOL | sudo sh configures
a machine; ?undo=1 reverses it. public-url sets
the address the page names, for example behind NAT.
GitLab: extra_hosts = ["pkgproxy:host-gateway"] in
config.toml, plus a mounted apt.conf.d file and
GOPROXY. GitHub: options: --add-host=pkgproxy:host-gateway,
then curl -fsSL http://pkgproxy:3142/setup/apt | sh.
-socket-path replaces the TCP listener with a UNIX socket
(mode 0777), for QEMU's guestfwd; -addr still
names the host:port the instructions print.
-lxd-tls-socket-path does the same for -lxd-tls-addr.
The snap runs with -content-socket-path and
-lxd-tls-content-socket-path under
$SNAP_COMMON/sockets, shared by the sockets slot
(content pkgproxy-sockets). The TCP listeners stay as they are. snapd auto-connects a plug with the same content from the same publisher; sideloaded (--dangerous) snaps need snap connect.
setup docker edits registry-mirrors in
daemon.json (Docker Hub, anonymous pulls).
/files/v1/fetch?url=…&sha256=… caches by checksum, over
http or https; files-allow-host,
files-allow-network and files-max-size (8G)
limit it.
snap set pkgproxy offline-mode=offline or
pkgproxyctl offline on; neither restarts pkgproxy.
store-overloaded backs off from a troubled upstream.
auto-offline=off on an isolated network.
Next: The setup page, for the scripts every machine fetches; Hosted CI runners, for jobs on the forge's runners; Offline mode, for runs that see only what is cached.
A job that runs on the machine itself, as a GitHub self-hosted runner does by default and GitLab's shell executor does, uses whatever the machine is set up to use. Set each tool up once:
pkgproxyctl setup apt | sudo sh pkgproxyctl setup snap | sudo sh pkgproxyctl setup docker | sudo sh sudo -u <runner user> sh -c 'pkgproxyctl setup go | sh'
setup go sets up the user who runs it, so run it as the user the runner runs its jobs as. On a machine that does not run pkgproxy itself, take the scripts from the one that does: curl -fsSL 'http://<host>:3142/setup/apt' | sudo sh.
A job container sees none of the runner machine's settings. Give it the host name pkgproxy for the machine, through Docker's host-gateway, and let the containers' networks reach port 3142 through the machine's firewall, for example sudo ufw allow from 172.16.0.0/12 to any port 3142 proto tcp.
GitLab's docker executor, in the runner's section of /etc/gitlab-runner/config.toml:
[[runners]]
environment = ["GOPROXY=http://pkgproxy:3142/go,direct"]
[runners.docker]
extra_hosts = ["pkgproxy:host-gateway"]
volumes = ["/cache", "/etc/gitlab-runner/pkgproxy-apt.conf:/etc/apt/apt.conf.d/99pkgproxy:ro"]
with the apt configuration every job container mounts:
echo 'Acquire::http::Proxy "http://pkgproxy:3142";' | sudo tee /etc/gitlab-runner/pkgproxy-apt.conf echo 'Acquire::Retries::Delay "false";' | sudo tee -a /etc/gitlab-runner/pkgproxy-apt.conf
The second line is for job images with apt older than 3.1.3, such as Ubuntu 24.04's. A GitHub job in a container:
jobs:
build:
runs-on: self-hosted
container:
image: ubuntu:24.04
options: --add-host=pkgproxy:host-gateway
steps:
- run: |
apt-get update && apt-get install -y curl
curl -fsSL http://pkgproxy:3142/setup/apt | sh
A tool that fetches large files by checksum gets http://pkgproxy:3142 in the job's environment; image-garden reads it from GARDEN_DL_PKGPROXY.
Run the same job twice, then run pkgproxyctl stats: every download of the second run counts as a hit. The access log says the same per request (outcome=hit), and so does the cache page. A job that does not show up there does not use pkgproxy.
On runners the forge hosts, pkgproxy runs inside the job and its cache lives in the CI cache between jobs; setup-pkgproxy does that on GitHub.
uses: zyga/setup-pkgproxy@v2. save-package-cache
is default-branch, always or never;
a job that added nothing saves nothing. Set up other tools with
pkgproxyctl setup TOOL -addr "${{ steps.pkgproxy.outputs.addr }}".
v1 workflows end with a step that stops pkgproxy.service
and a step with actions/cache/save. Caches saved by v1.0
and v1.1 were a plain directory, so the first run after moving from them
starts cold.
The job starts pkgproxy on 127.0.0.1:3142 and runs
pkgproxyctl setup against it. The CI cache restores the
whole cache at the start of each job and saves it when it changed.
Archive with tar --xattrs --xattrs-include='user.*' after
pkgproxy has stopped, and unpack the same way before it starts. A cache
from before 0.12 keeps its metadata in .meta.json files
instead, which any archive keeps.
PKGPROXY_CACHE_DIR under $CI_PROJECT_DIR,
cache: when: always, and after_script to save
even a failed job's downloads. Keep pkgproxy.log as an
artifact.
A public pkgproxy is an open proxy: no client authentication, plain
http:// to any host but its own and CONNECT to
port 443 of any of them, and files by checksum up to files-max-size each,
in a cache that deletes nothing. Run pkgproxy in the job instead.
Next: pkgproxy for CI, for runners you host; The store, for the cache on disk; Offline mode, for leaving a troubled mirror or store alone.
setup-pkgproxy is a GitHub Action that runs pkgproxy in the job. Both major versions:
go install on a miss, and install it as /usr/local/bin/pkgproxy with a pkgproxyctl link;/var/cache/pkgproxy, and start pkgproxy.service;configure-host is false;tar --xattrs (from v1.2.0 on), which the unit writes to /var/cache/pkgproxy-archive/cache.tar when pkgproxy stops.v2 (zyga/setup-pkgproxy@v2) | v1 (zyga/setup-pkgproxy@v1) | |
|---|---|---|
| saving the cache | the action's post step, at the end of the job, passed, failed or cancelled | the workflow's own steps: stop pkgproxy.service, hand the archive to the runner user, actions/cache/save |
| which runs save | save-package-cache: default-branch (default), always or never | wherever the workflow's save step runs, pull requests included |
| cache keys | a new key per saving job, <package-cache-key>-<arch>-<run>-…; the next job restores the newest | one key per CPU arch, <package-cache-key>-<arch>-success or -partial |
| a job that added nothing | saves nothing | the save step skips an exact hit (package-cache-hit) |
| pkgproxy | 0.11.0 | 0.11.0 |
on:
push:
branches: [main] # only the default branch saves the cache
pull_request:
# ...
steps:
- uses: zyga/setup-pkgproxy@v2
id: pkgproxy
- run: pkgproxyctl setup go -addr "${{ steps.pkgproxy.outputs.addr }}" | sh
| Input | Default | |
|---|---|---|
version | v0.11.0 | the pkgproxy version to build |
go-version | 1.27.1 | the Go that builds it on a binary-cache miss |
addr | :3142 | the address pkgproxy listens on |
binary-cache-key | pkgproxy-bin | key prefix of the built binary |
package-cache-key | pkgproxy-cache-v4 | key prefix of the package cache |
save-package-cache | default-branch | which runs save the package cache |
configure-host | true | run pkgproxyctl setup apt and setup snap on the runner |
| Output | |
|---|---|
addr | the address to pass to pkgproxyctl setup … -addr |
cache-dir | /var/cache/pkgproxy |
package-cache-restored-key | the key the cache was restored from, empty on a cold start |
package-cache-key | the key the post step saves under, empty when this run may not save |
Every run can restore the caches of its own ref, of its pull request's base branch and of the default branch. A cache a pull request saved helps only that pull request, and takes room the default branch's needs, so v2 saves on the default branch only by default. A workflow that runs only for pull requests then never saves the cache, so run it on pushes to the default branch too, as in the example above. In a matrix with several jobs on one CPU arch, give all but one save-package-cache: never.
- uses: zyga/setup-pkgproxy@v1
id: pkgproxy
# ... the job's own steps
- if: always()
run: |
sudo systemctl stop pkgproxy.service || true
sudo chown -R "$USER:$USER" ${{ steps.pkgproxy.outputs.cache-dir }}
- if: steps.pkgproxy.outputs.package-cache-hit != 'true'
uses: actions/cache/save@v6
with:
path: ${{ steps.pkgproxy.outputs.cache-dir }}
key: ${{ steps.pkgproxy.outputs.package-cache-key-success }}
In v1, cache-dir is /var/cache/pkgproxy-archive, and the outputs package-cache-hit, package-cache-key-success and package-cache-key-partial drive the save. To move to v2, change @v1 to @v2, delete the two save steps, and read the outputs v2 has instead. v1 stays maintained, on the action's releases/v1 branch. v1.2 saves the same archive under the same pkgproxy-cache-v4 prefix, so its cache warms v2; caches saved by v1.0 and v1.1 were the plain directory, and the first run after them starts cold.
pkgproxy keeps each cached file's metadata in the user.pkgproxy.meta extended attribute of the file. A CI cache that archives the directory itself may not keep extended attributes, and a file that lost its attribute is a miss: it is downloaded again, and pkgproxy logs once that it found such a file. Archive the cache yourself and let the CI cache keep that one file:
# stop pkgproxy first, so that nothing is half written tar --xattrs --xattrs-include='user.*' -cf pkgproxy-cache.tar .pkgproxy # before the next start tar --xattrs --xattrs-include='user.*' -xf pkgproxy-cache.tar
A cache directory on a file system without extended attributes keeps pkgproxy from starting, and its log says why.
A GitLab job runs in a container without a service manager: start pkgproxy in the background, with its cache in the project directory, and keep its archive in GitLab's cache. Its sockets stay in /run/pkgproxy, which keeps their paths short.
variables:
PKGPROXY_CACHE_DIR: $CI_PROJECT_DIR/.pkgproxy
PKGPROXY_ADDR: 127.0.0.1:3142
PKGPROXY_LXD_TLS_ADDR: "off"
build:
image: golang:1.25
cache:
key: pkgproxy
paths: [pkgproxy-cache.tar]
when: always
before_script:
- go install gitlab.com/zygoon/pkgproxy/cmd/pkgproxy@latest
- ln -s pkgproxy "$(go env GOPATH)/bin/pkgproxyctl"
- if [ -f pkgproxy-cache.tar ]; then tar --xattrs --xattrs-include='user.*' -xf pkgproxy-cache.tar; fi
- pkgproxy > pkgproxy.log 2>&1 & echo $! > pkgproxy.pid
- until curl -fs -o /dev/null http://127.0.0.1:3142/; do sleep 1; done
- pkgproxyctl setup apt -addr 127.0.0.1:3142 | sh
- pkgproxyctl setup go -addr 127.0.0.1:3142 | sh
script:
- ...
after_script:
- pid=$(cat pkgproxy.pid); kill "$pid"; while kill -0 "$pid" 2>/dev/null; do sleep 1; done
- tar --xattrs --xattrs-include='user.*' -cf pkgproxy-cache.tar .pkgproxy
Every setting is read from its PKGPROXY_ variable. From 0.12.0 on, a job can download a released pkgproxy and check it instead of building it with go install. Keep pkgproxy.log as an artifact to see what each request was served from.
| clients | no authentication: every client that reaches pkgproxy may use it |
| apt's side | passes plain http:// requests to any host but pkgproxy's own machine, and tunnels CONNECT to port 443 of any of them |
| files by checksum | downloads any public URL anyone names, up to files-max-size each |
| the cache | never deletes anything, so anyone can fill its disk |
Run pkgproxy in the job instead.
Every daemon setting is one name, set by a flag, an environment variable or a snap option, checked before it applies.
-revalidate-ttl 10m,
PKGPROXY_REVALIDATE_TTL=10m and snap set pkgproxy
revalidate-ttl=10m set one setting. Precedence: flag, then
environment, then default.
pkgproxy-env runs snapctl get for each key it
lists and exports PKGPROXY_*, then execs the daemon.
snap unset pkgproxy KEY restores the default.
The configure hook runs pkgproxy-env pkgproxy -check-config.
A bad value fails snap set; the daemon is not touched.
$SNAP_DATA/configured.env holds the last
PKGPROXY_* set minus PKGPROXY_OFFLINE_MODE. Same:
pkgproxyctl offline mode -keep-automatic. Different:
snapctl restart.
Not options: cache-dir, the socket paths,
check-config, offline, the upstream URLs, the
base paths, lxd-remote, go-sumdb.
snap set accepts any other key and ignores it; the Settings
table in DEPLOYMENT.md lists the options.
Next: Immutable or not, for the TTL settings; Offline mode, for the offline-mode setting; Statistics, for what the daemon reports.
Every setting is one name, such as revalidate-ttl. It is set by:
| Route | Form | Example |
|---|---|---|
| Command line | -name value | pkgproxy -revalidate-ttl 10m |
| Environment | PKGPROXY_ and the name in capitals, - as _ | PKGPROXY_REVALIDATE_TTL=10m |
| Snap option | the name, for the options listed below | sudo snap set pkgproxy revalidate-ttl=10m |
sudo snap unset pkgproxy NAME goes back to it, and snap get pkgproxy shows what you set.snap set accepts any key. A key that is not a snap option below is stored and has no effect.90s, 10m or 1h30m. Sizes like 8G or 500M.pkgproxy -check-config reads every flag and PKGPROXY_* variable, checks them and exits without starting the daemon. The snap's configure hook runs it on every snap set, so a bad value fails snap set and the running daemon is not touched. It refuses:
revalidate-ttl=ten, an unknown offline-mode or auto-offline word, or a bad size;addr or lxd-tls-addr that is not host:port or :port, a port outside 0 to 65535, or a host that is neither an IP address nor a host name (whether the name resolves is found only when the daemon starts);revalidate-ttl, or one side's TTL, of zero or less;trusted-networks entry that is neither an address nor a network, or a network of every address (0.0.0.0/0, ::/0);-offline together with an -offline-mode that names a different mode;The configure hook compares the settings the daemon would start with, leaving out offline-mode, with the copy it saved last time in /var/snap/pkgproxy/current/configured.env.
| Case | What happens |
|---|---|
| The daemon is not running | nothing; the new values apply when it starts |
Only offline-mode changed, or nothing | pkgproxyctl offline mode -keep-automatic MODE switches the running daemon; no restart |
| Anything else changed, the copy is missing, or the switch fails | the service restarts |
pkgproxyctl offline also writes offline-mode back as a snap option, without running the hook, so the mode survives a restart.
| Setting | Variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-addr | PKGPROXY_ADDR | addr | :3142 | address of the main HTTP port, for every client and the setup page |
-lxd-tls-addr | PKGPROXY_LXD_TLS_ADDR | lxd-tls-addr | :3143 | address of the HTTPS port lxc needs for image servers; off turns it off |
-socket-path | PKGPROXY_SOCKET_PATH | flag or environment only | (none) | listen on this UNIX socket instead of -addr; setup still names -addr |
-lxd-tls-socket-path | PKGPROXY_LXD_TLS_SOCKET_PATH | flag or environment only | (none) | serve HTTPS on this UNIX socket instead of -lxd-tls-addr |
-content-socket-path | PKGPROXY_CONTENT_SOCKET_PATH | pinned by the snap | (none); snap: $SNAP_COMMON/sockets/pkgproxy.sock | also serve the main port on this UNIX socket, to share the cache with another snap |
-lxd-tls-content-socket-path | PKGPROXY_LXD_TLS_CONTENT_SOCKET_PATH | pinned by the snap | (none); snap: $SNAP_COMMON/sockets/pkgproxy-lxd-tls.sock | the same for the HTTPS port |
-stats-socket-path | PKGPROXY_STATS_SOCKET_PATH | flag or environment only | /run/pkgproxy/stats.sock; snap: $SNAP_COMMON/stats.sock | UNIX socket that pkgproxyctl stats, open and setup read |
-admin-socket-path | PKGPROXY_ADMIN_SOCKET_PATH | flag or environment only | /run/pkgproxy/admin.sock; snap: $SNAP_COMMON/admin.sock | UNIX socket that pkgproxyctl cache and offline change pkgproxy through |
-setup-page | PKGPROXY_SETUP_PAGE | setup-page | true | serve the setup page at / and its scripts under /setup/; false also hides the applications menu entry |
-public-url | PKGPROXY_PUBLIC_URL | public-url | (the address the visitor used) | http://host[:port] the setup page tells visitors to use |
-trusted-networks | PKGPROXY_TRUSTED_NETWORKS | trusted-networks | (none: this machine only) | networks in CIDR notation, separated by commas, such as 192.168.0.0/24,fd00::/8, whose visitors see the cache page and what the cache holds as visitors on this machine do; see The cache page |
-read-header-timeout | PKGPROXY_READ_HEADER_TIMEOUT | read-header-timeout | 30s | how long a client may take to send its request headers |
-shutdown-grace | PKGPROXY_SHUTDOWN_GRACE | shutdown-grace | 10s | how long requests in progress get to finish when pkgproxy stops |
See One listener for what each port and socket answers.
| Setting | Variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-cache-dir | PKGPROXY_CACHE_DIR | pinned by the snap | /var/cache/pkgproxy; snap: $SNAP_COMMON/cache | directory that holds the cache and the LXD certificate; it must be on a file system that keeps extended attributes (ext4, XFS, btrfs, ZFS, tmpfs from Linux 6.6), or pkgproxy refuses to start; see The store |
-fill-drain-timeout | PKGPROXY_FILL_DRAIN_TIMEOUT | fill-drain-timeout | 10m | how long a failed-hash download of an apt package, snap or LXD image keeps going to clients already receiving it; Docker layers and files are cut short |
-upstream-idle-timeout | PKGPROXY_UPSTREAM_IDLE_TIMEOUT | upstream-idle-timeout | 1m | how long an upstream may send nothing before its request fails; 0 waits forever |
-stats-checkpoint-interval | PKGPROXY_STATS_CHECKPOINT_INTERVAL | stats-checkpoint-interval | 30s | how often statistics are saved; a crash loses what changed since |
-lxd-checkpoint-interval | PKGPROXY_LXD_CHECKPOINT_INTERVAL | lxd-checkpoint-interval | 30s | how often LXD image file hashes are saved |
See The store and Stalled upstreams.
| Setting | Variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-revalidate-ttl | PKGPROXY_REVALIDATE_TTL | revalidate-ttl | 5m | how long what changes is served before it is checked again, for every side without its own value |
-apt-revalidate-ttl | PKGPROXY_APT_REVALIDATE_TTL | apt-revalidate-ttl | revalidate-ttl | the same, apt indexes alone |
-snap-revalidate-ttl | PKGPROXY_SNAP_REVALIDATE_TTL | snap-revalidate-ttl | revalidate-ttl | the same, snap details and assertions alone |
-go-revalidate-ttl | PKGPROXY_GO_REVALIDATE_TTL | go-revalidate-ttl | revalidate-ttl | the same, Go module version lists alone |
-docker-revalidate-ttl | PKGPROXY_DOCKER_REVALIDATE_TTL | docker-revalidate-ttl | revalidate-ttl | the same, Docker image tags alone |
-lxd-revalidate-ttl | PKGPROXY_LXD_REVALIDATE_TTL | lxd-revalidate-ttl | revalidate-ttl | the same, LXD image lists alone |
-min-forced-revalidate-interval | PKGPROXY_MIN_FORCED_REVALIDATE_INTERVAL | min-forced-revalidate-interval | 30s | how soon after a check an apt index is still served from the cache when a client asks for a fresh copy |
Every TTL must be longer than zero. See Immutable or revalidated.
| Setting | Variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-offline-mode | PKGPROXY_OFFLINE_MODE | offline-mode | never | never, store-overloaded or offline; switch at run time with pkgproxyctl offline |
-offline | PKGPROXY_OFFLINE | flag or environment only | (unset) | short for -offline-mode offline; -offline=false is -offline-mode never |
-auto-offline | PKGPROXY_AUTO_OFFLINE | auto-offline | network on Linux, off elsewhere | off, network (no network, airplane mode included) or airplane-mode (all radios off) |
See Offline mode.
| Setting | Variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-snap-store-url | PKGPROXY_SNAP_STORE_URL | flag or environment only | https://api.snapcraft.io | Snap Store API that snaps and assertions come from |
-snap-external-url | PKGPROXY_SNAP_EXTERNAL_URL | flag or environment only | (the address the client used) | http://host[:port] put in the download links handed to snapd |
-go-upstream | PKGPROXY_GO_UPSTREAM | flag or environment only | https://proxy.golang.org | Go module proxy that modules come from |
-go-sumdb | PKGPROXY_GO_SUMDB | flag or environment only | sum.golang.org | checksum database modules are checked against before they are cached; off caches them unchecked |
-go-base-path | PKGPROXY_GO_BASE_PATH | flag or environment only | /go | URL path of the Go module proxy |
-docker-upstream | PKGPROXY_DOCKER_UPSTREAM | flag or environment only | https://registry-1.docker.io | registry Docker Hub images come from; off turns the mirror off |
-lxd-remote | PKGPROXY_LXD_REMOTE | flag or environment only | five remotes, see LXD image servers | name=url of an image server to cache; repeatable; name= drops a default; off caches none |
-lxd-base-path | PKGPROXY_LXD_BASE_PATH | flag or environment only | /lxd | URL path image servers are served under, each at /lxd/NAME |
apt has no upstream setting: apt names its mirror in each request.
| Setting | Variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-files-allow-host | PKGPROXY_FILES_ALLOW_HOST | files-allow-host | (empty: every host) | comma-separated hosts files may come from; .example.org allows every host under it; * allows all |
-files-allow-network | PKGPROXY_FILES_ALLOW_NETWORK | files-allow-network | (empty) | comma-separated private networks in CIDR form that files may come from; other private addresses are refused |
-files-max-size | PKGPROXY_FILES_MAX_SIZE | files-max-size | 8G | largest file fetched; 0 means no limit |
See Files by checksum.
How the cache page is built from a scan of the cache, the recorder's activity and the access log's live events, and who may see it.
withSetupPage serves /pkgproxy/cache,
/pkgproxy/cache/activity and
/pkgproxy/cache/events to local and trusted visitors from
setupPageCache: a cacheusage.Monitor,
stats.Recorder.Activity and the liveHub.
cacheusage.Scan sums allocated blocks, as du
does; Monitor.Status walks when the last report is older than
DefaultMaxAge (1 min) and waits DefaultWait (1 s).
Report.Blocks is cut to BlockLimits, five times
the SVG treemap's, which is what a browser without JavaScript gets.
stats.Recorder.Activity returns one Activity per
store key; newCacheActivity sums them by block, by the same
group and leaf IDs the layout gives, falling back to the group's rest,
then the kind's. activityTTL is 5 s.
reqlog.Wrap(next, logger, observers...) calls each
Observer with StepBegin,
StepOutcome and StepEnd, from the goroutine
serving the request, so an observer must not block.
reqlog.KindTunnel names a tunnel, its key the host and port.
liveHub.send never waits: a full queue
(liveBuffer) counts the event as lost. Lookups take at most
2 s, 8 at a time, and are kept 10 minutes; past 1,024 clients the old
ones are forgotten. An idle stream says it is alive every 20 s.
setupPageCache.mayView checks the listener,
forwardingHeaders, the unmapped remote address against
loopback, ownAddrs and -trusted-networks, and
the Host (namesMachine) against DNS rebinding.
The flag refuses 0.0.0.0/0 and ::/0; anyone
else is passed on to the gateway.
Next: The cache page, for what an administrator sees; A request in code, for where the events start; Store and keys, for the keys the blocks are sorted by.
All on the -addr listener, served by withSetupPage (only with -setup-page), for GET and HEAD, to a local visitor or one from -trusted-networks only. Anyone else's request goes on to the gateway and gets its 404. The handler name in the access log is setup-page.
| Path | Constant | Answer | Headers |
|---|---|---|---|
/pkgproxy/cache | cachePagePath | The cache page: HTML (cachepage.html.tmpl, with the map's JSON and cachepage.js) when Accept prefers it, otherwise the same figures as plain text through guide | Cache-Control: no-store, Vary: Accept, Accept-Language, Host, Content-Language |
/pkgproxy/cache/activity | cacheActivityPath | application/json: a cacheActivity, or null when the recorder has no activity | Cache-Control: no-store |
/pkgproxy/cache/events | cacheEventsPath | text/event-stream: one liveEvent per data: line; a HEAD gets the headers only; passed to the gateway when the hub is nil | Cache-Control: no-store |
/ | setupPagePath | The setup page; for a local or trusted visitor it also carries a cacheSummary and a link to the cache page | Cache-Control: no-cache, no-store for a local or trusted visitor |
setupPageCache.mayView(r) is true only when all of these hold, for the page, its activity and its events alike:
Forwarded, Via, X-Forwarded-For, X-Forwarded-Host, X-Forwarded-Proto, X-Forwarded-Port, X-Forwarded-Server, X-Real-Ip, True-Client-Ip, Cf-Connecting-Ip;isOwnAddr) or in a trusted network (isTrustedAddr);Host is an IP address, localhost, this machine's name or the host of -public-url (namesMachine), against DNS rebinding.-trusted-networks (trustedNetworksFlag; snap option trusted-networks; empty by default) takes CIDR prefixes separated by commas, a single address as a host prefix, through parseNetworks, which also takes an IPv4-mapped network as the IPv4 network it maps. A prefix of length 0 (0.0.0.0/0, ::/0) is refused. When any is set, the daemon logs showing the cache page to visitors from trusted networks with networks once at start. newSetupPageCache gets the prefixes and -public-url's host.
{"now": <Unix seconds>, "leaves": [<leafActivity>…]}. The series end with the minute, hour and UTC day now is in.
| JSON key | Field | Meaning |
|---|---|---|
g | Group | The group's ID, or the kind alone for what of a kind is not drawn as a group |
id | Leaf | The leaf's ID; absent for a group's or a kind's rest |
c | Content | Content rather than indexes or metadata |
l | Last | Last served, Unix seconds |
h | Hits | Hits in the last hour, day, week, all time |
s | Saved | Bytes saved in the same four windows |
mi, ho, d | Minutes, Hours, Days | Sparse series: pairs of an index and a count, over 60 minutes, 24 hours and 60 days |
IDs join a kind, then a label and a name per level, with \x1f (groupID, leafID): the same IDs the map's blocks carry, so the page finds a leaf's block however it was laid out.
cacheMapData, embedded in the page: kinds (kind, name, size, r, rest), groups (k, id, name, full, size, r, rest), blocks (k, g, id, name, size, r, meta), text (translated strings), activity (the first cacheActivity), live (cacheLiveData: events, upstreams, logos, text; absent without the hub), size, free. r is a rectangle's corners in a ground of cacheMapWorld (1000) units, to a hundredth.
t | Sent when | Other keys |
|---|---|---|
hi | the stream starts | at |
busy | already liveMaxWatchers pages watch; preceded by retry: 30000, and the stream ends | none |
b | reqlog.StepBegin | id, at, c (client address, or socket), n (name, once known), k (kind), g and l (block IDs) or h (a tunnel's host), u (upstream host) |
o | reqlog.StepOutcome | id, at, o |
e | reqlog.StepEnd | id, at, o (final), b (bytes) |
n | a client's reverse lookup found a name | c, n |
lost | the queue had been full and has just drained | lost: how many events were dropped |
An idle stream gets : keepalive every liveKeepalive. Nothing is encoded while no page watches.
| Name | Value | What it bounds |
|---|---|---|
liveBuffer | 512 | Events queued per watching page before they are lost |
liveMaxWatchers | 4 | Pages watching at once (a browser opens at most 6 connections to one host) |
liveBusyRetry | 30s | When a page told busy tries again |
liveKeepalive | 20s | Keepalive comment on an idle stream |
liveLookupTimeout | 2s | One reverse lookup of a client |
liveMaxLookups | 8 | Lookups at once |
liveNameTTL | 10m | How long a client's name is kept |
liveMaxClients | 1024 | Clients' names kept before the old ones are forgotten |
liveNamesTTL | 5s | How often the names of snaps, registry blobs and files are read again from the recorder |
activityTTL | 5s | How long one encoded activity answer is shared |
poll() in cachepage.js | 10s, 60s while the tab is hidden | How often the page fetches the activity |
cacheusage.DefaultMaxAge | 1m | Age of the last scan after which a page view starts another |
cacheusage.DefaultWait | 1s | How long a page view waits for a scan in progress |
MapBudget, MinKindBudget, MaxGroups, MinShareOf | 300, 32, 48, 8 | MapLimits, the SVG treemap a page without JavaScript shows: at most 800 rectangles |
BlockLimits | 5 × MapLimits | Report.Blocks, what the 3D map and the activity use |
| Name | What it is |
|---|---|
Scan(ctx, dir, names) (*Report, error) | Walks the cache directory, no symlinks, no other file systems, summing allocated blocks as du does; a cache from before the metadata attribute has each object counted with its .meta.json |
Tally, NewTally, Tally.Add, Tally.Report | The summing behind Scan |
Report | Total; Kinds within MapLimits, Blocks within BlockLimits, largest first |
Kind, Group, Leaf, Rest | A kind of download, an archive, snap, module, image, repository or file name, its members, and what a level leaves out (Count, Size) |
Limits | Budget, MinKindBudget, MaxGroups |
Monitor, NewMonitor, Monitor.Status, Monitor.Last | The last report, rescanned in the background under the daemon's context; Status has Report, Err, Age, Measuring |
Classify(kind, key, names) (group, leaf Ident) | A store key to its group and leaf |
Ident, Label | A name, or a label standing in for one: LabelIndexes, LabelUnknownSnap, LabelSnapMetadata, LabelUnknownRepo, LabelRegistryMetadata, LabelOther, LabelUnknownFile |
Names | What the recorder knows of objects kept by hash: Snaps, DockerRepos, Files |
DiskSpace(dir) (Space, error) | Free (for unprivileged users) and total space of the file system |
AptDir … FilesDir, KindOther | The store subdirectories, and the kind of everything else under the cache directory |
| Kind | Group | Leaf |
|---|---|---|
| apt | archive, <host>/<prefix> | source package of a pool file; dists/** under LabelIndexes |
| snap | snap name | revision; everything but blobs under LabelSnapMetadata |
| gomod | module path | version (its .info, .mod and .zip) |
| lxd | <remote>/<image path> | serial |
| docker | repository the blob was served through | blob; manifests and tags under LabelRegistryMetadata |
| files | name it was served under | <algo>:<short hex> |
How internal/fill lets one download serve every client that asks for the same file while it downloads, and what happens when the file fails its hash check.
A handler calls Group.Serve with a fill.Spec:
Fetch fills the Store (often a
fill.Download), and Cached,
Streaming and Failed are the side's own answers.
Small objects go through internal/flight instead.
fill.Group maps keys to *Fill under one mutex.
join joins a fill, or finds the object in the
Store, or starts run under
context.WithoutCancel. Commit happens before the fill leaves
the table.
An ObjectWriter that is also an io.ReaderAt can
be tailed. progressWriter advances written and
closes the changed channel; readers wait on it, then switch to
OpenRange on the committed object.
Serve hands httprange.Serve an
Entity whose Open(offset, length) returns a
tailing reader; its FlushHeaders sends the
status at once. With Spec.Size there are no validators yet,
so an If-Range fails closed and the whole body is sent.
Without a tail, serveAfterwards calls Cached.
join starts go g.run(context.WithoutCancel(ctx), …);
each reader selects on its own r.Context().
idletimeout.Transport bounds the fetch by inactivity
(-upstream-idle-timeout, default 60s), not total time.
Fill.Abandon(timeout) forgets the fill, wakes its readers and
drains. Apt, snap and LXD pass -fill-drain-timeout
(fill.DefaultDrainTimeout, 10m, when unset). Spec.HoldLastByte reads the last byte
only from the committed object.
Next: Store and keys, for the store a fill writes; How it is tested, for joins and timeouts in a bubble; Fetch once, for the behaviour from outside.
fill.Group maps keys to *Fill under one mutex; join holds it while it looks up the map and, on a miss, calls Store.Stat.
| State | How it got there | What a new request gets |
|---|---|---|
| Not in the map, not in the Store | nothing fetched it, or a fetch failed | join starts go g.run(context.WithoutCancel(ctx), …); this request is the starter (started) |
| Not in the map, committed in the Store | a fill committed and left the map | Cached(w, r, meta, true) |
| In the map, headers not in yet | the fetch has not called Begin | waits on ready; with Spec.Size > 0 it answers at once instead |
| In the map, tailable | Begin got a known size and a writer that is an io.ReaderAt | Streaming(w, r, meta, !started), then httprange.Serve reading from a tailing reader |
| In the map, not tailable | unknown size, or a writer that cannot be read back | waits for done, then Cached(…, !started) or Failed (serveAfterwards) |
| Abandoned | the fetch called Abandon: forgotten from the map, closing set | starts a new fill; the readers already following keep reading until the drain timeout |
| Done | run forgets the key, then finish closes done | as the first two rows: committed, or missing again |
A fill commits to the Store before it leaves the map, so a key found in neither is really missing.
| Name | Kind | What it does |
|---|---|---|
Group | type | The fills in progress for one Store, by key; zero value ready |
Group.Serve(w, r, Spec) | method | Answers from the Store, or joins or starts the fill and answers once the upstream's headers are in (200, 206 for a Range resume, 304) |
Spec | type | What Serve needs from a side (below) |
Fetch | func type | func(ctx, *Fill) error: download into the Store, Begin once the headers are in, write through Writer, commit only if it verifies; ctx is detached from every client |
Fill | type | One fill in progress |
Fill.Begin(meta, w) | method | Publishes the metadata (Size -1 if unknown) and, if w can be read back and the size is known, the pending bytes |
Fill.Writer(w) | method | Wraps the Store's writer so each write advances written and wakes the readers |
Fill.Err() | method | The fetch's error, once it is done |
Fill.Abandon(timeout) | method | Content not cacheable: forget the fill, let the readers already following get the bytes, wait for them up to timeout, then cut off stragglers |
Download | type | A Fetch for content-addressed objects at a plain URL |
Download.Fetch(ctx, f) | method | GET, Create, Begin, hash while copying, commit only if the hash (and size) match |
Passthrough(w, r, client, url) | func | A fresh, uncached, streamed GET for a client whose fill failed verification before anything was streamed |
BodyLength(r, meta) | func | How many bytes Serve streams for r from a fill in progress |
DefaultDrainTimeout | const | 10 minutes |
ErrHashMismatch | var | Content did not hash to what it was requested by; nothing is cached |
ErrSizeMismatch | var | Content not of Download.Size; nothing is cached |
ErrTooLarge | var | Content larger than Download.MaxSize; nothing is cached |
| Field | Meaning |
|---|---|
Store, Key | Where the object is, committed or pending |
Fetch | Starts the download when the object is neither stored nor being filled |
HoldLastByte | No follower gets the last byte before the commit: a bad object ends short |
Size | Known in advance: a miss is answered without waiting for the upstream's headers |
Cached(w, r, meta, hit) | Answers from the committed object; hit is false for the request that started the fill |
Streaming(w, r, meta, hit) | Sets the side's headers and records the serve before a response streams from the fill; Serve sets the status, Content-Length and Content-Range |
Failed(w, r, err) | Answers a fill that failed before anything was streamed, or a Store lookup that failed |
| Field | Meaning |
|---|---|
Client | Nil means http.DefaultClient |
URL, Store, Key | What to fetch and where to keep it |
Algo, NewHash, Hex | The hash's name for ContentHash, its constructor, and the digest the content must have |
Now | Stamps StoredAt; nil means time.Now |
DrainTimeout | Positive: on a mismatch, Abandon(DrainTimeout); zero leaves it to Spec.HoldLastByte |
MaxSize | Positive: a larger announced size is not read, and more bytes than that fail the fetch |
Size | Positive: the content must have exactly this size; also lets followers tail an upstream that announces none |
RecordOrigin | Store the URL after redirects as OriginURL |
| Side | Key | Fetch | Size | On a bad hash |
|---|---|---|---|---|
apt (proxy.serveFilled, pool files only) | cachekey key | fetchStoreAndIndex, checked against the index | none | Abandon(-fill-drain-timeout) in storeResponseBody; a client that got nothing yet is answered by a fresh passthrough |
snap (snapproxy) | sha3-384/<hex> | fill.Download, sha3-384, DrainTimeout from -fill-drain-timeout | from the store's answer | Abandon |
LXD (lxdproxy/blob.go) | <remote>/<path> | fill.Download, sha256 from the catalog, DrainTimeout from -fill-drain-timeout | from the catalog | Abandon |
Docker (registryproxy.serveFilled) | blobs/sha256/<hex> | fetchBlob, checked against the digest | none | HoldLastByte |
files (filesproxy.serveFill) | sha256/<hex> or sha512/<hex> | fill.Download with MaxSize (-files-max-size), Size (size=), RecordOrigin | none | HoldLastByte |
Go's .mod and .zip, Docker manifests and apt's dists/** files are not streamed: one shared fetch through flight.Group, served once it is stored.
Where pkgproxy's code and design live, and the conventions every change follows.
cmd/pkgproxy is the multi-call binary;
internal/ holds the sides (proxy is apt), the
shared path, the protocol parsers and the supporting packages. Nothing
under internal/ is importable from outside the module.
main.go routes on argv[0] with go-cmdr's
router.Cmd; ctl.go builds
pkgproxyctl's tree. Every leaf prints its help with
printUsage; tests use invoke(t, "pkgproxyctl", …).
docs/design.md holds §1, §12 and §18 and the index; each
area's file holds its own §, cited as docs/design/lxd.md §17.11;
go test ./docs guards the published site.
Issues and milestones live in GitLab (glab issue list), not
in a TODO file. git config core.hooksPath githooks, then
git commit -s; ./check.sh before pushing.
Next: A request in code, for one request through the packages; How it is tested, for what a change needs; Translations, for errors and messages that translate.
| Path | What it holds |
|---|---|
cmd/pkgproxy/ | The one multi-call binary: the daemon pkgproxy and the command line pkgproxyctl, a symlink to the same file |
internal/ | Every other package, one per concern; not importable from outside module gitlab.com/zygoon/pkgproxy |
internal/integration/ | Tests only: podman runs of real apt and go (skipped under -short or without podman), and snapd/, a nested Go module that drives snapd's own store client |
data/ | Assets embedded with //go:embed: the icon (also the snap's), the upstream logos of the cache page's live views, the Ubuntu mark |
docs/ | The design (design.md, design/*.md), statsapi.md, statsapi-internals.md, testing.md, troubleshooting.md, and docs_test.go, which guards the published site |
snap/ | snapcraft.yaml, the configure hook, local/bin/pkgproxy-env (snap options to PKGPROXY_* variables), the desktop file |
tests/ | The spread suite: tests/<client>/<task>/task.yaml, helpers in tests/lib/, fixtures for the snapcraft, rockcraft and charmcraft packs |
testdata/apt/ | Release, InRelease, Packages and Sources excerpts of Debian and Ubuntu, for the parsers' unit tests |
githooks/ | pre-commit, enabled with git config core.hooksPath githooks |
.dagger/ | The Dagger module whose checks ./check.sh and GitLab CI run |
spread.yaml, .image-garden.mk | The spread project and the VM images image-garden makes for it |
check.sh | exec dagger check |
DEPLOYMENT.md, README.md, NEWS | Published at https://zygoon.pl/pkgproxy/ with docs/ |
| Command | File |
|---|---|
pkgproxy (the daemon) | cmd_pkgproxy.go, with filesflags.go and envflags.go (PKGPROXY_* defaults) |
pkgproxyctl (the router) | ctl.go |
pkgproxyctl stats | cmd_stats.go |
pkgproxyctl setup | cmd_setup.go; one child each: cmd_setup_apt.go, _snap, _go, _docker, _lxd, _lxd_remote, _snapcraft, _rockcraft, _charmcraft, _status |
pkgproxyctl cache invalidate | cmd_cache.go |
pkgproxyctl offline | cmd_offline.go |
pkgproxyctl open | cmd_open.go |
pkgproxyctl version | cmd_version.go, with version.go |
Files without the cmd_ prefix hold what several commands share:
| File | What it holds |
|---|---|
main.go | The router.Cmd on argv[0], the locale and the translator |
usage.go | printUsage, the help layout of every leaf command (docs/design/cli.md §14.3) |
guide.go | The guide builder: translated paragraphs and verbatim code blocks |
instructions.go | Client configuration paths and the startup banner |
host.go | host, what commands take from outside, which tests replace |
setuppage.go, setuppage_cache.go | The setup page, its scripts under /setup/, and the cache page |
cachemap.go, cachelive.go, cachepage.js | The cache page's map, its activity and its live events |
lxdtls.go, lxdremote.go, lxdbridge.go, lxdhook.go | The LXD HTTPS listener and certificate, -lxd-remote, the bridge and profile of setup lxd |
statssocket.go, snapoption.go | Default socket and cache paths; reading snap options |
signals_*.go, umask_*.go | Platform parts: shutdown signals, the umask around socket creation |
| Package | Purpose |
|---|---|
proxy | apt: the forward proxy, cache key, store, origin and index wired into one handler (§7.5, §8) |
snapproxy | snap: a reverse relay snapd talks to through SNAPPY_FORCE_API_URL, caching snaps and assertions (§11) |
gomodproxy | Go: a GOPROXY origin chained to an upstream module proxy, with checksum database checks (§16) |
registryproxy | Docker: the read-only OCI distribution API for anonymous pulls (§19) |
lxdproxy | LXD: simplestreams image servers under <base>/<name> (§17) |
lxdapi | The part of LXD's REST API that LXD 6.10 and later download images through (§17.14) |
filesproxy | Files by checksum: GET /files/v1/fetch?url=…&sha256=… (§24) |
| Package | Purpose |
|---|---|
gateway | Dispatches each request to one side on the one listener |
reqlog | Wraps a listener's handler: one access log line per request, and events for observers (§23) |
offline | The offline switch: modes, per-host breakers, the transport, dial and middleware wrappers (§21) |
fill | Serves a large blob while it downloads into the store (§8.1) |
flight | singleflight with typed results and a joined bit |
revalidate | One detached revalidation per key, bounded wait, recent-failure memory |
store, store/fslocal, store/memstore | The store interface, its disk implementation, its in-memory one for tests (§7.1) |
cachekey | apt request URL to cache key and policy (§4, §7.3) |
index | Expected hashes of apt pool and by-hash paths from parsed indexes (§4.2, §7.4) |
lxdindex | Expected SHA256 of LXD blob paths from parsed catalogs, checkpointed to lxd-index.json |
httprange | Range, If-Range and conditional requests for apt, snap and LXD answers |
idletimeout | Fails an upstream request that makes no progress |
dialguard | Keeps apt's forward proxy from connecting to this machine for a client |
origin | Fetches from apt archives over HTTP |
| Package | Purpose |
|---|---|
apt | Release, InRelease, Packages and Sources: names, sizes and hashes only |
snap | The Snap Store API v2 knowledge the snap side acts on, and the assertion chain check |
gomod | The GOPROXY protocol's five endpoints: parsing, validation, cache keys |
simplestreams | LXD's stream index and product catalogs |
stats, statsapi | The counters and their read-only JSON API on the stats socket (§13) |
adminapi | The mutating API on the admin socket: cache invalidation, offline mode |
cacheusage | Measures the cache directory for the cache page (§20.7) |
treemap | Lays out the cache page's treemap |
hostnet | Follows whether the host is offline, for -auto-offline |
l10n | The translation catalog and the log handler that translates (§22) |
logkey, logmsg, httpheader, enumutil, relpath | Shared spellings: log attribute keys, log messages, header names, enum fallbacks, relative path validation |
docs/design.md holds §1, §12 and §18 and the index; each area has its own file under docs/design/, and the § numbers are shared by all of them. Cite a section as docs/design/lxd.md §17.11.
| § | File | Area |
|---|---|---|
| §1 | design.md | Goal and non-goals |
| §2–§8 | design/apt.md | What apt asks for, the forward proxy, normalization, policy, data model, Go interfaces (§7.1–§7.5), the fetch path (§8, §8.1) |
| §11 | design/snap.md | Snap packages and assertions (§11–§11.7) |
| §12 | design.md | Listener: TCP or UNIX socket |
| §13, §15 | design/stats.md | The stats API (§13.1–§13.6) and the default socket path |
| §14 | design/cli.md | pkgproxyctl as a multi-call binary, piped scripts, help layout (§14–§14.3) |
| §16–§16.4 | design/gomod.md | The Go module proxy and checksum database |
| §16.6–§16.10 | design/build-containers.md | LXD build containers and the admin API socket |
| §17 | design/lxd.md | LXD simplestreams image mirrors (§17–§17.14) |
| §18 | design.md | Tuning knobs: flags, environment, snap set |
| §19 | design/docker.md | The Docker Hub registry mirror (§19–§19.6.1) |
| §20 | design/setup-page.md | The setup page and the cache page (§20–§20.9) |
| §21 | design/offline.md | Offline mode |
| §22 | design/l10n.md | Localization |
| §23 | design/access-log.md | The access log and its outcomes |
| §24 | design/files.md | Files by checksum (§24–§24.6) |
A new area gets its own file, the next free § number and a line in the index in docs/design.md; a change to the design updates its document in the same commit. go test ./docs checks links and anchors, raw tags outside code, one title per file and the tables of contents.
| Rule | Where it is written |
|---|---|
Work starts from a GitLab issue in a milestone (glab issue list, glab issue create) | AGENTS.md |
Errors with l10n.D.Errorf, worded "cannot …"; log messages wrapped in i18n.MarkForTranslation | AGENTS.md, §22; see Translations |
Every commit signed off (git commit -s); CI runs dco-check | AGENTS.md, .gitlab-ci.yml |
Linters are tool dependencies in go.mod (go get -tool), run with go tool | AGENTS.md |
./check.sh before pushing; the pre-commit hook before each commit | AGENTS.md; see How it is tested |
Which packages one request passes through, from the listener to the access log line and the upstream transport.
reqlog.Wrap → offline.Middleware →
gateway.Handler → proxy.Handler:
Resolve, Store.Stat, then
fills.Serve, whose fetch commits through
Store.Create after the hash check.
route checks CONNECT or a non-empty
r.URL.Host first (apt), then the base paths, then
registryproxy.Matches, then snap's prefixes. A disabled side's
traffic gets 404 from the gateway, a registry-shaped path included.
Of the requests that share one fetch, only those that waited are
joined; the starter is fetched or
revalidated, as internal/flight reports.
offline.Switch.Transport around
idletimeout.Transport around http.DefaultTransport;
apt's base is dialguard.Guard.Transport (CONNECT dials
offlineSwitch.Dial(guard.Dialer().DialContext)), files' is
filesproxy.NewTransport. Stalls wrap
idletimeout.ErrStalled.
Next: internal/fill, for the fill a cache miss starts; The sides' packages, for what each handler does; Cache page code, for the requests watched live.
Outermost first. Each wrapper takes only its own paths and passes everything else, CONNECT and absolute-URL requests included, to the next.
| Listener | Chain |
|---|---|
-addr (or -socket-path), and -content-socket-path | reqlog.Wrap → withSetupPage (with -setup-page, the default) → withLxdCertDownload (when the LXD HTTPS listener is on) → offline.Middleware → gateway.Handler |
-lxd-tls-addr (or -lxd-tls-socket-path), and -lxd-tls-content-socket-path | reqlog.Wrap → withLxdAPI → offline.Middleware → gateway.Handler |
-stats-socket-path | statsapi.Handler: GET /pkgproxy/v1/stats, GET /pkgproxy/v1/info; no access log |
-admin-socket-path | adminapi.Handler: DELETE /pkgproxy/v1/cache, GET and PUT /pkgproxy/v1/offline; no access log |
Both traffic listeners pass the cache page's liveHub.observe to reqlog.Wrap as an observer. The snap side is wrapped once more, inside the gateway, by offline.MiddlewareAnswer(…, snapproxy.RefusedAnswer), because snapd retries a 503 for half a minute. offline.Middleware turns a 502 into a 503 when the fetch was refused for being offline.
| Wrapper | Match | Handler name |
|---|---|---|
withSetupPage | GET or HEAD of /, /setup/<tool>; for a local or trusted visitor only (setupPageCache.mayView), /pkgproxy/cache, /pkgproxy/cache/activity, /pkgproxy/cache/events | setup-page |
withLxdCertDownload | GET or HEAD of /pkgproxy-lxd-cert.pem | lxd-cert |
withLxdAPI | origin-form / or /1.0 and below (lxdapi.Matches) | lxd-api |
gateway.Handler.route, first match wins. The name is recorded with reqlog.SetHandler and is the internal/stats type, so an access log line and a stats row name the traffic alike.
| # | Match | Handler | Name |
|---|---|---|---|
| 1 | CONNECT, or a non-empty r.URL.Host (an absolute-URL request line) | Apt | apt |
| 2 | Go set and the path is -go-base-path (default /go) or below it | Go | gomod |
| 3 | Lxd set and the path is -lxd-base-path (default /lxd) or below it | Lxd | lxd |
| 4 | Files set and the path is /files or below it | Files | files |
| 5 | registryproxy.Matches: /v2/, /v2/<name>/manifests/<ref>, /blobs/<digest>, /referrers/<digest>, /tags/list, /v2/_catalog | Docker, or 404 when it is nil | docker |
| 6 | /v2, /api/v1 or /v1, or below one of them | Snap | snap |
| 7 | anything else | http.NotFound | none |
"Below" means the base path followed by /. Docker and Lxd are left a nil interface when -docker-upstream off or -lxd-remote off turned the side off; the Go and files sides are always set.
proxy.Handler.ServeHTTP:
| Request | What happens |
|---|---|
CONNECT | handleConnect: a blind tunnel, port 443 only, dialled through the dial guard |
| neither GET nor HEAD | 405 |
not an http:// absolute URL | 400 |
cachekey.PolicyImmutable | reqlog.SetObject(r, "apt", key), then serveCached; a pool miss streams through serveFilled |
cachekey.PolicyRevalidate | reqlog.SetObject, then serveCached with revalidation |
cachekey.PolicyPassthroughOnly | passthrough: relayed, not cached |
| Side | Round tripper, outermost first |
|---|---|
snap, Go, Docker, LXD (upstreamTransport) | offline.Switch.Transport → idletimeout.Transport → http.DefaultTransport |
apt (newAptHandler) | offline.Switch.Transport → idletimeout.Transport → dialguard.Guard.Transport(), inside origin.NewHTTPFetcher |
| apt CONNECT | offlineSwitch.Dial(guard.Dialer().DialContext) |
files (newFilesHandler) | offline.Switch.Transport → idletimeout.Transport → filesproxy.NewTransport(AddressPolicy{Allow: -files-allow-network}) |
The Go side's checksum database client uses the same upstreamTransport as its module fetches.
| Error | Package | Meaning |
|---|---|---|
idletimeout.ErrStalled | idletimeout | No progress for -upstream-idle-timeout (default 60s; 0 waits forever) |
offline.ErrOffline | offline | Refused: offline mode, or the host's breaker is open |
dialguard.ErrRefused | dialguard | Loopback, unspecified, or this machine's own address at -addr or -lxd-tls-addr's port |
filesproxy.ErrPrivateAddress | filesproxy | A private address outside -files-allow-network |
reqlog.Wrap(next, logger, observers...) logs one record per request, message handled request, once next returns. Level is Info, Error for a 5xx.
| Key | Value |
|---|---|
method | r.Method |
url | r.RequestURI, the target as the client sent it |
remote_addr | r.RemoteAddr |
handler | what SetHandler recorded, last call wins; absent when nothing did |
outcome | what SetOutcome recorded, last call wins; error for a 4xx or 5xx nothing labelled |
status | the response status, 200 when none was written (a hijacked tunnel) |
bytes | body bytes written, plus what AddBytes counted (a tunnel's copy) |
duration | how long next took |
| side attributes | what SetAttr added, such as assertions_fetched and chains_incomplete on a snap refresh |
response | 5xx only: the first 512 bytes of a textual, uncompressed body |
| Outcome | Meaning |
|---|---|
hit | Served from the cache without contacting the origin |
revalidated | The origin was asked about a cached entry, which was then served |
stale | The origin could not be reached; a cached entry due for revalidation was served |
fetched | Fetched, saved to the cache and served |
joined | Waited for or followed another request's fetch |
passthrough | Relayed from the origin, not cached |
synthesised | Answered by pkgproxy from what it recorded, without the origin |
tunnel | A blind CONNECT tunnel |
error | The request failed |
Only a request that named an object with SetObject or SetObjectFrom, or an open tunnel, has events. An Observer is called on the goroutine serving the request and must not block.
| Step | When | Fields set |
|---|---|---|
StepBegin | the first SetObject(r, kind, key) or SetObjectFrom(r, kind, key, origin) | ID, Time, RemoteAddr, Kind, Key, Origin |
StepOutcome | each SetOutcome after the begin | Outcome |
StepEnd | the handler returns, or panics (outcome error, panic re-raised) | final Outcome, Bytes, Status |
Kind is the stats type (apt, snap, gomod, lxd, docker, files) or reqlog.KindTunnel (tunnel), whose Key is the host and port. The cache page turns these into its live views: see Cache page code.
How each side is built from a store, a transport and shared helpers, and what the Go and Docker sides do inside.
newAptHandler … newFilesHandler in
cmd/pkgproxy, each with fslocal.New(cacheDir/<side>).
A disabled side's constructor returns nil, and the gateway field stays a
nil interface. Each handler names the object a request is for with
reqlog.SetObject, for the cache page's live views.
revalidate.Group.Do shares one detached fetch per key, bounded
by Policy.Wait (10 s) and Policy.RetryInterval
(1 min). The snap and Docker sides serve stale entries with their own code.
gomod.Request names the endpoint; Endpoint.Immutable
picks the path. fetchAndStoreImmutable runs under
flight, hashes with dirhash and commits after
verifyAgainstSumdb; serveTTL uses
revalidate.ServeDoc. See design §16.
tags/<name>/_/<tag>/<id> holds a digest;
manifests/sha256/… and blobs/sha256/… hold
immutable content. revalidateTag uses headTag;
a stale pointer is served when the registry cannot be asked.
Next: internal/fill, for the fills blob sides stream through; Store and keys, for the store every side writes; Go module proxy, for the Go side from outside.
RunFlag calls the constructors in this order, after opening the stats recorder, and before it listens. Each opens fslocal.New(-cache-dir/<dir>) and passes the logger, stats.Recorder and the offline switch.
| Side | Package and type | Constructor | Store dir | Gateway field |
|---|---|---|---|---|
| apt | proxy.Handler | newAptHandler | apt/ | Apt |
| snap | snapproxy.Handler | newSnapHandler | snap/ | Snap, inside offline.MiddlewareAnswer |
| Go | gomodproxy.Handler | newGoHandler | go/ | Go, at -go-base-path |
| LXD | lxdproxy.Handler | newLxdHandler | lxd/ | Lxd, at -lxd-base-path; nil when off |
| Docker | registryproxy.Handler | newDockerHandler | docker/ | Docker; nil when off |
| files | filesproxy.Handler | newFilesHandler | files/ | Files, at /files |
| Side | Upstream | Flag and default | Other settings |
|---|---|---|---|
| apt | whatever host the client's URL names | none | -apt-revalidate-ttl, -min-forced-revalidate-interval, -fill-drain-timeout |
| snap | the Snap Store API, and the CDN URLs it hands out | -snap-store-url https://api.snapcraft.io | -snap-external-url, -snap-revalidate-ttl, -fill-drain-timeout |
| Go | a module proxy | -go-upstream https://proxy.golang.org | -go-sumdb sum.golang.org (or off), -go-base-path /go, -go-revalidate-ttl |
| Docker | one registry | -docker-upstream https://registry-1.docker.io (off turns the side off) | -docker-revalidate-ttl |
| LXD | named simplestreams servers | -lxd-remote name=url, repeatable (off turns the side off) | -lxd-base-path /lxd, -lxd-revalidate-ttl, -lxd-checkpoint-interval, -lxd-tls-addr, -fill-drain-timeout |
| files | whatever URL the client names | none | -files-allow-host, -files-allow-network, -files-max-size (8G) |
Every -<side>-revalidate-ttl defaults to -revalidate-ttl, 5 minutes. The default LXD remotes: pkgproxy-images (images.lxd.canonical.com), pkgproxy-ubuntu and pkgproxy-ubuntu-daily (cloud-images.ubuntu.com /releases, /daily), and craft-providers' craft-com.ubuntu.cloud-buildd and craft-com.ubuntu.cloud-buildd-daily (/buildd/releases, /buildd/daily).
| Side | Kept forever (immutable) | Revalidated after its TTL | Not cached |
|---|---|---|---|
| apt | pool/**, dists/**/by-hash/**, checked against the hashes the parsed indexes list | other dists/**: Release, InRelease, Packages, Sources | URLs without dists or pool; CONNECT tunnels |
| snap | .snap files by sha3-384 | info, find, catalogue answers and assertions | answers to a user's macaroon that show private snaps; anything on a brand store's device session |
| Go | .mod and .zip of a version, checked against the checksum database before commit (unchecked with -go-sumdb off) | @v/list, @latest, .info | a path that is not one of the protocol's five endpoints |
| Docker | blobs and manifests by sha256 digest, checked against it | tags: a pointer to a digest per set of accepted media types, checked with a HEAD | methods other than GET and HEAD (405); tag lists and the catalog (registry 404) |
| LXD | image files by the sha256 their product catalog lists | streams/v1/index.json and product catalogs | a path no parsed catalog listed: 404, never fetched |
| files | the file by sha256 or sha512, whatever its URL | nothing | a private address outside -files-allow-network; a host outside -files-allow-host; a file over -files-max-size |
| Side | fill.Group (large blobs, streamed) | flight.Group (one shared fetch) | revalidate.Group (bounded revalidation) | Stale entries on upstream failure |
|---|---|---|---|---|
| apt | pool files | dists/**, and the passthrough fallback | yes, RevalidateWait, OriginRetryInterval | yes |
| snap | .snap files | relayed metadata fetches, and assertions fetched ahead | no | its own code |
| Go | no | .mod and .zip | yes, through revalidate.ServeDoc | yes |
| Docker | blobs | manifests | no | its own code: a stale tag pointer |
| LXD | image files | no | yes, through revalidate.ServeDoc | yes |
| files | files | no | no | not applicable |
revalidate.Policy: Wait defaults to revalidate.DefaultWait (10s), after which Do returns ErrSlow; RetryInterval to DefaultRetryInterval (1m), within which a host that failed gets ErrFailing without a fetch; Epoch is the offline switch's, so a change of mode forgets failures.
newDockerHandler returns nil for -docker-upstream off, newLxdHandler nil for -lxd-remote off. RunFlag assigns them to the gateway only when non-nil: a nil *lxdproxy.Handler stored in an http.Handler field would be a non-nil interface, and the gateway's h.Lxd != nil check would call through it. With Docker off, a registry-shaped path still matches and gets 404; with LXD off, /lxd falls through to the snap prefixes and then 404.
The store interface, its metadata and commit, and how apt's cache keys and policies reach it.
internal/store defines Store and
ObjectMeta; fslocal.New is called once per side
on -cache-dir/<side>; memstore serves the
tests. Keys: cachekey.PathResolver for apt, simple prefixes
such as sha3-384/ or blobs/sha256/ elsewhere.
store.Store: Stat, Open,
OpenRange, Create (an ObjectWriter
with Commit/Discard, optionally
io.ReaderAt), UpdateMeta, Delete.
fslocal on disk, memstore in tests.
store.ObjectMeta, marshalled with encoding/json
into the user.pkgproxy.meta xattr of
objects/<key>, gzipped above 3 KiB (64 KiB of JSON at
most; OriginURL is dropped if it would not fit). ContentHash is
<algo>:<digest> in the side's own terms.
fslocal: tmp/obj-<hex>,
setAttr, Sync, Close,
Rename to objects/<key>. A reader that
has the old file open keeps reading it. defer w.Discard() is
always safe. memstore copies its buffer on
Commit and copies out on read.
cachekey.PathResolver returns Resolved;
internal/apt parses Release and stanzas;
index.Store implements index.Lookup. The proxy
handler in internal/proxy wires them together.
PolicyImmutable commits Immutable: true;
PolicyRevalidate is checked against StoredAt and
-apt-revalidate-ttl (by default -revalidate-ttl), and touch refreshes it with
UpdateMeta on a 304. PolicyPassthroughOnly goes
to passthrough. cachekey.KeyPolicy serves
stats.IsContent.
Next: internal/fill, for clients reading while the store is written; The sides' packages, for each side's keys; The store, for the cache on disk.
| Method | Contract |
|---|---|
Open(ctx, key) (io.ReadCloser, ObjectMeta, error) | The body and metadata of a committed object; ErrNotFound otherwise |
OpenRange(ctx, key, offset, length) (io.ReadCloser, ObjectMeta, error) | Bytes [offset, offset+length); length < 0 reads to the end |
Stat(ctx, key) (ObjectMeta, error) | Metadata only; use it instead of opening and discarding the body |
Create(ctx, key) (ObjectWriter, error) | A writer whose object is visible to Open and Stat only after Commit; a Commit over an existing object switches body and metadata together, and a reader that has the old body keeps reading it |
Delete(ctx, key) error | Removes the object and its metadata; a reader that has it open keeps reading it |
UpdateMeta(ctx, key, meta) error | Replaces a committed object's metadata, body untouched (a 304); an error wrapping ErrNotFound if there is none |
| Other API | Contract |
|---|---|
ObjectWriter | io.Writer, Commit(ctx, meta) error, Discard() error; Discard after Commit is a no-op, so defer w.Discard() is always safe |
ObjectWriter that is also io.ReaderAt | Reads back the bytes written so far, concurrently with writes, for a fill in progress; may fail once Commit or Discard returned |
store.Put(ctx, st, key, data, meta) | Create, Write, Commit of a small in-memory object, discarding on failure; sets meta.Size |
store.ErrNotFound | Returned by Open, OpenRange and Stat for a key with no committed object |
| Field | Type | Set by | Meaning |
|---|---|---|---|
Size | int64 | every side | Body length |
ETag | string | sides that relay it | The origin's ETag |
LastModified | time.Time | sides that relay it | The origin's Last-Modified, sent back when revalidating |
StoredAt | time.Time | every side | When it was stored or last revalidated; TTLs count from it |
ContentHash | string | apt on every entry; the other sides when stored | <algorithm>:<digest> in the side's own terms: SHA256:<hex> (apt's index names), sha3-384:<hex> (snaps), sha256:<hex> (LXD, registry digests, files, Go's mutable endpoints), h1:<base64> (Go's dirhash) |
Immutable | bool | content-addressed entries | Never revalidated |
ContentType | string, omitempty | Docker manifests | The origin's media type, where the key cannot tell it |
MaxAge | time.Duration, omitempty | when the origin announces one | Cache-Control: max-age that differs from the side's TTL |
OriginURL | string, omitempty | files by checksum | The URL fetched, after redirects; dropped by fslocal when the metadata would not fit otherwise |
fslocal.New(dir) creates dir, objects/ and tmp/ (mode 0755), opens an os.Root on dir, so no key and no symlink can reach outside it, and probes for extended attributes by setting user.pkgproxy.meta on a tmp/probe-<hex> file; it fails on a file system without them (tmpfs before Linux 6.6, NFSv3, FAT, and every directory on a system other than Linux), and the daemon then refuses to start. The daemon calls it once per side, on -cache-dir/<side>.
Store.MissingMeta, when set, is called with the key of an object file that has neither the attribute nor a .meta.json: what a copy of the cache that dropped extended attributes leaves. cmd/pkgproxy sets it on every side's store to warnMissingMeta (cachestore.go), which logs once per store, so a restored cache that looks empty says why.
<-cache-dir>/ default /var/cache/pkgproxy
apt/ snap/ go/ docker/ lxd/ files/
objects/<key> the body; ObjectMeta in its user.pkgproxy.meta xattr
objects/<key>.meta.json ObjectMeta of a cache from before the xattr; read, never written
tmp/obj-<32 hex> a body being written
go/sumdb/ the checksum database client's own cache
lxd/tls/ the LXD HTTPS listener's certificate
stats.json the stats recorder's checkpoint
lxd-index.json the LXD blob hashes' checkpoint
| Operation | On disk |
|---|---|
Create | tmp/obj-<hex>, opened O_EXCL, mode 0644; ten tries at a fresh random name |
Commit | encodeMeta, setAttr on the temporary file, Sync, Close, MkdirAll of the key's directory, Rename to objects/<key>, then removes a leftover <key>.meta.json |
Discard | Closes and removes the temporary file; nothing after Commit |
Stat, Open, OpenRange | Open objects/<key> and read the metadata from that open file: the attribute, or <key>.meta.json when it has none; a file with neither, or not a regular file, is ErrNotFound, and a file with neither is also passed to MissingMeta |
UpdateMeta | Replaces the attribute in place on the open file (fsetxattr, atomic), body untouched, then removes a leftover <key>.meta.json; lost if a Commit replaced the file meanwhile, as the new object has its own |
Delete | Removes the object, then a leftover <key>.meta.json; a missing file is not an error |
The attribute holds JSON as it is up to maxMetaSize, 3072 bytes (ext4 keeps all of a file's attributes in one 4 KiB block); larger JSON, up to maxMetaJSON, 64 KiB, is compressed with gzip, whose magic bytes no JSON starts with. Metadata that still does not fit loses OriginURL, and is refused if it is still too large. getfattr --only-values -n user.pkgproxy.meta FILE | zcat shows a compressed value.
A key is a slash-separated relative path. cleanKey rejects one that relpath.Clean cleans to ., .., ../… or an absolute path, before os.Root enforces containment. memstore keeps committed objects in a map under a mutex, buffers a writer until Commit, and copies on the way in and out.
| Side | Store directory | Keys |
|---|---|---|
| apt | apt/ | <archive id>/dists/… and <archive id>/pool/…, from cachekey |
| snap | snap/ | sha3-384/<hex> (snaps), pending/sha3-384/<hex> (hash to CDN URL and size), info/<name>…, find/<hash>, assertions/<type>/<key>…, catalog/<hash>, refresh/… |
| Go | go/ | the GOPROXY URL space: <escaped module>/@v/<escaped version>.info, .mod, .zip, <module>/@v/list, <module>/@latest |
| Docker | docker/ | blobs/sha256/<hex>, manifests/sha256/<hex>, tags/<name>/_/<tag>/<id> (a digest per set of accepted media types) |
| LXD | lxd/ | <remote>/<path>, such as <remote>/images/<os>/<release>/<arch>/<variant>/<serial>/<file> |
| files | files/ | sha256/<hex> or sha512/<hex> |
cachekey.PathResolver{}.Resolve(u) finds the first dists or pool path segment. The archive id is the host (with its port unless it is 80) plus every non-empty segment before that one; the key is <archive id>/<archive path>.
| Path class | Policy | Store contract |
|---|---|---|
…/pool/** | PolicyImmutable | Committed with Immutable: true, only if it matches the hash the parsed indexes list (a path they do not list is cached with a warning); streamed through a fill on a miss |
…/dists/**/by-hash/<ALGO>/<hex> | PolicyImmutable | As above; the hash comes from the URL (KnownHash), no index lookup |
other …/dists/** (Release, InRelease, Packages, Sources, …) | PolicyRevalidate | Committed with the origin's LastModified; once StoredAt is older than -apt-revalidate-ttl (default -revalidate-ttl, 5m) a conditional GET goes out, and a 304 calls UpdateMeta |
no dists or pool segment | PolicyPassthroughOnly | No key; relayed, never stored |
| Example URL | Key |
|---|---|
http://archive.ubuntu.com/ubuntu/pool/main/a/apt/apt.deb | archive.ubuntu.com/ubuntu/pool/main/a/apt/apt.deb |
http://repo.example:8080/debian/dists/stable/Release | repo.example:8080/debian/dists/stable/Release |
http://host:80//dists/x/Release | host/dists/x/Release |
cachekey.Resolved field | Meaning |
|---|---|
Key | <ArchiveID>/<ArchivePath>; empty for passthrough |
ArchivePath | From dists/ or pool/ on; what index.Lookup and Release entries key on |
ArchiveID | Host (and port) plus path prefix, such as archive.ubuntu.com/ubuntu |
Policy | PolicyImmutable, PolicyRevalidate, PolicyPassthroughOnly (String(): immutable, revalidate, passthrough-only) |
OriginURL | The URL to fetch upstream |
KnownHash, HasKnownHash | The apt.Hash a by-hash path names |
cachekey.KeyPolicy(key) reads the policy back from a stored key; stats.IsContent uses it to tell content from indexes.
pkgproxy mirrors Docker Hub for anonymous pulls: it holds the token, rechecks tags, and keeps manifests and layers by digest.
pkgproxyctl setup docker | sudo sh edits
/etc/docker/daemon.json (or the Docker snap's) and restarts
the daemon. registry-mirrors takes no path, so the API is at
/v2/. Only docker.io images use it.
GET /v2/ is answered 200 by pkgproxy. On a miss, the
registry's 401 Bearer challenge is met with an anonymous
token, cached per repository. Pushes and authenticated pulls are not
supported.
A tag pointer (tags/<name>/_/<tag>/…) is trusted
for -docker-revalidate-ttl (default: -revalidate-ttl, 5m), then resolved
with HEAD. Manifests and blobs are keyed
manifests/sha256/<hex> and
blobs/sha256/<hex>, verified and immutable.
A blob miss is a fill shared by every pull. Its last byte is held back until the sha256 digest matches and the blob is committed; a mismatch is not cached and its responses end short.
-docker-upstream (default
https://registry-1.docker.io, or off, after which registry paths get a 404) and
-docker-revalidate-ttl (default: -revalidate-ttl, 5m; snap option
docker-revalidate-ttl). Only pulls are served; the cache is
<cache-dir>/docker.
Next: pkgproxy for CI, for build machines pulling the same images; Statistics, for layers counted per image.
Next: Immutable or not, for what is kept and what is rechecked; Snap Store relay, the other side under /v2; pkgproxyctl setup, for the other clients.
The mirror answers at the root of port 3142. These shapes are the registry's; any other path under /v2 belongs to the Snap Store relay. Only GET and HEAD are served; any other method is a 405.
| Path | What pkgproxy does | Cache policy |
|---|---|---|
/v2/ | Answers 200 itself, with an empty JSON body: no login needed | not cached; never asks the registry |
/v2/<name>/manifests/<tag> | Resolves the tag to a manifest digest with a HEAD to the registry, then serves that manifest | the tag's digest is revalidated after docker-revalidate-ttl; the manifest is kept forever |
/v2/<name>/manifests/sha256:<hex> | Serves the manifest, fetched once and checked against its digest | kept forever |
/v2/<name>/blobs/sha256:<hex> | Serves the layer or configuration blob; a miss is one download shared by every pull, Range included | kept forever, checked while it streams in; a HEAD on a miss asks the registry and caches nothing |
/v2/<name>/referrers/sha256:<hex> | Relays the request to the registry | not cached |
/v2/<name>/tags/list, /v2/_catalog | 404 | not served |
Manifests and blobs are shared by every repository and tag that names their digest. A cached digest is served under any repository name without asking the registry. With docker-upstream set to off, every path above is a 404.
| Registry's answer while resolving a tag | Client gets |
|---|---|
| Cannot be reached | The stale manifest, if the tag resolved before, with a warning in the log |
| 404 | 404: the tag is gone, and the client falls back to Docker Hub |
| 401 or 403, the tag resolved before | The stale manifest, asked again on every request |
| 401 or 403, nothing cached | 404 |
| A failed fetch of a manifest or blob not cached | 502 |
| Setting | Flag | Environment | Snap option | Default |
|---|---|---|---|---|
The registry to fetch from, or off to turn the mirror off | -docker-upstream | PKGPROXY_DOCKER_UPSTREAM | flag or environment only | https://registry-1.docker.io |
| How long a tag's digest is trusted | -docker-revalidate-ttl | PKGPROXY_DOCKER_REVALIDATE_TTL | docker-revalidate-ttl | revalidate-ttl, 5m |
Under the docker directory of the cache (/var/snap/pkgproxy/common/cache/docker in the snap, /var/cache/pkgproxy/docker otherwise), each object with its metadata in an extended attribute of its file:
docker/objects/ blobs/sha256/<hex> layers and configuration blobs manifests/sha256/<hex> manifests and image indexes, with their media type tags/<name>/_/<tag>/<id> the digest a tag last resolved to, per set of accepted media types
Nothing there is deleted on its own. To drop it all, stop pkgproxy (sudo snap stop pkgproxy), remove the directory and start it again. See The store.
A client asks pkgproxy for a large file by URL and checksum. The checksum is the cache key, and only bytes that match it are kept.
GET /files/v1/fetch?url=…&sha256=… (or
sha512, optional size). pkgproxy is the HTTPS
client and verifies the origin's certificate. X-Pkgproxy-Origin-URL
and X-Pkgproxy-Cache describe the answer.
The key is sha256/<hex> or sha512/<hex>
in <cache-dir>/files, whatever the URL. Entries are
immutable; a new file has a new checksum and misses.
A miss is a fill shared by every request, Range included. The last byte is held until the checksum, and the size if given, match; otherwise nothing is cached and every response ends short.
files-allow-host (empty: any host),
files-allow-network (empty: no private networks) and
files-max-size (8G, 0 for no limit),
as snap options or flags. The private-address check runs on every
connection, redirects included.
Next: Settings, for the files settings; The cache page, where files by checksum are a kind of their own, grouped by name.
Next: pkgproxy for CI, for many machines behind one cache; Offline mode, for files served with no network.
Next: Fetch once, for one download shared by every client; Offline mode, for files served with no network; The store, for where they are kept.
GET /files/v1/fetch?url=<URL>&sha256=<hex>[&size=<bytes>] GET /files/v1/fetch?url=<URL>&sha512=<hex>[&size=<bytes>] HEAD the same
Any other path under /files is a 404, and any method but GET and HEAD a 405.
| Parameter | Required | Value |
|---|---|---|
url | yes | An absolute http:// or https:// URL with a host, percent-encoded. No user name or password. |
sha256 | one of the two | 64 hexadecimal digits. |
sha512 | one of the two | 128 hexadecimal digits. Give one checksum, not both. |
size | no | The file's size in bytes, a positive number. Checked like the checksum, and lets clients follow a download whose server sends no length. |
| Header | Value |
|---|---|
Content-Type | application/octet-stream |
Content-Length | The file's size |
Accept-Ranges | bytes: Range requests and resumed downloads work, on a hit and while the file downloads |
X-Pkgproxy-Cache | hit from a complete cached file; miss while the file downloads, whether this request started the download or joined it |
X-Pkgproxy-Origin-URL | The URL the bytes were downloaded from, after redirects. On a hit it is the URL of the request that fetched the file. |
| Status | When |
|---|---|
| 200 or 206 | The file, from the cache or as it downloads. |
| 400 | A missing or malformed parameter; or, on a hit, a size other than the cached file's. |
| 403 | The host is not allowed by files-allow-host; size is larger than files-max-size; or a connection would go to a private address files-allow-network does not allow, after any redirect. |
| 404 | A HEAD for a file that is not cached (HEAD never contacts the origin). |
| 405 | A method other than GET or HEAD. |
| 502 | The download failed: the origin could not be fetched, more than 10 redirects, a file larger than files-max-size, or bytes that do not match the checksum or size. |
| 503 | Offline mode, for a file that is not cached; cached files are still served. |
A download that fails after its headers were sent ends short instead, so the client rejects it. On any error, the client downloads the file directly.
| Setting | Flag | Environment | Default | Meaning |
|---|---|---|---|---|
files-allow-host | -files-allow-host | PKGPROXY_FILES_ALLOW_HOST | empty: any host | Hosts clients may name, separated by commas: a host name, or a domain starting with . for every host under it. Empty or * allows every host. Checked on the URL the client names, not on redirects. |
files-allow-network | -files-allow-network | PKGPROXY_FILES_ALLOW_NETWORK | empty: none | Private networks, in CIDR notation and separated by commas, that files may be fetched from. |
files-max-size | -files-max-size | PKGPROXY_FILES_MAX_SIZE | 8G | Largest file fetched, such as 8G or 500M; 0 means no limit. |
All three are also snap options: sudo snap set pkgproxy files-max-size=20G. The private addresses refused unless allowed are loopback, private and unique local, link-local, unspecified, multicast, 0.0.0.0/8 and 100.64.0.0/10. With an HTTP proxy set in pkgproxy's environment, the proxy's address is the one checked.
Files are kept in the files directory of the cache, as files/objects/sha256/<hex> or files/objects/sha512/<hex>, each with its metadata, the origin URL among it, in an extended attribute of the file. The cache is /var/snap/pkgproxy/common/cache in the snap and /var/cache/pkgproxy otherwise. Files are never revalidated or deleted; see The store.
How one upstream download is shared by every request for a file, and what happens when it goes wrong.
Concurrent requests for one file share one upstream download. A fill
streams to every waiting client while it is written (apt's
pool/**, snaps, LXD image files, Docker blobs, files by
checksum); indexes, Go modules and manifests are served once complete.
A plain request gets 200 once the upstream's headers are in;
a Range or If-Range resume gets 206
from the same fill. With the size known in advance (snaps, LXD image
files) the headers go out at once.
A failed fetch ends every follower's response short and caches nothing;
the next request starts a new fill. -upstream-idle-timeout
(60s) bounds upstream inactivity, not total time.
Drain (apt, snap, LXD): followers get the origin's bytes, bounded by
-fill-drain-timeout (10m), for apt, snaps and LXD
images alike. Hold the last byte (Docker blobs, files): a
mismatch ends the response short.
Next: The store, for where a download is written; Stalled upstreams, for when an upstream goes silent; pool and dists, for apt's files.
Next: Stalled upstreams, for when an upstream goes silent; pool and dists, for apt's files; Immutable or not, for what is kept and what is rechecked.
Which entries are kept forever and which are rechecked after a short window, on every side.
Content addressed by hash or by version is immutable: fetched once, verified and cached forever. Indexes, version lists and tags are mutable and revalidated after a TTL. Files by checksum have no mutable part.
Inside the TTL a mutable entry is a hit. After it, apt sends
If-Modified-Since: 304 renews the entry,
200 replaces it. The other sides fetch again; Docker resolves
a tag with HEAD.
-revalidate-ttl (5m) applies to every side;
-apt-revalidate-ttl, -snap-revalidate-ttl,
-go-revalidate-ttl, -lxd-revalidate-ttl and
-docker-revalidate-ttl override it for one side each, with
snap options of the same names. Zero is refused. A snap store
max-age takes precedence; offline mode makes every window
endless.
Cache-Control: no-cache, max-age=N or
Pragma: no-cache force a conditional GET, except within
-min-forced-revalidate-interval (30s) of the last
check. A 304 keeps the entry; pkgproxyctl cache
invalidate removes one.
A failed revalidation serves the stale entry with a warning. apt, Go and
LXD wait at most 10 s for it and skip a failing host for 1 min. For apt, an origin
404/410 is relayed as 404; a hash
mismatch is passed through uncached.
Next: Offline mode, for freshness frozen; pool and dists, for apt in detail; Stalled upstreams, for when an upstream counts as stalled.
Kept forever means fetched once, checked against its hash where one is known, and never asked about again. Rechecked means served from the cache for the window in the last column, then checked with the upstream. Relayed means passed between client and upstream without being kept.
| Entry | Policy | Window |
|---|---|---|
anything under pool/, and dists/…/by-hash/… | kept forever, hash checked | none |
Release, InRelease, Release.gpg | rechecked with If-Modified-Since | apt-revalidate-ttl |
every other file under dists/ (Packages, Sources, Contents, Translation, diff indexes) | rechecked with If-Modified-Since | apt-revalidate-ttl |
a URL with no pool or dists path segment | relayed | none |
an https:// source, through a CONNECT tunnel | relayed, unseen | none |
An index with no Last-Modified from the mirror, or one a newer index already shows to be superseded, is fetched again in full instead of conditionally.
| Entry | Policy | Window |
|---|---|---|
| snap files, by their SHA3-384 | kept forever | none |
snap-revision assertions, and account, snap-declaration, store or validation-set assertions pinned to a sequence number | kept forever | none |
every other assertion, account-key included | rechecked | snap-revalidate-ttl |
| snap info, find, categories, sections, names, repairs, and the old v1 search and details | rechecked | the store's max-age, else snap-revalidate-ttl |
| refresh and install requests | relayed; each result recorded for answering offline | none |
| download redirects without a hash, such as resources | followed and streamed, not kept | none |
| logins, device sessions and serials, cohorts, metadata | relayed | none |
| any other store path | relayed | none |
The store's max-age wins whether it is shorter or longer than the window; a max-age of zero counts as none.
| Entry | Policy | Window |
|---|---|---|
.mod and .zip of a version | kept forever, after the checksum database check | none |
list, @latest and .info | fetched again in full | go-revalidate-ttl |
| a 404 from the upstream | not kept | none |
| Entry | Policy | Window |
|---|---|---|
| layers and configs, by digest | kept forever, digest checked | none |
| manifests and indexes, by digest | kept forever, digest checked | none |
| the digest a tag resolves to, per set of accepted media types | rechecked with a HEAD | docker-revalidate-ttl |
a HEAD of a layer not yet cached | answered from the registry's HEAD, not kept | none |
| referrers | relayed | none |
| Entry | Policy | Window |
|---|---|---|
streams/v1/index.json and every other .json catalog | fetched again in full | lxd-revalidate-ttl |
| an image file whose path a catalog listed | kept forever, hash checked | none |
| a path no catalog listed | answered 404, never fetched | none |
LXD's image API, on the HTTPS port, reads the same catalogs and files, so it follows the same policy.
Every file is kept forever under its checksum and never rechecked; there is nothing to revalidate.
| Setting | Default | Window for |
|---|---|---|
revalidate-ttl | 5m | every side whose own setting is unset |
apt-revalidate-ttl | revalidate-ttl | apt indexes |
snap-revalidate-ttl | revalidate-ttl | snap details and assertions without a max-age |
go-revalidate-ttl | revalidate-ttl | Go version lists, @latest, .info |
docker-revalidate-ttl | revalidate-ttl | Docker tags |
lxd-revalidate-ttl | revalidate-ttl | LXD catalogs |
min-forced-revalidate-interval | 30s | apt only: how long after a check a client's request for a fresh copy is still served from the cache |
Each is a -flag, a PKGPROXY_ variable and a snap option of the same name; see Settings. A window must be longer than zero: pkgproxy refuses 0 at start and in -check-config. In offline mode every window is endless; see Offline mode.
| Side | Cache-Control: no-cache, max-age=N, Pragma: no-cache |
|---|---|
| apt | a conditional request upstream before serving, unless the index was checked within min-forced-revalidate-interval; requests forcing one index share one check |
| snap, Go, Docker, LXD, files | ignored: the window decides |
A Pragma: no-cache counts only when there is no Cache-Control. A check confirms or replaces an index; it never deletes one. To throw a cached apt file away, use pkgproxyctl cache invalidate.
| Side | Upstream unreachable, stalled or answering 5xx | Upstream says the entry is gone |
|---|---|---|
| apt | cached copy served with a warning; waits at most 10 s, host skipped for 1 min | 404 or 410 relayed as 404 |
| Go, LXD | cached copy served with a warning; waits at most 10 s, host skipped for 1 min | cached copy served with a warning, as for any answer other than 200 |
| snap | cached copy served with a warning, also on a 429 from the store | upstream's answer relayed |
| Docker | stale tag served with a warning; a 401 or 403 also keeps a tag that resolved before | 404, and the client falls back to Docker Hub |
An outcome of stale in the access log marks each such answer.
The go command fetches modules from pkgproxy under /go. Module
files are checked against the checksum database and kept forever; version
lists are rechecked.
pkgproxyctl setup go | sh runs go env -w
GOPROXY=http://host:3142/go,direct for the user; export
GOPROXY=… does it for one CI job. ,direct is the
fallback on a 404 or 410.
@v/list, @latest and .info are
revalidated after -go-revalidate-ttl (default: -revalidate-ttl, 5m);
.mod and .zip are immutable. The upstream is
-go-upstream, https://proxy.golang.org by default.
A .mod or .zip is committed only after its
h1: hash matches sum.golang.org. A mismatch is a
502 and nothing is cached; a failed lookup is logged and cached.
-go-sumdb off skips the check.
Next: Settings, for the go flags and their defaults; The store, for where the modules are kept.
Next: pkgproxy for CI, for many machines behind one cache; Offline mode, for builds that see only what is cached.
Next: Offline mode, for builds from the cache with no network; pkgproxyctl setup, for the other clients.
Every request is a plain GET (or HEAD) on the main port, under -go-base-path, /go by default. Module paths are case-encoded as the protocol says: each capital letter is written as ! and the small letter, so github.com/BurntSushi/toml is asked for as github.com/!burnt!sushi/toml.
| Path | Answer | Cache policy |
|---|---|---|
/go/MODULE/@v/list | the known versions, one per line | revalidated after -go-revalidate-ttl |
/go/MODULE/@latest | the latest version, as JSON | revalidated after -go-revalidate-ttl |
/go/MODULE/@v/VERSION.info | a version's time and name, as JSON | revalidated after -go-revalidate-ttl |
/go/MODULE/@v/VERSION.mod | the version's go.mod | kept forever, after the checksum database check |
/go/MODULE/@v/VERSION.zip | the version's source, as a zip | kept forever, after the checksum database check |
any other path under /go | 404 | not cached |
A .mod or .zip must name a canonical version, such as v1.2.3 rather than v1.2; anything else is a 404. The checksum database's own proxy paths, /go/sumdb/…, are not served: they get a 404, and the go command then asks sum.golang.org directly, as it does with no proxy.
| Situation | Status the go command sees |
|---|---|
a method other than GET or HEAD | 405 |
| a path that is not one of the five requests | 404 |
.mod or .zip cached | 200 from the cache, also served to Range requests |
.mod or .zip not cached, the upstream answers 404 or 410 | the same 404 or 410, not cached, so ,direct takes over |
.mod or .zip whose hash differs from the checksum database, or the database fails its own integrity check | 502, nothing cached |
.mod or .zip, upstream unreachable | 502 |
| list, latest or info, cached and within the TTL | 200 from the cache |
| list, latest or info, past the TTL, upstream fails or answers an error | 200 with the cached copy, and a warning in the log |
| list, latest or info, never cached, upstream answers an error | the upstream's status, or 502 when it cannot be reached |
A 404 from the upstream is never cached, so the go command probing for a module boundary (asking for example.com before example.com/mod) reaches the upstream each time.
| Lookup result | What happens to the .mod or .zip |
|---|---|
the database lists the same h1: hash | cached |
| the database lists a different hash | not cached, 502 |
| the database's signature or tree does not verify | not cached, 502 |
| the database cannot be reached, or does not list the version yet | cached unchecked, a warning in the log |
-go-sumdb off | cached unchecked, no lookup |
Only sum.golang.org is supported, with the key the go command itself carries; any other -go-sumdb value stops pkgproxy at startup. The database is asked directly over HTTPS, and its tree head and lookups are kept under go/sumdb in the cache directory.
| Option | Environment variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-go-upstream | PKGPROXY_GO_UPSTREAM | flag or environment only | https://proxy.golang.org | the module proxy misses are fetched from |
-go-base-path | PKGPROXY_GO_BASE_PATH | flag or environment only | /go | the URL path of the module proxy, part of GOPROXY |
-go-sumdb | PKGPROXY_GO_SUMDB | flag or environment only | sum.golang.org | the checksum database, or off |
-go-revalidate-ttl | PKGPROXY_GO_REVALIDATE_TTL | go-revalidate-ttl | -revalidate-ttl, 5m | how long list, latest and info answers are served before they are checked again |
All settings are in Settings.
| Variable | Set by pkgproxyctl setup go | Meaning |
|---|---|---|
GOPROXY | http://HOST:3142/go,direct, with go env -w, for the user who runs it | ask pkgproxy first; on a 404 or 410, fetch from the module's source |
GOSUMDB | left as it is | the go command still checks every module itself |
GOPRIVATE | left as it is | modules it matches never go to any proxy |
pkgproxyctl setup go -undo runs go env -u GOPROXY, which removes whatever GOPROXY the user had set, not only pkgproxy's. For one CI job, export GOPROXY=http://HOST:3142/go,direct does the same without touching the user's settings. See Setting up clients.
How the upstream idle timeout fails a download that stops, without cutting off a slow one.
-upstream-idle-timeout (default 60s) fails a
request only after that long without progress. A 2 GB layer at 1 MB/s
takes its 33 minutes. 0 turns the check off.
The timer runs while the headers are awaited and while each read of the body waits for bytes, and is stopped between reads. Time package proxy spends passing the bytes on to a client is not the upstream's.
A shared download whose upstream stalls fails like on any upstream error and is removed from the downloads in progress. The next request for the file starts a new download rather than waiting on the old connection.
-read-header-timeout (30s),
-upstream-idle-timeout (60s) and
-shutdown-grace (10s), also as snap options of
the same names. Connecting upstream is bounded at 30s, the
TLS handshake at 10s.
Next: Fetch once, for the shared download; Immutable or not, for a stalled recheck; Offline mode, for leaving a failing upstream alone.
How errors, help, guides, pages and logs are translated through one catalog, with commands and paths kept out of it.
Prose is translated and reflowed; commands, paths and configuration stay
verbatim. TestCtlSetupApt_GuidePolish pins this output; the
setup page picks its language from Accept-Language.
l10n.D.Errorf keeps errors.Is identity for
sentinels. Error() is English; go-cmdr prints the
translation. Write "cannot …", not "failed to …" or "could not …".
g.para(…) takes prose from l10n.D.Tr;
g.code(…) takes lines that are never translated or reflowed.
The msgid has %s where the path goes.
l10n.NewLogHandler wraps the daemon's slog text
handler. A message from a library at run time passes through untranslated.
go tool i18n scan, merge -no-obsolete, translate
pl.po, then check -auto -S, all with -C
internal/l10n.
Next: How it is tested, for translated output in tests; The setup page, for the browser's language; The source tree, for the other conventions.
const: the scanner follows constants passed to Tr, TrN and Errorf.TRANSLATORS: right above the call; other comments stay out of the catalog.go tool i18n scan -C internal/l10n go tool i18n merge -C internal/l10n -no-obsolete # translate the new and fuzzy entries in internal/l10n/po/pl.po go tool i18n check -C internal/l10n -auto -S
go tool i18n is gitlab.com/zygoon/go-i18n/cmd/i18n, a tool dependency in go.mod. Scan and merge run at development time only; the binary embeds the result.
| Text | Call | Language |
|---|---|---|
| An error, sentinels included | l10n.D.Errorf("cannot …", …), never fmt.Errorf or errors.New | Error() English; go-cmdr's printer shows LocalizedError in the process's locale, wrapped causes included |
| Help, command output, page text | l10n.D.Tr(ctx, …), l10n.D.TrN(ctx, …, n) for plurals | The locale on ctx |
| A log message | logger.Info(i18n.MarkForTranslation("…"), …) | Translated by l10n.NewLogHandler as it is written |
| Setup guidance | g.para(l10n.D.Tr(ctx, …)) for prose, g.code(…) for commands, paths and configuration | Prose translated and wrapped to 74 columns (guideWidth); code verbatim, indented four spaces |
| A message a piped script prints | shellFail(l10n.D.Tr(ctx, …)) | Translated and shell-quoted; the rest of the script is not translated |
Wording: a failed action reads cannot <verb> …, not "failed to …", "could not …" or a gerund wrapper such as "creating X: %w". An error that describes a state ("malformed field line") stays as it is, and a package prefix such as apt: is kept. Daemon log messages and HTTP error bodies use the same wording. A msgid is one unwrapped paragraph with %s or %q where a path or name goes.
| Translated | Not translated |
|---|---|
pkgproxyctl and pkgproxy help, through go-cmdr's own catalog too | Error() of every error: logs and tests read the same in every locale |
Errors as pkgproxyctl and pkgproxy print them | HTTP error bodies, including snapd error results, the store-style error list of a refused snapd request, and offline.PlainAnswer's body |
| Setup guides on a terminal, and the daemon's startup banner | Scripts piped to sh, apart from what shellFail prints |
The setup page and the cache page, in the visitor's Accept-Language | Commands, paths, configuration and YAML: never part of a msgid |
| Daemon log messages, and error-valued log attributes | Log attribute keys and other values |
pkgproxyctl stats, offline, version and the other commands' output | A log message that comes from a library at run time, such as the checksum database client's |
The snap's daemon runs under systemd, normally in the C locale, so its log is in English unless the service is given a locale.
| Where | Locale |
|---|---|
main | i18n.DetectLocale(), the process's locale, on the base context; i18n.BindAll(ctx, l10n.D, cmdrl10n.Domain) binds pkgproxy's and go-cmdr's catalogs |
| Setup page, cache page, setup scripts | l10n.AcceptLanguage(r.Header.Get("Accept-Language")): the best match among English and the catalogs, English for none or a malformed header; set on a context used only to render, never to log |
l10n.NewLogHandler(ctx, next) | The record's context when it carries a locale, otherwise the daemon's |
| File | What it is |
|---|---|
internal/l10n/po/pkgproxy.pot | The template scan writes: every msgid, with its source locations |
internal/l10n/po/pl.po | Polish, with three plural forms |
internal/l10n/l10n.go | //go:embed po; registers domain D = gitlab.com/zygoon/pkgproxy; English is the source language; every <lang>.po in po is a language |
internal/l10n/log.go | NewLogHandler: translates each record's message and renders error attributes through D.LocalizedError, groups included |
A new language is a new po/<tag>.po: AcceptLanguage and the language list pick it up from the embedded directory.
| Test | What it checks |
|---|---|
cmd/pkgproxy TestCtlSetupApt_GuidePolish | setup apt in Polish: prose translated and rewrapped, configuration identical to English |
TestCtl_HelpPolish, TestCtlVersion_Polish, TestSetupPage_Polish | Help, version and the setup page in Polish |
internal/l10n TestAcceptLanguage | Header to locale |
internal/l10n TestLogHandler_* | Messages and error attributes translated, groups, the record's locale winning, unknown messages passing through |
internal/registryproxy TestErrors_Localize | The side's errors localize |
Command tests run in English through invoke; invokeIn(t, locale, argv...) runs them in another locale.
The port or socket every client connects to, the other sockets the daemon opens, and what answers a request before the sides.
-addr (default :3142) is the TCP listener.
-socket-path replaces it with a UNIX socket, for example for
QEMU's guestfwd. Setup instructions always name
-addr.
-lxd-tls-addr (:3143) is HTTPS for lxc.
stats.sock and admin.sock live in
/var/snap/pkgproxy/common in the snap, /run/pkgproxy
otherwise. The snap adds content sockets under
$SNAP_COMMON/sockets.
bind(2) applies the umask; connect(2) needs
write permission. The umask is cleared around every bind, so each socket
is 0777. A stale socket file is removed first; this is not a
check for a second daemon.
-addr and -lxd-tls-addr must be
host:port with a numeric port. pkgproxy
-check-config validates flags and PKGPROXY_*
variables and exits; the snap's configure hook runs it.
Main port: access log, setup page (/, /setup/,
/pkgproxy/cache), LXD certificate (/pkgproxy-lxd-
cert.pem), offline mode, then the sides. HTTPS port: access log,
LXD image API, offline mode, the same sides.
Next: Settings, for the address and socket path; LXD images, for the HTTPS port; The cache page, for what the main port shows a local visitor.
Plain HTTP on -addr, or on the UNIX socket -socket-path names instead. A request is offered to each row in turn; the first that claims it answers. The name in the last column is what the access log records as the handler.
| Order | Request | Answered by | Local or trusted only | Log name |
|---|---|---|---|---|
| 1 | GET or HEAD / | the setup page | no | setup-page |
| 1 | GET or HEAD /setup/TOOL, ?undo=1 for the undo script | a setup script for sh | no | setup-page |
| 1 | GET or HEAD /pkgproxy/cache | the cache page | yes | setup-page |
| 1 | GET or HEAD /pkgproxy/cache/activity | the cache page's figures, as JSON | yes | setup-page |
| 1 | GET or HEAD /pkgproxy/cache/events | the cache page's live stream | yes | setup-page |
| 2 | GET or HEAD /pkgproxy-lxd-cert.pem | the HTTPS port's certificate, in PEM | no | lxd-cert |
| 3 | CONNECT host:port, or an absolute URL such as GET http://archive.ubuntu.com/… | the apt forward proxy | no | apt |
| 4 | /go and below | the Go module proxy | no | gomod |
| 5 | /lxd/NAME/… | an LXD image server, by simplestreams | no | lxd |
| 6 | /files/v1/fetch?… | files by checksum | no | files |
| 7 | /v2/, /v2/REPO/manifests/REF, /v2/REPO/blobs/DIGEST, /v2/REPO/referrers/DIGEST | the Docker Hub mirror | no | docker |
| 7 | /v2/REPO/tags/list, /v2/_catalog | the Docker Hub mirror, which answers 404 | no | docker |
| 8 | any other path under /v2, /api/v1 or /v1 | the Snap Store relay | no | snap |
| 9 | anything else | 404, sent nowhere | no | (none) |
GET and HEAD; other methods fall through. Row 1 goes away with setup-page set to false, row 2 when the HTTPS port is off.trusted-networks (-trusted-networks, empty by default); without a reverse proxy's header such as Via or X-Forwarded-For; and with a Host that is an IP address, localhost, the machine's name or the host of public-url.-go-base-path and -lxd-base-path. A side that is turned off claims nothing: its paths reach row 9, and Docker-shaped paths get a 404 rather than going to the Snap Store.HTTPS on -lxd-tls-addr, with the self-signed certificate from the cache directory, or on the UNIX socket -lxd-tls-socket-path names. It exists for lxc, which talks to image servers only over HTTPS. It is off when -lxd-tls-addr is off or no LXD remote is configured.
| Order | Request | Answered by | Log name |
|---|---|---|---|
| 1 | GET / | LXD's API root, as LXD asks when it adds a remote | lxd-api |
| 1 | GET /1.0 | the server description | lxd-api |
| 1 | GET /1.0/images?project=NAME | the images of the remote NAME | lxd-api |
| 1 | GET /1.0/images/aliases/ALIAS?project=NAME | an alias of that remote | lxd-api |
| 1 | GET /1.0/images/FINGERPRINT?project=NAME | one image | lxd-api |
| 1 | GET /1.0/images/FINGERPRINT/export?project=NAME | the image's files, as LXD 6.10 and later download them | lxd-api |
| 2 | anything else | the same sides as the main port, from row 3 on | as on the main port |
Under /1.0, other paths get LXD's 404 and other methods a 405. The setup page, the cache page and the certificate download are on the main port only. The lxc remote add commands that pkgproxyctl setup lxd-remote prints use https://HOST:3143/lxd/NAME with simplestreams.
| Socket | Snap | Without the snap | Serves | Used by |
|---|---|---|---|---|
| stats | /var/snap/pkgproxy/common/stats.sock | /run/pkgproxy/stats.sock | GET /pkgproxy/v1/stats, GET /pkgproxy/v1/info | pkgproxyctl stats, open and setup |
| admin | /var/snap/pkgproxy/common/admin.sock | /run/pkgproxy/admin.sock | DELETE /pkgproxy/v1/cache with the apt URL in a JSON body, GET and PUT /pkgproxy/v1/offline | pkgproxyctl cache invalidate, pkgproxyctl offline, the snap's configure hook |
| content, main | /var/snap/pkgproxy/common/sockets/pkgproxy.sock | where -content-socket-path points | the main port's table, without the cache page's rows | another snap, through the sockets content interface |
| content, HTTPS | /var/snap/pkgproxy/common/sockets/pkgproxy-lxd-tls.sock | where -lxd-tls-content-socket-path points | the HTTPS port's table | the same |
-socket-path and -lxd-tls-socket-path replace it.0777: anyone on the machine may connect, so anyone may switch offline mode or drop an apt file. The stats socket only reads.See Statistics for the stats API, pkgproxyctl for the commands and Settings for the flags.
pkgproxy serves LXD's simplestreams image servers over its own HTTPS port. It rechecks the catalogs and keeps every image file it has checked.
lxc remote add NAME https://host:3143/lxd/NAME --protocol=simplestreams.
lxc refuses a simplestreams remote that is not https://, so
pkgproxy has a TLS listener, -lxd-tls-addr, with a
self-signed certificate kept under lxd/tls in the cache
directory.
--accept-certificate is not read for simplestreams remotes.
pkgproxyctl setup lxd-remote | sh adds each remote and
fetches /pkgproxy-lxd-cert.pem into lxc's
servercerts/NAME.crt (under ~/snap/lxd/common/config for the LXD snap, else ~/.config/lxc). It refuses sudo, which
would configure root's lxc instead.
The defaults: pkgproxy-images, pkgproxy-ubuntu,
pkgproxy-ubuntu-daily, and craft-providers'
craft-com.ubuntu.cloud-buildd and
craft-com.ubuntu.cloud-buildd-daily, which setup converges
onto pkgproxy and -undo restores.
streams/v1/index.json and the catalogs revalidate after
-lxd-revalidate-ttl (default: -revalidate-ttl, 5m). Image files are allowlisted by the
catalogs, SHA-256 verified and cached forever; the allowlist is saved in
lxd-index.json every -lxd-checkpoint-interval
(30s) and on shutdown.
On LXD 6.10 setup lxd-remote adds an lxd image
registry per remote over one public cluster link, which pins pkgproxy's
certificate. pkgproxy serves LXD's image API, read-only, on the HTTPS
port. Registries belong to the LXD server, so they serve every user of it.
-lxd-remote name=url (repeatable; name= drops a
default; off), -lxd-tls-addr (:3143),
-lxd-base-path (/lxd),
-lxd-revalidate-ttl (default: -revalidate-ttl, 5m), -lxd-checkpoint-interval
(30s).
Next: Build containers, for what runs inside them; pkgproxy for CI, for many machines behind one cache.
Next: Build containers, for what runs inside them; Fetch once, for one shared image download; Offline mode, for launching with no network.
With no -lxd-remote at all, these five are served. The address lxc uses is https://HOST:3143/lxd/NAME.
| Remote | Real image server | Launch with |
|---|---|---|
pkgproxy-images | https://images.lxd.canonical.com | lxc launch pkgproxy-images:debian/12 |
pkgproxy-ubuntu | https://cloud-images.ubuntu.com/releases | lxc launch pkgproxy-ubuntu:24.04 |
pkgproxy-ubuntu-daily | https://cloud-images.ubuntu.com/daily | lxc launch pkgproxy-ubuntu-daily:24.04 |
craft-com.ubuntu.cloud-buildd | https://cloud-images.ubuntu.com/buildd/releases | used by snapcraft, rockcraft and charmcraft |
craft-com.ubuntu.cloud-buildd-daily | https://cloud-images.ubuntu.com/buildd/daily | used by snapcraft, rockcraft and charmcraft |
-lxd-remote NAME=URL adds a remote or replaces a default of that name, -lxd-remote NAME= drops a default, and -lxd-remote off turns the LXD side off. The option repeats, once per remote. A name may not be empty, . or .., or hold /, :, ,, ?, #, %, white space or a control character.
Paths are relative to a remote, so /lxd/pkgproxy-ubuntu/streams/v1/index.json is fetched from https://cloud-images.ubuntu.com/releases/streams/v1/index.json. Only GET and HEAD are answered; anything else is a 405.
Path under /lxd/NAME/ | What it is | Cache policy |
|---|---|---|
streams/v1/index.json | the index of the remote's catalogs | revalidated after -lxd-revalidate-ttl; not verified, as nothing lists its hash |
any other path ending in .json | a product catalog, such as streams/v1/images.json | revalidated after -lxd-revalidate-ttl; read for the SHA-256 of every file it lists |
| a file a catalog listed | an image file: metadata tarball, squashfs root, VM disk | fetched once, checked against the catalog's SHA-256, kept forever |
| a file no catalog listed | — | 404, never fetched |
| a remote that is not configured, or no path after the name | — | 404 |
a path that climbs out with .. | — | 400 |
A failed recheck of a catalog serves the cached copy, as on the other sides. A HEAD for an image file that is not cached yet is answered from the catalog, with its size, without fetching it. An image file that fails its hash check is not cached, but the clients already receiving it get the upstream's bytes unchanged, for at most -fill-drain-timeout; lxc checks the image itself. A client's Authorization header is never sent upstream, so private image servers are not supported.
| Path | Main port, 3142 | HTTPS port, 3143 |
|---|---|---|
/lxd/NAME/… | yes, over plain HTTP, for curl and scripts | yes; the only form lxc accepts |
/pkgproxy-lxd-cert.pem | yes: the certificate, to pin | no |
/, /1.0, /1.0/images/… | no | yes: LXD's image API, for LXD 6.10 |
LXD 6.10 downloads images through an image registry, which asks the HTTPS port this read-only part of LXD's own API. ?project=NAME names the remote; the answers come from the same cached catalogs and image files.
| Request | Answer |
|---|---|
GET /, GET /1.0 | a public server with no API extensions |
GET /1.0/images, with or without ?recursion=1 | the remote's images for the requester's architecture |
GET /1.0/images/aliases/ALIAS | the image the alias names: the container image, or the virtual machine image when there is no container image |
GET /1.0/images/FINGERPRINT | the image, by its fingerprint or a unique prefix of it |
GET /1.0/images/FINGERPRINT/export | its files: one for a unified image, or metadata and then rootfs or rootfs.img as a multipart answer |
The requester's architecture comes from LXD's User-Agent. These requests appear in the access log with handler=lxd-api.
Under the cache directory, /var/snap/pkgproxy/common/cache in the snap and /var/cache/pkgproxy otherwise:
| Path | What it holds |
|---|---|
lxd/objects/NAME/… | the cached catalogs and image files, one directory per remote, each file with its metadata in an extended attribute |
lxd/tls/cert.pem | the self-signed certificate of the HTTPS port: ECDSA P-256, valid for ten years |
lxd/tls/key.pem | its private key, readable by its owner only |
lxd-index.json | the SHA-256 of every image file the catalogs listed, saved every -lxd-checkpoint-interval and on shutdown |
Keep lxd/tls when moving or clearing the cache: a new certificate breaks every remote pinned to the old one until pkgproxyctl setup lxd-remote runs again. Without lxd-index.json, a request for an image file whose catalog has not been read since the restart gets a 404.
| Option | Environment variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-lxd-remote | PKGPROXY_LXD_REMOTE | flag or environment only | the five above | remotes to serve, as NAME=URL; off serves none |
-lxd-base-path | PKGPROXY_LXD_BASE_PATH | flag or environment only | /lxd | the URL path the remotes are served under |
-lxd-tls-addr | PKGPROXY_LXD_TLS_ADDR | lxd-tls-addr | :3143 | address of the HTTPS port, or off |
-lxd-tls-socket-path | PKGPROXY_LXD_TLS_SOCKET_PATH | flag or environment only | none | serve the HTTPS port on this UNIX socket instead; setup scripts still name -lxd-tls-addr |
-lxd-tls-content-socket-path | PKGPROXY_LXD_TLS_CONTENT_SOCKET_PATH | flag or environment only | none; the snap sets /var/snap/pkgproxy/common/sockets/pkgproxy-lxd-tls.sock | also serve the HTTPS port on this UNIX socket, to share it with another snap |
-lxd-revalidate-ttl | PKGPROXY_LXD_REVALIDATE_TTL | lxd-revalidate-ttl | -revalidate-ttl, 5m | how long index.json and catalogs are served before they are checked again |
-lxd-checkpoint-interval | PKGPROXY_LXD_CHECKPOINT_INTERVAL | lxd-checkpoint-interval | 30s | how often the learned image hashes are saved |
With -lxd-tls-addr off, pkgproxyctl setup lxd-remote prints no script, since lxc has no HTTPS address to use. All settings are in Settings; what the setup script changes is in Setting up clients.
The three offline modes, what works and what fails without the internet, and how the mode is switched.
-offline-mode is never (default),
offline or store-overloaded. -offline
is short for -offline-mode offline; given both, they must
agree.
Hits are served with every TTL frozen. Upstream requests and
CONNECT dials are refused before they start. A miss is a
503 (pkgproxy is offline, no
Retry-After); snapd gets a 424.
Never-cached content fails offline: HTTPS apt sources, new modules and
images, unseen snap searches. A dists/** entry the index knows
is superseded is refused online or off. Snaps seen online install and
refresh.
One breaker per host: open after 3 consecutive failures, or at once on
429/503 with Retry-After; 30 s,
doubled up to 10 min; then one probe. A 500 or
501 is not a failure.
pkgproxyctl offline (status), on,
off and mode MODE, or snap set pkgproxy
offline-mode=…: no restart either way. In the snap the switch is
saved as the option.
-auto-offline is network (default on Linux),
airplane-mode or off (the only value elsewhere).
Changes settle for 5 s; offline-mode is not changed.
Next: pkgproxy for CI, for builds that see only the cache; Immutable or
not, for what offline freezes; pkgproxyctl, for the offline
command.
Next: Immutable or not, for what offline freezes;
pkgproxyctl, for the offline command; Snap Store relay, for
snaps without a network.
offline-mode | Upstreams | Windows | What the cache cannot answer |
|---|---|---|---|
never (default) | contacted as needed | as configured | fetched |
store-overloaded | contacted, except a host whose breaker is open | as configured | from a host left alone: 503 with Retry-After, the time until its next try; snapd gets 424 |
offline | never contacted, no DNS lookup or connection | endless: every cached entry is a hit | 503 without Retry-After; snapd gets 424 |
The breaker of store-overloaded, one per upstream host: it opens after 3 failures in a row (a transport error, a timeout, 429, 502, 503, 504), or at once on a 429 or 503 with Retry-After (honoured up to 10 min). It stays open 30 s, doubling each time it opens again, up to 10 min; then one probe request decides. A 500 or 501 is not a failure.
| Side | Still served | Refused |
|---|---|---|
| apt | cached pool/ files and dists/ indexes, as they were | anything not cached; an index a newer Release shows to be superseded; https:// sources, whose tunnels are refused |
| Snap Store | cached snap info, assertions and snap files; install, download and refresh of snaps that were answered online; searches made online | searches never made online and actions never answered online, as 424 |
| Go modules | cached .mod, .zip, list, @latest and .info | modules and versions never fetched |
| Docker Hub | tags resolved before, cached manifests and layers | images never pulled |
| LXD images | cached catalogs and image files, through lxc and through LXD's image API | images never fetched |
| Files by checksum | files already fetched | files never fetched |
| Client | Status | Body |
|---|---|---|
| apt, go, Docker, LXD, curl | 503 | pkgproxy is offline: the content is not in the cache |
the same, in store-overloaded | 503, Retry-After | pkgproxy is not contacting the upstream for a while, as it looked overloaded: the content is not in the cache |
| snapd | 424 | a store error list: pkgproxy is offline and the content is not in its cache |
snapd, in store-overloaded | 424 | pkgproxy is giving the store a rest, as it looked overloaded, and the content is not in its cache |
The bodies are in English whatever the daemon's language. snapd does not retry a 424, where it would retry a 5xx for half a minute.
auto-offline | pkgproxy is offline while | Reported reason |
|---|---|---|
network (default on Linux) | the machine has no usable default route | no-network |
airplane-mode | the machine has at least one radio and every radio is off | airplane-mode |
off (the only value outside Linux) | never by itself | none |
A change counts once it has held for 5 s. The configured offline-mode is kept and applies again once the machine is back. A switch by hand wins until the machine next goes offline.
| Command | Does |
|---|---|
pkgproxyctl offline or pkgproxyctl offline status | shows the mode in effect, why if automatic, and each upstream being left alone with its next try |
pkgproxyctl offline on | mode offline |
pkgproxyctl offline off | mode never (not the mode before) |
pkgproxyctl offline mode MODE | mode never, store-overloaded or offline |
pkgproxyctl offline mode -keep-automatic MODE | sets the mode for when the machine is back, staying offline while it is not; the snap's configure hook uses it |
sudo snap set pkgproxy offline-mode=MODE | the same setting, without a restart |
sudo snap set pkgproxy auto-offline=off | stop going offline with the machine |
Each command takes -socket-path, the admin socket. In the snap a switch is saved as the offline-mode option and survives a restart; outside it a switch lasts until the daemon restarts, and pkgproxyctl says so. Any change of mode forgets the breakers and the hosts whose recheck failed.
| Setting | Flag | Variable | Snap option | Default |
|---|---|---|---|---|
| start mode | -offline-mode | PKGPROXY_OFFLINE_MODE | offline-mode | never |
short for offline | -offline | PKGPROXY_OFFLINE | flag or environment only | off |
| going offline with the machine | -auto-offline | PKGPROXY_AUTO_OFFLINE | auto-offline | network on Linux, else off |
Given both -offline and -offline-mode, they must name the same mode, or pkgproxy refuses to start.
GET and PUT of /pkgproxy/v1/offline on the admin socket read and set the mode; pkgproxyctl offline is its front end. The stats socket's /pkgproxy/v1/info reports the same, read-only.
| Field | Meaning |
|---|---|
mode | the mode in effect: never, store-overloaded or offline |
configured_mode | while offline automatically: the mode that applies again |
automatic | while offline automatically: no-network or airplane-mode |
auto_offline | the auto-offline setting |
keep_automatic | in a PUT: keep an automatic override |
upstreams | hosts whose breaker is not closed: host, state (open or half-open), retry_in_seconds |
restart_mode | the mode after a restart |
persistent | whether a switch survives a restart |
persist_error | from a PUT: why the mode could not be saved |
One request through pkgproxy, from the client to the cache on disk and back.
Five kinds of client share one daemon and one port, 3142; LXD
image servers also need the HTTPS port 3143. Each download is
fetched upstream once and served from disk after that.
apt is recognized by its absolute request line. The others are told apart
by path: /go, /lxd, /files, Docker's
/v2/…/manifests|blobs|referrers|tags/list,
/v2/_catalog and /v2/, and the rest of
/v2 for snapd. Anything else is answered with 404.
The key names the archive and the path inside it. The store has one directory per kind of download, and a write becomes visible only on commit: a temporary file renamed into place.
Concurrent requests for one key share one upstream fetch, a fill. It runs detached from the clients, streams to them while it writes, and is committed only after the content checks out.
Immutable content is cached forever without revalidation. Mutable indexes are served for a short TTL and then revalidated with a conditional GET. Nothing is ever evicted.
Next: One listener, for the ports and sockets; The store, for the cache on disk; Settings, for the flags and snap options.
Next: pkgproxy for CI, for many machines behind one cache; Fetch once, for one download shared by many jobs; Offline mode, for builds that see only what was cached.
Next: pkgproxyctl setup, for pointing your machines at it;
Fetch once, for one download shared by every client; Immutable or not,
for what is kept and what is rechecked.
pkgproxyctl is the pkgproxy binary under a second name, with a few commands that reach the daemon over two sockets.
pkgproxyctl is a symlink to pkgproxy, which
dispatches on the name it was invoked as. In the snap, if the shell cannot
find pkgproxyctl, use pkgproxy.pkgproxyctl.
Without the snap, every release from 0.12.0 has the binary for Linux on amd64
and arm64 on the releases page.
pkgproxyctl -h lists the six commands; pkgproxyctl
setup -h and pkgproxyctl offline -h list theirs.
stats.sock serves stats, version,
open and setup; admin.sock serves
cache and offline. Each command's
-socket-path points it elsewhere.
Usage: … [OPTIONS] MODE, a description, then
Available options. A stray argument is an error, not
something to guess around.
Next: pkgproxyctl setup, to set up clients; Statistics, for what pkgproxyctl stats reports; Offline mode, for what pkgproxyctl offline switches.
| Command | Arguments | Options | Socket | What it does |
|---|---|---|---|---|
stats | none | -json, -limit N (default 20), -r WHEN, -socket-path | stats | shows what is cached and how much downloading it saved |
setup TOOL | none | see below | stats, to read -addr | prints how to point a tool at pkgproxy: a guide on a terminal, a script for sh when piped |
cache invalidate | URL | -socket-path | admin | throws away the cached copy of one apt file |
offline | none | -socket-path | admin | the same as offline status |
offline status | none | -socket-path | admin | shows the mode and each upstream left alone for now |
offline on | none | -socket-path | admin | serves only from the cache: the mode offline |
offline off | none | -socket-path | admin | contacts upstreams as needed: the mode never |
offline mode | never, store-overloaded or offline | -keep-automatic, -socket-path | admin | sets the mode |
open | none | -socket-path | stats | opens the setup page in the desktop's browser |
version | none | -socket-path | stats | shows the version of pkgproxyctl and of the running daemon, and in the snap the snap's version and revision |
-h at any level lists what is below it; pkgproxyctl setup -h lists the tools.-r takes a Unix timestamp, an RFC 3339 time or how long ago, such as 30m or 2h.-keep-automatic sets the mode that applies once the machine is online again, without ending an automatic offline; the snap's configure hook uses it.-socket-path defaults to /var/snap/pkgproxy/common/stats.sock or admin.sock in the snap, and to /run/pkgproxy/stats.sock or admin.sock otherwise.cache invalidate takes the http:// URL apt fetches the file from, with dists/ or pool/ in its path, such as http://deb.debian.org/debian/dists/bookworm/InRelease. Other kinds of download cannot be thrown away one by one.version says so when the running daemon is another build than pkgproxyctl, and how to restart it.pkgproxy.open and the applications menu entry run pkgproxyctl open. If the shell cannot find pkgproxyctl, use pkgproxy.pkgproxyctl.| Tool | Sets up | Run the script as | Options besides -addr and -undo |
|---|---|---|---|
apt | apt's proxy | root | none |
snap | snapd's store, through a systemd drop-in; restarts snapd | root | none |
go | GOPROXY for the user | the user | -go-base-path (also -base-path), default /go |
docker | Docker Hub mirror; restarts Docker | root | none |
lxd | a pkgproxy profile for LXD containers, or a project's default profile | the user, in the lxd group | -project NAME |
lxd-remote | lxc image remotes through the HTTPS port, and their certificate | the user who runs lxc, not with sudo | -lxd-tls-addr, -lxd-base-path (also -base-path), -lxd-remote |
snapcraft | the snapcraft LXD project's build containers | the user, in the lxd group | none |
rockcraft | the rockcraft LXD project's build containers | the user, in the lxd group | none |
charmcraft | the charmcraft LXD project's build containers | the user, in the lxd group | none |
status | nothing: prints a script that reports what is set up | the user | -lxd-tls-addr, -go-base-path, -lxd-base-path, -lxd-remote; no -undo |
-addr and -lxd-tls-addr are read from the running daemon when not given; -go-base-path, -lxd-base-path and -lxd-remote are not, and default to the daemon's defaults.pkgproxyctl setup apt | sudo sh, or | sh for the tools run as the user. -undo prints the script that removes the setup.exit 1 to the pipe, so the pipe fails.setup status reports, for each other tool: set up, not set up, set up for another address, needs setup again, not installed, or cannot check.See Setting up clients for what each script writes.
cannot … error, with exit status 1. Asking for help exits with status 64.One pkgproxyctl setup command per client, printed as a guide or piped as a script, safe to run again and to undo.
On a terminal pkgproxyctl setup apt prints a guide; piped, it
prints a script. Either way apt ends up with an
Acquire::http::Proxy line in
/etc/apt/apt.conf.d/99pkgproxy.
Each piped script is one { … } block. It runs only where its
tool is, warns unless it runs as root, and chains its steps with
&& so that a failed step fails the script.
apt, snap and docker change system
files and need root. go, lxd,
lxd-remote, snapcraft, rockcraft and
charmcraft change your own settings and run as you.
-addr defaults to what /pkgproxy/v1/info reports,
else :3142. A wildcard host becomes the host name. A bad
address prints the error on stderr and exit 1 to the pipe.
pkgproxyctl setup X -undo | sh reverses setup X
(with sudo sh for apt, snap and docker). Both can be run
again safely.
pkgproxyctl setup status | sh reports configuration only, one
line per setup command. It runs as you, without sudo.
Next: Build containers, for what setup lxd and the craft tools set up; LXD images, for the remotes setup lxd-remote adds; Statistics, to check the builds use the cache.
Next: The setup page, for machines without pkgproxy; Statistics, to check that a client uses the cache; pkgproxyctl, for the other commands.
Every script runs only where its tool is; elsewhere it says there is nothing to do and succeeds. <host> is the address the setup names, as in pkgproxy:3142.
| Command | Runs as | What it changes |
|---|---|---|
setup apt | root (warns otherwise) | Writes /etc/apt/apt.conf.d/99pkgproxy with Acquire::http::Proxy "http://<host>";. Where apt is older than 3.1.3 it adds Acquire::Retries::Delay "false";, so apt does not hang while pkgproxy is offline. |
setup snap | root (warns otherwise) | Writes /etc/systemd/system/snapd.service.d/pkgproxy.conf setting SNAPPY_FORCE_API_URL=http://<host>, then systemctl daemon-reload and, where snapd is installed, restarts it. Needs systemd. |
setup docker | root (fails otherwise) | Puts http://<host> first in registry-mirrors in /var/snap/docker/current/config/daemon.json (Docker snap) or /etc/docker/daemon.json, creating the file if needed and keeping other settings and mirrors, then restarts Docker. A file it cannot understand is left alone, with an error. |
setup go | you | go env -w GOPROXY=http://<host>/go,direct, in your own Go environment file. |
setup lxd | you, in the lxd group | Creates or replaces the LXD profile pkgproxy: environment.GOPROXY, environment.SNAPPY_FORCE_API_URL and two container hooks (below). Add the profile to containers yourself. |
setup lxd -project P | you, in the lxd group | The same settings in the default profile of LXD project P, as the craft setups do. |
setup snapcraft, setup rockcraft, setup charmcraft | you, in the lxd group | setup lxd for the LXD project of the same name: creates the project if missing (copying the default profile of the default project), deletes its base-instance containers so the next build starts from a fresh base, and sets the three settings on the project's default profile. |
setup lxd-remote | you; refuses to run under sudo | For each image remote, lxc remote add <name> https://<host-name>:3143/lxd/<name> --protocol=simplestreams, with pkgproxy's certificate from http://<host>/pkgproxy-lxd-cert.pem saved as servercerts/<name>.crt in your lxc configuration directory (~/snap/lxd/common/config or ~/.config/lxc). A remote of the same name at another address is replaced. On an LXD with image registries it also adds a cluster link named pkgproxy-… and an image registry per remote. |
setup status | you | Changes nothing: prints a report (see below). |
The profile's hooks run on the LXD host, around each container that uses it, so nothing needs installing in the image.
| Hook | What it does in the container's root file system |
|---|---|
| before start | Writes /etc/apt/apt.conf.d/99pkgproxy-lxd (Acquire::http::Proxy and Acquire::Retries::Delay "false") and /etc/systemd/system/snapd.service.d/pkgproxy-lxd.conf (SNAPPY_FORCE_API_URL). Rewrites https:// to http:// for the archive, security, ports and old-releases hosts of ubuntu.com in /etc/apt/sources.list and /etc/apt/sources.list.d/*.sources. |
| after stop | Removes the two files it wrote. |
Containers reach pkgproxy at the address the setup names, or, when that address is this machine itself (a loopback address), at the address of the LXD bridge the project's default profile uses.
| Command | -undo |
|---|---|
setup apt | Removes /etc/apt/apt.conf.d/99pkgproxy. |
setup snap | Removes the snapd drop-in, reloads systemd and restarts snapd. |
setup docker | Removes only pkgproxy's entry from registry-mirrors, and the key when no mirror is left, then restarts Docker. |
setup go | go env -u GOPROXY, if you have a Go environment file. |
setup lxd | Deletes the pkgproxy profile. |
setup lxd -project P and the craft setups | Unsets environment.GOPROXY, environment.SNAPPY_FORCE_API_URL and raw.lxc on the project's default profile. The project is kept. |
setup lxd-remote | Removes each remote that points at pkgproxy and its certificate, and puts craft-com.ubuntu.cloud-buildd and craft-com.ubuntu.cloud-buildd-daily back at https://cloud-images.ubuntu.com/buildd/releases and …/buildd/daily. Removes pkgproxy's image registries, the images LXD downloaded through them, and its cluster links. |
| Option | Commands | Meaning |
|---|---|---|
-addr | all | pkgproxy's listen address. Taken from the running daemon when not given, else :3142. A wildcard host becomes this machine's host name. |
-undo | all but status | Print the commands that reverse the setup. |
-go-base-path, -base-path | go, status (-go-base-path only) | pkgproxy's Go base path, /go by default; not taken from the daemon. |
-project | lxd | LXD project whose default profile to change, instead of a pkgproxy profile. |
-lxd-base-path, -base-path | lxd-remote, status (-lxd-base-path only) | pkgproxy's LXD base path, /lxd by default; not taken from the daemon. |
-lxd-remote | lxd-remote, status | The image remotes, as given to the daemon; not taken from it. By default craft-com.ubuntu.cloud-buildd, craft-com.ubuntu.cloud-buildd-daily, pkgproxy-images, pkgproxy-ubuntu and pkgproxy-ubuntu-daily. |
-lxd-tls-addr | lxd-remote, status | pkgproxy's HTTPS address for LXD images. Taken from the daemon when not given, else :3143. |
pkgproxyctl setup status | sh prints one line per setup command with one of these states, and notes under it.
| State | Meaning |
|---|---|
| set up | Set up for the pkgproxy at -addr. |
| not set up | Nothing of this setup is there. |
| set up for another address | Set up, but for another pkgproxy address. |
| needs setup again | There, but it cannot work as it is, such as an LXD remote at an old address; run that setup again. |
| not installed | The tool is not on this machine. |
| cannot check | The script could not tell, with the reason in a note. |
pkgproxyctl is the pkgproxy binary under another name. The snap installs both. Without the snap, every release from 0.12.0 on publishes the binary for Linux on amd64 and arm64, with a SHA256SUMS file to check it against, on the releases page. To install the newest release, with arch set to the machine's architecture:
arch=amd64 url=https://gitlab.com/zygoon/pkgproxy/-/releases/permalink/latest/downloads curl -fL -O "$url/pkgproxy-linux-$arch" -O "$url/SHA256SUMS" sha256sum --check --ignore-missing SHA256SUMS sudo install -m 755 "pkgproxy-linux-$arch" /usr/local/bin/pkgproxy sudo ln -sf pkgproxy /usr/local/bin/pkgproxyctl
Or build it with Go: go install gitlab.com/zygoon/pkgproxy/cmd/pkgproxy@latest, then ln -s pkgproxy "$(go env GOPATH)/bin/pkgproxyctl".
A machine without pkgproxyctl gets the same scripts from the daemon's setup page (The setup page):
| URL | What it serves |
|---|---|
http://<host>:3142/setup/<tool> | #!/bin/sh and the lines pkgproxyctl setup <tool> prints when piped, as text/plain |
http://<host>:3142/setup/<tool>?undo=1 | The -undo script |
The tools are apt, snap, go, docker (unless the Docker mirror is off), lxd, lxd-remote (only with image remotes and the HTTPS listener on), snapcraft, rockcraft and charmcraft; status is not offered. The scripts name the address the visitor reached pkgproxy at, or the public-url setting. Pipe those for apt, snap and docker to sudo sh, and the others to sh.
The page at the proxy's own address that sets up any machine, with the desktop entry, the link to the cache page and the offline notice.
GET / is HTML for a browser and plain text for curl. Each
tool's script is at /setup/<tool>, its undo at
?undo=1. -setup-page=false turns it off.
/setup/apt is #!/bin/sh plus what
pkgproxyctl setup apt | sh would run, served as
text/plain. Minimal Debian and Ubuntu images have no curl:
paste the commands instead.
-public-url http://host[:port] (snap option
public-url) beats the request's Host, which
beats -addr.
pkgproxy.open runs pkgproxyctl open: it reads
/pkgproxy/v1/info, opens http://localhost:3142/
with xdg-open, and on failure says why on stderr.
The line links to /pkgproxy/cache, at the address the visitor
used. A request from another machine, unless
trusted-networks names its network, or carrying Forwarded,
Via or X-Forwarded-*, gets a 404.
Shown in the offline mode offline, while the machine has no
network or is in airplane mode, and in store-overloaded while
some upstream host is left alone. It is rendered without JavaScript.
Next: The cache page, for what it shows; Offline mode, for what fails while offline; pkgproxyctl setup, for the same scripts from the command line.
Next: pkgproxyctl setup, for the same scripts on the proxy's machine; Offline mode, for what fails while it is offline.
snapd uses pkgproxy as its store over plain HTTP. The store's answers are kept, rechecked or only relayed.
pkgproxyctl setup snap | sudo sh writes a systemd drop-in
that sets SNAPPY_FORCE_API_URL and restarts snapd. snapd then
uses pkgproxy as its store over HTTP; pkgproxy relays to the real store over
HTTPS. No TLS is intercepted.
Every object with a url and a sha3-384 gets its
URL rewritten to /v2/download/sha3-384/<hex>; the real
URL is recorded under pending/. The host is the request's own,
or -snap-external-url.
A miss is one shared fill from the store's URL. The snap is committed as
sha3-384/<hex> only when it matches the hash, and is
never revalidated. A mismatch is not cached; snapd rejects it too.
snap-revision and sequence-pinned assertions, other than
account-key, are permanent.
Other assertions, info and find are revalidated
after the store's max-age, else after
-snap-revalidate-ttl (default: -revalidate-ttl, 5m).
POST /v2/snaps/refresh is relayed, and each action's result
recorded; offline, snapd's install is answered from a record whose snap
and assertion chain are cached. Auth, device sessions and unknown paths are
relayed uncached; private answers are never kept.
Next: Immutable or not, for what is kept forever and what is
rechecked; Offline mode, for installs with no network; pkgproxyctl
setup, for the other clients.
snapd sends every store request to the address in SNAPPY_FORCE_API_URL, so the relay answers paths under /v2, /api/v1 and /v1 on port 3142, after the Docker registry's shapes (Docker Hub registry mirror). pkgproxy relays to https://api.snapcraft.io.
| Path | What pkgproxy does | Cache policy |
|---|---|---|
GET /v2/download/sha3-384/<hex> | Serves the snap (or delta) a rewritten link names, fetched once from the real URL recorded for the hash; Range included | kept forever, checked against the hash; a HEAD on a miss is answered from the record without fetching |
GET /v2/assertions/<type>/<key> | Serves an assertion | see the next table |
GET /v2/snaps/info/<name> | Relays, rewriting download links | revalidated |
GET /v2/snaps/find | Relays a search | revalidated; a search of private snaps is relayed uncached |
POST /v2/snaps/refresh | Relays installs, refreshes and downloads, rewriting download links | the answer is not cached; each action's result is recorded |
GET /v2/snaps/categories, /v2/snaps/category/<name>, /v2/repairs/… | Relays | revalidated |
GET /api/v1/snaps, /api/v1/snaps/sections, /api/v1/snaps/names, /api/v1/snaps/search, /api/v1/snaps/details/<name> | The older search API, relayed with download links rewritten | revalidated |
/api/v1/snaps/download/…, /api/v1/snaps/dm-verity/download/…, /api/v1/snaps/resources/…, /api/v1/resources/download/…, /api/v1/media/download/… | Follows the store's redirect to its CDN and streams the body | not cached: there is no hash to key it by |
/api/v1/snaps/auth/…, /api/v1/snaps/metadata, /v1/…, POST /v2/cohorts | Relays untouched: logins, device sessions and serials | not cached |
any other path under /v2 | Relays, rewriting download links | not cached |
| Assertion type | Without ?sequence=N | With ?sequence=N |
|---|---|---|
snap-revision | kept forever | kept forever |
account, snap-declaration, store, validation-set | revalidated | kept forever |
account-key | revalidated | revalidated |
| any other type | revalidated | revalidated |
Revalidated means served from the cache for the max-age the store announced for the entry, else for snap-revalidate-ttl, then asked for again. When the store cannot answer a revalidation (no connection, a 429 or a 5xx), the expired entry is served, with a warning in the log.
In every relayed JSON answer, an object with both a url and a sha3-384 gets its url replaced by http://<address>/v2/download/sha3-384/<hex>, and assertion-stream-urls are re-based to the same address. The older API's download_url and anon_download_url are rewritten the same way. <address> is the one snapd reached pkgproxy at, or snap-external-url.
| Request | Served from the cache | Kept |
|---|---|---|
| Anonymous, or a device session on the public store | yes | yes |
| A logged-in user | yes | only answers whose snaps all say "private": false; the catalogue is relayed uncached |
| A device session on a brand store | no | nothing, and no refresh results are recorded |
While pkgproxy is not contacting the store (Offline mode), a POST /v2/snaps/refresh is answered from the recorded results, with an X-Pkgproxy-Synthesised header. An action is answered only when its snap file and every assertion that vouches for it are cached; otherwise it gets an error of its own, and the other actions still succeed. Actions with a cohort, ignore-validation or validation sets are never recorded or answered.
| Setting | Flag | Environment | Snap option | Default |
|---|---|---|---|---|
| Snap Store API to relay to | -snap-store-url | PKGPROXY_SNAP_STORE_URL | flag or environment only | https://api.snapcraft.io |
Address put in rewritten links, as http://host[:port] | -snap-external-url | PKGPROXY_SNAP_EXTERNAL_URL | flag or environment only | the address the client used |
How long details and assertions are trusted when the store announces no max-age | -snap-revalidate-ttl | PKGPROXY_SNAP_REVALIDATE_TTL | snap-revalidate-ttl | revalidate-ttl, 5m |
| How long clients already following a snap download that failed its hash check keep receiving it | -fill-drain-timeout | PKGPROXY_FILL_DRAIN_TIMEOUT | fill-drain-timeout | 10m |
Under the snap directory of the cache (/var/snap/pkgproxy/common/cache/snap in the snap, /var/cache/pkgproxy/snap otherwise), each object with its metadata in an extended attribute of its file:
snap/objects/ sha3-384/<hex> snaps and deltas pending/sha3-384/<hex> the real URL and size a hash stands for, kept after the snap is cached assertions/<type>/<key> assertions info/<name>… snap details, per query and selecting headers find/<hash> searches catalog/<hash> categories, repairs and the older search API refresh/… recorded install and refresh results
See The store.
pkgproxy counts what every cached file saved and reports it through pkgproxyctl stats and a read-only socket.
Per cache key: hits, fetches, the size, and
bytes-saved, what the hits sent. A joined request is a hit;
the one that started the fetch is a fetch.
Each entry has content: true or false, decided by its key
when the report is made. The totals add only content.
pkgproxyctl stats, -r 1h for the last hour,
-limit N (20 by default), -json for the
entries as JSON.
-stats-socket-path, by default
$SNAP_COMMON/stats.sock in the snap and
/run/pkgproxy/stats.sock otherwise. The reference is
docs/statsapi.md.
-stats-checkpoint-interval (snap option
stats-checkpoint-interval, default 30s) writes
<cache-dir>/stats.json. A crash loses at most one
interval.
Next: The cache page, for the counts as a live map; The access log, for one line per request; Immutable or not, for why metadata is rechecked.
Next: pkgproxy for CI, to measure one build; The cache page, to see a build's downloads live; The access log, for one line per request.
Next: Fetch once, for why a joined request is a hit; Immutable or not, for why metadata is rechecked; pkgproxyctl, for the other commands.
| Option | Default | Does |
|---|---|---|
-limit N | 20 | show at most N entries, those that saved the most |
-r WHEN | all time | only changes since WHEN: a Unix timestamp, an RFC 3339 time, or how long ago, such as 30m or 2h |
-json | off | print the entries as JSON instead of a report |
-socket-path PATH | the stats socket below | where to ask |
| How pkgproxy runs | Socket |
|---|---|
| the pkgproxy snap | /var/snap/pkgproxy/common/stats.sock |
| bundled in another snap | /var/snap/<snap>/common/stats.sock |
| on its own | /run/pkgproxy/stats.sock |
Moved with -stats-socket-path or PKGPROXY_STATS_SOCKET_PATH; it is not a snap option. Every local user can read it; nothing can write through it. Ask it with any HTTP client that speaks to a UNIX socket; the host name in the URL is ignored:
curl --unix-socket /run/pkgproxy/stats.sock 'http://localhost/pkgproxy/v1/stats?limit=10' curl --unix-socket /run/pkgproxy/stats.sock "http://localhost/pkgproxy/v1/stats?since=$(date -d '1 hour ago' +%s)" curl --unix-socket /run/pkgproxy/stats.sock http://localhost/pkgproxy/v1/info
| Path | Method | Answers |
|---|---|---|
/pkgproxy/v1/stats | GET, HEAD | the entries that saved the most, biggest first |
/pkgproxy/v1/info | GET, HEAD | the listen addresses, the version and the offline state |
| Status | When |
|---|---|
200 | a JSON answer |
400 | since is not a whole number |
404 | any other path, a trailing / included |
405 | any method but GET or HEAD |
Errors are plain text.
| Parameter | Default | Meaning |
|---|---|---|
limit=N | 200 | at most N entries; anything but a positive whole number means the default |
since=T | all time | count only from T, in Unix seconds; empty is the same as absent |
Without since the answer is an array of entries; with it, an object {"since": …, "entries": […]} whose since is the start really used, in the daemon's time zone. An empty answer is [] or "entries":[], never null. Only entries hit at least once appear.
| Field | Meaning |
|---|---|
type | apt, snap, gomod, lxd, docker or files |
key | what was downloaded; see below |
content | true for content, false for metadata |
size-bytes | its size |
hits | times served without going upstream, joined requests included |
fetches | times pkgproxy went upstream for it, to download or to recheck |
bytes-saved | what the hits sent: the whole file for a GET, the range asked for, nothing for a HEAD or a 304 |
type | key | Content | Metadata |
|---|---|---|---|
apt | archive and path, such as deb.debian.org/debian/pool/main/h/hello/hello_2.10-3_amd64.deb | under pool/, and dists/…/by-hash/… | every other file under dists/ |
gomod | module file, such as golang.org/x/text/@v/v0.3.0.zip | .zip, .mod | list, @latest, .info |
lxd | <remote>/<path>, such as pkgproxy-ubuntu/streams/v1/index.json | image files | anything ending in .json |
docker | <repository>@<digest>; a layer pulled through two repositories shows twice | every entry | none counted |
files | <file name>@<algorithm>:<hex>, the name being the URL's last part | every entry | none counted |
snap | the snap's name, one entry for all its revisions; "" for downloads not matched to a name | every entry | none counted |
For a snap, size-bytes is the revision served last, fetches counts every download of the snap, cached or not, and bytes-saved adds up all revisions. What the cache saved is the sum of bytes-saved over the entries whose content is true; the ranking puts both kinds together.
since within | Counts start at the beginning of its |
|---|---|
| the last hour | minute |
| the last 2 days | hour |
| the last 60 days | day |
| further back | moved forward to 60 days ago |
All-time counts are never trimmed.
| Field | Meaning |
|---|---|
addr | the client address, as given to -addr (:3142) |
lxd_tls_addr | the HTTPS address for LXD, as given to -lxd-tls-addr (:3143), or off |
setup_page | whether / serves the setup page; absent before 0.9.0, read as true |
version | the daemon's version, as pkgproxyctl version shows it |
offline.mode | never, store-overloaded or offline, read at the time of asking |
offline.automatic | while offline with the machine: no-network or airplane-mode |
offline.configured_mode | while offline with the machine: the mode that applies again |
offline.upstreams | hosts being left alone, each with host, state (open or half-open) and retry_in_seconds |
pkgproxyctl setup, open, stats and version read this endpoint to find the daemon's ports, its setup page, its offline mode and its build. See Offline mode.
| Setting | Flag | Snap option | Default |
|---|---|---|---|
| how often counts are saved | -stats-checkpoint-interval | stats-checkpoint-interval | 30s |
| where the socket is | -stats-socket-path | flag or environment only | see above |
The counts and their history are saved to stats.json in the cache directory, /var/snap/pkgproxy/common/cache/stats.json in the snap, at every interval and on a clean shutdown. A crash loses what changed since the last save.
Where the cache lives on disk, how a download is committed, and how to free space.
-cache-dir: /var/snap/pkgproxy/common/cache in
the snap, /var/cache/pkgproxy otherwise. Each kind has
objects/ and tmp/; stats.json,
lxd-index.json and lxd/tls sit beside them.
The body is objects/<key>; its metadata is JSON in
the file's user.pkgproxy.meta extended attribute. pkgproxy
refuses to start on a file system without them (tmpfs before Linux 6.6,
NFSv3, FAT); copy the cache with cp -a, rsync
-aX or tar --xattrs.
The metadata is set on the file in tmp/, which is flushed
and renamed into objects/: body and metadata switch
together. A 304 replaces only the
user.pkgproxy.meta attribute.
Nothing is evicted. Stop pkgproxy, remove one kind's directory, start it
again. After removing lxd, run pkgproxyctl setup
lxd-remote | sh; keep stats.json to keep the
statistics.
pkgproxyctl cache invalidate <url> sends
DELETE /pkgproxy/v1/cache over admin.sock. It
works for apt only; other kinds are verified before they are stored.
Next: Statistics, for what the cache holds and saved; The cache page, for what is cached and served now; Settings, for the cache directory.
| What | In the snap | Without the snap |
|---|---|---|
Cache directory (-cache-dir) | /var/snap/pkgproxy/common/cache | /var/cache/pkgproxy |
| Statistics socket | /var/snap/pkgproxy/common/stats.sock | /run/pkgproxy/stats.sock |
| Admin socket | /var/snap/pkgproxy/common/admin.sock | /run/pkgproxy/admin.sock |
| Content sockets, shared with other snaps | /var/snap/pkgproxy/common/sockets/ | only where -content-socket-path and -lxd-tls-content-socket-path point |
| Settings the configure hook last applied | /var/snap/pkgproxy/current/configured.env | (none) |
The snap's common directory survives snap updates and refreshes.
/var/snap/pkgproxy/common/
├── cache/
│ ├── apt/
│ │ ├── objects/
│ │ │ └── archive.ubuntu.com/ubuntu/
│ │ │ ├── dists/noble/InRelease
│ │ │ └── pool/main/h/hello/hello_2.10-3_amd64.deb
│ │ └── tmp/ downloads in progress
│ ├── snap/
│ │ ├── objects/
│ │ │ ├── sha3-384/HEX snap files, by hash
│ │ │ ├── info/NAME snap details
│ │ │ ├── find/… refresh/… catalog/…
│ │ │ ├── assertions/…
│ │ │ └── pending/sha3-384/HEX where a rewritten download really points
│ │ └── tmp/
│ ├── go/
│ │ ├── objects/
│ │ │ └── MODULE/@v/ list, VERSION.info, VERSION.mod, VERSION.zip
│ │ ├── sumdb/ checksum database tiles and notes
│ │ └── tmp/
│ ├── docker/
│ │ ├── objects/
│ │ │ ├── blobs/sha256/HEX layers and configs
│ │ │ ├── manifests/sha256/HEX
│ │ │ └── tags/REPOSITORY/_/TAG/ACCEPT
│ │ └── tmp/
│ ├── lxd/
│ │ ├── objects/
│ │ │ └── REMOTE/… the image server's own paths
│ │ ├── tmp/
│ │ └── tls/
│ │ ├── cert.pem the HTTPS port's self-signed certificate
│ │ └── key.pem and its private key
│ ├── files/
│ │ ├── objects/
│ │ │ ├── sha256/HEX
│ │ │ └── sha512/HEX
│ │ └── tmp/
│ ├── stats.json saved statistics
│ └── lxd-index.json saved hashes of LXD image files
├── stats.sock
├── admin.sock
└── sockets/
├── pkgproxy.sock the main port, for another snap
└── pkgproxy-lxd-tls.sock the HTTPS port, for another snap
Without the snap, what the snap keeps in cache/ is in /var/cache/pkgproxy, and stats.sock and admin.sock are in /run/pkgproxy. Every cached object is one file, with its metadata in an extended attribute of that file. A kind's directory is made when its side starts, so a side turned off, such as Docker with docker-upstream set to off, has none; lxd/tls exists only while the HTTPS port is on.
| Kind | Key under objects/ | Example |
|---|---|---|
| apt | mirror host (with a port other than 80), the path before dists or pool, then the archive path | archive.ubuntu.com/ubuntu/pool/main/h/hello/hello_2.10-3_amd64.deb |
| snap | sha3-384/ and the snap's hash; details, searches, refreshes and assertions under their own prefixes | sha3-384/1f2e… |
| go | the module path and version, case-escaped as the Go module proxy protocol names them | golang.org/x/mod/@v/v0.20.0.zip |
| docker | blobs/sha256/, manifests/sha256/, or tags/ with the repository, tag and a short hash of the client's accepted types | tags/library/ubuntu/_/24.04/any |
| lxd | the remote's name, then the image server's path | pkgproxy-ubuntu/server/releases/noble/… |
| files | the checksum's algorithm and its hex digest, whatever URL the file came from | sha256/9f86… |
Each object's metadata is JSON in the user.pkgproxy.meta extended attribute of its file, so it is set and replaced together with the file. getfattr -n user.pkgproxy.meta FILE shows it. Metadata of more than 3 KiB is kept compressed with gzip, up to 64 KiB of JSON: getfattr --only-values -n user.pkgproxy.meta FILE | zcat shows it.
| Field | Meaning |
|---|---|
Size | size of the object in bytes |
ETag, LastModified | the upstream's validators, sent back when the entry is rechecked |
StoredAt | when the object was stored or last rechecked |
ContentHash | the hash of the object as stored, as ALGORITHM:DIGEST in the kind's own terms: SHA256:HEX (or the index's other algorithm) for apt, sha3-384:HEX for snaps, sha256:HEX for LXD images, Docker digests and files, h1:BASE64 for Go module zips and go.mod files, and sha256:HEX of what was stored for Go's version lists and .info files; empty for the rest. Where the kind has a published hash to check against, the object matched it before it was stored; an apt file no index lists and Go's version lists are stored with the hash unchecked |
Immutable | the entry is kept forever and never rechecked |
ContentType | the media type to serve it with, for sides that keep one |
MaxAge | a lifetime the upstream announced, for sides that honour one |
OriginURL | the URL the object was downloaded from, for sides that record it; left out if the metadata would not fit otherwise |
A file without the attribute is a miss and is fetched again; pkgproxy logs the first one it finds in each kind's directory. Anything under tmp/ is never served and can be removed while pkgproxy is stopped.
A cache made by an older pkgproxy keeps each object's metadata in a <file>.meta.json beside it. pkgproxy still reads that file for an object without the attribute, and removes it the next time it writes or deletes the object.
| file systems that keep extended attributes | ext4, XFS, btrfs, ZFS; tmpfs from Linux 6.6 |
| file systems that do not | tmpfs before Linux 6.6, NFSv3, FAT: pkgproxy refuses to start with its cache there |
| copying or backing up the cache | cp -a, rsync -aX or tar --xattrs, which keep extended attributes |
| a copy that dropped them | every file reads as a miss and is downloaded again the next time a client needs it; pkgproxy logs the first such file it finds in each kind's directory |
| To | Do |
|---|---|
| see what each kind takes | du -sh /var/snap/pkgproxy/common/cache/* |
| free one kind | sudo snap stop pkgproxy, sudo rm -rf /var/snap/pkgproxy/common/cache/KIND, sudo snap start pkgproxy |
| free the LXD kind | as above, then pkgproxyctl setup lxd-remote again on every client, because the certificate is new |
| throw away one apt file | pkgproxyctl cache invalidate URL, with the http:// URL apt fetches it from |
| keep the statistics | leave stats.json in place |
See pkgproxyctl for cache invalidate and Statistics for what stats.json holds.
Unit tests, command tests, synctest bubbles, exploratory runs and the spread suite, and the checks before a commit.
Index parsing, cache keys, hash checks and the Store are
plain unit tests. ./check.sh is dagger check:
fmt, vet, golint, build, staticcheck, goconst, errcheck, wsl, ineffassign,
go test -race -short on two Go versions, and reuse lint.
invoke(t, "pkgproxyctl", "setup", …) wraps go-cmdr's
cmdtest; assert the whole stdout, with
<IGNORE> where it varies. noDaemon(t)
points SNAP_COMMON at an empty directory;
unixServer fakes a daemon.
synctest.Test runs the test in a bubble;
synctest.Wait() returns when everything else is blocked.
Timeouts and TTLs pass on a fake clock. Use
httptest.NewTestServer and its Client(), not a
real socket.
Exploratory apt runs in a podman container against the
pkgproxy on the host; no CI job snapshots a mirror. A bug found this way
becomes a narrow unit test of the logic that failed.
tests/<client>/: setup,
setup-setup, undo, undo-undo,
undo-first, smoke.
tests/lib/invariants.sh runs after every task. CI runs the
LXD 6 system; LXD 5.21 is left to local runs.
git config core.hooksPath githooks enables
githooks/pre-commit, which also runs
go mod tidy -diff in every module. Tests, golint and reuse lint are left to
./check.sh and CI. Commit with git commit -s
for the DCO check.
Next: internal/fill, for the code synctest tests drive; The source tree, for the conventions; pkgproxy for CI, for the spread suite as a farm.
| Kind | Where | How to run |
|---|---|---|
| Unit tests of pure logic | *_test.go beside the code under internal/; fixtures in testdata/apt/ | go test ./internal/..., or one package |
| Command tests | cmd/pkgproxy/*_test.go, one beside each cmd_*.go | go test ./cmd/pkgproxy |
| Synctest tests (shared fetches, timeouts, TTLs) | in the packages they test: fill, flight, idletimeout, proxy, snapproxy, lxdproxy, reqlog, cmd/pkgproxy and others | as their package |
| Documentation checks | docs/docs_test.go: TestTablesOfContents, TestLinks, TestNoRawTags, TestHeader | go test ./docs |
| Podman runs of real apt and go | internal/integration/ | go test ./internal/integration; skipped under -short, without podman, or when a container cannot reach the host |
| snapd's own store client against the snap side | internal/integration/snapd/, a nested module | cd internal/integration/snapd && go test ./... |
| Exploratory apt runs | by hand: a Debian or Ubuntu container under podman with its apt pointed at the host's pkgproxy | not automated; a bug found becomes a narrow unit test |
| Spread suite | spread.yaml, tests/<suite>/<task>/task.yaml, tests/lib/ | spread garden: (below) |
| Everything CI runs but spread | .dagger/main.go | ./check.sh |
| Helper | File | What it does |
|---|---|---|
TestMain | main_test.go | Started as pkgproxy or pkgproxyctl (a symlink), the test binary runs main, so tests can run the snap's shell scripts against the real commands |
invoke(t, argv...) | main_test.go | Runs the real command tree in English with go-cmdr's cmdtest; its buffers make every setup command print its piped script |
invokeIn(t, locale, argv...) | main_test.go | The same in another locale |
invokeOn(t, h, argv...), invokeWith | main_test.go | With a host of test doubles, or another command tree |
check(t, errs...) | main_test.go | Reports every failed expectation of one run |
noDaemon(t) | main_test.go | Points SNAP_COMMON, and with it the default stats and admin sockets, at an empty directory |
unixServer(t, handler) | cmd_stats_test.go | Serves a fake daemon on a UNIX socket |
Expectations compare the whole output: inv.ExpectStdout, ExpectStderr, ExpectStderrTemplate, with <IGNORE> or <VARIABLE name> where output varies (host names, temporary paths, time).
synctest.Test; synctest.Wait() returns once every other goroutine in the bubble is blocked.time.Sleep or poll: timeouts, TTLs and sleeps pass on the bubble's fake clock.httptest.NewTestServer and its Client(): a goroutine blocked on a real socket keeps the clock from moving../check.sh is exec dagger check; GitLab CI's dagger-check job runs the same module and then dagger call coverage export --path=coverage. It needs the dagger CLI and docker or podman. The container is golang:1.27 with shared module and build caches.
| Check | Runs |
|---|---|
go-fmt | go fmt ./... prints nothing |
go-vet | go vet ./... |
go-lint | go tool golang.org/x/lint/golint ./... |
go-build | go build ./... |
static-check | go tool honnef.co/go/tools/cmd/staticcheck ./... |
go-const | goconst, ignoring immutable, revalidate, tcp, unix, json, limit |
err-check | go tool github.com/kisielk/errcheck ./... |
wsl | go tool github.com/bombsimon/wsl/v5/cmd/wsl ./... |
ineff-assign | go tool github.com/gordonklaus/ineffassign ./... |
test | go test -race -short -coverprofile=coverage.txt -covermode=atomic ./..., then go tool cover -func |
go-test-latest | go test -race -short ./... with GOTOOLCHAIN set to the latest stable Go from go.dev |
snapd-test | go vet and go test -race -count=1 ./... in internal/integration/snapd |
snapd-lint | gofmt, staticcheck, errcheck and ineffassign in internal/integration/snapd, built with -modfile=/src/go.mod |
reuse-lint | reuse lint in fsfe/reuse:latest |
| Command | Does |
|---|---|
dagger check -l | List the checks |
dagger check go-vet static-check | Run some by name |
dagger check --failfast | Stop at the first failure |
dagger call coverage export --path=coverage | coverage.txt, coverage-func.txt, Cobertura coverage.xml |
Enable once per clone (all its worktrees share it): git config core.hooksPath githooks. It needs no container and runs, with the tools go.mod pins:
| Check | Command |
|---|---|
| gofmt | gofmt -l on every tracked .go file outside .dagger/ |
| go-vet | go vet ./... |
| staticcheck | go tool honnef.co/go/tools/cmd/staticcheck ./... |
| goconst | on a copy of the files CI sees, with CI's ignore list |
| errcheck | go tool github.com/kisielk/errcheck ./... |
| wsl | go tool github.com/bombsimon/wsl/v5/cmd/wsl ./... |
| ineffassign | go tool github.com/gordonklaus/ineffassign ./... |
| go mod tidy | go mod tidy -diff in every module but .dagger, the nested ones included |
It also reminds you to regenerate .dagger/dagger.gen.go when a staged .dagger Go file changed without it. Tests, golint and REUSE lint are left to ./check.sh and CI.
Every commit needs Signed-off-by:: commit with git commit -s, repair with git commit --amend -s or git rebase --signoff. CI's dco job runs dco-check on merge requests and the default branch.
| System | LXD | Workers | Suites |
|---|---|---|---|
ubuntu-cloud-26.04-at-lxd6 | 6 | 5 | every suite but lxd-upgrade; the only system CI runs |
ubuntu-cloud-26.04-at-lxd5 | 5.21 | 3 | lxd, lxd-remote, snapcraft, rockcraft, charmcraft, lxd-upgrade |
| Suite | Tasks |
|---|---|
tests/apt/, tests/snap/ | setup, setup-setup, undo, undo-undo, undo-first, smoke, offline, store-overloaded |
tests/docker/, tests/go/ | setup, setup-setup, undo, undo-undo, undo-first, smoke, offline |
tests/lxd-remote/ | the six setup tasks, offline, refuses-sudo |
tests/lxd/ | the six setup tasks |
tests/files/ | smoke, offline |
tests/install/ | daemon, snap-option (one variant per snap option), offline-switch, auto-offline |
tests/snapcraft/, tests/rockcraft/, tests/charmcraft/ | pack, twice, the second from the cache |
tests/lxd-upgrade/ | setup-again, on -at-lxd5: LXD 5.21 to 6 |
tests/build/ | snap: packs the snap under test; manual, with PKGPROXY_SPREAD_BUILD=1 |
| Task | Asserts |
|---|---|
setup | The setup command, piped to sh, configures the client |
setup-setup | Twice leaves the machine as once |
undo | Setup then -undo leaves the machine as it was |
undo-undo | A second -undo succeeds and changes nothing |
undo-first | -undo on a machine never set up succeeds and changes nothing |
smoke | The client downloads through pkgproxy; the second download is a cache hit |
tests/lib/invariants.sh runs after every task: no proxy setting left behind, offline mode never, auto-offline=off, the default routes. tests/lib/pkgproxy.sh has the pp_* helpers (pp_not for a negative, pp_user to run as ubuntu, pp_hits_delta, pp_content_re, pp_assert_no_upstream, …); tests/lib/workload/<client>.sh has the wl_* functions the setup tasks share.
rm -f pkgproxy_*.snap && snapcraft pack # the one snap the suite installs, in the main checkout spread -list spread garden: spread garden:ubuntu-cloud-26.04-at-lxd6:tests/apt/ spread garden:ubuntu-cloud-26.04-at-lxd6:tests/apt/smoke spread -debug ... # a shell in the VM when a task fails spread -reuse ... # keep the VMs for the next run
It needs image-garden and spread (the image-garden snap has both), KVM and about 6 GiB per VM, and network access. In CI, spread-snap packs the snap in a VM from a git bundle, and spread runs the -at-lxd6 tasks against it in five VMs.
pkgproxy caches what apt, snapd, Go, Docker and LXD download, for every machine on your network. This topic covers the problem, the solution, the setup and the limits.
Every machine, VM, container and CI job downloads the same packages and images from the internet on its own, and an empty build environment does it again on every run.
One pkgproxy for the whole network: the first request for a file downloads it once, and every later one is served from its disk. Without the internet, it serves what is cached.
sudo snap install pkgproxy starts the service on
3142 (and 3143 for LXD).
pkgproxyctl setup <tool> prints a script; other machines
fetch it from http://<host>:3142/setup/<tool>.
-undo prints the way back.
No TLS termination: https apt sources go through an uncached
CONNECT tunnel to port 443. Files by checksum are the
exception. No signature checks (apt does them) and no garbage collection.
Next: pkgproxy for CI, for many runners behind one cache; A request, for one request end to end; The store, for where the cache lives.
Next: A request, for one request end to end; pkgproxyctl
setup, for pointing your tools at it; Offline mode, for working
without the internet.