pkgproxy design: LXD images

Caching LXD simplestreams image mirrors. Part of the pkgproxy design, whose index says which file holds each § number.

17. LXD simplestreams image-mirror caching

17. LXD simplestreams image-mirror caching

The code is in internal/simplestreams (protocol parsing), internal/lxdindex (the learned hashes) and internal/lxdproxy (the handler), wired into the daemon by cmd/pkgproxy.

17.1 Why this needs a different (but familiar) shape

LXD/Incus clients (lxc/incus) fetch container and VM images from remote “image servers” via simplestreams, a read-only, GET-only protocol: a JSON index describing products, and content-addressed blob files (root filesystem squashfs images, metadata tarballs, VM disk images). Full server-to-server --protocol=lxd remotes (mTLS client certificates, a stateful REST and events API) are out of scope: that is a management API, not a content-distribution protocol, and a poor fit for blob caching.

Like the Go module proxy (§16) and unlike the snap side (§11), the client addresses pkgproxy as an origin (lxc remote add images https://host:3143/lxd/images --protocol=simplestreams), so the handler is a path-preserving relay and no response body needs rewriting: every path a simplestreams document names is relative to the mirror root (observed on images.lxd.canonical.com and cloud-images.ubuntu.com, and so in lxc’s own shared/simplestreams package).

The caching policy is apt’s (§4.2/§4.3): streams/v1/index.json and the product catalogs are small, mutable metadata, revalidated after a short TTL and parsed on the way through to learn each blob’s path -> SHA256; the blobs they describe are immutable, hash-verified and cached forever, like pool/**.

17.2 Protocol surface

streams/v1/index.json (fixed path, relative to a mirror’s root): a top-level index keyed by content-id, each entry giving a path to that content-id’s product catalog (e.g. streams/v1/images.json) and, optionally, a sha256 of that file’s own bytes.
A product catalog’s products map holds, per product/version, an items map of files, each with path (relative to the mirror root), sha256, size, and ftype (the file’s role: a metadata tarball, a squashfs rootfs, a qcow2/raw VM disk image, …).
combined_sha256/combined_sha256_squashfs/etc fields on a version exist for split-image pairs, but are deliberately not consumed: they verify a pair of files together for the client’s own combined- stream assembly, while each file’s own sha256 is independently sufficient for this proxy’s per-file cache verification.
No query params, no required headers, GET-only — as simple and well-behaved a target as the GOPROXY protocol.

17.3 Multi-remote routing, cache keys, and the strict blob allowlist

The LXD side proxies several named remotes at once, so that LXD’s own image servers and craft-providers’ build-base remotes (§17.11) are all cached by one instance. A request path <LxdBasePath>/<remote>/<path> maps to <remote's base URL>/<path>; an unconfigured remote name is refused with 404 and never falls through to another side. Each remote is an independent cache namespace with no cross-remote alias heuristics: remote names are explicit, trusted configuration, never inferred from a request, unlike apt’s mirrors (§4.1).

Cache keys are "<remote>/<path>", hash-verified on write for blobs, as apt’s pool/** (§4.2), not a content-addressed store. A blob’s origin URL is always derivable from the remote and the path, so, unlike snap’s CDN URLs, nothing about the origin has to be learned or kept.

A blob is only ever fetched or cached if its path was learned from a parsed product catalog. A path never seen listed is refused with 404, and nothing is fetched upstream. The apt-like alternative (classify by path shape and fetch-and-cache with a logged gap when the hash is unknown) is rejected: refusing to relay an arbitrary path under a trusted remote’s namespace is a stronger safety property, and since a client reads the catalog before it asks for a blob, the allowlist costs nothing, as long as it survives restarts (§17.12).

17.4 Caching policy table

Mirrors §5’s apt table, paths relative to a remote’s root:

Path pattern	Mutability	Policy
`streams/v1/index.json`	mutable	revalidate after TTL; unverified trust root (as apt’s InRelease, §4.2)
anything ending in `.json`	mutable	revalidate after TTL; parsed best-effort to learn blob path→SHA256 mappings; opportunistically checked against `index.json`’s declared hash when one is known (never load-bearing — nothing verifies `index.json` itself)
anything else, path previously learned from a parsed product catalog	immutable	fetch once, hash-verify against the learned SHA256, cache forever, never revalidate
anything else, path never learned	—	404, never fetched (§17.3)

A JSON document that fails to parse is still cached and served, raw and logged, rather than failing the request: it is still a valid response to relay, it just teaches the index nothing (as the snap side does with an unparseable JSON body).

17.5 Cache layout and blob delivery

The LXD side has its own fslocal.Store at <cache-dir>/lxd, next to apt, snap and go, as each side does (§11.4).

Blob delivery is the same fill as snap’s (§11.4, internal/fill), since VM disk images can reach low-GB sizes: every client of a miss, Range resumes included, follows the download as it is written, and the headers, with a correct Content-Length already known from the catalog, go out before the upstream has answered. A blob that fails its SHA256 check is never cached (logged as an error), but clients still get the upstream’s bytes unmodified: lxc/incus verifies image content itself, as snapd does. A HEAD on a miss is answered from the index without fetching.

17.6 CLI flags

The flags and their defaults are in pkgproxy -help and DEPLOYMENT.md; the design points are:

-lxd-remote name=url (repeatable) sets, overrides or (name=) removes a remote, and -lxd-remote off disables the whole side: a string sentinel, as -go-sumdb off (§16.3), rather than a second bool flag. With no -lxd-remote at all, the five remotes of §17.11 are configured, so pkgproxy is useful for LXD with no flags.
-lxd-tls-addr (off to disable) is the HTTPS listener every real lxc/Incus client needs (§17.10).
-lxd-tls-socket-path binds that listener to a UNIX socket instead, and -lxd-tls-content-socket-path adds one, as -socket-path and -content-socket-path do for the plain listener: for a bundling consumer that bridges a socket into a guest (QEMU’s guestfwd unix: form) rather than binding a host TCP port that can collide with another pkgproxy. -lxd-tls-addr still supplies the port of the printed setup URL. The certificate’s names need not match the bridged hostname, since lxc pins the exact certificate (§17.10).
-lxd-checkpoint-interval is how often the learned blob hashes are saved (§17.12).

17.7 Client enablement

pkgproxyctl setup lxd-remote prints the certificate-pinning script of §17.10 for every configured remote, or its -undo reversal, with the same TTY-vs-piped split as every other setup subcommand (§14.1). The daemon’s startup banner prints it only when a remote is configured and -lxd-tls-addr is not off, since this side, unlike apt, snap and go, can be disabled.

It is not called setup lxd: that configures LXD containers as clients of pkgproxy (§16.6/§16.7), the opposite direction from LXD itself using pkgproxy as an image server.

17.8 Explicit v1 scope boundaries

Full mTLS --protocol=lxd remotes (§17.1), split-image combined_sha256* verification (§17.2), cross-remote deduplication (§17.3), GC (a project-wide non-goal, §1).
Private simplestreams servers: the handler builds its own outbound request and does not forward a client’s Authorization header.
index.json signature verification: there is no trust chain to check, the same accepted gap as apt’s InRelease trust root (§4.2) and the Go side’s sumdb-lookup failures (§16.3).
CA-signed certificates (e.g. ACME) for -lxd-tls-addr: they need a public DNS name and port 80/443 reachability, which a LAN-local instance typically lacks; pinning (§17.10) covers the common case.

17.10 HTTPS and per-remote certificate pinning

lxc remote add --protocol=simplestreams requires an https:// URL, rejected client-side with no override flag — so unlike every other pkgproxy side, LXD needs a real TLS listener. --accept-certificate doesn’t help here: lxc’s own “fast track simplestreams” add path never reads it, so a self-signed certificate added that way is never trusted, and a later lxc image list fails with x509: certificate signed by unknown authority. (Both observed with a real lxc.)

What does work: lxc reads a per-remote trusted certificate from <lxc config dir>/servercerts/<name>.crt if present, and forwards it to the local LXD daemon as Source.Certificate on image download too, so pinning one file covers both the client’s own metadata fetches and the daemon’s blob download, with no system CA changes or daemon restart. Installing a CA into the system trust store and restarting the daemon also works, but needs root and the restart.

pkgproxy generates one self-signed certificate, persisted under <cache-dir>/lxd/tls so it survives restarts (regenerating it would invalidate every pinned copy), and serves the LXD side over HTTPS with it on -lxd-tls-addr. The certificate’s PEM is also served unauthenticated at /pkgproxy-lxd-cert.pem on the plain listener, so a setup script can fetch it; over the HTTPS endpoint that would be circular.

The setup script detects the client’s lxc config directory ($HOME/.config/lxc, or $HOME/snap/lxd/common/config for the snap client, told apart by which config.yml exists, since a shell script cannot see the snap wrapper’s own $HOME/$LXD_CONF redirection), then for each remote converges and pins (§17.11). -undo reverses it for the remotes that point at pkgproxy. With -lxd-tls-addr off no script is printed, as there is no working URL; the operator is pointed at fronting the plain listener with their own TLS termination instead.

17.11 Default remote names and convergence

LXD’s built-in remotes (images, ubuntu, ubuntu-daily, ubuntu-minimal(-daily), local) are permanent, unremovable Static entries: lxc remote add/remote remove both refuse to touch them, regardless of protocol. So the defaults use renamed equivalents instead — pkgproxy-images, pkgproxy-ubuntu, pkgproxy-ubuntu-daily — which need lxc launch pkgproxy-ubuntu:24.04 rather than lxc launch ubuntu:24.04.

craft-providers (snapcraft/rockcraft/charmcraft’s LXD provisioner) doesn’t use those names at all: it hardcodes its own, craft-com.ubuntu.cloud-buildd and craft-com.ubuntu.cloud-buildd-daily, and only creates them if absent, never checking the existing address. Neither name is LXD-static, so pre-registering them, pointing at pkgproxy, makes every craft-provider build cached with no tool-side configuration. Both are among the defaults.

A machine that has run snapcraft already has these two registered against the real upstream. The setup script converges such a remote rather than skipping it, since skipping forfeits caching in exactly the case that matters most: it queries the existing address (lxc remote list --format csv), removes the remote if the address differs from pkgproxy’s, and adds it pointing at pkgproxy, then pins pkgproxy’s certificate for it. This is safe because pkgproxy knows the upstream it proxies, so undo can point the remote back. A remote already pointing at pkgproxy is left alone, so a repeat run is a no-op.

-undo touches only a remote that points at pkgproxy: one whose URL is https://, on any host and port, with the path pkgproxy serves that remote at (<-lxd-base-path>/<name>), so a remote added while pkgproxy listened on another address is found too. It removes such a remote and the certificate setup pinned for it, except a craft-providers remote (craft-com.ubuntu.cloud-buildd*), which it points back at its real upstream, as craft-providers itself would add it. A remote that does not exist, or points anywhere else, is left alone, its certificate included. So undoing before setting up, or twice, changes nothing, and after setup and undo only the craft-providers remotes remain, on their upstream. Undo never adds a pkgproxy-* remote the user did not have.

A transparent, name-independent alternative (HTTPS_PROXY, or LXD’s own core.proxy_https) is rejected: both are CONNECT-tunnel forward proxies, giving pkgproxy the destination hostname but never the plaintext, the same limitation the project accepts for HTTPS-only apt archives (§3). Caching would require MITM-decrypting traffic to a domain pkgproxy doesn’t own, which the project rules out. Named, converged remotes are the only way to cache within that constraint.

17.12 Persisting the blob allowlist across restarts

The learned blob hashes behind §17.3’s allowlist are checkpointed to <cache-dir>/lxd-index.json, as internal/stats checkpoints its counters (§13.4), every -lxd-checkpoint-interval and on shutdown. They must outlive a restart because LXD does not refetch a remote’s catalog before each blob download: it relies on what it learned earlier, so after a restart with an empty allowlist a blob request comes alone and would get a 404 (observed with snapcraft pack).

Only the product catalogs’ blob hashes are kept, not the parsed index.json: the index’s hash is short-lived, revalidated metadata and only opportunistically checked (§17.4), so nothing depends on it surviving a restart.

apt’s internal/index stays in memory: apt’s policy for an unknown pool/** file is to fetch it with a logged gap rather than refuse it, so an empty index after a restart costs verification, not a 404.

17.13 `setup lxd-remote` refuses to run as root

setup apt, setup snap and setup docker write root-owned system files and need sudo. setup lxd-remote does not: lxc remote add/remove and the pinned certificate under servercerts only touch the invoking user’s lxc client configuration. Piped to sudo sh, the script would succeed without error but register the remotes in root’s ~/.config/lxc, since sudo resets $HOME, where the operator’s own lxc and snapcraft never see them.

A system-wide registration is not possible: a remote:image name is a client-side alias that lxc resolves before talking to the daemon (the daemon has no notion of remotes), lxc remote add has no system-wide form and lxc reads no configuration under /etc, and craft-providers runs lxc with the caller’s environment. Every user who wants cached LXD images registers the remotes in their own account.

Because the failure is silent, the script’s first line, in both the apply and the undo direction, refuses to run as root, before the configuration directory is looked up. The TTY guidance says so, and setup lxd and setup snapcraft point at setup lxd-remote as the separate step that caches LXD’s own base images.