LXD image servers
lxd for users for administrators for CI operators
pkgproxy serves LXD's simplestreams image servers over its own HTTPS port, with a pinned self-signed certificate, rechecking the catalogs and keeping every image file it has checked.
On the map: lxd
01 What this topic covers
pkgproxy serves LXD's simplestreams image servers over its own HTTPS port. It rechecks the catalogs and keeps every image file it has checked.
pkgproxy's LXD image servers let lxc launch images from pkgproxy's cache instead of the internet. First you see the image server it runs on an HTTPS port, and how you pin its certificate. Then come the five remotes it serves by default, why catalogs are rechecked and image files are kept, and what changed with LXD 6.10. Administrators also see the LXD settings.
02 An image server on an HTTPS port
lxc remote add NAME https://host:3143/lxd/NAME --protocol=simplestreams.
lxc refuses a simplestreams remote that is not https://, so
pkgproxy has a TLS listener, -lxd-tls-addr, with a
self-signed certificate kept under lxd/tls in the cache
directory.
LXD downloads images from image servers that speak simplestreams: JSON catalogs of products, and the image files they list. pkgproxy serves the same protocol, so lxc adds it as a remote with the simplestreams protocol, at an HTTPS address on port 3143, under /lxd, followed by the remote's name. Lxc accepts only HTTPS for such a remote, so pkgproxy runs a second listener with TLS. Its certificate is self signed, made once and kept in the cache directory. Behind each remote name is one real image server.
03 Pinning the certificate, as yourself
--accept-certificate is not read for simplestreams remotes.
pkgproxyctl setup lxd-remote | sh adds each remote and
fetches /pkgproxy-lxd-cert.pem into lxc's
servercerts/NAME.crt (under ~/snap/lxd/common/config for the LXD snap, else ~/.config/lxc). It refuses sudo, which
would configure root's lxc instead.
Lxc remote add has an accept certificate option, but lxc ignores it for image servers, so a self signed certificate added that way is never trusted. Lxc does trust a certificate it finds in its configuration directory, under servercerts, named after the remote. pkgproxyctl setup lxd-remote prints a script that adds each remote and downloads pkgproxy's certificate there, from the plain port, 3142. Run it as yourself, not with sudo: it changes your own lxc configuration, and refuses to run under sudo.
04 Five remotes served by default
The defaults: pkgproxy-images, pkgproxy-ubuntu,
pkgproxy-ubuntu-daily, and craft-providers'
craft-com.ubuntu.cloud-buildd and
craft-com.ubuntu.cloud-buildd-daily, which setup converges
onto pkgproxy and -undo restores.
With no remote flag, pkgproxy serves five remotes. LXD's own names, images, ubuntu and ubuntu daily, are built into lxc and cannot be replaced, so three of the five carry a pkgproxy prefix: you launch from pkgproxy-ubuntu instead of ubuntu. The other two are the names snapcraft, rockcraft and charmcraft use for their build images. Setup adds them too, or points them at pkgproxy where snapcraft already made them, so every craft build is cached with no change to the tool. Undo points them back at the real server.
05 Catalogs rechecked, image files kept
streams/v1/index.json and the catalogs revalidate after
-lxd-revalidate-ttl (default: -revalidate-ttl, 5m). Image files are allowlisted by the
catalogs, SHA-256 verified and cached forever; the allowlist is saved in
lxd-index.json every -lxd-checkpoint-interval
(30s) and on shutdown.
Lxc first reads index.json, which names the product catalogs, then a catalog such as images.json. These lists change, so they are served from the cache for five minutes and then checked again. On the way through, pkgproxy reads each catalog and learns the SHA-256 of every image file it lists. An image file is fetched once, checked against that hash and kept forever. A path no catalog listed is refused with a 404 and never fetched. The learned hashes are saved to disk every thirty seconds, so they survive a restart.
06 LXD 6.10 and image registries
On LXD 6.10 setup lxd-remote adds an lxd image
registry per remote over one public cluster link, which pins pkgproxy's
certificate. pkgproxy serves LXD's image API, read-only, on the HTTPS
port. Registries belong to the LXD server, so they serve every user of it.
LXD 6.10 changed how the server downloads an image. It no longer uses the address of the lxc remote but an image registry, and a simplestreams registry trusts only the system's certificate authorities, not a self signed certificate. So on 6.10 the setup script also makes one public cluster link to pkgproxy, which pins its certificate, and one image registry for each remote, named after it. pkgproxy answers the read only part of LXD's API that downloads images, from the same cache. An LXD upgraded after setup needs setup run again.
07 The LXD settings for administrators
-lxd-remote name=url (repeatable; name= drops a
default; off), -lxd-tls-addr (:3143),
-lxd-base-path (/lxd),
-lxd-revalidate-ttl (default: -revalidate-ttl, 5m), -lxd-checkpoint-interval
(30s).
The LXD side has a handful of settings. The -lxd-remote flag adds or replaces a remote as a name equal to a URL, an empty URL drops a default, and off turns the whole side off. The -lxd-tls-addr sets the HTTPS port, 3143 by default, or off. Remotes are served under /lxd, the -lxd-base-path. Image lists are rechecked after the -lxd-revalidate-ttl, five minutes, and the learned hashes are saved every -lxd-checkpoint-interval, thirty seconds. The TLS address, the TTL and the checkpoint interval are also snap options.
08 Where to go next for CI operators in place of the general version
Next: Build containers, for what runs inside them; pkgproxy for CI, for many machines behind one cache.
Lxc now launches images through pkgproxy, and each image file comes from the internet once. To point the build containers themselves at pkgproxy, watch Build containers. To put many build machines behind one cache, watch pkgproxy for CI.
09 Where to go next general version
Next: Build containers, for what runs inside them; Fetch once, for one shared image download; Offline mode, for launching with no network.
Lxc now launches images through pkgproxy, and each image file comes from the internet once. To point apt, snapd and Go inside the containers at pkgproxy too, watch Build containers. To see how one download of a large image is shared by every client, watch Fetch once. To launch from the cache with no network, watch Offline mode.
Read more
The default remotes
With no -lxd-remote at all, these five are served. The address lxc uses is https://HOST:3143/lxd/NAME.
| Remote | Real image server | Launch with |
|---|---|---|
pkgproxy-images | https://images.lxd.canonical.com | lxc launch pkgproxy-images:debian/12 |
pkgproxy-ubuntu | https://cloud-images.ubuntu.com/releases | lxc launch pkgproxy-ubuntu:24.04 |
pkgproxy-ubuntu-daily | https://cloud-images.ubuntu.com/daily | lxc launch pkgproxy-ubuntu-daily:24.04 |
craft-com.ubuntu.cloud-buildd | https://cloud-images.ubuntu.com/buildd/releases | used by snapcraft, rockcraft and charmcraft |
craft-com.ubuntu.cloud-buildd-daily | https://cloud-images.ubuntu.com/buildd/daily | used by snapcraft, rockcraft and charmcraft |
-lxd-remote NAME=URL adds a remote or replaces a default of that name, -lxd-remote NAME= drops a default, and -lxd-remote off turns the LXD side off. The option repeats, once per remote. A name may not be empty, . or .., or hold /, :, ,, ?, #, %, white space or a control character.
Paths under /lxd
Paths are relative to a remote, so /lxd/pkgproxy-ubuntu/streams/v1/index.json is fetched from https://cloud-images.ubuntu.com/releases/streams/v1/index.json. Only GET and HEAD are answered; anything else is a 405.
Path under /lxd/NAME/ | What it is | Cache policy |
|---|---|---|
streams/v1/index.json | the index of the remote's catalogs | revalidated after -lxd-revalidate-ttl; not verified, as nothing lists its hash |
any other path ending in .json | a product catalog, such as streams/v1/images.json | revalidated after -lxd-revalidate-ttl; read for the SHA-256 of every file it lists |
| a file a catalog listed | an image file: metadata tarball, squashfs root, VM disk | fetched once, checked against the catalog's SHA-256, kept forever |
| a file no catalog listed | — | 404, never fetched |
| a remote that is not configured, or no path after the name | — | 404 |
a path that climbs out with .. | — | 400 |
A failed recheck of a catalog serves the cached copy, as on the other sides. A HEAD for an image file that is not cached yet is answered from the catalog, with its size, without fetching it. An image file that fails its hash check is not cached, but the clients already receiving it get the upstream's bytes unchanged, for at most -fill-drain-timeout; lxc checks the image itself. A client's Authorization header is never sent upstream, so private image servers are not supported.
Which port serves what
| Path | Main port, 3142 | HTTPS port, 3143 |
|---|---|---|
/lxd/NAME/… | yes, over plain HTTP, for curl and scripts | yes; the only form lxc accepts |
/pkgproxy-lxd-cert.pem | yes: the certificate, to pin | no |
/, /1.0, /1.0/images/… | no | yes: LXD's image API, for LXD 6.10 |
LXD's image API
LXD 6.10 downloads images through an image registry, which asks the HTTPS port this read-only part of LXD's own API. ?project=NAME names the remote; the answers come from the same cached catalogs and image files.
| Request | Answer |
|---|---|
GET /, GET /1.0 | a public server with no API extensions |
GET /1.0/images, with or without ?recursion=1 | the remote's images for the requester's architecture |
GET /1.0/images/aliases/ALIAS | the image the alias names: the container image, or the virtual machine image when there is no container image |
GET /1.0/images/FINGERPRINT | the image, by its fingerprint or a unique prefix of it |
GET /1.0/images/FINGERPRINT/export | its files: one for a unified image, or metadata and then rootfs or rootfs.img as a multipart answer |
The requester's architecture comes from LXD's User-Agent. These requests appear in the access log with handler=lxd-api.
for administrators for CI operators
Files on disk
Under the cache directory, /var/snap/pkgproxy/common/cache in the snap and /var/cache/pkgproxy otherwise:
| Path | What it holds |
|---|---|
lxd/objects/NAME/… | the cached catalogs and image files, one directory per remote, each file with its metadata in an extended attribute |
lxd/tls/cert.pem | the self-signed certificate of the HTTPS port: ECDSA P-256, valid for ten years |
lxd/tls/key.pem | its private key, readable by its owner only |
lxd-index.json | the SHA-256 of every image file the catalogs listed, saved every -lxd-checkpoint-interval and on shutdown |
Keep lxd/tls when moving or clearing the cache: a new certificate breaks every remote pinned to the old one until pkgproxyctl setup lxd-remote runs again. Without lxd-index.json, a request for an image file whose catalog has not been read since the restart gets a 404.
for administrators for CI operators
Settings
| Option | Environment variable | Snap option | Default | Meaning |
|---|---|---|---|---|
-lxd-remote | PKGPROXY_LXD_REMOTE | flag or environment only | the five above | remotes to serve, as NAME=URL; off serves none |
-lxd-base-path | PKGPROXY_LXD_BASE_PATH | flag or environment only | /lxd | the URL path the remotes are served under |
-lxd-tls-addr | PKGPROXY_LXD_TLS_ADDR | lxd-tls-addr | :3143 | address of the HTTPS port, or off |
-lxd-tls-socket-path | PKGPROXY_LXD_TLS_SOCKET_PATH | flag or environment only | none | serve the HTTPS port on this UNIX socket instead; setup scripts still name -lxd-tls-addr |
-lxd-tls-content-socket-path | PKGPROXY_LXD_TLS_CONTENT_SOCKET_PATH | flag or environment only | none; the snap sets /var/snap/pkgproxy/common/sockets/pkgproxy-lxd-tls.sock | also serve the HTTPS port on this UNIX socket, to share it with another snap |
-lxd-revalidate-ttl | PKGPROXY_LXD_REVALIDATE_TTL | lxd-revalidate-ttl | -revalidate-ttl, 5m | how long index.json and catalogs are served before they are checked again |
-lxd-checkpoint-interval | PKGPROXY_LXD_CHECKPOINT_INTERVAL | lxd-checkpoint-interval | 30s | how often the learned image hashes are saved |
With -lxd-tls-addr off, pkgproxyctl setup lxd-remote prints no script, since lxc has no HTTPS address to use. All settings are in Settings; what the setup script changes is in Setting up clients.
In the documentation
Roads from here
CORE · the path every side shares
- Immutable or revalidated — why catalogs are rechecked and image files are not
- Fetch once — one download of a large image shared by every client
- Offline mode — launching from the cache with no network
LXD
- LXD build containers — point the build containers themselves at pkgproxy
SETUP AND THE COMMAND LINE
- Setting up clients — the other setup commands and -undo
OPERATIONS
- pkgproxy for CI — many machines building against one cache