LXD image servers

lxd for users for administrators for CI operators

pkgproxy serves LXD's simplestreams image servers over its own HTTPS port, with a pinned self-signed certificate, rechecking the catalogs and keeping every image file it has checked.

On the map: lxd

▶ Watch this deck — 9 scenes, ~5 min

01 What this topic covers

ABOUT THIS TOPIC LXD image servers lxc launches images from pkgproxy's cache instead of the internet 1 An image server on an HTTPS port 2 Pinning the certificate, as yourself 3 Five remotes served by default 4 Catalogs rechecked, image files kept 5 LXD 6.10 and image registries 6 The LXD settings
The topic title, LXD image servers, and its six scenes in order: the HTTPS remote, pinning the certificate, five default remotes, catalogs and images, LXD 6.10 registries, the settings

pkgproxy serves LXD's simplestreams image servers over its own HTTPS port. It rechecks the catalogs and keeps every image file it has checked.

pkgproxy's LXD image servers let lxc launch images from pkgproxy's cache instead of the internet. First you see the image server it runs on an HTTPS port, and how you pin its certificate. Then come the five remotes it serves by default, why catalogs are rechecked and image files are kept, and what changed with LXD 6.10. Administrators also see the LXD settings.

02 An image server on an HTTPS port

HTTPS REMOTE lxc cloud-images.ubuntu.com /releases JSON catalogs and image files $ lxc remote add pkgproxy-ubuntu \ https://host:3143/lxd/pkgproxy-ubuntu \ --protocol=simplestreams pkgproxy TLS on :3143 https only self-signed certificate made once, kept in the cache directory under lxd/tls per name
lxc adds pkgproxy as a simplestreams remote at an https address on port 3143; pkgproxy answers over TLS with a self-signed certificate kept on disk and relays each remote name to one real image server

lxc remote add NAME https://host:3143/lxd/NAME --protocol=simplestreams. lxc refuses a simplestreams remote that is not https://, so pkgproxy has a TLS listener, -lxd-tls-addr, with a self-signed certificate kept under lxd/tls in the cache directory.

LXD downloads images from image servers that speak simplestreams: JSON catalogs of products, and the image files they list. pkgproxy serves the same protocol, so lxc adds it as a remote with the simplestreams protocol, at an HTTPS address on port 3143, under /lxd, followed by the remote's name. Lxc accepts only HTTPS for such a remote, so pkgproxy runs a second listener with TLS. Its certificate is self signed, made once and kept in the cache directory. Behind each remote name is one real image server.

03 Pinning the certificate, as yourself

PINNING $ lxc remote add … --accept-certificate ignored for image servers lxc trusts a certificate here, one per remote $LXC_CONF_DIR/servercerts/pkgproxy-ubuntu.crt pkgproxy :3142 /pkgproxy-lxd-cert.pem plain HTTP $ pkgproxyctl setup lxd-remote | sh lxc remote add pkgproxy-ubuntu https://host:3143/lxd/pkgproxy-ubuntu --protocol=simplestreams curl -fsS http://host:3142/pkgproxy-lxd-cert.pem -o "$LXC_CONF_DIR/servercerts/pkgproxy-ubuntu.crt" run it as yourself, not with sudo: it refuses sudo would change root's lxc configuration, which your lxc and snapcraft never read
An lxc remote add with --accept-certificate is struck out; instead the setup script downloads pkgproxy's certificate over plain HTTP into lxc's servercerts directory, one file per remote, and refuses to run under sudo

--accept-certificate is not read for simplestreams remotes. pkgproxyctl setup lxd-remote | sh adds each remote and fetches /pkgproxy-lxd-cert.pem into lxc's servercerts/NAME.crt (under ~/snap/lxd/common/config for the LXD snap, else ~/.config/lxc). It refuses sudo, which would configure root's lxc instead.

Lxc remote add has an accept certificate option, but lxc ignores it for image servers, so a self signed certificate added that way is never trusted. Lxc does trust a certificate it finds in its configuration directory, under servercerts, named after the remote. pkgproxyctl setup lxd-remote prints a script that adds each remote and downloads pkgproxy's certificate there, from the plain port, 3142. Run it as yourself, not with sudo: it changes your own lxc configuration, and refuses to run under sudo.

04 Five remotes served by default

DEFAULT REMOTES no flags needed: five remotes images: ubuntu: ubuntu-daily: built into lxc, cannot be replaced pkgproxy-images images.lxd.canonical.com pkgproxy-ubuntu cloud-images.ubuntu.com/releases pkgproxy-ubuntu-daily cloud-images.ubuntu.com/daily $ lxc launch pkgproxy-ubuntu:24.04 my-container the names snapcraft, rockcraft and charmcraft use for build images craft-com.ubuntu.cloud-buildd cloud-images.ubuntu.com/buildd/releases craft-com.ubuntu.cloud-buildd-daily cloud-images.ubuntu.com/buildd/daily setup adds them, or points existing ones at pkgproxy -undo points them back
lxc's built-in remotes images, ubuntu and ubuntu-daily cannot be replaced; pkgproxy's five default remotes map pkgproxy-images, pkgproxy-ubuntu, pkgproxy-ubuntu-daily and the two craft-com.ubuntu.cloud-buildd remotes to their real image servers

The defaults: pkgproxy-images, pkgproxy-ubuntu, pkgproxy-ubuntu-daily, and craft-providers' craft-com.ubuntu.cloud-buildd and craft-com.ubuntu.cloud-buildd-daily, which setup converges onto pkgproxy and -undo restores.

With no remote flag, pkgproxy serves five remotes. LXD's own names, images, ubuntu and ubuntu daily, are built into lxc and cannot be replaced, so three of the five carry a pkgproxy prefix: you launch from pkgproxy-ubuntu instead of ubuntu. The other two are the names snapcraft, rockcraft and charmcraft use for their build images. Setup adds them too, or points them at pkgproxy where snapcraft already made them, so every craft build is cached with no change to the tool. Undo points them back at the real server.

05 Catalogs rechecked, image files kept

CATALOGS AND IMAGES streams/v1/index.json streams/v1/images.json names the catalogs lists every image file rechecked after 5 minutes …/noble/…-amd64.squashfssha256 9c1e… …/noble/…-lxd.tar.xzsha256 41b7… learned on the way through: path → SHA-256 …/noble/…-amd64.squashfs fetched once, hash checked, kept forever some/unlisted/path 404, never fetched lxd-index.json saved every 30 seconds: survives a restart
index.json leads to images.json, both rechecked after five minutes; the catalog teaches pkgproxy a SHA-256 for each image file path; a listed image file is fetched once, checked and kept forever; an unlisted path gets 404; the learned hashes are saved to lxd-index.json

streams/v1/index.json and the catalogs revalidate after -lxd-revalidate-ttl (default: -revalidate-ttl, 5m). Image files are allowlisted by the catalogs, SHA-256 verified and cached forever; the allowlist is saved in lxd-index.json every -lxd-checkpoint-interval (30s) and on shutdown.

Lxc first reads index.json, which names the product catalogs, then a catalog such as images.json. These lists change, so they are served from the cache for five minutes and then checked again. On the way through, pkgproxy reads each catalog and learns the SHA-256 of every image file it lists. An image file is fetched once, checked against that hash and kept forever. A path no catalog listed is refused with a 404 and never fetched. The learned hashes are saved to disk every thirty seconds, so they survive a restart.

06 LXD 6.10 and image registries

LXD 6.10 lxc LXD 6.10 server simplestreams registry trusts the system's CAs only: no self-signed certificate registry pkgproxy-ubuntu public cluster link, pinned pkgproxy :3143 LXD image API, read only, same cache $ pkgproxyctl setup lxd-remote | sh # again, after upgrading LXD
lxc asks an LXD 6.10 server, which downloads through an image registry over a public cluster link that pins pkgproxy's certificate; a simplestreams registry is struck out because it trusts the system's CAs only; pkgproxy answers LXD's read-only image API on port 3143

On LXD 6.10 setup lxd-remote adds an lxd image registry per remote over one public cluster link, which pins pkgproxy's certificate. pkgproxy serves LXD's image API, read-only, on the HTTPS port. Registries belong to the LXD server, so they serve every user of it.

LXD 6.10 changed how the server downloads an image. It no longer uses the address of the lxc remote but an image registry, and a simplestreams registry trusts only the system's certificate authorities, not a self signed certificate. So on 6.10 the setup script also makes one public cluster link to pkgproxy, which pins its certificate, and one image registry for each remote, named after it. pkgproxy answers the read only part of LXD's API that downloads images, from the same cache. An LXD upgraded after setup needs setup run again.

07 The LXD settings for administrators

LXD SETTINGS -lxd-remote name=url add or replace a remote; repeatable -lxd-remote name= drop a default remote -lxd-remote off no LXD side at all -lxd-tls-addr :3143 the HTTPS port, or off -lxd-base-path /lxd remotes at /lxd/NAME -lxd-revalidate-ttl 5m recheck image lists -lxd-checkpoint-interval 30s save learned hashes also snap options: lxd-tls-addr, lxd-revalidate-ttl, lxd-checkpoint-interval $ sudo snap set pkgproxy lxd-revalidate-ttl=10m
The LXD flags with their defaults: -lxd-remote, -lxd-tls-addr :3143, -lxd-base-path /lxd, -lxd-revalidate-ttl 5m and -lxd-checkpoint-interval 30s, and which of them are snap options

-lxd-remote name=url (repeatable; name= drops a default; off), -lxd-tls-addr (:3143), -lxd-base-path (/lxd), -lxd-revalidate-ttl (default: -revalidate-ttl, 5m), -lxd-checkpoint-interval (30s).

The LXD side has a handful of settings. The -lxd-remote flag adds or replaces a remote as a name equal to a URL, an empty URL drops a default, and off turns the whole side off. The -lxd-tls-addr sets the HTTPS port, 3143 by default, or off. Remotes are served under /lxd, the -lxd-base-path. Image lists are rechecked after the -lxd-revalidate-ttl, five minutes, and the learned hashes are saved every -lxd-checkpoint-interval, thirty seconds. The TLS address, the TTL and the checkpoint interval are also snap options.

08 Where to go next for CI operators in place of the general version

WHERE NEXT lxc launches images through pkgproxy, and each image file comes from the internet once you are here LXD image servers Build containers apt, snapd and Go inside the containers pkgproxy for CI many build machines behind one cache
You are here, LXD image servers; next: Build containers, pkgproxy for CI

Next: Build containers, for what runs inside them; pkgproxy for CI, for many machines behind one cache.

Lxc now launches images through pkgproxy, and each image file comes from the internet once. To point the build containers themselves at pkgproxy, watch Build containers. To put many build machines behind one cache, watch pkgproxy for CI.

09 Where to go next general version

WHERE NEXT lxc launches images through pkgproxy, and each image file comes from the internet once you are here LXD image servers Build containers apt, snapd and Go inside the containers Fetch once one download of a large image, shared Offline mode launching from the cache with no network
You are here, LXD image servers; next: Build containers, Fetch once, Offline mode

Next: Build containers, for what runs inside them; Fetch once, for one shared image download; Offline mode, for launching with no network.

Lxc now launches images through pkgproxy, and each image file comes from the internet once. To point apt, snapd and Go inside the containers at pkgproxy too, watch Build containers. To see how one download of a large image is shared by every client, watch Fetch once. To launch from the cache with no network, watch Offline mode.

Read more

The default remotes

With no -lxd-remote at all, these five are served. The address lxc uses is https://HOST:3143/lxd/NAME.

RemoteReal image serverLaunch with
pkgproxy-imageshttps://images.lxd.canonical.comlxc launch pkgproxy-images:debian/12
pkgproxy-ubuntuhttps://cloud-images.ubuntu.com/releaseslxc launch pkgproxy-ubuntu:24.04
pkgproxy-ubuntu-dailyhttps://cloud-images.ubuntu.com/dailylxc launch pkgproxy-ubuntu-daily:24.04
craft-com.ubuntu.cloud-builddhttps://cloud-images.ubuntu.com/buildd/releasesused by snapcraft, rockcraft and charmcraft
craft-com.ubuntu.cloud-buildd-dailyhttps://cloud-images.ubuntu.com/buildd/dailyused by snapcraft, rockcraft and charmcraft

-lxd-remote NAME=URL adds a remote or replaces a default of that name, -lxd-remote NAME= drops a default, and -lxd-remote off turns the LXD side off. The option repeats, once per remote. A name may not be empty, . or .., or hold /, :, ,, ?, #, %, white space or a control character.

Paths under /lxd

Paths are relative to a remote, so /lxd/pkgproxy-ubuntu/streams/v1/index.json is fetched from https://cloud-images.ubuntu.com/releases/streams/v1/index.json. Only GET and HEAD are answered; anything else is a 405.

Path under /lxd/NAME/What it isCache policy
streams/v1/index.jsonthe index of the remote's catalogsrevalidated after -lxd-revalidate-ttl; not verified, as nothing lists its hash
any other path ending in .jsona product catalog, such as streams/v1/images.jsonrevalidated after -lxd-revalidate-ttl; read for the SHA-256 of every file it lists
a file a catalog listedan image file: metadata tarball, squashfs root, VM diskfetched once, checked against the catalog's SHA-256, kept forever
a file no catalog listed—404, never fetched
a remote that is not configured, or no path after the name—404
a path that climbs out with ..—400

A failed recheck of a catalog serves the cached copy, as on the other sides. A HEAD for an image file that is not cached yet is answered from the catalog, with its size, without fetching it. An image file that fails its hash check is not cached, but the clients already receiving it get the upstream's bytes unchanged, for at most -fill-drain-timeout; lxc checks the image itself. A client's Authorization header is never sent upstream, so private image servers are not supported.

Which port serves what

PathMain port, 3142HTTPS port, 3143
/lxd/NAME/…yes, over plain HTTP, for curl and scriptsyes; the only form lxc accepts
/pkgproxy-lxd-cert.pemyes: the certificate, to pinno
/, /1.0, /1.0/images/…noyes: LXD's image API, for LXD 6.10

LXD's image API

LXD 6.10 downloads images through an image registry, which asks the HTTPS port this read-only part of LXD's own API. ?project=NAME names the remote; the answers come from the same cached catalogs and image files.

RequestAnswer
GET /, GET /1.0a public server with no API extensions
GET /1.0/images, with or without ?recursion=1the remote's images for the requester's architecture
GET /1.0/images/aliases/ALIASthe image the alias names: the container image, or the virtual machine image when there is no container image
GET /1.0/images/FINGERPRINTthe image, by its fingerprint or a unique prefix of it
GET /1.0/images/FINGERPRINT/exportits files: one for a unified image, or metadata and then rootfs or rootfs.img as a multipart answer

The requester's architecture comes from LXD's User-Agent. These requests appear in the access log with handler=lxd-api.

for administrators for CI operators

Files on disk

Under the cache directory, /var/snap/pkgproxy/common/cache in the snap and /var/cache/pkgproxy otherwise:

PathWhat it holds
lxd/objects/NAME/…the cached catalogs and image files, one directory per remote, each file with its metadata in an extended attribute
lxd/tls/cert.pemthe self-signed certificate of the HTTPS port: ECDSA P-256, valid for ten years
lxd/tls/key.pemits private key, readable by its owner only
lxd-index.jsonthe SHA-256 of every image file the catalogs listed, saved every -lxd-checkpoint-interval and on shutdown

Keep lxd/tls when moving or clearing the cache: a new certificate breaks every remote pinned to the old one until pkgproxyctl setup lxd-remote runs again. Without lxd-index.json, a request for an image file whose catalog has not been read since the restart gets a 404.

for administrators for CI operators

Settings

OptionEnvironment variableSnap optionDefaultMeaning
-lxd-remotePKGPROXY_LXD_REMOTEflag or environment onlythe five aboveremotes to serve, as NAME=URL; off serves none
-lxd-base-pathPKGPROXY_LXD_BASE_PATHflag or environment only/lxdthe URL path the remotes are served under
-lxd-tls-addrPKGPROXY_LXD_TLS_ADDRlxd-tls-addr:3143address of the HTTPS port, or off
-lxd-tls-socket-pathPKGPROXY_LXD_TLS_SOCKET_PATHflag or environment onlynoneserve the HTTPS port on this UNIX socket instead; setup scripts still name -lxd-tls-addr
-lxd-tls-content-socket-pathPKGPROXY_LXD_TLS_CONTENT_SOCKET_PATHflag or environment onlynone; the snap sets /var/snap/pkgproxy/common/sockets/pkgproxy-lxd-tls.sockalso serve the HTTPS port on this UNIX socket, to share it with another snap
-lxd-revalidate-ttlPKGPROXY_LXD_REVALIDATE_TTLlxd-revalidate-ttl-revalidate-ttl, 5mhow long index.json and catalogs are served before they are checked again
-lxd-checkpoint-intervalPKGPROXY_LXD_CHECKPOINT_INTERVALlxd-checkpoint-interval30show often the learned image hashes are saved

With -lxd-tls-addr off, pkgproxyctl setup lxd-remote prints no script, since lxc has no HTTPS address to use. All settings are in Settings; what the setup script changes is in Setting up clients.

Roads from here

CORE · the path every side shares

LXD

SETUP AND THE COMMAND LINE

OPERATIONS