pkgproxySnap StoreSource
Contents · Build containers

pkgproxy design: build containers

Pointing LXD build containers at pkgproxy (setup lxd, setup snapcraft), and the admin API socket that lets that run without root. Part of the pkgproxy design, whose index says which file holds each § number.

16.6 LXD build containers: environment.GOPROXY

pkgproxyctl setup lxd points apt, snapd and the go command in LXD build containers at pkgproxy. Go has no system-wide config file (§16.4), and in a build container neither go env -w (it needs the build user to have run go already) nor /etc/environment (read by pam_env only, and a build step run by lxc exec or craft-providers is not a login session) reliably reaches the build process.

Decision: LXD’s own environment.<NAME> profile config key, environment.GOPROXY=http://<host>:<port>/go,direct. LXD itself puts it in the environment of every process in the container, whatever the shell, session or user.

It is applied unconditionally, as apt and snapd are: a container that does not build Go never reads it, so there is no opt-out flag.

The same key carries SNAPPY_FORCE_API_URL=http://<host>:<port>, for the snap command run in the container rather than for snapd, which the drop-in configures. craft-parts fetches a part’s stage-snaps with snap download, and that client asks the store itself, reading the variable from its own environment: without it, rockcraft’s pebble part, in every rock, and any stage-snaps of a snapcraft or charmcraft part, went straight to the store. Undo unsets it with GOPROXY.

16.7 setup lxd: one application path (piped | sh), not two

Every setup subcommand applies its configuration one way: by piping its printed output through sh. There is no Go-side --apply; where the naive command falls short, the printed script is fixed instead:

  • Standalone profile (-project unset): lxc profile edit does not create a missing profile, so the script runs lxc profile show local:pkgproxy >/dev/null 2>&1 || lxc profile create local:pkgproxy first.
  • -project (the project’s default profile, which craft-providers uses): lxc profile edit is a full-document PUT, and LXD refuses to drop the devices of a profile an instance relies on (At least one instance relies on this profile's root disk device). The script therefore sends a PATCH to LXD’s API (lxc query -X PATCH -d '...' 'local:/1.0/profiles/default?project=P'), which merges the given config and devices keys and leaves the rest of the profile alone (setup snapcraft, setup rockcraft and setup charmcraft are this, for the project each tool builds in, which craft-application names after the tool). It also deletes stale *-base-instance-* containers, so the next build does not reuse a base image made before the profile changed.
  • A craft tool’s project before its first build: craft-providers creates the project of snapcraft, rockcraft or charmcraft on its first build, so on a machine where the tool never ran the PATCH finds no project. The script creates it the way craft-providers does (craft_providers.lxd.project.create_with_default_profile): lxc project create local:P, with no config, so LXD gives it its default features, then the default project’s default profile, as lxc profile show prints it, written to the new project’s default profile with lxc profile edit (a new project’s default profile is empty, with no root disk and no nic). A later build finds the project as it would have made it, and leaves it alone. The profile goes through a shell variable, not a pipe, so that a failing lxc profile show stops the script rather than writing an empty profile. This is done for any project named with -project, as setup snapcraft names snapcraft: a project the script must change, and that does not exist yet, is created as the craft tools would create it.

Every lxc command names the local: remote (lxc profile edit local:pkgproxy, lxc list local: --project P, lxc delete --project P -f local:<name>). The admin socket, the socket check and the provisioning files are always on this machine, and without it the lxc commands would go to the user’s default remote, which may be another server. Sending the PATCH with lxc query rather than curl on LXD’s socket keeps one client and one remote for the whole script, and prints LXD’s own error.

16.8 Disk devices and a raw.lxc pre-start hook, not cloud-init

Nothing may depend on an agent inside the container: craft-providers’ “buildd-base” images have no cloud-init at all, so cloud-config (user.user-data) silently does nothing there.

apt/snapd config → profile disk devices. Two profile disk devices (pkgproxy-apt-conf, pkgproxy-snapd-conf) bind-mount host files written under lxdprovision.Dir() onto the apt and snapd config paths in the container. LXD mounts them before the container’s init runs, so apt and snapd see them from the first boot, with nothing installed inside and no snapd restart. The apt file also sets Acquire::Retries::Delay "false", whatever the container’s apt: an apt older than 3.1.3, such as Ubuntu 24.04’s and that of snapcraft’s build bases, can hang in apt update while pkgproxy is offline (LP: #2003851), and a newer one only retries at once instead of backing off. The file is the same for every container, so it cannot ask, as setup apt does, which apt it is for.

https→http apt sources → a raw.lxc pre-start hook. Ubuntu 24.04+ ships https:// URIs in its deb822 ubuntu.sources, and an https:// source is CONNECT-tunnelled and never cached (§3). A disk device can only mount a static file, so a raw.lxc profile key names an lxc.hook.pre-start script (also under Dir()). LXC runs it on the host with $LXC_ROOTFS_PATH, the unpacked rootfs before the container’s init starts, so a host-side sed -i rewrites Ubuntu’s own archive mirrors to http://, again with no agent inside.

Rejected alternatives:

  • snapcraft’s “external fetch-service” hook (CRAFT_USE_EXTERNAL_FETCH_SERVICE): it also sets Acquire::https::Proxy, but that traffic is only cacheable because the real fetch-service is a TLS MITM. Pointed at pkgproxy it is the same uncached CONNECT tunnel, and becoming a MITM is ruled out (§3, §1).
  • A full replacement sources file bind-mounted over the container’s: it would hardcode the Ubuntu series, and go stale whenever craft-providers moves its base image on; a sed needs no such knowledge.

The hook never fails. LXC treats a non-zero pre-start hook exit as fatal (lxc launch hangs until timeout), so a broken hook would stop every container on the profile from starting. Every step of the script is best-effort, with no set -e, and it always exits 0.

Dir() follows how LXD is installed, not pkgproxy. The hook and the disk-device sources must be visible to the LXD daemon. A snap LXD’s mount namespace has the host’s /var/snap but not /var/lib/<name> (only via /var/lib/snapd/hostfs), and a missing hook fails the container’s start with run_buffer: Script exited with status 127. So lxdprovision.Dir() is $SNAP_COMMON/lxd when pkgproxy is a snap; else /var/snap/pkgproxy/common/lxd when the snap LXD’s socket exists; else (a deb LXD, in the host namespace) /var/lib/pkgproxy/lxd. The files and the profile’s references both go through Dir(), so they always agree.

16.9 A mutating admin API socket, so provisioning needs no root

The provisioning files of §16.8 live in a root-owned directory, while the LXD API calls only need lxd group membership. Decision: the pkgproxy daemon, which can already write there, writes them on request: internal/adminapi’s POST/DELETE /pkgproxy/v1/lxd-provision, on the admin socket (-admin-socket-path, default $SNAP_COMMON/admin.sock, else /run/pkgproxy/admin.sock). The piped scripts ask the daemon (curl --unix-socket <admin-socket> -X POST http://localhost/pkgproxy/v1/lxd-provision -d '{"host":...,"port":...}') instead of writing files, so setup lxd/setup snapcraft need no sudo, apply or undo. Every admin route is under /pkgproxy/v1/ (adminapi.Prefix), as the stats socket’s are (§13.5).

A socket of its own, not a route on the stats socket. Read-only stats are safe to leave broadly reachable (every app in a bundling snap, no plug); a mutating API is a different trust shape, and keeping it apart keeps the stats API GET-only by construction (§13.1). The admin API also serves cache invalidate and offline mode (§21). Nothing restricts who can reach either socket beyond UNIX file permissions, an accepted gap: lxd group membership already grants more (arbitrary containers and host bind-mounts) than this endpoint’s bounded rewrite of three known files.

The LXD API calls stay in the printed script. pkgproxy and LXD each run in their own confined snap, and pkgproxy’s has no interface to LXD’s socket, so the daemon cannot drive LXD at all. lxdprovision.WriteFiles only touches pkgproxy’s own $SNAP_COMMON, and every LXD-facing command is run by the operator’s unconfined shell, the one thing with a path to both.

The request carries the bridge address the script found, the same one it puts in environment.GOPROXY, rather than detecting it again in the daemon, so there is one detection path, not two that could disagree. DELETE takes no body.

The script asks LXD for the bridge, not pkgproxyctl. The snap’s pkgproxyctl has no access to LXD’s socket, and the lxd interface that would give it needs a store grant and gives root-equivalent access; an attempt is denied and logged by AppArmor. The printed script runs as the user, unconfined, in the lxd group, so it asks with lxc on local: (lxdBridgeDetectScript), in POSIX sh without jq. pkgproxyctl only renders the address as ${host}.

The bridge is the one LXD manages that the containers are on. For project P (-project, snapcraft, or default for the standalone profile, and default when P does not exist yet, as craft-providers copies its default profile), the script takes each nic device of P’s default profile, its network (or parent, for nictype: bridged), and keeps it when lxc network show reports managed: true and type: bridge, so an unmanaged bridge such as Docker’s docker0 is never taken. The address is the bridge’s ipv4.address, else its ipv6.address in brackets, without the prefix length, and must consist of hex digits, dots and colons. When no nic is on such a bridge, nics are on different ones, or the bridge has no usable address, the script stops with a cannot ... message saying what to do (lxd init, or one bridge on the profile) before it changes anything.

“No root” means the lxd group. The admin socket is world-writable (listenAt clears the umask), while LXD’s socket (and the snap’s lxd-user socket) is srw-rw---- root lxd. So the scripts need what lxc needs, membership in lxd. sudo is harmless (nothing is written to a home directory, and profiles live on the LXD server), so unlike setup lxd-remote (§17.13) the scripts do not refuse it. Each piped script, apply and undo, starts with [ -w <lxd socket> ] || { echo ...; exit 1; }, naming the group and how to join it (test -w on a socket checks connect permission in dash and bash, and root passes it). curl runs with -sSf, so a failure is shown, and the apply script stops (|| exit 1) when the admin POST fails, so a profile is never pointed at provisioning files that were not written (a container whose disk-device source is missing does not start). Undo is best-effort: it goes on past a key or device that is already gone.

cache invalidate handles apt only: DELETE /pkgproxy/v1/cache resolves the URL with apt’s cachekey.PathResolver against the apt store. The other kinds of download are verified against a hash before they are cached (snap sha3-384, the Go checksum database, LXD and Docker sha256), their mutable lists revalidate after their *-revalidate-ttl, and removing a kind’s directory while pkgproxy is stopped drops it entirely. Invalidating one of their entries by URL would need a resolver per side (snap and Docker keys depend on request headers), which is not worth it while nothing needs it.