Why the craft components keep no cache

the pkgproxy-url input for CI operators

The craft components keep nothing between jobs. A pkgproxy at an address the VM can reach, named by the pkgproxy-url input or the runner's PKGPROXY_URL, with GARDEN_DL_PKGPROXY for the cloud image, makes the second build fast; setup scripts point apt, snapd and Go at it.

On the map: built on it

▶ Watch this deck — 6 scenes, ~5 min

Worth seeing first

01 Every job starts from nothing

FROM NOTHING job 1 job 2 cloud image boot the image snapcraft snaps, debs, Go cloud image boot the image snapcraft snaps, debs, Go kept between them: nothing scratch directory deleted, container gone, no GitLab cache pkgproxy, on the runners' network job 2's downloads come from the cache only the boot and the pack are done again
Two jobs in a row each download the cloud image, make and boot the VM's image, install snapcraft and fetch snaps, packages and Go modules; the component deletes its scratch directory, the container is gone and there is no GitLab cache; a pkgproxy on the runners' network serves the downloads of the second job from its cache, leaving only the boot and the pack

Each job downloads the cloud image, snapcraft, snaps, packages and Go modules again. The component keeps no cache between jobs; a pkgproxy on the runners' network serves them from its cache after the first time.

Run the same snap build twice, and look at what the second job does. Exactly what the first one did. It downloads the Ubuntu cloud image, makes and boots the VM's image, installs snapcraft, then pulls the snaps, packages and Go modules your build needs. The component keeps no cache of its own: the scratch directory is deleted, the container is gone, and the job has no GitLab cache either. What makes the second build fast is pkgproxy, a caching proxy on your runners' network. After the first job, nearly everything comes from its cache, and only the work itself is done again.

Details and sources

Details

  • "Each job starts from nothing: image-garden downloads the Ubuntu cloud image and makes the VM's image from it, and the VM downloads snapcraft, the snaps and packages the build needs, and its Go modules. The component keeps no cache of its own between jobs: pkgproxy is what makes the second build fast." (snapcraft-gitlab-ci README.md:203-211).
  • The work directory with the prepared image is removed at the end of every job (snapcraft-vm:287-290); the job has no cache: key (templates/build.yml:115-157).
  • rockcraft- and charmcraft-gitlab-ci say the same in their READMEs and work the same way.
  • "Only the work itself" means the provisioning boot and the pack, which run in every job (derived).

02 Two kinds of download, two settings

TWO PATHS the job container the VM image-garden: the cloud image GARDEN_DL_PKGPROXY set on the runner; by checksum apt, snapd, Go in the VM pkgproxy-url: … empty: the runner's PKGPROXY_URL · off: none export PKGPROXY_URL=http://HOST:PORT; image-garden allocate --spread … the VM's image is made already set up for pkgproxy GARDEN_DL_PKGPROXY=http://pkgproxy.lan:3142 PKGPROXY_URL=http://pkgproxy.lan:3142 the runner's environment
Two download paths: image-garden in the job container fetches the cloud image through pkgproxy by checksum when the runner sets GARDEN_DL_PKGPROXY; the VM's downloads follow the pkgproxy-url input, empty for the runner's PKGPROXY_URL and off for none; the script exports the chosen URL as PKGPROXY_URL before allocate so the image is made set up for pkgproxy; two variables in the runner's environment cover every job

The cloud image goes through GARDEN_DL_PKGPROXY, set on the runner. The VM's downloads follow pkgproxy-url: empty means the runner's PKGPROXY_URL, off means none; the URL is exported as PKGPROXY_URL for allocate.

Two different things download, and each finds pkgproxy its own way. The cloud image is fetched by image-garden, in the job container. It goes through pkgproxy, by its checksum, when the runner sets GARDEN_DL_PKGPROXY. Everything inside the VM follows the pkgproxy-url input instead. Leave it empty, and the job uses the runner's PKGPROXY_URL, if the runner sets one; set it to off, and the build runs without pkgproxy. The script exports the URL it picked as PKGPROXY_URL before allocate, so image-garden makes the VM's image already set up for pkgproxy. So two variables in the runner's environment cover every job.

Details and sources

Details

  • image-garden downloads the Ubuntu cloud image through pkgproxy "when the runner sets GARDEN_DL_PKGPROXY (pkgproxy 0.12 or later)"; the VM downloads through it "when the job has a pkgproxy URL that the VM reaches", pkgproxy-url or the runner's PKGPROXY_URL, off for none (snapcraft-gitlab-ci README.md:213-219).
  • pick_pkgproxy: empty takes $PKGPROXY_URL, off none; anything else must match http://HOST[:PORT] and loses a trailing slash (snapcraft-vm:215-229). The URL is exported as PKGPROXY_URL (or PKGPROXY_URL is unset) before image-garden allocate (snapcraft-vm:356-366).
  • With GARDEN_DL_PKGPROXY set, image-garden asks pkgproxy for files/v1/fetch?url=…&sha256=… and falls back to the origin (image-garden:281-327).
  • From image-garden v0.8.0, an unset GARDEN_DL_PKGPROXY defaults to PKGPROXY_URL (image-garden:31-38), so the URL the script exports would carry the image download too. The components' images are built on v0.7.2, which reads only GARDEN_DL_PKGPROXY; that is why the README asks the runner to set it.
  • How a guest of an image made with PKGPROXY_URL reaches the proxy: how a guest reaches pkgproxy.
  • pkgproxy.lan on the slide is an example name for your pkgproxy machine.

03 An address the VM can reach

A REACHABLE ADDRESS the runner machine the job container the VM, behind QEMU's user network the pkgproxy machine http://pkgproxy.lan:3142 its address on your network host-gateway the container's name; the VM has none image-garden resolves the host when it makes the image: the name must resolve in the job container too
The VM sits behind QEMU's user network inside the job container on the runner machine and reaches pkgproxy through the container's network; the URL must name the pkgproxy machine's address on your network, such as http://pkgproxy.lan:3142; host-gateway is the container's name and the VM has none; image-garden resolves the host when it makes the image, so the name must resolve in the container too

pkgproxy-url must be http://HOST[:PORT] at an address that routes from the job container, such as the pkgproxy machine's address on your network. host-gateway is the container's name, not the VM's.

The address is where most setups go wrong. The VM sits behind QEMU's user network, inside the job's container, so it reaches the world through the container's network. The URL has to name something that routes from there: the pkgproxy machine's address on your network, written as http, a host and a port. Docker's host-gateway won't do. That name belongs to the job container, and the VM doesn't have it; in a VM that runs Docker itself, the same address is the VM's own bridge. And image-garden turns the host name into an address when it makes the image, so the name must resolve in the job container too.

Details and sources

Details

  • "The VM reaches the network through the job container's, so the address must be one that routes from there to pkgproxy: the pkgproxy machine's address on your network. host-gateway, Docker's bridge address, is not one: it is the job container's name for the machine, which the VM does not have, and in a VM with Docker in it, the VM's own bridge." (snapcraft-gitlab-ci README.md:218-224; the input's description says the same, templates/build.yml:93-99).
  • When it makes the image, image-garden takes the host from PKGPROXY_URL, resolves it with getent ahostsv4 and writes that IPv4 address into the guest's /etc/hosts as pkgproxy; a host that does not resolve gives the warning cannot use pkgproxy at "…": want http://host[:port] with a host that resolves to IPv4; guests get no pkgproxy (010-qemu.mk:24-29).
  • The setup scripts the craft script runs use the URL as given, inside the VM (snapcraft-vm:394-396), so the host must also resolve and route from the VM (derived).
  • pkgproxy.lan on the slide is an example name.

04 Before the build, the setup scripts

SETUP SCRIPTS in the VM, as root, before snapcraft https://archive.ubuntu.com → http://archive.ubuntu.com (and security, ports) pkgproxy caches http archives; https only tunnels through curl -fsSL http://pkgproxy.lan:3142/setup/apt | sh curl -fsSL http://pkgproxy.lan:3142/setup/snap | sh curl -fsSL http://pkgproxy.lan:3142/setup/go | sh pkgproxy-setup: apt snap go a script whose tool is not installed does nothing snap install snapcraft pkgproxy and every Go module the build fetches
As root in the VM, before snapcraft is installed, the script rewrites Ubuntu's https archives to http because pkgproxy caches only http; then for apt, snap and go it runs curl URL/setup/TOOL piped to sh, each a no-op where the tool is missing; snapcraft, the snaps and Go modules then come through pkgproxy

As root in the VM: Ubuntu's https:// archives become http://, then curl -fsSL URL/setup/TOOL | sh for each tool in pkgproxy-setup (apt snap go). snapcraft, snaps, packages and Go modules then come through pkgproxy.

With a URL, there's one more step before snapcraft goes in. Running as root, because snapcraft builds as root, the script switches Ubuntu's https archives to plain http, since pkgproxy caches only http ones; anything over https just tunnels through. Then, for each tool named in the pkgproxy-setup input, apt, snap and go by default, it fetches pkgproxy's own setup script from the proxy and runs it. Each script does nothing where its tool isn't installed. So when snapd installs snapcraft a moment later, that comes through pkgproxy too, and so does every Go module the build fetches.

Details and sources

Details

  • With a pkgproxy, the section "Setting the VM up for pkgproxy at URL" runs, as root: a sed over /etc/apt that turns https://archive.ubuntu.com, security.ubuntu.com and ports.ubuntu.com into http:// ("pkgproxy caches http:// archives only; an https:// one is a tunnel"), then curl -fsSL URL/setup/TOOL | sh for each tool (snapcraft-vm:388-398).
  • "As root: snapcraft builds as root, with root's Go settings. Each script does nothing where its tool is not installed." (snapcraft-vm:392-393; README.md:226-233).
  • pkgproxy-setup defaults to apt snap go; each name must match ^[a-z][a-z-]*$ (templates/build.yml:100-105, snapcraft-vm:307-309).
  • The image itself was already made for pkgproxy, because PKGPROXY_URL was set: its cloud-init adds pkgproxy to /etc/hosts, runs the apt setup at bootcmd on Debian and Ubuntu and the snap setup at runcmd, and sets GOPROXY in /etc/environment (010-qemu.mk:127-180). The script's own run covers root's Go settings and the https archives.
  • image-garden's user-data also turns https:// into http:// in every .list and deb822 .sources file under /etc/apt, at the end of the first boot (010-qemu.mk:244). The components' images carry image-garden v0.7.2, whose rewrite covers .list files only (mk/010-qemu.mk:216 at the v0.7.2 tag), while Ubuntu 24.04 and later keep their archives in .sources; the script's sed is what switches those (derived).

05 When the proxy setting is wrong

WHEN IT GOES WRONG resolve start the VM pkgproxy setup pack pkgproxy-url: https://pkgproxy.lan snapcraft-vm: cannot use the pkgproxy "https://pkgproxy.lan": its URL must be http://HOST[:PORT] stops at once, before any VM starts pkgproxy-url: http://pkgproxy.lan:3142 down, or not resolvable in the VM fails in the pkgproxy setup section the cloud image: image-garden falls back to the origin pkgproxy-url: off a build today, without pkgproxy
A malformed pkgproxy URL stops the job before any VM starts with cannot use the pkgproxy, its URL must be http://HOST[:PORT]; a well-formed URL the VM cannot use fails later, in the section that sets the VM up for pkgproxy; the cloud image download falls back to the origin; pkgproxy-url: off builds without pkgproxy

A malformed URL stops the job at once: cannot use the pkgproxy "…": its URL must be http://HOST[:PORT]. An unreachable one fails the pkgproxy section. Image downloads fall back to the origin; pkgproxy-url: off builds without.

Two mistakes are common, and each fails in its own place. A URL that isn't http, a host and an optional port stops the job before any VM starts, with cannot use the pkgproxy, its URL must be http, host, port. A well-formed URL that the VM can't use, because the proxy is down or the name doesn't resolve there, fails a few minutes later, in the section that sets the VM up for pkgproxy, when the first setup script won't download. The cloud image is more forgiving: image-garden falls back to downloading it directly. And when you need a build today, set pkgproxy-url to off.

Details and sources

Details

  • A URL that does not match ^http://[^/:@[:space:]]+(:[0-9]+)?/?$ fails in the first section with snapcraft-vm: cannot use the pkgproxy "URL": its URL must be http://HOST[:PORT] (snapcraft-vm:215-229). So does an https:// URL.
  • The setup scripts run as bash -c "set -euo pipefail; curl -fsSL URL/setup/TOOL | sh" in the VM, and snapcraft-vm runs with set -e, so a curl that cannot connect or resolve ends the job in the section "Setting the VM up for pkgproxy at URL" (snapcraft-vm:245-256, snapcraft-vm:388-398). Derived from the code, not run.
  • image-garden asks pkgproxy for the image once and, when that fails, prints "Cannot download through pkgproxy, downloading URL directly" and downloads from the origin (image-garden:281-327); v0.7.2, in the components' images, falls back the same way.
  • pkgproxy-url: off builds with no pkgproxy, even when the runner sets PKGPROXY_URL (snapcraft-vm:215-221). The runner's GARDEN_DL_PKGPROXY still applies to the image download.

06 Where to go from here

ROADS no cache, pkgproxy Why pkgproxy download once KVM in CI the runner's environment Reaching pkgproxy a guest and a URL Image downloads by checksum GitHub caching save the cache instead
Five roads from the craft components' pkgproxy: why pkgproxy, the runner's environment in KVM in CI, how a guest reaches pkgproxy, image downloads through pkgproxy, and how the GitHub action caches

Next: why pkgproxy, the runner's environment, how a guest reaches pkgproxy, image downloads, and the GitHub action's caches.

So the craft components keep nothing between jobs, and a pkgproxy on your network does the remembering. The runner's environment names it, the VM reaches it at an address that routes from the job container, and the setup scripts point apt, snapd and Go at it. Next, the pkgproxy deck explains what it caches and why. The container's CI deck shows where the runner's variables go. How a guest reaches a proxy at a URL, and how images come through it, each have a deck. And the GitHub action solves the same problem by saving pkgproxy's cache between jobs.

Details and sources

Roads from here

BUILT ON IT · CI integrations

PKGPROXY · caching for guests

CONTAINER · OCI image

pkgproxy map ↗