Why the craft components keep no cache
the pkgproxy-url input for CI operators
The craft components keep nothing between jobs. A pkgproxy at an address the VM can reach, named by the pkgproxy-url input or the runner's PKGPROXY_URL, with GARDEN_DL_PKGPROXY for the cloud image, makes the second build fast; setup scripts point apt, snapd and Go at it.
On the map: built on it
▶ Watch this deck — 6 scenes, ~5 min
Worth seeing first
- Building snaps, rocks and charms in a VM — the job whose downloads this deck caches
01 Every job starts from nothing
Each job downloads the cloud image, snapcraft, snaps, packages and Go modules again. The component keeps no cache between jobs; a pkgproxy on the runners' network serves them from its cache after the first time.
Run the same snap build twice, and look at what the second job does. Exactly what the first one did. It downloads the Ubuntu cloud image, makes and boots the VM's image, installs snapcraft, then pulls the snaps, packages and Go modules your build needs. The component keeps no cache of its own: the scratch directory is deleted, the container is gone, and the job has no GitLab cache either. What makes the second build fast is pkgproxy, a caching proxy on your runners' network. After the first job, nearly everything comes from its cache, and only the work itself is done again.
Details and sources
Details
- "Each job starts from nothing: image-garden downloads the Ubuntu cloud image and makes the VM's image from it, and the VM downloads snapcraft, the snaps and packages the build needs, and its Go modules. The component keeps no cache of its own between jobs: pkgproxy is what makes the second build fast." (snapcraft-gitlab-ci README.md:203-211).
- The work directory with the prepared image is removed at the end of every job (snapcraft-vm:287-290); the job has no
cache:key (templates/build.yml:115-157). - rockcraft- and charmcraft-gitlab-ci say the same in their READMEs and work the same way.
- "Only the work itself" means the provisioning boot and the pack, which run in every job (derived).
Read more
02 Two kinds of download, two settings
The cloud image goes through GARDEN_DL_PKGPROXY, set on the runner. The VM's downloads follow pkgproxy-url: empty means the runner's PKGPROXY_URL, off means none; the URL is exported as PKGPROXY_URL for allocate.
Two different things download, and each finds pkgproxy its own way. The cloud image is fetched by image-garden, in the job container. It goes through pkgproxy, by its checksum, when the runner sets GARDEN_DL_PKGPROXY. Everything inside the VM follows the pkgproxy-url input instead. Leave it empty, and the job uses the runner's PKGPROXY_URL, if the runner sets one; set it to off, and the build runs without pkgproxy. The script exports the URL it picked as PKGPROXY_URL before allocate, so image-garden makes the VM's image already set up for pkgproxy. So two variables in the runner's environment cover every job.
Details and sources
Details
- image-garden downloads the Ubuntu cloud image through pkgproxy "when the runner sets
GARDEN_DL_PKGPROXY(pkgproxy 0.12 or later)"; the VM downloads through it "when the job has a pkgproxy URL that the VM reaches",pkgproxy-urlor the runner'sPKGPROXY_URL,offfor none (snapcraft-gitlab-ci README.md:213-219). pick_pkgproxy: empty takes$PKGPROXY_URL,offnone; anything else must matchhttp://HOST[:PORT]and loses a trailing slash (snapcraft-vm:215-229). The URL is exported asPKGPROXY_URL(orPKGPROXY_URLis unset) beforeimage-garden allocate(snapcraft-vm:356-366).- With
GARDEN_DL_PKGPROXYset, image-garden asks pkgproxy forfiles/v1/fetch?url=…&sha256=…and falls back to the origin (image-garden:281-327). - From image-garden v0.8.0, an unset
GARDEN_DL_PKGPROXYdefaults toPKGPROXY_URL(image-garden:31-38), so the URL the script exports would carry the image download too. The components' images are built on v0.7.2, which reads onlyGARDEN_DL_PKGPROXY; that is why the README asks the runner to set it. - How a guest of an image made with
PKGPROXY_URLreaches the proxy: how a guest reaches pkgproxy. pkgproxy.lanon the slide is an example name for your pkgproxy machine.
03 An address the VM can reach
pkgproxy-url must be http://HOST[:PORT] at an address that routes from the job container, such as the pkgproxy machine's address on your network. host-gateway is the container's name, not the VM's.
The address is where most setups go wrong. The VM sits behind QEMU's user network, inside the job's container, so it reaches the world through the container's network. The URL has to name something that routes from there: the pkgproxy machine's address on your network, written as http, a host and a port. Docker's host-gateway won't do. That name belongs to the job container, and the VM doesn't have it; in a VM that runs Docker itself, the same address is the VM's own bridge. And image-garden turns the host name into an address when it makes the image, so the name must resolve in the job container too.
Details and sources
Details
- "The VM reaches the network through the job container's, so the address must be one that routes from there to pkgproxy: the pkgproxy machine's address on your network.
host-gateway, Docker's bridge address, is not one: it is the job container's name for the machine, which the VM does not have, and in a VM with Docker in it, the VM's own bridge." (snapcraft-gitlab-ci README.md:218-224; the input's description says the same, templates/build.yml:93-99). - When it makes the image, image-garden takes the host from
PKGPROXY_URL, resolves it withgetent ahostsv4and writes that IPv4 address into the guest's/etc/hostsaspkgproxy; a host that does not resolve gives the warningcannot use pkgproxy at "…": want http://host[:port] with a host that resolves to IPv4; guests get no pkgproxy(010-qemu.mk:24-29). - The setup scripts the craft script runs use the URL as given, inside the VM (snapcraft-vm:394-396), so the host must also resolve and route from the VM (derived).
pkgproxy.lanon the slide is an example name.
04 Before the build, the setup scripts
As root in the VM: Ubuntu's https:// archives become http://, then curl -fsSL URL/setup/TOOL | sh for each tool in pkgproxy-setup (apt snap go). snapcraft, snaps, packages and Go modules then come through pkgproxy.
With a URL, there's one more step before snapcraft goes in. Running as root, because snapcraft builds as root, the script switches Ubuntu's https archives to plain http, since pkgproxy caches only http ones; anything over https just tunnels through. Then, for each tool named in the pkgproxy-setup input, apt, snap and go by default, it fetches pkgproxy's own setup script from the proxy and runs it. Each script does nothing where its tool isn't installed. So when snapd installs snapcraft a moment later, that comes through pkgproxy too, and so does every Go module the build fetches.
Details and sources
Details
- With a pkgproxy, the section "Setting the VM up for pkgproxy at URL" runs, as root: a
sedover/etc/aptthat turnshttps://archive.ubuntu.com,security.ubuntu.comandports.ubuntu.comintohttp://("pkgproxy caches http:// archives only; an https:// one is a tunnel"), thencurl -fsSL URL/setup/TOOL | shfor each tool (snapcraft-vm:388-398). - "As root: snapcraft builds as root, with root's Go settings. Each script does nothing where its tool is not installed." (snapcraft-vm:392-393; README.md:226-233).
pkgproxy-setupdefaults toapt snap go; each name must match^[a-z][a-z-]*$(templates/build.yml:100-105, snapcraft-vm:307-309).- The image itself was already made for pkgproxy, because
PKGPROXY_URLwas set: its cloud-init addspkgproxyto/etc/hosts, runs the apt setup at bootcmd on Debian and Ubuntu and the snap setup at runcmd, and setsGOPROXYin/etc/environment(010-qemu.mk:127-180). The script's own run covers root's Go settings and the https archives. - image-garden's user-data also turns
https://intohttp://in every.listand deb822.sourcesfile under/etc/apt, at the end of the first boot (010-qemu.mk:244). The components' images carry image-garden v0.7.2, whose rewrite covers.listfiles only (mk/010-qemu.mk:216at the v0.7.2 tag), while Ubuntu 24.04 and later keep their archives in.sources; the script'ssedis what switches those (derived).
05 When the proxy setting is wrong
A malformed URL stops the job at once: cannot use the pkgproxy "…": its URL must be http://HOST[:PORT]. An unreachable one fails the pkgproxy section. Image downloads fall back to the origin; pkgproxy-url: off builds without.
Two mistakes are common, and each fails in its own place. A URL that isn't http, a host and an optional port stops the job before any VM starts, with cannot use the pkgproxy, its URL must be http, host, port. A well-formed URL that the VM can't use, because the proxy is down or the name doesn't resolve there, fails a few minutes later, in the section that sets the VM up for pkgproxy, when the first setup script won't download. The cloud image is more forgiving: image-garden falls back to downloading it directly. And when you need a build today, set pkgproxy-url to off.
Details and sources
Details
- A URL that does not match
^http://[^/:@[:space:]]+(:[0-9]+)?/?$fails in the first section withsnapcraft-vm: cannot use the pkgproxy "URL": its URL must be http://HOST[:PORT](snapcraft-vm:215-229). So does anhttps://URL. - The setup scripts run as
bash -c "set -euo pipefail; curl -fsSL URL/setup/TOOL | sh"in the VM, and snapcraft-vm runs withset -e, so acurlthat cannot connect or resolve ends the job in the section "Setting the VM up for pkgproxy at URL" (snapcraft-vm:245-256, snapcraft-vm:388-398). Derived from the code, not run. - image-garden asks pkgproxy for the image once and, when that fails, prints "Cannot download through pkgproxy, downloading URL directly" and downloads from the origin (image-garden:281-327); v0.7.2, in the components' images, falls back the same way.
pkgproxy-url: offbuilds with no pkgproxy, even when the runner setsPKGPROXY_URL(snapcraft-vm:215-221). The runner'sGARDEN_DL_PKGPROXYstill applies to the image download.
Read more
06 Where to go from here
Next: why pkgproxy, the runner's environment, how a guest reaches pkgproxy, image downloads, and the GitHub action's caches.
So the craft components keep nothing between jobs, and a pkgproxy on your network does the remembering. The runner's environment names it, the VM reaches it at an address that routes from the job container, and the setup scripts point apt, snapd and Go at it. Next, the pkgproxy deck explains what it caches and why. The container's CI deck shows where the runner's variables go. How a guest reaches a proxy at a URL, and how images come through it, each have a deck. And the GitHub action solves the same problem by saving pkgproxy's cache between jobs.
Details and sources
Roads from here
BUILT ON IT · CI integrations
- How the action caches — how the GitHub action keeps the same cache between jobs instead
PKGPROXY · caching for guests
- What pkgproxy gives a garden — what pkgproxy caches and why it is the cache
- How a guest reaches pkgproxy — how a guest reaches a pkgproxy at a URL
- Cloud images through pkgproxy — cloud images fetched through pkgproxy by checksum
CONTAINER · OCI image
- Running the container in CI — setting PKGPROXY_URL in the runner's environment
pkgproxy map ↗
- pkgproxy for CI — on the pkgproxy map, a pkgproxy for your own runners