Building snaps, rocks and charms in a VM

snap, rock and charm builds for CI operators

snapcraft-, rockcraft- and charmcraft-gitlab-ci pick an Ubuntu system from the project's base, allocate it in a scratch directory, log in as --query reports, copy the tree in over ssh, pack in destructive mode, copy the result back and discard the VM.

On the map: built on it

▶ Watch this deck — 7 scenes, ~5 min

Worth seeing first

01 Packing in a throwaway VM

PACKING IN A VM a GitLab job, in a container no LXD, no Launchpad include: - component: $CI_SERVER_FQDN/zygoon/snapcraft-gitlab-ci/build@1 inputs: tags: [x86_64, kvm] a throwaway Ubuntu VM that matches your base copy in pack copy out snapcraft-vm · rockcraft-vm · charmcraft-vm one script per component, the same steps
A GitLab job runs in a container with no LXD and no Launchpad; including snapcraft-gitlab-ci/build@1 with the tags of a KVM runner gives a job that starts a throwaway Ubuntu VM of the snap's base, copies the project in, packs it and copies the result out; rockcraft-vm and charmcraft-vm do the same

Include snapcraft-gitlab-ci/build@1 with the tag of a KVM runner: its job packs the snap in a throwaway Ubuntu VM of the snap's base. rockcraft- and charmcraft-gitlab-ci do the same for rocks and charms.

Say you keep a snap on GitLab and want every merge request to pack it. snapcraft wants to build on the Ubuntu release of your snap's base, and a GitLab job runs in a container, with no LXD and no Launchpad at hand. The craft components fill that gap with image-garden. You include one, give it the tag of a runner with KVM, and its job starts a throwaway Ubuntu VM that matches your base, copies your project in, packs it there and copies the result back. The rockcraft and charmcraft components work the same way. This deck follows snapcraft-vm, the script behind the job, from start to finish.

Details and sources

Details

  • The component packs a snap "in a throwaway virtual machine of the snap's base, on your own runners with KVM", with no LXD, no Launchpad and no container image with a snapcraft for your base: it starts an Ubuntu cloud image with image-garden, installs snapcraft, runs snapcraft pack --destructive-mode and copies the snaps back (snapcraft-gitlab-ci README.md:9-21).
  • The usage example includes $CI_SERVER_FQDN/zygoon/snapcraft-gitlab-ci/build@1 with tags: [x86_64, kvm]; on a runner without /dev/kvm the job fails before it starts the VM (README.md:25-35).
  • The job's only script line is snapcraft-vm; its artifacts are the snaps, the components and snapcraft-vm-logs/, uploaded when: always (templates/build.yml:143-157).
  • rockcraft-gitlab-ci runs rockcraft-vm and charmcraft-gitlab-ci runs charmcraft-vm, the same script with RCV_ and CCV_ variables instead of SCV_. This deck quotes snapcraft-gitlab-ci at 91b5e29; rockcraft-gitlab-ci was read at e8fd725, charmcraft-gitlab-ci at 6fc8543.

02 The system comes from your base

WHICH SYSTEM base: core24 ubuntu-cloud-24.04 core20, 22, 26 alike + .$(uname -m) ubuntu-cloud-24.04.x86_64 build-base: ubuntu@24.04 rockcraft and charmcraft: the same result system: … the input overrides the base snapcraft-vm: cannot start a VM without /dev/kvm: … no KVM: the job fails here, before any VM starts
The snap's base core24 picks ubuntu-cloud-24.04, core20, core22 and core26 alike; the runner's uname -m adds the architecture, giving ubuntu-cloud-24.04.x86_64; rockcraft and charmcraft read a build base such as ubuntu@24.04; the system input overrides the base; without /dev/kvm the job fails before any VM starts

core24 picks ubuntu-cloud-24.04, and the runner's uname -m completes it: ubuntu-cloud-24.04.x86_64. The system input overrides the base; without /dev/kvm the job stops here.

First the script works out which machine to start. It reads your snap's definition for its build base, or else its base. core24 means Ubuntu 24.04, so the system is ubuntu-cloud-24.04, and core20, core22 and core26 map the same way. Then it adds the runner's own architecture, from uname -m, which gives ubuntu-cloud-24.04.x86_64. rockcraft and charmcraft read their build base, such as Ubuntu 24.04, to the same effect. The system input overrides all of this. And before anything starts, the script checks for /dev/kvm. Without it, the job fails right there, rather than build in a VM many times slower.

Details and sources

Details

  • snapcraft-vm reads build-base, or base, from the first of snap/snapcraft.yaml, snapcraft.yaml, .snapcraft.yaml and build-aux/snap/snapcraft.yaml it finds (snapcraft-vm:152-178).
  • core20, core22, core24 and core26 map to ubuntu-cloud-20.04 … ubuntu-cloud-26.04; any other base fails with "cannot pick a system for the base … set the system input" (snapcraft-vm:180-189). snapcraft comes from latest/stable, or 8.x/stable for core20 (snapcraft-vm:191-198).
  • A system without an architecture suffix gets .$SCV_ARCH, which defaults to uname -m (snapcraft-vm:64, snapcraft-vm:299-305). The snap is built for the runner's architecture.
  • rockcraft-vm reads build-base, or base (a bare rock must set build-base), and maps ubuntu@20.04 to ubuntu@26.04 (and the deprecated ubuntu:20.04, ubuntu:22.04); devel needs the system input (rockcraft-vm:165-192). charmcraft-vm also looks in platforms and the legacy bases, and asks for the platform or system input when a charm has several (charmcraft-vm:174-228).
  • Without the KVM device: "cannot start a VM without /dev/kvm: run the job on a runner of your own with KVM, picked with the tags input; GitLab.com's hosted runners have none" (snapcraft-vm:319-321).

03 allocate, in a directory of its own

ALLOCATE your checkout no gigabytes here /var/tmp/snapcraft-vm.XXXXXX/ .image-garden/ .image-garden.mk QEMU_MEM_OPTION="-m 2048" QEMU_SMP_OPTION="-smp 2" the memory and cpus inputs $ image-garden allocate --spread ubuntu-cloud-24.04.x86_64 every job: the download, then the provisioning boot <ADDRESS localhost:PORT> the script keeps the port
Your checkout stays clean; a scratch directory /var/tmp/snapcraft-vm.XXXXXX holds .image-garden/ and a copy of your .image-garden.mk; QEMU_MEM_OPTION=-m 2048 and QEMU_SMP_OPTION=-smp 2 size the VM; image-garden allocate --spread ubuntu-cloud-24.04.x86_64 runs there, downloading and provisioning in every job, and answers with ADDRESS localhost:PORT

A scratch directory under /var/tmp holds .image-garden/ and your .image-garden.mk; QEMU_MEM_OPTION and QEMU_SMP_OPTION size the VM; image-garden allocate --spread answers with <ADDRESS localhost:PORT>.

image-garden keeps its images and logs in the directory it runs in, and the script doesn't want gigabytes in your checkout. So it makes a scratch directory under /var/tmp, with a .image-garden directory inside, and copies your own .image-garden.mk there, if you gave one. It sizes the VM through QEMU_MEM_OPTION and QEMU_SMP_OPTION: two gigabytes and two CPUs by default. Then, in that directory, it runs image-garden allocate --spread with the system. In every job, that means the download and the provisioning boot. What comes back is one line, ADDRESS in angle brackets, and the script keeps the port.

Details and sources

Details

  • "image-garden keeps its images, logs and a spread.yaml lock file in the directory it runs in. Not the checkout: what it leaves there would be in the job's working tree." The work directory is mktemp -d "${TMPDIR:-/var/tmp}"/snapcraft-vm.XXXXXX, unless SCV_WORK_DIR names one; the script creates .image-garden/ in it and copies the image-garden-mk input there as .image-garden.mk (snapcraft-vm:337-349).
  • It exports PKGPROXY_URL (or unsets it), QEMU_MEM_OPTION="-m $SCV_MEMORY" and QEMU_SMP_OPTION="-smp $SCV_CPUS", then runs (cd "$work" && image-garden allocate --spread "$system") and takes the address from the line <ADDRESS …>; without one it fails with "image-garden printed no address" (snapcraft-vm:356-373). The defaults are 2048 MiB and 2 CPUs; cpus: auto takes nproc, capped by the job cgroup's CPU quota (snapcraft-vm:125-150).
  • allocate makes the .run and .qcow2 first, in .image-garden/ when that directory exists (image-garden:1057-1063); with --spread it prints <ADDRESS localhost:PORT> once the guest's SSH banner answers (image-garden:1167-1203).
  • The scratch directory is new in every job, and so is the container's download cache, so every job downloads the cloud image (through pkgproxy, when there is one) and boots it once to provision it (derived from the above).
  • The components' images are built on image-garden v0.7.2; this map describes v0.8.0, whose allocate behaves the same for these steps.

04 Logging in with what --query reports

LOGGING IN $ .image-garden/ubuntu-cloud-24.04.x86_64.run --query vm.memory_mb=2048 system.users.0.username=root system.users.0.password=root system.users.1.username=ubuntu system.users.1.password=ubuntu the first user is the one the script reads ssh -p PORT root@localhost … password supplied, host keys ignored a first user other than root: each command under sudo -H each command's output, streamed into the job log in folded sections
The script runs .image-garden/ubuntu-cloud-24.04.x86_64.run --query, which prints vm.memory_mb and the users, root with password root first and ubuntu second; every step is then an ssh command as root to localhost on the allocated port with the password supplied and host keys ignored; a user other than root would need sudo; output streams into folded sections of the job log

.image-garden/SYSTEM.run --query names system.users.0: root, password root, on Ubuntu cloud images. Every step is then ssh -p PORT root@localhost, its output streamed into the job log.

To log in, the script needs a user and a password, and it doesn't guess. It asks the .run script in the scratch directory, with --query, and reads the first user from the answer. For an Ubuntu cloud image, that's root, with password root, and the ubuntu user comes second. From then on, every step is an ssh command to localhost on that port, with the password supplied for it and host keys ignored, since the machine is brand new. If the first user weren't root, each command would run under sudo. And each command's output streams straight into the job log, in folded sections.

Details and sources

Details

  • The script runs "$work"/.image-garden/"$system".run --query and reads system.users.0.username and system.users.0.password; with no user it fails with "its image names no user"; a user other than root gets sudo -H in front of every command (snapcraft-vm:374-381).
  • The .run script answers --query with vm.memory_mb=… and, for systems of rulegen.py's default root distro, system.users.0 root/root and system.users.1 the distribution's user with its own name as password (010-qemu.mk:57-90, rulegen.py:127). For ubuntu-cloud-24.04.x86_64 that is root/root, then ubuntu/ubuntu. Only Ubuntu Core systems report ubuntu first (only:ubuntu).
  • Each guest command is SSHPASS=… sshpass -e ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o ConnectTimeout=30 -o ServerAliveInterval=15 -o ServerAliveCountMax=8 -p PORT USER@localhost, without a terminal, so stdout and stderr stay apart (snapcraft-vm:231-256).
  • The sections are GitLab's collapsible sections; outside GitLab they are plain == heading lines (snapcraft-vm:91-110).
  • The components' own tests use a stand-in image-garden whose .run reports ubuntu/ubuntu as user 0 (testdata/bin/image-garden:20); the real cloud image reports root first.

05 Copy in, pack, copy out

COPY IN, PACK, COPY OUT the job container the VM snap install snapcraft --classic your working tree .git in; .image-garden and the output stay out tar over ssh /root/build snapcraft pack --destructive-mode *.snap *.comp tar back snaps/ snapcraft-vm.env: SNAP_FILE, SNAP_FILES, SNAP_VERSION, SNAP_COMPONENTS for later jobs, such as one that installs the snap and tries it
In the VM the script waits for snapd's seeding and installs snapcraft; the job's working tree, with .git but without .image-garden or the job's output, streams in as tar over ssh into /root/build; snapcraft pack --destructive-mode runs in the VM; the snaps and components stream back into snaps/; a dotenv report names them for later jobs

snapcraft is installed in the VM, the working tree streams in as tar over ssh (no .image-garden), snapcraft pack --destructive-mode runs there, and the snaps stream back into snaps/, named in a dotenv report.

Now the build. Inside the VM, the script waits for snapd to finish seeding, then installs snapcraft from its channel. Your working tree goes in as a tar stream over the same ssh connection, git history included, but without any .image-garden directory and without the job's own output. snapcraft packs in destructive mode, right on the VM's own system, which is fine, since the VM exists only for this build. Then the snaps and components come back the same way, as a tar stream, into the job's snaps directory. A small report names them for later jobs, such as one that installs the snap and tries it.

Details and sources

Details

  • snap wait system seed.loaded, then snap install snapcraft --classic --channel=…; when that fails, a folded section shows what the VM's network reaches before the job fails with "cannot install snapcraft in the VM" (snapcraft-vm:400-416).
  • The tree is tar -cf - --no-wildcards --exclude=.image-garden --exclude=./OUTPUT …, followed by a second archive of what git tracks inside .image-garden directories, streamed uncompressed into /root/build in the VM; the job token is removed from origin's URL and git in the VM ignores snapcraft's work directories (snapcraft-vm:420-453).
  • The optional prepare input runs as root before the pack; then snapcraft pack --destructive-mode with --platform or --build-for and the snapcraft-args input (snapcraft-vm:455-467). Files the pack left that git does not ignore are listed, and fail the job with require-clean-tree: "true" (snapcraft-vm:469-482).
  • *.snap and *.comp come back as a tar stream into output-dir (snaps); the dotenv report snapcraft-vm.env sets SNAP_FILE, SNAP_FILES, SNAP_VERSION and SNAP_COMPONENTS (snapcraft-vm:484-506; README.md:93-109).
  • rockcraft-vm packs *.rock with rockcraft pack --destructive-mode (rockcraft-vm:447, rockcraft-vm:468-469); charmcraft-vm packs *.charm (charmcraft-vm:489, charmcraft-vm:510-511).

Read more

06 Discard, keep the logs, delete the rest

ON THE WAY OUT a clean build, a failed pack or an interrupted job: one cleanup fetch snapcraft's logs from the VM $ image-garden discard localhost:PORT QEMU stops .image-garden/*.log snapcraft-vm-logs/ serial console and QEMU's output: crashed inside, or killed? rm -rf /var/tmp/snapcraft-vm.XXXXXX the prepared image, gigabytes artifacts, even when the job fails: snaps/*.snap, *.comp, snapcraft-vm-logs/
On success, failure or interrupt one cleanup runs: it fetches snapcraft's logs from the VM, runs image-garden discard localhost:PORT so QEMU stops, copies .image-garden/*.log into snapcraft-vm-logs/, deletes the scratch directory with its gigabytes of prepared image, and the job uploads the snaps and logs as artifacts even when it fails

An exit trap fetches snapcraft's logs, runs image-garden discard localhost:PORT, copies .image-garden/*.log into snapcraft-vm-logs/ and deletes the scratch directory. Artifacts are uploaded when: always.

Whatever happens, a clean build, a failed pack or an interrupted job, one cleanup runs on the way out. It fetches snapcraft's own logs from the VM, then runs image-garden discard with the address, from the scratch directory, which stops QEMU. It copies the serial console and QEMU's output from .image-garden into the logs directory, because they tell a VM that crashed inside from one that was killed from outside. Last, it deletes the scratch directory, prepared image and all: it's gigabytes, and nothing would reuse it. The job uploads the snaps and the logs as artifacts, even when it fails.

Details and sources

Details

  • trap cleanup EXIT, and trap 'exit 143' TERM HUP INT, so the cleanup also runs when the job is interrupted (snapcraft-vm:294-295).
  • The cleanup copies /root/.local/state/snapcraft/log out of the VM, runs (cd "$work" && image-garden discard "$address"), copies "$work"/.image-garden/*.log into the log directory ("The VM's serial console and qemu's output: what tells a VM that died inside from one killed from outside") and removes a work directory it made ("The VM's image is gigabytes: not left behind where nothing reuses it"); the exit status is the build's (snapcraft-vm:264-293).
  • discard takes localhost:PORT, kills the QEMU named by its pid file and removes the pid file (image-garden:1254-1268). The logs it leaves are SYSTEM.PORT.serial.log, .stdout.log and .stderr.log from allocate (image-garden:1153-1160), and the provisioning boot's SYSTEM.qcow2.log.
  • The job's artifacts: OUTPUT/*.snap, OUTPUT/*.comp and snapcraft-vm-logs/, when: always, kept for expire-in (1 week), and the dotenv report (templates/build.yml:145-157).

07 Where to go from here

ROADS craft components Why no cache pkgproxy for the second build KVM in CI runners and tags allocate before the ADDRESS line .run --query users and memory The contract what the script relies on GitLab vs GitHub container vs snap
Six roads from the craft components: why they keep no cache, KVM in CI, allocate, the --query output, the contract, and GitLab versus GitHub

Next: why the craft components keep no cache, KVM in CI, allocate, --query, the contract, and GitLab versus GitHub.

So a craft job picks the system from your base, allocates it in a scratch directory, logs in as --query says, copies your tree in, packs, copies the result out and discards the VM. Next, the pkgproxy deck explains why nothing is cached between jobs, and what makes the second build fast. KVM in CI shows how to set up the runners. allocate has its own deck, and so does the query output. The contract deck lists exactly what this script expects from image-garden. And the comparison deck sets GitLab beside GitHub.

Details and sources

Read more

Basis of the statements on this page

Read in snapcraft-gitlab-ci at 91b5e29, rockcraft-gitlab-ci at e8fd725 and charmcraft-gitlab-ci at 6fc8543 ("code"): templates/build.yml, bin/*craft-vm and the READMEs. What image-garden does in each step was read in image-garden v0.8.0; the components' images are built on image-garden v0.7.2. The user that --query reports for an Ubuntu cloud image is derived from rulegen.py's default. Nothing was run for this page.

An example

.gitlab-ci.yml:

include: - component: $CI_SERVER_FQDN/zygoon/snapcraft-gitlab-ci/build@1 inputs: tags: [x86_64, kvm] memory: "4096" cpus: auto

The steps of snapcraft-vm

StepWhat happenssnapcraft-vm
Resolvebase to system, .$(uname -m), pkgproxy URL, cpus, /dev/kvm check, work directory:297-354
Starting SYSTEMPKGPROXY_URL, QEMU_MEM_OPTION, QEMU_SMP_OPTION; image-garden allocate --spread; <ADDRESS>; .run --query:356-382
The VM/etc/os-release, uname -m, nproc, free -m in the guest:384-386
pkgproxyhttps archives to http; curl URL/setup/TOOL piped to sh:388-398
snapcraftsnap wait system seed.loaded; snap install snapcraft --classic:400-416
Copying the sourcetar over ssh into /root/build, no .image-garden:418-453
Preparingthe prepare input, as root:455-460
Packingsnapcraft pack --destructive-mode:462-482
Copying the snaps outtar back into output-dir; dotenv:484-508
Discardinglogs, image-garden discard, .image-garden/*.log, rm -rf the work directory:264-295

rockcraft-vm and charmcraft-vm have the same steps, about 20 lines earlier and 20 lines later respectively.

The build inputs

The three templates share these inputs; the variable prefix is SCV_, RCV_ or CCV_.

InputDefaultMeaning
job-name, stagesnap-build (rock-build, charm-build), buildthe job
imagethe component's own image, same versionmust have image-garden, ssh, sshpass and the script
tags[]runners that give jobs /dev/kvm
timeout1hthe job's timeout
project-dir.where the project's yaml file is
systememptythe image-garden system; empty picks it from the base
memory2048the VM's memory in MiB, as QEMU_MEM_OPTION
cpus2the VM's CPUs, or auto, as QEMU_SMP_OPTION
prepareemptyshell run as root in the VM before the pack
require-clean-tree"false"fail when the pack leaves untracked files
image-garden-mkemptyan .image-garden.mk for the VM's image
pkgproxy-urlemptyhttp://HOST[:PORT]; empty for the runner's PKGPROXY_URL, off for none
pkgproxy-setupapt snap gopkgproxy setup scripts the VM runs
output-dirsnaps (rocks, charms)where the packed files go
expire-in1 weekhow long artifacts are kept

Sources: snapcraft-gitlab-ci templates/build.yml:13-113, and the same file in rockcraft-gitlab-ci and charmcraft-gitlab-ci. Each component also has its own inputs, such as snapcraft-channel and platform.

Base to system

snapcraft baserockcraft and charmcraft build basesystem
core20ubuntu@20.04ubuntu-cloud-20.04
core22ubuntu@22.04ubuntu-cloud-22.04
core24ubuntu@24.04ubuntu-cloud-24.04
core26ubuntu@26.04ubuntu-cloud-26.04

Then .$(uname -m) is appended, unless the system already ends in an architecture.

Roads from here

RUNTIME · make rules and the CLI

BUILT ON IT · CI integrations

CONTAINER · OCI image