Building snaps, rocks and charms in a VM
snap, rock and charm builds for CI operators
snapcraft-, rockcraft- and charmcraft-gitlab-ci pick an Ubuntu system from the project's base, allocate it in a scratch directory, log in as --query reports, copy the tree in over ssh, pack in destructive mode, copy the result back and discard the VM.
On the map: built on it
▶ Watch this deck — 7 scenes, ~5 min
Worth seeing first
- Running the container in CI — runners with /dev/kvm and how jobs pick them
01 Packing in a throwaway VM
Include snapcraft-gitlab-ci/build@1 with the tag of a KVM runner: its job packs the snap in a throwaway Ubuntu VM of the snap's base. rockcraft- and charmcraft-gitlab-ci do the same for rocks and charms.
Say you keep a snap on GitLab and want every merge request to pack it. snapcraft wants to build on the Ubuntu release of your snap's base, and a GitLab job runs in a container, with no LXD and no Launchpad at hand. The craft components fill that gap with image-garden. You include one, give it the tag of a runner with KVM, and its job starts a throwaway Ubuntu VM that matches your base, copies your project in, packs it there and copies the result back. The rockcraft and charmcraft components work the same way. This deck follows snapcraft-vm, the script behind the job, from start to finish.
Details and sources
Details
- The component packs a snap "in a throwaway virtual machine of the snap's base, on your own runners with KVM", with no LXD, no Launchpad and no container image with a snapcraft for your base: it starts an Ubuntu cloud image with image-garden, installs snapcraft, runs
snapcraft pack --destructive-modeand copies the snaps back (snapcraft-gitlab-ci README.md:9-21). - The usage example includes
$CI_SERVER_FQDN/zygoon/snapcraft-gitlab-ci/build@1withtags: [x86_64, kvm]; on a runner without/dev/kvmthe job fails before it starts the VM (README.md:25-35). - The job's only script line is
snapcraft-vm; its artifacts are the snaps, the components andsnapcraft-vm-logs/, uploadedwhen: always(templates/build.yml:143-157). - rockcraft-gitlab-ci runs
rockcraft-vmand charmcraft-gitlab-ci runscharmcraft-vm, the same script withRCV_andCCV_variables instead ofSCV_. This deck quotes snapcraft-gitlab-ci at91b5e29; rockcraft-gitlab-ci was read ate8fd725, charmcraft-gitlab-ci at6fc8543.
Read more
02 The system comes from your base
core24 picks ubuntu-cloud-24.04, and the runner's uname -m completes it: ubuntu-cloud-24.04.x86_64. The system input overrides the base; without /dev/kvm the job stops here.
First the script works out which machine to start. It reads your snap's definition for its build base, or else its base. core24 means Ubuntu 24.04, so the system is ubuntu-cloud-24.04, and core20, core22 and core26 map the same way. Then it adds the runner's own architecture, from uname -m, which gives ubuntu-cloud-24.04.x86_64. rockcraft and charmcraft read their build base, such as Ubuntu 24.04, to the same effect. The system input overrides all of this. And before anything starts, the script checks for /dev/kvm. Without it, the job fails right there, rather than build in a VM many times slower.
Details and sources
Details
- snapcraft-vm reads
build-base, orbase, from the first ofsnap/snapcraft.yaml,snapcraft.yaml,.snapcraft.yamlandbuild-aux/snap/snapcraft.yamlit finds (snapcraft-vm:152-178). core20,core22,core24andcore26map toubuntu-cloud-20.04…ubuntu-cloud-26.04; any other base fails with "cannot pick a system for the base … set the system input" (snapcraft-vm:180-189). snapcraft comes fromlatest/stable, or8.x/stablefor core20 (snapcraft-vm:191-198).- A system without an architecture suffix gets
.$SCV_ARCH, which defaults touname -m(snapcraft-vm:64, snapcraft-vm:299-305). The snap is built for the runner's architecture. - rockcraft-vm reads
build-base, orbase(a bare rock must setbuild-base), and mapsubuntu@20.04toubuntu@26.04(and the deprecatedubuntu:20.04,ubuntu:22.04);develneeds thesysteminput (rockcraft-vm:165-192). charmcraft-vm also looks inplatformsand the legacybases, and asks for theplatformorsysteminput when a charm has several (charmcraft-vm:174-228). - Without the KVM device: "cannot start a VM without /dev/kvm: run the job on a runner of your own with KVM, picked with the tags input; GitLab.com's hosted runners have none" (snapcraft-vm:319-321).
Read more
03 allocate, in a directory of its own
A scratch directory under /var/tmp holds .image-garden/ and your .image-garden.mk; QEMU_MEM_OPTION and QEMU_SMP_OPTION size the VM; image-garden allocate --spread answers with <ADDRESS localhost:PORT>.
image-garden keeps its images and logs in the directory it runs in, and the script doesn't want gigabytes in your checkout. So it makes a scratch directory under /var/tmp, with a .image-garden directory inside, and copies your own .image-garden.mk there, if you gave one. It sizes the VM through QEMU_MEM_OPTION and QEMU_SMP_OPTION: two gigabytes and two CPUs by default. Then, in that directory, it runs image-garden allocate --spread with the system. In every job, that means the download and the provisioning boot. What comes back is one line, ADDRESS in angle brackets, and the script keeps the port.
Details and sources
Details
- "image-garden keeps its images, logs and a spread.yaml lock file in the directory it runs in. Not the checkout: what it leaves there would be in the job's working tree." The work directory is
mktemp -d "${TMPDIR:-/var/tmp}"/snapcraft-vm.XXXXXX, unlessSCV_WORK_DIRnames one; the script creates.image-garden/in it and copies theimage-garden-mkinput there as.image-garden.mk(snapcraft-vm:337-349). - It exports
PKGPROXY_URL(or unsets it),QEMU_MEM_OPTION="-m $SCV_MEMORY"andQEMU_SMP_OPTION="-smp $SCV_CPUS", then runs(cd "$work" && image-garden allocate --spread "$system")and takes the address from the line<ADDRESS …>; without one it fails with "image-garden printed no address" (snapcraft-vm:356-373). The defaults are 2048 MiB and 2 CPUs;cpus: autotakesnproc, capped by the job cgroup's CPU quota (snapcraft-vm:125-150). - allocate makes the
.runand.qcow2first, in.image-garden/when that directory exists (image-garden:1057-1063); with--spreadit prints<ADDRESS localhost:PORT>once the guest's SSH banner answers (image-garden:1167-1203). - The scratch directory is new in every job, and so is the container's download cache, so every job downloads the cloud image (through pkgproxy, when there is one) and boots it once to provision it (derived from the above).
- The components' images are built on image-garden v0.7.2; this map describes v0.8.0, whose allocate behaves the same for these steps.
Read more
04 Logging in with what --query reports
.image-garden/SYSTEM.run --query names system.users.0: root, password root, on Ubuntu cloud images. Every step is then ssh -p PORT root@localhost, its output streamed into the job log.
To log in, the script needs a user and a password, and it doesn't guess. It asks the .run script in the scratch directory, with --query, and reads the first user from the answer. For an Ubuntu cloud image, that's root, with password root, and the ubuntu user comes second. From then on, every step is an ssh command to localhost on that port, with the password supplied for it and host keys ignored, since the machine is brand new. If the first user weren't root, each command would run under sudo. And each command's output streams straight into the job log, in folded sections.
Details and sources
Details
- The script runs
"$work"/.image-garden/"$system".run --queryand readssystem.users.0.usernameandsystem.users.0.password; with no user it fails with "its image names no user"; a user other than root getssudo -Hin front of every command (snapcraft-vm:374-381). - The
.runscript answers--querywithvm.memory_mb=…and, for systems of rulegen.py's defaultroot distro,system.users.0root/root andsystem.users.1the distribution's user with its own name as password (010-qemu.mk:57-90, rulegen.py:127). Forubuntu-cloud-24.04.x86_64that is root/root, then ubuntu/ubuntu. Only Ubuntu Core systems reportubuntufirst (only:ubuntu). - Each guest command is
SSHPASS=… sshpass -e ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o ConnectTimeout=30 -o ServerAliveInterval=15 -o ServerAliveCountMax=8 -p PORT USER@localhost, without a terminal, so stdout and stderr stay apart (snapcraft-vm:231-256). - The sections are GitLab's collapsible sections; outside GitLab they are plain
== headinglines (snapcraft-vm:91-110). - The components' own tests use a stand-in image-garden whose
.runreportsubuntu/ubuntuas user 0 (testdata/bin/image-garden:20); the real cloud image reports root first.
05 Copy in, pack, copy out
snapcraft is installed in the VM, the working tree streams in as tar over ssh (no .image-garden), snapcraft pack --destructive-mode runs there, and the snaps stream back into snaps/, named in a dotenv report.
Now the build. Inside the VM, the script waits for snapd to finish seeding, then installs snapcraft from its channel. Your working tree goes in as a tar stream over the same ssh connection, git history included, but without any .image-garden directory and without the job's own output. snapcraft packs in destructive mode, right on the VM's own system, which is fine, since the VM exists only for this build. Then the snaps and components come back the same way, as a tar stream, into the job's snaps directory. A small report names them for later jobs, such as one that installs the snap and tries it.
Details and sources
Details
snap wait system seed.loaded, thensnap install snapcraft --classic --channel=…; when that fails, a folded section shows what the VM's network reaches before the job fails with "cannot install snapcraft in the VM" (snapcraft-vm:400-416).- The tree is
tar -cf - --no-wildcards --exclude=.image-garden --exclude=./OUTPUT …, followed by a second archive of what git tracks inside.image-gardendirectories, streamed uncompressed into/root/buildin the VM; the job token is removed fromorigin's URL and git in the VM ignores snapcraft's work directories (snapcraft-vm:420-453). - The optional
prepareinput runs as root before the pack; thensnapcraft pack --destructive-modewith--platformor--build-forand thesnapcraft-argsinput (snapcraft-vm:455-467). Files the pack left that git does not ignore are listed, and fail the job withrequire-clean-tree: "true"(snapcraft-vm:469-482). *.snapand*.compcome back as a tar stream intooutput-dir(snaps); the dotenv reportsnapcraft-vm.envsetsSNAP_FILE,SNAP_FILES,SNAP_VERSIONandSNAP_COMPONENTS(snapcraft-vm:484-506; README.md:93-109).- rockcraft-vm packs
*.rockwithrockcraft pack --destructive-mode(rockcraft-vm:447, rockcraft-vm:468-469); charmcraft-vm packs*.charm(charmcraft-vm:489, charmcraft-vm:510-511).
Read more
- The component's README on what the VM gets and a clean tree.
06 Discard, keep the logs, delete the rest
An exit trap fetches snapcraft's logs, runs image-garden discard localhost:PORT, copies .image-garden/*.log into snapcraft-vm-logs/ and deletes the scratch directory. Artifacts are uploaded when: always.
Whatever happens, a clean build, a failed pack or an interrupted job, one cleanup runs on the way out. It fetches snapcraft's own logs from the VM, then runs image-garden discard with the address, from the scratch directory, which stops QEMU. It copies the serial console and QEMU's output from .image-garden into the logs directory, because they tell a VM that crashed inside from one that was killed from outside. Last, it deletes the scratch directory, prepared image and all: it's gigabytes, and nothing would reuse it. The job uploads the snaps and the logs as artifacts, even when it fails.
Details and sources
Details
trap cleanup EXIT, andtrap 'exit 143' TERM HUP INT, so the cleanup also runs when the job is interrupted (snapcraft-vm:294-295).- The cleanup copies
/root/.local/state/snapcraft/logout of the VM, runs(cd "$work" && image-garden discard "$address"), copies"$work"/.image-garden/*.loginto the log directory ("The VM's serial console and qemu's output: what tells a VM that died inside from one killed from outside") and removes a work directory it made ("The VM's image is gigabytes: not left behind where nothing reuses it"); the exit status is the build's (snapcraft-vm:264-293). - discard takes
localhost:PORT, kills the QEMU named by its pid file and removes the pid file (image-garden:1254-1268). The logs it leaves areSYSTEM.PORT.serial.log,.stdout.logand.stderr.logfrom allocate (image-garden:1153-1160), and the provisioning boot'sSYSTEM.qcow2.log. - The job's artifacts:
OUTPUT/*.snap,OUTPUT/*.compandsnapcraft-vm-logs/,when: always, kept forexpire-in(1 week), and the dotenv report (templates/build.yml:145-157).
07 Where to go from here
Next: why the craft components keep no cache, KVM in CI, allocate, --query, the contract, and GitLab versus GitHub.
So a craft job picks the system from your base, allocates it in a scratch directory, logs in as --query says, copies your tree in, packs, copies the result out and discards the VM. Next, the pkgproxy deck explains why nothing is cached between jobs, and what makes the second build fast. KVM in CI shows how to set up the runners. allocate has its own deck, and so does the query output. The contract deck lists exactly what this script expects from image-garden. And the comparison deck sets GitLab beside GitHub.
Details and sources
Read more
Basis of the statements on this page
Read in snapcraft-gitlab-ci at 91b5e29, rockcraft-gitlab-ci at e8fd725 and charmcraft-gitlab-ci at 6fc8543 ("code"): templates/build.yml, bin/*craft-vm and the READMEs. What image-garden does in each step was read in image-garden v0.8.0; the components' images are built on image-garden v0.7.2. The user that --query reports for an Ubuntu cloud image is derived from rulegen.py's default. Nothing was run for this page.
An example
.gitlab-ci.yml:
include: - component: $CI_SERVER_FQDN/zygoon/snapcraft-gitlab-ci/build@1 inputs: tags: [x86_64, kvm] memory: "4096" cpus: auto
The steps of snapcraft-vm
| Step | What happens | snapcraft-vm |
|---|---|---|
| Resolve | base to system, .$(uname -m), pkgproxy URL, cpus, /dev/kvm check, work directory | :297-354 |
| Starting SYSTEM | PKGPROXY_URL, QEMU_MEM_OPTION, QEMU_SMP_OPTION; image-garden allocate --spread; <ADDRESS>; .run --query | :356-382 |
| The VM | /etc/os-release, uname -m, nproc, free -m in the guest | :384-386 |
| pkgproxy | https archives to http; curl URL/setup/TOOL piped to sh | :388-398 |
| snapcraft | snap wait system seed.loaded; snap install snapcraft --classic | :400-416 |
| Copying the source | tar over ssh into /root/build, no .image-garden | :418-453 |
| Preparing | the prepare input, as root | :455-460 |
| Packing | snapcraft pack --destructive-mode | :462-482 |
| Copying the snaps out | tar back into output-dir; dotenv | :484-508 |
| Discarding | logs, image-garden discard, .image-garden/*.log, rm -rf the work directory | :264-295 |
rockcraft-vm and charmcraft-vm have the same steps, about 20 lines earlier and 20 lines later respectively.
The build inputs
The three templates share these inputs; the variable prefix is SCV_, RCV_ or CCV_.
| Input | Default | Meaning |
|---|---|---|
job-name, stage | snap-build (rock-build, charm-build), build | the job |
image | the component's own image, same version | must have image-garden, ssh, sshpass and the script |
tags | [] | runners that give jobs /dev/kvm |
timeout | 1h | the job's timeout |
project-dir | . | where the project's yaml file is |
system | empty | the image-garden system; empty picks it from the base |
memory | 2048 | the VM's memory in MiB, as QEMU_MEM_OPTION |
cpus | 2 | the VM's CPUs, or auto, as QEMU_SMP_OPTION |
prepare | empty | shell run as root in the VM before the pack |
require-clean-tree | "false" | fail when the pack leaves untracked files |
image-garden-mk | empty | an .image-garden.mk for the VM's image |
pkgproxy-url | empty | http://HOST[:PORT]; empty for the runner's PKGPROXY_URL, off for none |
pkgproxy-setup | apt snap go | pkgproxy setup scripts the VM runs |
output-dir | snaps (rocks, charms) | where the packed files go |
expire-in | 1 week | how long artifacts are kept |
Sources: snapcraft-gitlab-ci templates/build.yml:13-113, and the same file in rockcraft-gitlab-ci and charmcraft-gitlab-ci. Each component also has its own inputs, such as snapcraft-channel and platform.
Base to system
| snapcraft base | rockcraft and charmcraft build base | system |
|---|---|---|
core20 | ubuntu@20.04 | ubuntu-cloud-20.04 |
core22 | ubuntu@22.04 | ubuntu-cloud-22.04 |
core24 | ubuntu@24.04 | ubuntu-cloud-24.04 |
core26 | ubuntu@26.04 | ubuntu-cloud-26.04 |
Then .$(uname -m) is appended, unless the system already ends in an architecture.
Roads from here
RUNTIME · make rules and the CLI
- image-garden allocate — what allocate does before it prints the address
- Asking a .run script about its system — the users and memory a .run script reports
BUILT ON IT · CI integrations
- Why the craft components keep no cache — why nothing is cached between jobs, and what makes the second build fast
- What the integrations rely on — exactly which image-garden behaviour the script relies on
- Container on GitLab, snap on GitHub — the same kind of job on GitHub, with the snap
CONTAINER · OCI image
- Running the container in CI — setting up runners with /dev/kvm and tags