Running the container in CI
runners with /dev/kvm for CI operators
Hosted GitLab runners give jobs no /dev/kvm, so image-garden jobs run on runners of your own that pass the device into the job container and are picked with tags, one runner per host architecture. A pkgproxy set in the runner's environment replaces a CI cache of images.
On the map: container
▶ Watch this deck — 6 scenes, ~4 min
Worth seeing first
- Running image-garden in a container — what the image carries and how it is run by hand
Part of journeys
- From your laptop to a git forge, step 3 of 6 · previous: Running image-garden in a container · next: spread-gitlab-ci
01 No KVM on hosted runners
GitLab.com's hosted runners have no /dev/kvm: guests would be emulated, many times slower. Run image-garden jobs on runners of your own.
Your tests pass on your laptop, and now you'd like GitLab to run them on every push. The container is the obvious way in: the job's image already has everything. There's one catch. GitLab's hosted runners don't give jobs the KVM device. image-garden still works there, but QEMU has to emulate every instruction, many times slower, and a run of tests may not fit in the job's time limit at all. So image-garden jobs run on runners of your own, machines that can hand the KVM device to the job's container. This deck shows how they're set up and picked.
Details and sources
Details
- image-garden needs
/dev/kvmto run machines at full speed; without it QEMU emulates the CPU (TCG), many times slower. GitLab.com's hosted runners do not give jobs/dev/kvm(README.md:778-785). - The same holds for every job built on image-garden, such as spread-gitlab-ci and snapcraft-gitlab-ci (README.md:785-787).
image-garden makewarns when/dev/kvmexists but is not writable (image-garden:670-671).
02 A runner that passes the device
In the runner's config.toml, [runners.docker] takes devices = ["/dev/kvm"]. Tag the runner, for example kvm, and the jobs with the same tag.
On a machine of yours, you register a runner with the docker executor, and in its config.toml, under the docker section, you add one line: devices equals /dev/kvm. Every job container on that runner now gets the device, with nothing to do in the job itself. Then give the runner a tag, such as kvm, and give the image-garden jobs the same tag, so GitLab sends them there and nowhere else. A job without the tag can still land on a hosted runner, and crawl.
Details and sources
Details
- The README's recipe: runners of your own that pass the device into the job's container,
devices = ["/dev/kvm"]in the docker executor'sconfig.toml, picked withtags(README.md:782-785). - Derived, from GitLab Runner's configuration:
devicesbelongs to the[runners.docker]section of a runner that uses the docker executor; the job itself needs no option. The tag namekvmis this map's example; image-garden's own pipeline useskvmandkvm-large(.gitlab-ci.yml:49, .gitlab-ci.yml:627-630).
Read more
03 One image, two kinds of host
The published image covers x86_64 and aarch64 hosts. Tag jobs with the host architecture too, such as [kvm, x86_64]: KVM helps only guests of the host's own architecture.
The image is built twice, natively, on an x86_64 runner and on an arm sixty-four one, and published as one name. Each runner pulls the variant for its own processor. KVM only speeds up guests of the host's own architecture, so an x86_64 runner runs x86_64 systems at full speed, and an arm sixty-four runner runs arm sixty-four ones. A tag per architecture picks the right host. Other guests, such as risk-five or s390x, are emulated anywhere. Give the runners meant for that slow, emulated work a tag of their own, so it doesn't hold up your short jobs.
Details and sources
Details
docker-archbuilds the image natively on anx86_64and anaarch64runner;docker-manifestjoins them as the release tag andlatest(.gitlab-ci.yml:1231-1278).- Derived: KVM accelerates only guests of the host's own architecture; a
.runscript asks forkvm:tcgwhen KVM is there and falls back to TCG (010-qemu-x86_64.mk:17). - Derived: a separate tag for emulated work keeps long, slow jobs off the runners that short jobs use; image-garden's own pipeline does this (.gitlab-ci.yml:663-678).
for contributors
In image-garden's CI
- image-garden's own test jobs pick hosts with
tags: [linux, x86_64, kvm](.gitlab-ci.yml:623-633) andtags: [aarch64, kvm, linux](.gitlab-ci.yml:649-659). - Emulated work runs on x86_64 runners tagged
qemu-emulated, "on slots of their own, never holding up short jobs", where each image boots only once (GARDEN_TEST_BOOT_ONLY) and a failure does not fail the pipeline (.gitlab-ci.yml:663-678).
Read more
04 A job, end to end
image: registry.gitlab.com/zygoon/image-garden:v0.8.0, tags: [kvm, x86_64], QEMU_MEM_OPTION and QEMU_SMP_OPTION in variables, then image-garden make.
Here's a whole job. Its image is the image-garden container, at a release tag, and its tags ask for KVM on x86_64. A couple of variables keep the machine small: two CPU cores and a gigabyte of memory, through the same QEMU option variables you'd use at home. The script makes the system and runs the tests: first the .run script, so the first boot is over, then spread on your project. To skip writing such a job, include spread-gitlab-ci, which brings spread along.
Details and sources
Details
- The job on the slide is illustrative, not from image-garden's tree, and was not run for this map. Its parts are: the published image (.gitlab-ci.yml:1257-1278), tags as in image-garden's own jobs (.gitlab-ci.yml:623-633), and the
QEMU_SMP_OPTION: -smp 2,QEMU_MEM_OPTION: -m 1024variables those jobs set (.gitlab-ci.yml:568-569). - image-garden's own test jobs install image-garden from source in
ubuntu:rollingrather than use the container (.gitlab-ci.yml:275-316), then runtests/ci/run-systems.shfor a cluster of systems (.gitlab-ci.yml:542-593); for each system it runsimage-garden make "$system.$arch.run"and, inspread-demo,make spread-"$system"(tests/ci/run-systems.sh:58-113). - The image has no spread; spread-gitlab-ci's image adds it.
Read more
05 A pkgproxy instead of a CI cache
No CI cache of images: set PKGPROXY_URL (or GARDEN_DL_PKGPROXY) in the runner's config.toml environment, and every job downloads through one pkgproxy.
Every job starts from an empty container, so the cloud image is downloaded again each time. The tempting fix is GitLab's cache, but zipping, uploading and restoring hundreds of megabytes per job costs more than it saves. So skip it. Instead, run a pkgproxy next to your runners, and set its address in the runner's environment, in config.toml. Every job then downloads images, packages and snaps through it, by checksum, from the internet once. If the proxy is down, jobs download directly and still pass.
Details and sources
Details
- image-garden's test jobs use no CI cache: zipping, uploading and restoring hundreds of MB per job costs more than it saves; a runner with a pkgproxy, 0.12 or later, sets
GARDEN_DL_PKGPROXYin its environment (config.toml), and image-garden downloads each image, and the spread binary, through it, by checksum (.gitlab-ci.yml:550-556). PKGPROXY_URLdoes the same for image downloads and also gives the guests a pkgproxy; one pkgproxy downloads each image once for every machine and CI job, and an unreachable one is skipped (README.md:234-255).- The address on the slide,
10.0.0.5, is an example. Derived: in GitLab Runner'sconfig.toml,environmentunder[[runners]]sets variables for every job of that runner. The address must be reachable from the job container's network.
Read more
06 Where to go from here
Next: spread-gitlab-ci, the craft components, why pkgproxy, and the container on GitLab against the snap on GitHub.
So a CI job needs a runner of yours with the KVM device, a tag per host architecture, the image at a release tag, and a pkgproxy in the runner's environment. Next, spread-gitlab-ci gives you a finished spread job in this image, and the craft components build snaps, rocks and charms in a VM. The pkgproxy deck explains the cache. And on GitHub, the action installs the snap instead: the comparison deck sets the two side by side.
Details and sources
Read more
Read more
Basis of the statements on this page
Read in the README and .gitlab-ci.yml of image-garden at the pinned commit ("code"). The runner configuration and the job below are examples, not run for this map ("derived").
A runner with KVM
[[runners]] name = "garden-x86_64" executor = "docker" environment = ["PKGPROXY_URL=http://10.0.0.5:3142"] [runners.docker] devices = ["/dev/kvm"]
Register it with the tags kvm and its architecture, x86_64 or aarch64.
A job
garden-tests: image: registry.gitlab.com/zygoon/image-garden:v0.8.0 tags: [kvm, x86_64] variables: QEMU_MEM_OPTION: -m 1024 QEMU_SMP_OPTION: -smp 2 script: - image-garden make ubuntu-cloud-24.04.x86_64.run
Choices and their reasons
| Choice | Why |
|---|---|
| runners of your own | hosted GitLab.com runners give no /dev/kvm |
devices = ["/dev/kvm"] | the job container gets the device without asking |
| a tag per host architecture | KVM helps only guests of the host's architecture |
a release tag of the image, not latest | the job stays the same until you change it |
PKGPROXY_URL in the runner's environment | images and packages are downloaded once, with no CI cache |
| no GitLab cache of images | hundreds of MB per job cost more to move than to fetch through pkgproxy |
Roads from here
ARCHITECTURES · machines and firmware
- Six architectures, six QEMU machines — which guests run with KVM on which host
BUILT ON IT · CI integrations
- spread-gitlab-ci — a ready-made job that runs spread in this image
- Building snaps, rocks and charms in a VM — ready-made jobs that build snaps, rocks and charms in a VM
- Container on GitLab, snap on GitHub — the same job with the snap on GitHub
PKGPROXY · caching for guests
- What pkgproxy gives a garden — why pkgproxy, not a CI cache, keeps jobs fast