Running the container in CI

runners with /dev/kvm for CI operators

Hosted GitLab runners give jobs no /dev/kvm, so image-garden jobs run on runners of your own that pass the device into the job container and are picked with tags, one runner per host architecture. A pkgproxy set in the runner's environment replaces a CI cache of images.

On the map: container

▶ Watch this deck — 6 scenes, ~4 min

Worth seeing first

Part of journeys

01 No KVM on hosted runners

KVM IN CI your laptop: passes a GitLab job in the image-garden container a hosted runner /dev/kvm QEMU emulates: many times slower may not fit the job's time limit a runner of your own /dev/kvm → the job container guests at full speed next: how such a runner is set up, and how jobs find it
Tests pass on your laptop; in GitLab CI the job runs the image-garden container; hosted runners have no /dev/kvm, so QEMU emulates and a run may not fit the time limit; runners of your own pass /dev/kvm into the job container

GitLab.com's hosted runners have no /dev/kvm: guests would be emulated, many times slower. Run image-garden jobs on runners of your own.

Your tests pass on your laptop, and now you'd like GitLab to run them on every push. The container is the obvious way in: the job's image already has everything. There's one catch. GitLab's hosted runners don't give jobs the KVM device. image-garden still works there, but QEMU has to emulate every instruction, many times slower, and a run of tests may not fit in the job's time limit at all. So image-garden jobs run on runners of your own, machines that can hand the KVM device to the job's container. This deck shows how they're set up and picked.

Details and sources

Details

  • image-garden needs /dev/kvm to run machines at full speed; without it QEMU emulates the CPU (TCG), many times slower. GitLab.com's hosted runners do not give jobs /dev/kvm (README.md:778-785).
  • The same holds for every job built on image-garden, such as spread-gitlab-ci and snapcraft-gitlab-ci (README.md:785-787).
  • image-garden make warns when /dev/kvm exists but is not writable (image-garden:670-671).

02 A runner that passes the device

A RUNNER WITH KVM config.toml on your machine [[runners]] executor = "docker" [runners.docker] devices = ["/dev/kvm"] every job container gets /dev/kvm your pipeline configuration garden-tests: tags: [kvm] runner tagged kvm, job tagged kvm: GitLab sends it there no tag: it may land on a hosted runner, and crawl
The runner's config.toml, under runners.docker, has devices = /dev/kvm and a kvm tag; a job with tags kvm goes to that runner and gets /dev/kvm in its container; a job without the tag may land on a hosted runner and crawl

In the runner's config.toml, [runners.docker] takes devices = ["/dev/kvm"]. Tag the runner, for example kvm, and the jobs with the same tag.

On a machine of yours, you register a runner with the docker executor, and in its config.toml, under the docker section, you add one line: devices equals /dev/kvm. Every job container on that runner now gets the device, with nothing to do in the job itself. Then give the runner a tag, such as kvm, and give the image-garden jobs the same tag, so GitLab sends them there and nowhere else. A job without the tag can still land on a hosted runner, and crawl.

Details and sources

Details

  • The README's recipe: runners of your own that pass the device into the job's container, devices = ["/dev/kvm"] in the docker executor's config.toml, picked with tags (README.md:782-785).
  • Derived, from GitLab Runner's configuration: devices belongs to the [runners.docker] section of a runner that uses the docker executor; the job itself needs no option. The tag name kvm is this map's example; image-garden's own pipeline uses kvm and kvm-large (.gitlab-ci.yml:49, .gitlab-ci.yml:627-630).

03 One image, two kinds of host

ONE IMAGE, TWO HOSTS registry.gitlab.com/zygoon/image-garden x86_64 runner aarch64 runner ubuntu-cloud-24.04.x86_64 KVM: full speed ubuntu-cloud-24.04.aarch64 KVM: full speed tags: [kvm, x86_64] tags: [kvm, aarch64] riscv64, s390x, ppc64, loongarch64 guests: emulated on any host give runners for emulated work a tag of their own, so short jobs don't wait
The image is built natively on an x86_64 and an aarch64 runner and published as one name; each runner pulls its own variant; KVM speeds up guests of the host's architecture; tags such as kvm and x86_64 pick the host; riscv64, s390x and other guests are emulated, and runners for that slow work deserve a tag of their own

The published image covers x86_64 and aarch64 hosts. Tag jobs with the host architecture too, such as [kvm, x86_64]: KVM helps only guests of the host's own architecture.

The image is built twice, natively, on an x86_64 runner and on an arm sixty-four one, and published as one name. Each runner pulls the variant for its own processor. KVM only speeds up guests of the host's own architecture, so an x86_64 runner runs x86_64 systems at full speed, and an arm sixty-four runner runs arm sixty-four ones. A tag per architecture picks the right host. Other guests, such as risk-five or s390x, are emulated anywhere. Give the runners meant for that slow, emulated work a tag of their own, so it doesn't hold up your short jobs.

Details and sources

Details

  • docker-arch builds the image natively on an x86_64 and an aarch64 runner; docker-manifest joins them as the release tag and latest (.gitlab-ci.yml:1231-1278).
  • Derived: KVM accelerates only guests of the host's own architecture; a .run script asks for kvm:tcg when KVM is there and falls back to TCG (010-qemu-x86_64.mk:17).
  • Derived: a separate tag for emulated work keeps long, slow jobs off the runners that short jobs use; image-garden's own pipeline does this (.gitlab-ci.yml:663-678).

for contributors

In image-garden's CI

  • image-garden's own test jobs pick hosts with tags: [linux, x86_64, kvm] (.gitlab-ci.yml:623-633) and tags: [aarch64, kvm, linux] (.gitlab-ci.yml:649-659).
  • Emulated work runs on x86_64 runners tagged qemu-emulated, "on slots of their own, never holding up short jobs", where each image boots only once (GARDEN_TEST_BOOT_ONLY) and a failure does not fail the pipeline (.gitlab-ci.yml:663-678).

04 A job, end to end

A JOB garden-tests: image: registry.gitlab.com/zygoon/image-garden:v0.8.0 tags: [kvm, x86_64] variables: QEMU_MEM_OPTION: -m 1024 QEMU_SMP_OPTION: -smp 2 the same variables as on your machine script: - image-garden make ubuntu-cloud-24.04.x86_64.run - your tests, for example spread first the .run, so the first boot is over, then spread or include spread-gitlab-ci, spread included
A whole GitLab job: image registry.gitlab.com/zygoon/image-garden:v0.8.0, tags kvm and x86_64, variables QEMU_MEM_OPTION -m 1024 and QEMU_SMP_OPTION -smp 2, and a script that runs image-garden make and then the tests; spread-gitlab-ci is a finished job of this kind

image: registry.gitlab.com/zygoon/image-garden:v0.8.0, tags: [kvm, x86_64], QEMU_MEM_OPTION and QEMU_SMP_OPTION in variables, then image-garden make.

Here's a whole job. Its image is the image-garden container, at a release tag, and its tags ask for KVM on x86_64. A couple of variables keep the machine small: two CPU cores and a gigabyte of memory, through the same QEMU option variables you'd use at home. The script makes the system and runs the tests: first the .run script, so the first boot is over, then spread on your project. To skip writing such a job, include spread-gitlab-ci, which brings spread along.

Details and sources

Details

  • The job on the slide is illustrative, not from image-garden's tree, and was not run for this map. Its parts are: the published image (.gitlab-ci.yml:1257-1278), tags as in image-garden's own jobs (.gitlab-ci.yml:623-633), and the QEMU_SMP_OPTION: -smp 2, QEMU_MEM_OPTION: -m 1024 variables those jobs set (.gitlab-ci.yml:568-569).
  • image-garden's own test jobs install image-garden from source in ubuntu:rolling rather than use the container (.gitlab-ci.yml:275-316), then run tests/ci/run-systems.sh for a cluster of systems (.gitlab-ci.yml:542-593); for each system it runs image-garden make "$system.$arch.run" and, in spread-demo, make spread-"$system" (tests/ci/run-systems.sh:58-113).
  • The image has no spread; spread-gitlab-ci's image adds it.

05 A pkgproxy instead of a CI cache

NO CI CACHE each job: empty GitLab cache: zip, upload, restore hundreds of MB per job: costs more than it saves so skip it config.toml on the runner environment = [ "PKGPROXY_URL=http://10.0.0.5:3142"] one pkgproxy for every job images, packages, snaps: by checksum, from the internet once proxy down: jobs download directly, and still pass
Each job starts from an empty container; a GitLab cache of images means zipping, uploading and restoring hundreds of megabytes per job, so skip it; a pkgproxy next to the runners, set in the runner's environment, serves every job's images, packages and snaps, fetched from the internet once; when it is down, jobs download directly

No CI cache of images: set PKGPROXY_URL (or GARDEN_DL_PKGPROXY) in the runner's config.toml environment, and every job downloads through one pkgproxy.

Every job starts from an empty container, so the cloud image is downloaded again each time. The tempting fix is GitLab's cache, but zipping, uploading and restoring hundreds of megabytes per job costs more than it saves. So skip it. Instead, run a pkgproxy next to your runners, and set its address in the runner's environment, in config.toml. Every job then downloads images, packages and snaps through it, by checksum, from the internet once. If the proxy is down, jobs download directly and still pass.

Details and sources

Details

  • image-garden's test jobs use no CI cache: zipping, uploading and restoring hundreds of MB per job costs more than it saves; a runner with a pkgproxy, 0.12 or later, sets GARDEN_DL_PKGPROXY in its environment (config.toml), and image-garden downloads each image, and the spread binary, through it, by checksum (.gitlab-ci.yml:550-556).
  • PKGPROXY_URL does the same for image downloads and also gives the guests a pkgproxy; one pkgproxy downloads each image once for every machine and CI job, and an unreachable one is skipped (README.md:234-255).
  • The address on the slide, 10.0.0.5, is an example. Derived: in GitLab Runner's config.toml, environment under [[runners]] sets variables for every job of that runner. The address must be reachable from the job container's network.

06 Where to go from here

ROADS The container in CI spread-gitlab-ci a finished spread job *craft-gitlab-ci snaps, rocks, charms in a VM Why pkgproxy download once GitLab vs GitHub container vs snap
Four roads from Running the container in CI: spread-gitlab-ci, the craft components, why pkgproxy, and GitLab against GitHub

Next: spread-gitlab-ci, the craft components, why pkgproxy, and the container on GitLab against the snap on GitHub.

So a CI job needs a runner of yours with the KVM device, a tag per host architecture, the image at a release tag, and a pkgproxy in the runner's environment. Next, spread-gitlab-ci gives you a finished spread job in this image, and the craft components build snaps, rocks and charms in a VM. The pkgproxy deck explains the cache. And on GitHub, the action installs the snap instead: the comparison deck sets the two side by side.

Details and sources

Read more

Basis of the statements on this page

Read in the README and .gitlab-ci.yml of image-garden at the pinned commit ("code"). The runner configuration and the job below are examples, not run for this map ("derived").

A runner with KVM

[[runners]] name = "garden-x86_64" executor = "docker" environment = ["PKGPROXY_URL=http://10.0.0.5:3142"] [runners.docker] devices = ["/dev/kvm"]

Register it with the tags kvm and its architecture, x86_64 or aarch64.

A job

garden-tests: image: registry.gitlab.com/zygoon/image-garden:v0.8.0 tags: [kvm, x86_64] variables: QEMU_MEM_OPTION: -m 1024 QEMU_SMP_OPTION: -smp 2 script: - image-garden make ubuntu-cloud-24.04.x86_64.run

Choices and their reasons

ChoiceWhy
runners of your ownhosted GitLab.com runners give no /dev/kvm
devices = ["/dev/kvm"]the job container gets the device without asking
a tag per host architectureKVM helps only guests of the host's architecture
a release tag of the image, not latestthe job stays the same until you change it
PKGPROXY_URL in the runner's environmentimages and packages are downloaded once, with no CI cache
no GitLab cache of imageshundreds of MB per job cost more to move than to fetch through pkgproxy

Roads from here

ARCHITECTURES · machines and firmware

BUILT ON IT · CI integrations

PKGPROXY · caching for guests