Running image-garden in a container

/dev/kvm and a volume for users for CI operators

The image-garden container carries a patched QEMU for six architectures, swtpm and Secure Boot firmware. docker or podman run it with /dev/kvm passed in and the cache on a volume; image-garden make and allocate then work as on the host, in your project's directory, with pkgproxy named by PKGPROXY_URL.

On the map: container

▶ Watch this deck — 7 scenes, ~5 min

Part of journeys

01 The same commands, nothing installed

THE CONTAINER your machine a CI runner x86_64 or aarch64, with /dev/kvm registry.gitlab.com/zygoon/image-garden image-garden, QEMU, make, curl, xorriso everything it runs, already inside $ docker run … image-garden:latest … podman takes the same options $ image-garden make ubuntu-cloud-24.04.x86_64.run $ image-garden allocate --spread ubuntu-cloud-24.04.x86_64 the same commands as on any other install
Your machine or a CI runner, on x86_64 or aarch64, runs the image-garden container with docker or podman; inside are image-garden and its tools; the same commands, image-garden make and image-garden allocate, are typed inside

The container is image-garden with all its tools, for x86_64 and aarch64 hosts. Run it with docker or podman and use the usual commands inside.

Say you'd like image-garden without installing QEMU and its friends on your machine, or you want every CI runner to have exactly the same tools. That's what the container is for. It's one image, built for x86_64 and for arm sixty-four hosts, with image-garden and everything it runs already inside. You start it with docker or podman, hand it the KVM device and a place to keep downloads, and then type the same commands as anywhere else: image-garden make, and image-garden allocate. This deck shows what's in the image, how to run it on your own project, and what it leaves out.

Details and sources

Details

  • The final image is FROM ubuntu:latest with image-garden, QEMU and the tools it runs (Dockerfile:73-104).
  • It is built natively on an x86_64 and an aarch64 runner and joined into one image, tagged with the release (such as v0.8.0) and latest, at registry.gitlab.com/zygoon/image-garden (.gitlab-ci.yml:1231-1278). Docker and podman pull the variant of the host's architecture.
  • The commands inside are the same as on any install: see image-garden make and allocate.

02 What the image carries

WHAT IS INSIDE FROM ubuntu:latest QEMU, built from source, with the guestfwd patch x86_64 aarch64 riscv64 s390x ppc64 loongarch64 qemu-img image-garden + its .mk make curl xorriso xz git swtpm and OVMF, for the +tpm systems not inside: spread pkgproxy an entrypoint name the command you want to run
The image is built from Ubuntu; QEMU is compiled with the guestfwd patch for x86_64, aarch64, riscv64, s390x, ppc64 and loongarch64; qemu-img, image-garden and its makefile, make, curl, xorriso, xz and git are next to it, with swtpm and OVMF for TPM systems; spread, pkgproxy and an entrypoint are not included

QEMU for six architectures with the guestfwd patch, qemu-img, image-garden, make, curl, xorriso, xz, git, swtpm and OVMF. No spread, no pkgproxy, no entrypoint.

Let's look inside. The image starts from Ubuntu and builds QEMU from source, with the same small patch the snap carries, for six guest architectures: x86_64, arm sixty-four, risk-five, s390x, ppc64 and loongarch64. Next to it sit qemu-img, the image-garden script and its makefile, and the tools they call: make, curl, xorriso, xz and git. swtpm and the OVMF package are there for the Ubuntu Core systems with a TPM. What isn't there matters too: no spread, no pkgproxy, and no entrypoint. You always name the command you want to run.

Details and sources

Details

  • QEMU 11.1.3 is built from its release tarball, checked by sha256, with qemu-patches/guestfwd-unix-socket.patch applied, for x86_64, aarch64, riscv64, s390x, ppc64 and loongarch64 system emulation, with tools (qemu-img) (Dockerfile:5-54).
  • image-garden is built, checked and installed in its own stage, then copied into /usr/local (Dockerfile:58-71, Dockerfile:103).
  • The runtime packages: ca-certificates, curl, git, git-restore-mtime, libcurl3t64-gnutls, libfdt1, libslirp0, make, ovmf, swtpm, xorriso, xz-utils (Dockerfile:78-91).
  • GARDEN_X86_64_PREFIX, GARDEN_AARCH64_PREFIX, GARDEN_RISCV64_PREFIX and GARDEN_LOONGARCH64_PREFIX, all /usr/local, point image-garden at the EDK2 firmware built with QEMU (Dockerfile:99-102); s390x and ppc64 need no firmware file.
  • The Dockerfile sets no ENTRYPOINT or CMD, and installs no spread and no pkgproxy.

03 Two things to pass in

TWO THINGS TO PASS IN $ docker run --rm -it \ --device /dev/kvm \ --volume ~/.cache/image-garden:/var/cache/image-garden \ registry.gitlab.com/zygoon/image-garden:v0.8.0 \ image-garden make ubuntu-cloud-24.04.x86_64.run KVM: seconds without it: emulated, many minutes a cache that outlives it the next container finds the images podman run takes the very same options
docker run with --device /dev/kvm, without which QEMU emulates and boots take minutes; with a volume mounting ~/.cache/image-garden at /var/cache/image-garden so downloads are kept; then the command image-garden make ubuntu-cloud-24.04.x86_64.run; podman takes the same options

--device /dev/kvm for speed, --volume ~/.cache/image-garden:/var/cache/image-garden to keep downloads, then image-garden make SYSTEM.run.

Two things go on the command line. The first is the KVM device, with --device /dev/kvm. Without it QEMU emulates every instruction, and a boot that takes seconds takes many minutes. The second is the cache. The image keeps downloads in a volume at /var/cache/image-garden, so mount a directory of yours there, and the next container finds the images the last one fetched. Then comes the command itself: image-garden make, with the system you want. Podman takes the very same options.

Details and sources

Details

  • The image sets GARDEN_CACHE_DIR=/var/cache/image-garden and declares it a VOLUME (Dockerfile:96-97).
  • The README's example passes --device /dev/kvm and --volume $HOME/.cache/image-garden:/var/cache/image-garden, for docker and podman alike (README.md:758-776). Without /dev/kvm, QEMU emulates the CPU (TCG), many times slower (README.md:780-781).
  • The project's own smoke test runs the image the same way, with image-garden make -C /tmp ubuntu-cloud-24.04.$(uname -m).run as the command (Makefile:137-157).
  • The README's examples run image-garden make ubuntu-cloud-24.04.x86_64.run (README.md:768, README.md:775), as the smoke test does. The image has no entrypoint, so the command has to be image-garden make: plain make would run GNU make without image-garden's makefile (derived from the Dockerfile).

04 Working in your project

YOUR PROJECT $ docker run … -v "$PWD:/src" -w /src … image-garden make ubuntu-cloud-24.04.x86_64.run /src, your project spread.yaml tests/ .image-garden/ ubuntu-cloud-24.04.x86_64.run ubuntu-cloud-24.04.x86_64.qcow2 ….qcow2.log .image-garden.mk your systems /var/cache/image-garden/dl the disk's backing file, in the volume keep one cache for the project
Your project directory is mounted at /src and made the working directory; image-garden make writes the .run script, the prepared disk and logs there, or in .image-garden/; .image-garden.mk is read as on the host; the prepared disk refers to its download in the cache volume

Mount the project with -v "$PWD:/src" -w /src. Files are made there, or in .image-garden/, and .image-garden.mk is read as on the host.

For your own tests, mount the project too, and make it the working directory, for example /src. image-garden make works where it's run: the .run script, the prepared disk and the logs land in your project, or in its .image-garden directory when that exists. Your .image-garden.mk is picked up the same way, so the systems you defined for your tests exist inside the container as well. The prepared disk refers back to its download in the cache volume, so keep using the same cache directory for the same project.

Details and sources

Details

  • image-garden make runs make in .image-garden/ when the project has that directory, and in the current directory otherwise (image-garden:982-993).
  • The project's .image-garden.mk is included from GARDEN_PROJECT_DIR, the directory image-garden was started in (999-local.mk:4-5).
  • -v "$PWD:/src" -w /src is the usual docker way to make the project the working directory; image-garden's notes use it to run a CI component's image on a checkout (proxy/README.md:277-279).
  • A prepared .qcow2 is an overlay whose backing file is the download in the cache directory: with another cache, see rebase.

05 allocate inside the container

ALLOCATE INSIDE the container $ image-garden allocate ubuntu-cloud-24.04.x86_64 QEMU, the guest's SSH 127.0.0.1:5123, the container's loopback ssh or spread in the same container how the CI components run it $ image-garden discard localhost:5123 or let the container exit
Inside the container, allocate starts QEMU and forwards 127.0.0.1:5123 to the guest's SSH; that loopback belongs to the container, so ssh or spread must run in the same container; discard stops the VM, or the container's exit takes QEMU with it

allocate forwards 127.0.0.1:PORT inside the container. Run ssh or spread in the same container, then image-garden discard.

allocate works too, with one thing to keep in mind. It forwards SSH from a port on the loopback address, and that's the loopback address of the container, not of your machine. So whatever logs in to the VM has to run in the same container: an ssh command in the same shell, or spread, started there after allocate. That's exactly how the CI components use it. When the tests are done, discard the machine before the container exits, or simply let the container go and take QEMU with it.

Details and sources

Details

  • allocate picks a port between 5000 and 9999 (image-garden:1100) and forwards 127.0.0.1:PORT to the guest's port 22 (image-garden:1107); it starts the .run script with -snapshot and -daemonize (image-garden:1153-1163).
  • Derived: in a container, 127.0.0.1 is the container's own loopback, so the forwarded port is reachable from inside it only, unless the port is published, which allocate's random port makes awkward.
  • The CI components run allocate and the program that logs in, ssh or spread, in the same job container (findings/research-integrations.md).
  • When the container exits, its processes, QEMU included, end with it.

06 pkgproxy by URL

PKGPROXY BY URL $ docker run … -e PKGPROXY_URL=http://pkgproxy.example.lan:3142 … the container image-garden images, checksum lists the guest packages, snaps, Go modules pkgproxy, port 3142 reachable from the container the name becomes an address when the image is made; the guest reaches it through QEMU's user network unreachable: downloads go to their origin
docker run -e PKGPROXY_URL names a pkgproxy; image-garden asks it for cloud images and checksum lists; guests send packages, snaps and Go modules there; the host name is turned into an address when the image is made and the guest reaches it through QEMU's user network; an unreachable proxy means direct downloads

-e PKGPROXY_URL=http://pkgproxy.example.lan:3142 sends image downloads and guest packages through a pkgproxy that the container can reach.

The container has no pkgproxy of its own, so name one with the PKGPROXY_URL variable, passed in with -e. image-garden then asks that pkgproxy for every cloud image and checksum list, and the guests it prepares send their packages, snaps and Go modules there too. The address has to work from inside the container. The host's name is turned into an address when an image is made, and the guest reaches it through QEMU's user network. If the proxy can't be reached, downloads go straight to their origin, so nothing fails because of it.

Details and sources

Details

  • With PKGPROXY_URL set, make takes the host and port from it and resolves the host to an IPv4 address when it makes a system's files; a host that does not resolve leaves guests without pkgproxy, with a warning (010-qemu.mk:24-30). The guest then knows that address as pkgproxy.
  • The same variable has image-garden download images through that pkgproxy, unless GARDEN_DL_PKGPROXY says otherwise (image-garden:31-38).
  • One pkgproxy on the local network downloads each image once for every machine and CI job; when it cannot be reached or cannot give an image, image-garden downloads it directly (README.md:250-255).
  • The socket modes, with QEMU's guestfwd to 10.0.2.100, are set up by the snap; outside it, PKGPROXY_URL is the way (proxy/README.md:7-12, proxy/README.md:117-127).

07 Where to go from here

ROADS In a container KVM in CI runners with /dev/kvm spread-gitlab-ci spread added to this image Three ways in snap, container, packages image-garden make the command line allocate a VM in the background Reaching the host 10.0.2.100 or a URL
Six roads from Running image-garden in a container: the container in CI, spread-gitlab-ci, three ways in, image-garden make, allocate and reaching pkgproxy

Next: the container in CI, spread-gitlab-ci, the three ways in, image-garden make, allocate and reaching pkgproxy.

So the container is image-garden with its tools, the KVM device and a cache volume passed in, your project mounted as the working directory, and a pkgproxy named by URL. Next, the CI deck runs the same image on GitLab runners. spread-gitlab-ci adds spread to it. To compare it with the snap and native packages, take the three ways in. And for the commands themselves, there's image-garden make, allocate, and how a guest reaches pkgproxy.

Details and sources

Details

  • The deck's commands in one place are in the reading section below.

Read more

Basis of the statements on this page

Read in the Dockerfile, Makefile, README and .gitlab-ci.yml of image-garden at the pinned commit ("code"). The commands were not run for this map.

Running the container

docker run --rm -it \ --device /dev/kvm \ --volume $HOME/.cache/image-garden:/var/cache/image-garden \ --volume "$PWD:/src" --workdir /src \ --env PKGPROXY_URL=http://pkgproxy.example.lan:3142 \ registry.gitlab.com/zygoon/image-garden:v0.8.0 \ image-garden make ubuntu-cloud-24.04.x86_64.run

podman run takes the same options. Leave out PKGPROXY_URL when you have no pkgproxy.

OptionWhy
--device /dev/kvmhardware acceleration; without it QEMU emulates the CPU, many times slower
--volume …:/var/cache/image-gardenkeeps downloaded images; the image sets GARDEN_CACHE_DIR there and declares it a volume
--volume "$PWD:/src" --workdir /srcmakes your project the working directory, so .image-garden.mk is read and files land there
--env PKGPROXY_URL=…a pkgproxy for image downloads and for the guests
-itan interactive terminal, for a .run script's serial console

What the image contains

PartWhere
QEMU 11.1.3 with the guestfwd patch, six system emulators and qemu-img/usr/local
image-garden and image-garden.mk/usr/local
make, curl, xorriso, xz, git, git-restore-mtimeUbuntu packages
swtpm and OVMF (Secure Boot firmware)Ubuntu packages
the cache/var/cache/image-garden, a volume

Not included: spread, pkgproxy, an entrypoint.

Tags

The image is published at registry.gitlab.com/zygoon/image-garden with each release's tag, such as v0.8.0, and latest for the newest release, for x86_64 and aarch64 hosts (.gitlab-ci.yml:1257-1278).

Roads from here

START HERE

RUNTIME · make rules and the CLI

BUILT ON IT · CI integrations

PKGPROXY · caching for guests

CONTAINER · OCI image