Running image-garden in a container
/dev/kvm and a volume for users for CI operators
The image-garden container carries a patched QEMU for six architectures, swtpm and Secure Boot firmware. docker or podman run it with /dev/kvm passed in and the cache on a volume; image-garden make and allocate then work as on the host, in your project's directory, with pkgproxy named by PKGPROXY_URL.
On the map: container
▶ Watch this deck — 7 scenes, ~5 min
Part of journeys
- From your laptop to a git forge, step 2 of 6 · previous: The adhoc backend: allocate and discard · next: Running the container in CI
01 The same commands, nothing installed
The container is image-garden with all its tools, for x86_64 and aarch64 hosts. Run it with docker or podman and use the usual commands inside.
Say you'd like image-garden without installing QEMU and its friends on your machine, or you want every CI runner to have exactly the same tools. That's what the container is for. It's one image, built for x86_64 and for arm sixty-four hosts, with image-garden and everything it runs already inside. You start it with docker or podman, hand it the KVM device and a place to keep downloads, and then type the same commands as anywhere else: image-garden make, and image-garden allocate. This deck shows what's in the image, how to run it on your own project, and what it leaves out.
Details and sources
Details
- The final image is
FROM ubuntu:latestwith image-garden, QEMU and the tools it runs (Dockerfile:73-104). - It is built natively on an x86_64 and an aarch64 runner and joined into one image, tagged with the release (such as
v0.8.0) andlatest, atregistry.gitlab.com/zygoon/image-garden(.gitlab-ci.yml:1231-1278). Docker and podman pull the variant of the host's architecture. - The commands inside are the same as on any install: see image-garden make and allocate.
Read more
02 What the image carries
QEMU for six architectures with the guestfwd patch, qemu-img, image-garden, make, curl, xorriso, xz, git, swtpm and OVMF. No spread, no pkgproxy, no entrypoint.
Let's look inside. The image starts from Ubuntu and builds QEMU from source, with the same small patch the snap carries, for six guest architectures: x86_64, arm sixty-four, risk-five, s390x, ppc64 and loongarch64. Next to it sit qemu-img, the image-garden script and its makefile, and the tools they call: make, curl, xorriso, xz and git. swtpm and the OVMF package are there for the Ubuntu Core systems with a TPM. What isn't there matters too: no spread, no pkgproxy, and no entrypoint. You always name the command you want to run.
Details and sources
Details
- QEMU 11.1.3 is built from its release tarball, checked by sha256, with
qemu-patches/guestfwd-unix-socket.patchapplied, forx86_64,aarch64,riscv64,s390x,ppc64andloongarch64system emulation, with tools (qemu-img) (Dockerfile:5-54). - image-garden is built, checked and installed in its own stage, then copied into
/usr/local(Dockerfile:58-71, Dockerfile:103). - The runtime packages:
ca-certificates,curl,git,git-restore-mtime,libcurl3t64-gnutls,libfdt1,libslirp0,make,ovmf,swtpm,xorriso,xz-utils(Dockerfile:78-91). GARDEN_X86_64_PREFIX,GARDEN_AARCH64_PREFIX,GARDEN_RISCV64_PREFIXandGARDEN_LOONGARCH64_PREFIX, all/usr/local, point image-garden at the EDK2 firmware built with QEMU (Dockerfile:99-102); s390x and ppc64 need no firmware file.- The Dockerfile sets no
ENTRYPOINTorCMD, and installs no spread and no pkgproxy.
Read more
- The guestfwd patch, Secure Boot and TPM, spread-gitlab-ci, which adds spread.
03 Two things to pass in
--device /dev/kvm for speed, --volume ~/.cache/image-garden:/var/cache/image-garden to keep downloads, then image-garden make SYSTEM.run.
Two things go on the command line. The first is the KVM device, with --device /dev/kvm. Without it QEMU emulates every instruction, and a boot that takes seconds takes many minutes. The second is the cache. The image keeps downloads in a volume at /var/cache/image-garden, so mount a directory of yours there, and the next container finds the images the last one fetched. Then comes the command itself: image-garden make, with the system you want. Podman takes the very same options.
Details and sources
Details
- The image sets
GARDEN_CACHE_DIR=/var/cache/image-gardenand declares it aVOLUME(Dockerfile:96-97). - The README's example passes
--device /dev/kvmand--volume $HOME/.cache/image-garden:/var/cache/image-garden, for docker and podman alike (README.md:758-776). Without/dev/kvm, QEMU emulates the CPU (TCG), many times slower (README.md:780-781). - The project's own smoke test runs the image the same way, with
image-garden make -C /tmp ubuntu-cloud-24.04.$(uname -m).runas the command (Makefile:137-157). - The README's examples run
image-garden make ubuntu-cloud-24.04.x86_64.run(README.md:768, README.md:775), as the smoke test does. The image has no entrypoint, so the command has to beimage-garden make: plainmakewould run GNU make without image-garden's makefile (derived from the Dockerfile).
Read more
04 Working in your project
Mount the project with -v "$PWD:/src" -w /src. Files are made there, or in .image-garden/, and .image-garden.mk is read as on the host.
For your own tests, mount the project too, and make it the working directory, for example /src. image-garden make works where it's run: the .run script, the prepared disk and the logs land in your project, or in its .image-garden directory when that exists. Your .image-garden.mk is picked up the same way, so the systems you defined for your tests exist inside the container as well. The prepared disk refers back to its download in the cache volume, so keep using the same cache directory for the same project.
Details and sources
Details
image-garden makeruns make in.image-garden/when the project has that directory, and in the current directory otherwise (image-garden:982-993).- The project's
.image-garden.mkis included fromGARDEN_PROJECT_DIR, the directory image-garden was started in (999-local.mk:4-5). -v "$PWD:/src" -w /srcis the usual docker way to make the project the working directory; image-garden's notes use it to run a CI component's image on a checkout (proxy/README.md:277-279).- A prepared
.qcow2is an overlay whose backing file is the download in the cache directory: with another cache, see rebase.
Read more
05 allocate inside the container
allocate forwards 127.0.0.1:PORT inside the container. Run ssh or spread in the same container, then image-garden discard.
allocate works too, with one thing to keep in mind. It forwards SSH from a port on the loopback address, and that's the loopback address of the container, not of your machine. So whatever logs in to the VM has to run in the same container: an ssh command in the same shell, or spread, started there after allocate. That's exactly how the CI components use it. When the tests are done, discard the machine before the container exits, or simply let the container go and take QEMU with it.
Details and sources
Details
- allocate picks a port between 5000 and 9999 (image-garden:1100) and forwards
127.0.0.1:PORTto the guest's port 22 (image-garden:1107); it starts the.runscript with-snapshotand-daemonize(image-garden:1153-1163). - Derived: in a container,
127.0.0.1is the container's own loopback, so the forwarded port is reachable from inside it only, unless the port is published, which allocate's random port makes awkward. - The CI components run allocate and the program that logs in,
sshor spread, in the same job container (findings/research-integrations.md). - When the container exits, its processes, QEMU included, end with it.
Read more
06 pkgproxy by URL
-e PKGPROXY_URL=http://pkgproxy.example.lan:3142 sends image downloads and guest packages through a pkgproxy that the container can reach.
The container has no pkgproxy of its own, so name one with the PKGPROXY_URL variable, passed in with -e. image-garden then asks that pkgproxy for every cloud image and checksum list, and the guests it prepares send their packages, snaps and Go modules there too. The address has to work from inside the container. The host's name is turned into an address when an image is made, and the guest reaches it through QEMU's user network. If the proxy can't be reached, downloads go straight to their origin, so nothing fails because of it.
Details and sources
Details
- With
PKGPROXY_URLset, make takes the host and port from it and resolves the host to an IPv4 address when it makes a system's files; a host that does not resolve leaves guests without pkgproxy, with a warning (010-qemu.mk:24-30). The guest then knows that address aspkgproxy. - The same variable has image-garden download images through that pkgproxy, unless
GARDEN_DL_PKGPROXYsays otherwise (image-garden:31-38). - One pkgproxy on the local network downloads each image once for every machine and CI job; when it cannot be reached or cannot give an image, image-garden downloads it directly (README.md:250-255).
- The socket modes, with QEMU's guestfwd to
10.0.2.100, are set up by the snap; outside it,PKGPROXY_URLis the way (proxy/README.md:7-12, proxy/README.md:117-127).
Read more
07 Where to go from here
Next: the container in CI, spread-gitlab-ci, the three ways in, image-garden make, allocate and reaching pkgproxy.
So the container is image-garden with its tools, the KVM device and a cache volume passed in, your project mounted as the working directory, and a pkgproxy named by URL. Next, the CI deck runs the same image on GitLab runners. spread-gitlab-ci adds spread to it. To compare it with the snap and native packages, take the three ways in. And for the commands themselves, there's image-garden make, allocate, and how a guest reaches pkgproxy.
Details and sources
Details
- The deck's commands in one place are in the reading section below.
Read more
Basis of the statements on this page
Read in the Dockerfile, Makefile, README and .gitlab-ci.yml of image-garden at the pinned commit ("code"). The commands were not run for this map.
Running the container
docker run --rm -it \ --device /dev/kvm \ --volume $HOME/.cache/image-garden:/var/cache/image-garden \ --volume "$PWD:/src" --workdir /src \ --env PKGPROXY_URL=http://pkgproxy.example.lan:3142 \ registry.gitlab.com/zygoon/image-garden:v0.8.0 \ image-garden make ubuntu-cloud-24.04.x86_64.run
podman run takes the same options. Leave out PKGPROXY_URL when you have no pkgproxy.
| Option | Why |
|---|---|
--device /dev/kvm | hardware acceleration; without it QEMU emulates the CPU, many times slower |
--volume …:/var/cache/image-garden | keeps downloaded images; the image sets GARDEN_CACHE_DIR there and declares it a volume |
--volume "$PWD:/src" --workdir /src | makes your project the working directory, so .image-garden.mk is read and files land there |
--env PKGPROXY_URL=… | a pkgproxy for image downloads and for the guests |
-it | an interactive terminal, for a .run script's serial console |
What the image contains
| Part | Where |
|---|---|
QEMU 11.1.3 with the guestfwd patch, six system emulators and qemu-img | /usr/local |
image-garden and image-garden.mk | /usr/local |
| make, curl, xorriso, xz, git, git-restore-mtime | Ubuntu packages |
| swtpm and OVMF (Secure Boot firmware) | Ubuntu packages |
| the cache | /var/cache/image-garden, a volume |
Not included: spread, pkgproxy, an entrypoint.
Tags
The image is published at registry.gitlab.com/zygoon/image-garden with each release's tag, such as v0.8.0, and latest for the newest release, for x86_64 and aarch64 hosts (.gitlab-ci.yml:1257-1278).
Roads from here
START HERE
- Snap, container or native package — how the container compares with the snap and native packages
RUNTIME · make rules and the CLI
- image-garden make and the other commands — the image-garden commands you run inside
- image-garden allocate — allocate, and why spread runs in the same container
BUILT ON IT · CI integrations
- spread-gitlab-ci — spread added to this image, as a GitLab CI component
PKGPROXY · caching for guests
- How a guest reaches pkgproxy — how a guest reaches a pkgproxy named by a URL
CONTAINER · OCI image
- Running the container in CI — the same image in GitLab CI, on runners with /dev/kvm