Container on GitLab, snap on GitHub
container vs snap for CI operators
On GitLab the job runs inside an image built on the image-garden container, on your own KVM runners, with a long-lived pkgproxy named by PKGPROXY_URL; on GitHub the action installs the snaps on a fresh runner every job, runs pkgproxy beside them and keeps its cache in the GitHub cache.
On the map: built on it
▶ Watch this deck — 6 scenes, ~4 min
Worth seeing first
- Projects built on image-garden — the five projects compared here
01 One test suite, two CI systems
spread-gitlab-ci on GitLab and image-garden-action on GitHub run the same spread.yaml, with the same allocate and discard. They differ in four ways: where image-garden comes from, KVM, pkgproxy and what persists.
Say one spread suite runs on ubuntu-cloud-24.04 at home, and you want it in CI on GitLab and on GitHub. On GitLab you include spread-gitlab-ci; on GitHub you add image-garden-action to a workflow. Underneath both, spread runs your garden backend, which calls the same allocate and discard, and your spread.yaml doesn't change. Around that, nearly everything differs: where image-garden comes from, how the job gets KVM, where pkgproxy runs, and what's still there for the next job. This deck sets the two side by side, one question at a time.
Details and sources
Details
- spread-gitlab-ci runs
spreadwith thebackendinput,gardenby default (spread.yml:27-31); image-garden-action runsspreadwithspread-backend, alsogardenby default, and the requiredgarden-system(action.yaml:42-47). - Neither defines the backend: it is the
type: adhocbackend of yourspread.yaml, whose hooks runimage-garden allocate --spreadandimage-garden discard(README.md:740-751). - The craft components on GitLab call allocate and discard themselves, in the same container; what this deck says about the GitLab side holds for them too.
- Running one suite on both is an example; the projects do not require it.
Read more
02 Where image-garden comes from
GitLab: an image built FROM the image-garden container at a tag, with spread added; nothing is installed in the job. GitHub: snapd, pkgproxy and the image-garden snap installed in every job, from latest/stable unless a channel input says otherwise.
On GitLab, the job runs inside a container image, and the component's image is built from image-garden's own container, at a fixed tag. QEMU and every tool image-garden needs are already inside, and spread-gitlab-ci adds spread on top. The job installs nothing. On GitHub, the job gets a whole Ubuntu virtual machine instead, so the action installs things on it, in every job: snapd, the pkgproxy snap, and the image-garden snap from the stable channel. spread comes from the snap. So GitLab gives you exactly the image you pinned, and GitHub gives you the latest release, unless you name a channel.
Details and sources
Details
- spread-gitlab-ci's image is
FROM registry.gitlab.com/zygoon/image-garden:v0.7.0with checksum-pinned spread, spread-plus and spread-gitlab-report added (spread-gitlab-ci Dockerfile:20-52); the component's defaultimageis its own image of the same version (spread.yml:18-22). The craft components build onv0.7.2. - Known discrepancy: spread-gitlab-ci builds on an older image-garden. Its base is v0.7.0, older than the craft components' v0.7.2; none of the four uses v0.8.0 yet.
- The action installs or refreshes snapd from
snapd-channel, installs the pkgproxy snap frompkgproxy-channeland the image-garden snap, in devmode, fromimage-garden-channel, alllatest/stableby default, and aliasesimage-garden.spreadasspread(action.yaml:9-17, action.yaml:209-232, action.yaml:366-397). - What the container carries: running image-garden in a container; the snap: the image-garden snap.
Read more
03 Getting KVM
GitLab: your own runners with devices = ["/dev/kvm"], picked with tags. GitHub: the action runs chmod 666 /dev/kvm when the device exists, and otherwise continues, emulated.
Next, the KVM device. GitLab's hosted runners don't give jobs one. So on GitLab you run runners of your own, pass /dev/kvm into the job container in the runner's config.toml, and pick those runners with a tag. The craft components refuse to start without it. On GitHub, the action checks the runner for the device and opens it to every user. Where it's missing, the action only notes that it carries on without hardware acceleration, and every guest runs emulated, much more slowly. Either way, a guest of the host's own architecture is the fast one.
Details and sources
Details
- "GitLab.com's hosted runners do not give jobs
/dev/kvm, so run image-garden jobs on runners of your own that pass the device into the job's container (devices = ["/dev/kvm"]in the docker executor'sconfig.toml), and pick them withtags" (README.md:778-787). - The craft scripts stop with "cannot start a VM without /dev/kvm" (snapcraft-vm:319-321); spread-gitlab-ci has no such check and runs emulated.
- The action:
sudo chmod -v 666 /dev/kvmwhen it exists, otherwise the notice "/dev/kvm is not available, continuing without hardware acceleration" (action.yaml:153-160); without it, tests are "much, much slower" (README.md:367-373). - KVM only speeds up guests of the host's architecture: KVM in CI, six architectures.
04 Where pkgproxy runs
GitLab: a long-lived pkgproxy named by PKGPROXY_URL, reached by its address. GitHub: the pkgproxy snap on the runner, pkgproxy=snap, reached through QEMU's guestfwd to its socket; or an existing one on port 3142 by URL.
pkgproxy sits in different places too. On GitLab, it's a machine of yours next to the runners, running all the time. The runner's environment names it with PKGPROXY_URL, and each guest connects to its address through QEMU's user network. On GitHub, pkgproxy is a snap on the runner itself, installed by the action. image-garden is connected to its sockets, and QEMU forwards a fixed address in each guest straight to pkgproxy's socket on the host. If something already listens on pkgproxy's port, as when setup-pkgproxy ran first, the action uses that one, by its URL, instead.
Details and sources
Details
- With
PKGPROXY_URLset, image-garden resolves its host to IPv4 and the guest connects there; otherwise, with the sockets of the bundled pkgproxy or of a pkgproxy snap, QEMU getsguestfwd=tcp:10.0.2.100:3142-unix:…/pkgproxy.sock(010-qemu.mk:20-44). - The action uses a pkgproxy that already listens on port 3142 ("external" mode), otherwise installs the snap ("snap" mode) (action.yaml:173-188); it connects
image-garden:pkgproxy-socketsand setspkgproxy=snap, or setspkgproxy=http://HOST-IP:3142(action.yaml:376-393). setup-pkgproxy must come before the action (README.md:229-239). - On GitLab the craft components take the URL from
pkgproxy-urlor the runner'sPKGPROXY_URL: the craft components' pkgproxy.
05 What the next job finds
GitLab: jobs keep nothing but spread's rerun list; pkgproxy's cache stays on its own machine. GitHub: the action saves pkgproxy's cache in two GitHub caches and restores it; prepared images are opt-in.
And when the job ends? On GitLab, the job container is thrown away, and the components keep nothing: the craft scripts delete their scratch directory, and spread-gitlab-ci keeps only its list of tasks to rerun. The memory lives in pkgproxy, whose cache simply stays on its machine. On GitHub, the whole runner is gone, pkgproxy included. So the action packs pkgproxy's cache into the GitHub cache, images and packages apart, and the next job unpacks it. Prepared images can be cached too, if you opt in. Same idea both ways: keep the downloads, and redo the rest.
Details and sources
Details
- The craft scripts remove their work directory (snapcraft-vm:287-290) and their job has no
cache:; spread-gitlab-ci's only cache is.spread-rerun/, keyed per pipeline and job (spread.yml:115-121). - The action's two pkgproxy caches (action.yaml:189-208) and the opt-in prepared-image cache (action.yaml:116-123): how the action caches.
- "Redo the rest" means the provisioning boot, the tests or the pack (derived).
06 Where to go from here
Next: spread-gitlab-ci, KVM in CI, the GitHub action, its caches, the craft components' pkgproxy and the snap's pkgproxy setting.
So on GitLab you pin an image, bring your own KVM runners and run one pkgproxy that remembers everything. On GitHub the action builds the same setup on a fresh runner every time and carries pkgproxy's cache along. For the GitLab side in full, take spread-gitlab-ci, and KVM in CI for the runners. For the GitHub side, the action's deck, and its caching deck. The craft components' pkgproxy deck shows the GitLab address rules, and the snap's pkgproxy setting explains the GitHub one.
Details and sources
Read more
Basis of the statements on this page
Read in spread-gitlab-ci at 3d522d4, the craft components at the commits in integrations.commits, image-garden-action at 2891c7a and image-garden v0.8.0 ("code"). Nothing was run for this page.
Side by side
| Question | GitLab (spread-gitlab-ci, craft components) | GitHub (image-garden-action) |
|---|---|---|
| Where the job runs | in a container image built FROM the image-garden container | on a fresh Ubuntu runner |
| image-garden | the container at a fixed tag (v0.7.0 or v0.7.2) | the snap, image-garden-channel, latest/stable by default, installed every job |
| spread | added to the image by spread-gitlab-ci | the snap's spread component, aliased as spread |
| KVM | your own runners, devices = ["/dev/kvm"], picked with tags | chmod 666 /dev/kvm when present; otherwise emulated |
| pkgproxy | a long-lived pkgproxy, PKGPROXY_URL (and GARDEN_DL_PKGPROXY) in the runner's environment | the pkgproxy snap on the runner, pkgproxy=snap; or an existing one on port 3142 by URL |
| How a guest reaches it | the URL's host, resolved to IPv4, through QEMU's user network | guestfwd from 10.0.2.100:3142 to pkgproxy's socket |
| What persists | pkgproxy's own disk; spread-gitlab-ci's rerun list | pkgproxy's cache in two GitHub caches; prepared images opt-in |
| A new image-garden release | when the component's image moves to it | in the next job |
Sources: the image-garden README's CI section, 010-qemu.mk:20-44, spread-gitlab-ci Dockerfile:20-52, image-garden-action action.yaml.
Roads from here
BUILT ON IT · CI integrations
- spread-gitlab-ci — the GitLab side, spread-gitlab-ci, in full
- image-garden-action for GitHub — the GitHub side, image-garden-action, in full
- How the action caches — how the GitHub side keeps pkgproxy's cache
- Why the craft components keep no cache — the GitLab side's pkgproxy, from a craft job
SNAP · snap-only features
- The pkgproxy snap setting — pkgproxy=snap and pkgproxy=URL in the snap
CONTAINER · OCI image
- Running the container in CI — runners with /dev/kvm and pkgproxy in the runner's environment