Container on GitLab, snap on GitHub

container vs snap for CI operators

On GitLab the job runs inside an image built on the image-garden container, on your own KVM runners, with a long-lived pkgproxy named by PKGPROXY_URL; on GitHub the action installs the snaps on a fresh runner every job, runs pkgproxy beside them and keeps its cache in the GitHub cache.

On the map: built on it

▶ Watch this deck — 6 scenes, ~4 min

Worth seeing first

01 One test suite, two CI systems

GITLAB AND GITHUB spread.yaml · ubuntu-cloud-24.04 GitLab: include spread-gitlab-ci GitHub: uses image-garden-action the same allocate and discard around it, four differences: image-garden KVM pkgproxy the next job side by side, one question at a time
One spread suite for ubuntu-cloud-24.04: on GitLab spread-gitlab-ci runs it, on GitHub image-garden-action does; both run your spread.yaml's garden backend with the same allocate and discard; four things differ: where image-garden comes from, KVM, where pkgproxy runs, and what the next job finds

spread-gitlab-ci on GitLab and image-garden-action on GitHub run the same spread.yaml, with the same allocate and discard. They differ in four ways: where image-garden comes from, KVM, pkgproxy and what persists.

Say one spread suite runs on ubuntu-cloud-24.04 at home, and you want it in CI on GitLab and on GitHub. On GitLab you include spread-gitlab-ci; on GitHub you add image-garden-action to a workflow. Underneath both, spread runs your garden backend, which calls the same allocate and discard, and your spread.yaml doesn't change. Around that, nearly everything differs: where image-garden comes from, how the job gets KVM, where pkgproxy runs, and what's still there for the next job. This deck sets the two side by side, one question at a time.

Details and sources

Details

  • spread-gitlab-ci runs spread with the backend input, garden by default (spread.yml:27-31); image-garden-action runs spread with spread-backend, also garden by default, and the required garden-system (action.yaml:42-47).
  • Neither defines the backend: it is the type: adhoc backend of your spread.yaml, whose hooks run image-garden allocate --spread and image-garden discard (README.md:740-751).
  • The craft components on GitLab call allocate and discard themselves, in the same container; what this deck says about the GitLab side holds for them too.
  • Running one suite on both is an example; the projects do not require it.

02 Where image-garden comes from

WHERE IMAGE-GARDEN COMES FROM GitLab GitHub the job's container image FROM …/zygoon/image-garden:TAG QEMU and image-garden's tools + spread, from the component the job installs nothing a fresh Ubuntu runner snapd the pkgproxy snap image-garden snap, latest/stable, with spread installed in every job the image you pinned the latest release, unless you name a channel
On GitLab the job runs in the component's image, built FROM the image-garden container at a fixed tag with QEMU and its tools inside and spread added, and installs nothing; on GitHub the action installs snapd, the pkgproxy snap and the image-garden snap from latest/stable on a fresh Ubuntu runner in every job, and spread comes from the snap; GitLab gives the pinned image, GitHub the latest release unless a channel is named

GitLab: an image built FROM the image-garden container at a tag, with spread added; nothing is installed in the job. GitHub: snapd, pkgproxy and the image-garden snap installed in every job, from latest/stable unless a channel input says otherwise.

On GitLab, the job runs inside a container image, and the component's image is built from image-garden's own container, at a fixed tag. QEMU and every tool image-garden needs are already inside, and spread-gitlab-ci adds spread on top. The job installs nothing. On GitHub, the job gets a whole Ubuntu virtual machine instead, so the action installs things on it, in every job: snapd, the pkgproxy snap, and the image-garden snap from the stable channel. spread comes from the snap. So GitLab gives you exactly the image you pinned, and GitHub gives you the latest release, unless you name a channel.

Details and sources

Details

  • spread-gitlab-ci's image is FROM registry.gitlab.com/zygoon/image-garden:v0.7.0 with checksum-pinned spread, spread-plus and spread-gitlab-report added (spread-gitlab-ci Dockerfile:20-52); the component's default image is its own image of the same version (spread.yml:18-22). The craft components build on v0.7.2.
  • Known discrepancy: spread-gitlab-ci builds on an older image-garden. Its base is v0.7.0, older than the craft components' v0.7.2; none of the four uses v0.8.0 yet.
  • The action installs or refreshes snapd from snapd-channel, installs the pkgproxy snap from pkgproxy-channel and the image-garden snap, in devmode, from image-garden-channel, all latest/stable by default, and aliases image-garden.spread as spread (action.yaml:9-17, action.yaml:209-232, action.yaml:366-397).
  • What the container carries: running image-garden in a container; the snap: the image-garden snap.

03 Getting KVM

KVM GitLab GitHub hosted runners: no /dev/kvm [runners.docker] devices = ["/dev/kvm"] tags: [x86_64, kvm] craft components: no KVM, no build sudo chmod -v 666 /dev/kvm when the runner has the device no device: a notice, then emulation much more slowly fast: a guest of the host's own architecture
GitLab's hosted runners give no /dev/kvm, so you run your own runners with devices = /dev/kvm in config.toml and pick them with a tag, and the craft components refuse to start without it; on GitHub the action chmods /dev/kvm when present and otherwise continues emulated, much slower; either way the host's own architecture is the fast guest

GitLab: your own runners with devices = ["/dev/kvm"], picked with tags. GitHub: the action runs chmod 666 /dev/kvm when the device exists, and otherwise continues, emulated.

Next, the KVM device. GitLab's hosted runners don't give jobs one. So on GitLab you run runners of your own, pass /dev/kvm into the job container in the runner's config.toml, and pick those runners with a tag. The craft components refuse to start without it. On GitHub, the action checks the runner for the device and opens it to every user. Where it's missing, the action only notes that it carries on without hardware acceleration, and every guest runs emulated, much more slowly. Either way, a guest of the host's own architecture is the fast one.

Details and sources

Details

  • "GitLab.com's hosted runners do not give jobs /dev/kvm, so run image-garden jobs on runners of your own that pass the device into the job's container (devices = ["/dev/kvm"] in the docker executor's config.toml), and pick them with tags" (README.md:778-787).
  • The craft scripts stop with "cannot start a VM without /dev/kvm" (snapcraft-vm:319-321); spread-gitlab-ci has no such check and runs emulated.
  • The action: sudo chmod -v 666 /dev/kvm when it exists, otherwise the notice "/dev/kvm is not available, continuing without hardware acceleration" (action.yaml:153-160); without it, tests are "much, much slower" (README.md:367-373).
  • KVM only speeds up guests of the host's architecture: KVM in CI, six architectures.

04 Where pkgproxy runs

PKGPROXY GitLab GitHub a pkgproxy machine, always on PKGPROXY_URL=http://pkgproxy.lan:3142 the runner's environment guest → its address, via QEMU's network the runner the pkgproxy snap, beside image-garden guestfwd 10.0.2.100:3142 → socket a fixed address in each guest pkgproxy=http://HOST-IP:3142 something already on port 3142, such as setup-pkgproxy: used by its URL
On GitLab pkgproxy is a long-lived machine next to the runners, named by PKGPROXY_URL in the runner's environment, and each guest connects to its address through QEMU's user network; on GitHub pkgproxy is a snap on the runner, connected to image-garden's sockets plug, and QEMU forwards a fixed guest address to its socket; with a pkgproxy already on port 3142 the action uses it by URL

GitLab: a long-lived pkgproxy named by PKGPROXY_URL, reached by its address. GitHub: the pkgproxy snap on the runner, pkgproxy=snap, reached through QEMU's guestfwd to its socket; or an existing one on port 3142 by URL.

pkgproxy sits in different places too. On GitLab, it's a machine of yours next to the runners, running all the time. The runner's environment names it with PKGPROXY_URL, and each guest connects to its address through QEMU's user network. On GitHub, pkgproxy is a snap on the runner itself, installed by the action. image-garden is connected to its sockets, and QEMU forwards a fixed address in each guest straight to pkgproxy's socket on the host. If something already listens on pkgproxy's port, as when setup-pkgproxy ran first, the action uses that one, by its URL, instead.

Details and sources

Details

  • With PKGPROXY_URL set, image-garden resolves its host to IPv4 and the guest connects there; otherwise, with the sockets of the bundled pkgproxy or of a pkgproxy snap, QEMU gets guestfwd=tcp:10.0.2.100:3142-unix:…/pkgproxy.sock (010-qemu.mk:20-44).
  • The action uses a pkgproxy that already listens on port 3142 ("external" mode), otherwise installs the snap ("snap" mode) (action.yaml:173-188); it connects image-garden:pkgproxy-sockets and sets pkgproxy=snap, or sets pkgproxy=http://HOST-IP:3142 (action.yaml:376-393). setup-pkgproxy must come before the action (README.md:229-239).
  • On GitLab the craft components take the URL from pkgproxy-url or the runner's PKGPROXY_URL: the craft components' pkgproxy.

05 What the next job finds

THE NEXT JOB GitLab GitHub the job container: thrown away craft: scratch directory deleted spread-gitlab-ci: only its rerun list pkgproxy's cache stays on its machine the runner: gone, pkgproxy too GitHub cache: images · packages packed at the end, unpacked by the next job prepared images: opt-in both ways: keep the downloads, redo the rest
On GitLab the job container is thrown away, the craft scripts delete their scratch directory and spread-gitlab-ci keeps only its rerun list, while pkgproxy's cache stays on its own machine; on GitHub the whole runner is gone, so the action packs pkgproxy's cache into two GitHub caches that the next job unpacks, with prepared images opt-in; both keep the downloads and redo the rest

GitLab: jobs keep nothing but spread's rerun list; pkgproxy's cache stays on its own machine. GitHub: the action saves pkgproxy's cache in two GitHub caches and restores it; prepared images are opt-in.

And when the job ends? On GitLab, the job container is thrown away, and the components keep nothing: the craft scripts delete their scratch directory, and spread-gitlab-ci keeps only its list of tasks to rerun. The memory lives in pkgproxy, whose cache simply stays on its machine. On GitHub, the whole runner is gone, pkgproxy included. So the action packs pkgproxy's cache into the GitHub cache, images and packages apart, and the next job unpacks it. Prepared images can be cached too, if you opt in. Same idea both ways: keep the downloads, and redo the rest.

Details and sources

Details

06 Where to go from here

ROADS GitLab vs GitHub spread-gitlab-ci the GitLab side KVM in CI your runners GitHub action the GitHub side GitHub caching pkgproxy's cache saved Craft and pkgproxy the GitLab address rules pkgproxy= the snap's setting
Six roads from GitLab versus GitHub: spread-gitlab-ci, KVM in CI, the GitHub action, how the action caches, the craft components' pkgproxy, and the snap's pkgproxy setting

Next: spread-gitlab-ci, KVM in CI, the GitHub action, its caches, the craft components' pkgproxy and the snap's pkgproxy setting.

So on GitLab you pin an image, bring your own KVM runners and run one pkgproxy that remembers everything. On GitHub the action builds the same setup on a fresh runner every time and carries pkgproxy's cache along. For the GitLab side in full, take spread-gitlab-ci, and KVM in CI for the runners. For the GitHub side, the action's deck, and its caching deck. The craft components' pkgproxy deck shows the GitLab address rules, and the snap's pkgproxy setting explains the GitHub one.

Details and sources

Read more

Basis of the statements on this page

Read in spread-gitlab-ci at 3d522d4, the craft components at the commits in integrations.commits, image-garden-action at 2891c7a and image-garden v0.8.0 ("code"). Nothing was run for this page.

Side by side

QuestionGitLab (spread-gitlab-ci, craft components)GitHub (image-garden-action)
Where the job runsin a container image built FROM the image-garden containeron a fresh Ubuntu runner
image-gardenthe container at a fixed tag (v0.7.0 or v0.7.2)the snap, image-garden-channel, latest/stable by default, installed every job
spreadadded to the image by spread-gitlab-cithe snap's spread component, aliased as spread
KVMyour own runners, devices = ["/dev/kvm"], picked with tagschmod 666 /dev/kvm when present; otherwise emulated
pkgproxya long-lived pkgproxy, PKGPROXY_URL (and GARDEN_DL_PKGPROXY) in the runner's environmentthe pkgproxy snap on the runner, pkgproxy=snap; or an existing one on port 3142 by URL
How a guest reaches itthe URL's host, resolved to IPv4, through QEMU's user networkguestfwd from 10.0.2.100:3142 to pkgproxy's socket
What persistspkgproxy's own disk; spread-gitlab-ci's rerun listpkgproxy's cache in two GitHub caches; prepared images opt-in
A new image-garden releasewhen the component's image moves to itin the next job

Sources: the image-garden README's CI section, 010-qemu.mk:20-44, spread-gitlab-ci Dockerfile:20-52, image-garden-action action.yaml.

Roads from here

BUILT ON IT · CI integrations

SNAP · snap-only features

CONTAINER · OCI image