Six architectures, six QEMU machines

a QEMU machine for each for users for contributors

Each architecture has its own QEMU machine and CPU: q35 and max on x86_64, virt on aarch64, riscv64 and loongarch64, s390-ccw-virtio on s390x, pseries with POWER9 on ppc64. The .run script uses KVM when QEMU offers it and TCG emulation otherwise, and the host's own architecture gets four CPUs, the others one.

On the map: architectures

▶ Watch this deck — 7 scenes, ~5 min

Worth seeing first

01 One machine per architecture

ONE MACHINE PER ARCHITECTURE a guest needs a board of its own architecture x86_64 aarch64 riscv64 s390x ppc64 loongarch64 QEMU_SYSTEM_X86_64 ?= qemu-system-x86_64 QEMU_SYSTEM_X86_64_MACHINE = q35 QEMU_CPU_X86_64 ?= max QEMU_SMP_X86_64 = 1 each .run script ubuntu-cloud-24.04.x86_64.run, in short qemu-system-x86_64 -cpu max -machine q35,accel=… -m 2048 -smp … "$@"
Six sets of settings, one per architecture, each name the QEMU program, the machine, the CPU model and the CPU count; the x86_64 one names qemu-system-x86_64, q35, max and 1, and they go into each system's .run script, so ubuntu-cloud-24.04.x86_64.run runs qemu-system-x86_64 on a q35 machine

Each architecture has its own settings: the QEMU program, machine, CPU and default CPU count that go into the .run script.

An Arm guest can't boot on a PC's virtual motherboard, and an s390x guest needs a mainframe's. So for each of its six architectures, image-garden carries a small makefile that names the QEMU program, the machine type, the CPU model and the default number of CPUs, and writes them into every system's .run script. Build ubuntu-cloud-24.04.x86_64 and you get QEMU's x86 emulator on a q35 machine. This deck shows each architecture's choices, when a guest runs at full speed and when it's emulated, and how many CPUs it gets.

Details and sources

Details

02 Six machines, six CPUs

SIX MACHINES, SIX CPUS -machine -cpu x86_64 q35 max aarch64 virt,gic-version=max max riscv64 virt,acpi=off max loongarch64 virt max s390x s390-ccw-virtio qemu ppc64 pseries power9 amazonlinux-cloud-2 and -2023 on aarch64: -cpu neoverse-n1
Machine and CPU per architecture: x86_64 q35 with max; aarch64 virt,gic-version=max with max; riscv64 virt,acpi=off with max; loongarch64 virt with max; s390x s390-ccw-virtio with the qemu CPU; ppc64 pseries with power9. Amazon Linux on aarch64 uses neoverse-n1

x86_64 q35, aarch64 virt,gic-version=max, riscv64 virt,acpi=off, loongarch64 virt, all with -cpu max; s390x s390-ccw-virtio with qemu, ppc64 pseries with power9.

Here are the six. x86_64 runs on q35, a modern PC chipset, with the CPU model called max: every feature QEMU can offer. aarch64 uses the virt board with the newest interrupt controller it supports, riscv64 the virt board with ACPI turned off, and loongarch64 the virt board too, all three with CPU max. s390x gets the s390-ccw-virtio machine with the CPU model called qemu, and ppc64 a pseries machine with a POWER9 processor. One family changes this: Amazon Linux on aarch64 asks for a Neoverse N1 instead of max.

Details and sources

Details

ArchitectureMachineCPUSource
x86_64q35 (q35,smm=on for the +tpm systems)maxmk/010-qemu-x86_64.mk:5-6
aarch64virt,gic-version=maxmaxmk/010-qemu-aarch64.mk:4-5
riscv64virt,acpi=offmaxmk/010-qemu-riscv64.mk:4-5
loongarch64virtmaxmk/010-qemu-loongarch64.mk:4-5
s390xs390-ccw-virtioqemumk/010-qemu-s390x.mk:4-5
ppc64pseriespower9mk/010-qemu-ppc64.mk:4-5
  • Amazon Linux 2 and 2023 on aarch64 set QEMU_CPU_AARCH64=neoverse-n1 for their .qcow2 and .run targets (mk/050-amazonlinux-2.mk:10-11).
  • The README suggests QEMU_CPU_X86_64=Haswell or newer for Rocky 10 and other RHEL 10 derivatives where max is not enough (README.md:677-681).
  • What max, gic-version=max and acpi=off mean is QEMU's documentation (derived).

03 KVM when it can, TCG when it must

KVM OR TCG can the host's processor run the guest's code itself? .run starts $ qemu-system-x86_64 -accel help kvm listed: accel=kvm:tcg KVM, emulation as fallback otherwise: accel=tcg emulation only QEMU_ACCEL_MACHINE_PROP=,accel=tcg ./SYSTEM.run your choice wins WARNING … because /dev/kvm exists, but you do not have write access to the device. $ sudo usermod -aG kvm $LOGNAME
Each time a .run script starts it runs its QEMU with -accel help; if kvm is listed it uses accel=kvm:tcg, KVM with emulation as the fallback, otherwise accel=tcg. QEMU_ACCEL_MACHINE_PROP overrides the choice. When /dev/kvm exists but is not writable, image-garden warns and suggests sudo usermod -aG kvm $LOGNAME

The .run script checks -accel help when it starts and uses accel=kvm:tcg or accel=tcg; QEMU_ACCEL_MACHINE_PROP overrides it. A /dev/kvm you cannot write gets a warning.

Speed comes down to one question: can the host's processor run the guest's code itself? When it can, QEMU uses KVM, the kernel's hardware virtualisation. Each .run script asks its QEMU every time it starts, with -accel help. If KVM is in the list, it asks for kvm:tcg: KVM first, with emulation as the fallback. If not, it's tcg alone. You can set the choice yourself with QEMU_ACCEL_MACHINE_PROP. And when /dev/kvm exists but you can't write to it, image-garden prints a large warning, with the command that adds you to the kvm group.

Details and sources

Details

  • The .run script computes default_qemu_accel_machine_prop when it runs: ,accel=kvm:tcg when QEMU -accel help lists kvm, else ,accel=tcg (mk/010-qemu-x86_64.mk:63); QEMU_ACCEL_MACHINE_PROP from the environment wins (mk/010-qemu-x86_64.mk:72) and is appended to -machine (mk/010-qemu-x86_64.mk:96). The provisioning boot uses the same test (mk/010-qemu-x86_64.mk:17). Every architecture's file does the same.
  • kvm:tcg tells QEMU to try KVM and fall back to TCG (derived from QEMU's documentation).
  • image-garden make and allocate run kvm_check: when /dev/kvm exists but is not writable, it prints a banner, then sudo chmod 666 /dev/kvm as advice in CI (CI set) or sudo usermod -aG kvm $LOGNAME elsewhere (image-garden:669-709).
  • The README and the man page document QEMU_ACCEL_MACHINE_PROP with the same default (README.md:943-945, image-garden.1:670-677).

04 Emulation works, slowly

EMULATION IS SLOW a PC, x86_64 x86_64 guest KVM the CPU aarch64 guest TCG translates every instruction many times slower README: cross-architecture emulation works but is slow, not for routine use a CI runner without /dev/kvm: all TCG give emulated systems time, and run them where time is cheap
On a PC, an x86_64 guest runs on the CPU through KVM while an aarch64 guest is translated instruction by instruction by TCG, many times slower; the README calls cross-architecture emulation slow and not for routine use, and hosted CI runners without /dev/kvm emulate everything. Give emulated systems time

Without KVM, as for every foreign architecture, QEMU emulates the CPU with TCG, many times slower. Builds and tests of such systems need longer timeouts.

KVM helps only when the guest and the host share an architecture. Build an aarch64 system on a PC and there's no hardware to lend: QEMU translates every guest instruction itself, with TCG, its tiny code generator. That's what lets one PC boot all six architectures. But it's many times slower, and the README says so: cross-architecture emulation works, but it's slow, and not for routine use. The same happens when KVM is missing altogether, as on hosted CI runners without /dev/kvm. So give emulated systems time, and run them where time is cheap.

Details and sources

Details

  • The README on make all: "Cross-architecture emulation works but is slow and not recommended for routine use" (README.md:529-534).
  • The README on CI: without /dev/kvm, QEMU emulates the CPU (TCG), many times slower; GitLab.com's hosted runners do not give jobs /dev/kvm (README.md:778-785).
  • The README's host tiers: x86_64 tier 1, aarch64 tier 2, riscv64, s390x and ppc64le hosts tier 3, "Native where KVM is available" and "May timeout in CI"; guests of another architecture are emulated (README.md:384-396).
  • A QEMU offers KVM only for its own architecture on a matching host, so a foreign architecture always runs under TCG (derived).

05 Four CPUs for your own architecture

CPUS PER GUEST the count depends on the host, not on the guest QEMU_SMP_X86_64 = 1, QEMU_SMP_AARCH64 = 1 … QEMU_SMP_$(GARDEN_HOST_ARCH) = 4 on a PC ubuntu-cloud-24.04.x86_64 -smp 4 an aarch64 system -smp 1 on an Arm host an x86_64 system -smp 1 an aarch64 system -smp 4 default_qemu_smp_option="-smp 4" written into the .run script by make $ QEMU_SMP_OPTION="-smp 2" ./ubuntu-cloud-24.04.x86_64.run
Each architecture's makefile sets one CPU, and the host's own architecture is raised to four: on a PC ubuntu-cloud-24.04.x86_64 gets -smp 4 and an aarch64 system -smp 1, on an Arm host the other way round. The value is written into the .run script; QEMU_SMP_OPTION="-smp 2" overrides it for one run

The host's architecture gets -smp 4, every other one -smp 1, written into the .run script. QEMU_SMP_OPTION="-smp 2" overrides it for one run.

How many CPUs does a guest get? That depends on your host, not on the guest. Each architecture's makefile sets one, and then the common QEMU makefile raises your host's own architecture to four, because native cores are cheaper than emulated ones. So on a PC, ubuntu-cloud-24.04.x86_64 gets four CPUs and an aarch64 system gets one; on an Arm host it's the other way round. The number is written into the .run script when make creates it. To change it for one run, set QEMU_SMP_OPTION, for example to -smp 2.

Details and sources

Details

  • Every per-architecture file sets QEMU_SMP_<ARCH> = 1, such as mk/010-qemu-x86_64.mk:7; mk/010-qemu.mk sets QEMU_SMP_$(shell echo $(GARDEN_HOST_ARCH) | tr a-z- A-Z_) = 4, with the comment "Try to give the native arch some more cores as those are usually cheaper than tcg" (mk/010-qemu.mk:5-6).
  • The .run script records default_qemu_smp_option="-smp N" and uses QEMU_SMP_OPTION from the environment when set (mk/010-qemu-x86_64.mk:65, :74). A .run script made on one host keeps that host's count when copied elsewhere (derived).
  • GARDEN_HOST_ARCH is uname -m with ppc64le named ppc64, so a POWER host raises QEMU_SMP_PPC64 (mk/000-garden.mk:14-17).
  • The README documents all six, QEMU_SMP_X86_64, _AARCH64, _RISCV64, _LOONGARCH64, _S390X and _PPC64, as "4 native, 1 cross-arch" (README.md:890-895), and its host tier table gives every host 4 vCPUs for its own architecture (README.md:389-396).
  • allocate's memory check counts memory only, not CPUs; spread's backend can pass QEMU_SMP_OPTION (the adhoc backend).

06 How the default is set for contributors

IN MAKE image-garden.mk, in order mk/010-qemu-x86_64.mk … QEMU_SMP_X86_64 = 1 mk/010-qemu.mk QEMU_SMP_$(shell echo $(GARDEN_HOST_ARCH) | tr a-z- A-Z_) = 4 "-" sorts before ".": the per-arch files come first default_qemu_smp_option="-smp $(QEMU_SMP_X86_64)" expanded when the recipe runs: 4 mk/999-local.mk -include .image-garden.mk $ image-garden make QEMU_SMP_X86_64=2 SYSTEM.run the command line beats all ppc64le host: GARDEN_HOST_ARCH = ppc64, so ppc64 guests get 4
image-garden.mk concatenates the per-architecture makefiles, which set QEMU_SMP_X86_64 = 1 and the like, then mk/010-qemu.mk, which sets the variable of the host architecture to 4, then mk/999-local.mk with the project's .image-garden.mk; the last assignment wins and the .run recipe reads it late; the make command line beats all. GARDEN_HOST_ARCH is uname -m with ppc64le named ppc64, so a POWER host sets QEMU_SMP_PPC64 to 4

mk/010-qemu-ARCH.mk sets QEMU_SMP_ARCH = 1; mk/010-qemu.mk, concatenated after them, sets the host's to 4, named from uname -m with ppc64le as ppc64. .image-garden.mk and the command line can override it.

Here's how that works in make. The six per-architecture makefiles each set their CPU count to one, with a plain assignment, such as QEMU_SMP_X86_64 equals one. The common QEMU makefile sorts after them in image-garden.mk, because a dash sorts before a dot. It assigns four to the variable named after your host's architecture, upper-cased. The last assignment wins, and the value is read late, when the .run recipe runs. A project's .image-garden.mk comes later still, and the make command line beats them all. The host's architecture is uname -m, except that ppc64le counts as ppc64, so a POWER host gives its ppc64 guests four too.

Details and sources

Details

  • image-garden.mk is data/000-systems.mk, then $(sort ...) of mk/*.mk, then the generated files (Makefile:48-50). Byte order puts mk/010-qemu-aarch64.mk and its siblings before mk/010-qemu.mk, as - (0x2d) sorts before . (0x2e).
  • QEMU_SMP_X86_64 = 1 (mk/010-qemu-x86_64.mk:7) is a recursive assignment; the later = 4 replaces it (mk/010-qemu.mk:6). The .run recipe expands $(QEMU_SMP_X86_64) when it runs (mk/010-qemu-x86_64.mk:65), after every file is read.
  • mk/999-local.mk includes the project's .image-garden.mk (mk/999-local.mk:4-5); a variable given on the make command line overrides assignments in makefiles (GNU make, derived). image-garden make passes its arguments to make (image-garden:982-993).
  • The variable is named after GARDEN_HOST_ARCH, which is uname -m with ppc64le named ppc64 (mk/000-garden.mk:14-17). On a POWER host the variable set to 4 is QEMU_SMP_PPC64, the one the ppc64 .run rules read; the same name makes GARDEN_ARCH default to ppc64, so those systems join all:: too.

07 Where to go from here

ROADS Six architectures The .run script memory, CPUs and more The container in CI runners without KVM The support matrix systems per architecture Boot types efi, bios, platform Firmware the UEFI each machine loads loongarch64 OpenCloudOS 9
Roads from six architectures: the .run script's defaults, the container in CI without KVM, the support matrix, boot types, firmware and loongarch64

Next: changing a .run script's defaults, runners without KVM, the support matrix, boot types, firmware and loongarch64.

So each architecture is a machine, a CPU and a CPU count, with KVM where the host allows it. From here, the run script deck shows every variable you can change before a boot, memory and CPUs included. The container in CI deck covers runners without KVM. The support matrix shows which systems each architecture has, the boot deck what efi, bios and platform change, and the firmware deck what UEFI each machine loads. loongarch64 has a deck of its own.

Details and sources

Read more

Basis of the statements on this page

Read in mk/010-qemu-*.mk, mk/010-qemu.mk and mk/050-amazonlinux-2.mk of image-garden at the tag in garden.commit ("code"). What the accelerator and CPU names do inside QEMU, and how the defaults play out on a given host, are derived from those rules and QEMU's documentation. Nothing was run for this page.

Per architecture

ArchitectureQEMU programMachineCPUCPUs on this host / elsewhereBoot typesSnap component
x86_64qemu-system-x86_64q35max4 / 1efi, biosqemu-x86-64
aarch64qemu-system-aarch64virt,gic-version=maxmax (neoverse-n1 for Amazon Linux)4 / 1efiqemu-aarch64
riscv64qemu-system-riscv64virt,acpi=offmax4 / 1efiqemu-riscv64
loongarch64qemu-system-loongarch64virtmax4 / 1efiqemu-loongarch64
s390xqemu-system-s390xs390-ccw-virtioqemu4 / 1platformqemu-s390x
ppc64qemu-system-ppc64pseriespower94 / 1 (a ppc64le host counts as ppc64)platformqemu-ppc64

Every architecture: -m 2048, -nographic, a virtio-blk disk, -device virtio-rng-pci, and -netdev user with a virtio-net-pci card.

The knobs

KnobKindEffect
QEMU_ACCEL_MACHINE_PROPenvironment, at run timereplaces ,accel=kvm:tcg or ,accel=tcg
QEMU_SMP_OPTIONenvironment, at run timereplaces -smp N, such as QEMU_SMP_OPTION="-smp 2"
QEMU_SMP_X86_64, _AARCH64, _RISCV64, _S390X, _PPC64, _LOONGARCH64make variablethe N written into new .run scripts
QEMU_CPU_X86_64 and siblingsmake variablethe -cpu model

Make variables go on the command line (image-garden make QEMU_SMP_X86_64=2 SYSTEM.run) or in .image-garden.mk; a .run script that exists is not remade for a new value until a makefile it depends on changes. See the .run script.

Roads from here

SYSTEMS · distributions

ARCHITECTURES · machines and firmware

RUNTIME · make rules and the CLI

CONTAINER · OCI image