Six architectures, six QEMU machines
a QEMU machine for each for users for contributors
Each architecture has its own QEMU machine and CPU: q35 and max on x86_64, virt on aarch64, riscv64 and loongarch64, s390-ccw-virtio on s390x, pseries with POWER9 on ppc64. The .run script uses KVM when QEMU offers it and TCG emulation otherwise, and the host's own architecture gets four CPUs, the others one.
On the map: architectures
▶ Watch this deck — 7 scenes, ~5 min
Worth seeing first
- The support matrix — which systems each architecture has
01 One machine per architecture
Each architecture has its own settings: the QEMU program, machine, CPU and default CPU count that go into the .run script.
An Arm guest can't boot on a PC's virtual motherboard, and an s390x guest needs a mainframe's. So for each of its six architectures, image-garden carries a small makefile that names the QEMU program, the machine type, the CPU model and the default number of CPUs, and writes them into every system's .run script. Build ubuntu-cloud-24.04.x86_64 and you get QEMU's x86 emulator on a q35 machine. This deck shows each architecture's choices, when a guest runs at full speed and when it's emulated, and how many CPUs it gets.
Details and sources
Details
- Each
mk/010-qemu-ARCH.mkstarts with the program (QEMU_SYSTEM_X86_64, in the snap the snap'sqemu-system-x86-64app, elseqemu-system-x86_64fromPATH), the machine, the CPU and the CPU count (mk/010-qemu-x86_64.mk:3-7, mk/010-qemu-aarch64.mk:4-6, mk/010-qemu-s390x.mk:4-6). - They are joined into
image-garden.mkwith the othermk/*.mkfiles (Makefile:48-50), and the run and provisioning rules of each system call that architecture's command line (rulegen.py:593-724). - The one-line command on the slide is a shortened view: the
.runscript builds the command as an array, with storage, firmware, RNG and network options too (mk/010-qemu-x86_64.mk:59-108).
Read more
02 Six machines, six CPUs
x86_64 q35, aarch64 virt,gic-version=max, riscv64 virt,acpi=off, loongarch64 virt, all with -cpu max; s390x s390-ccw-virtio with qemu, ppc64 pseries with power9.
Here are the six. x86_64 runs on q35, a modern PC chipset, with the CPU model called max: every feature QEMU can offer. aarch64 uses the virt board with the newest interrupt controller it supports, riscv64 the virt board with ACPI turned off, and loongarch64 the virt board too, all three with CPU max. s390x gets the s390-ccw-virtio machine with the CPU model called qemu, and ppc64 a pseries machine with a POWER9 processor. One family changes this: Amazon Linux on aarch64 asks for a Neoverse N1 instead of max.
Details and sources
Details
| Architecture | Machine | CPU | Source |
|---|---|---|---|
| x86_64 | q35 (q35,smm=on for the +tpm systems) | max | mk/010-qemu-x86_64.mk:5-6 |
| aarch64 | virt,gic-version=max | max | mk/010-qemu-aarch64.mk:4-5 |
| riscv64 | virt,acpi=off | max | mk/010-qemu-riscv64.mk:4-5 |
| loongarch64 | virt | max | mk/010-qemu-loongarch64.mk:4-5 |
| s390x | s390-ccw-virtio | qemu | mk/010-qemu-s390x.mk:4-5 |
| ppc64 | pseries | power9 | mk/010-qemu-ppc64.mk:4-5 |
- Amazon Linux 2 and 2023 on aarch64 set
QEMU_CPU_AARCH64=neoverse-n1for their.qcow2and.runtargets (mk/050-amazonlinux-2.mk:10-11). - The README suggests
QEMU_CPU_X86_64=Haswellor newer for Rocky 10 and other RHEL 10 derivatives wheremaxis not enough (README.md:677-681). - What
max,gic-version=maxandacpi=offmean is QEMU's documentation (derived).
Read more
03 KVM when it can, TCG when it must
The .run script checks -accel help when it starts and uses accel=kvm:tcg or accel=tcg; QEMU_ACCEL_MACHINE_PROP overrides it. A /dev/kvm you cannot write gets a warning.
Speed comes down to one question: can the host's processor run the guest's code itself? When it can, QEMU uses KVM, the kernel's hardware virtualisation. Each .run script asks its QEMU every time it starts, with -accel help. If KVM is in the list, it asks for kvm:tcg: KVM first, with emulation as the fallback. If not, it's tcg alone. You can set the choice yourself with QEMU_ACCEL_MACHINE_PROP. And when /dev/kvm exists but you can't write to it, image-garden prints a large warning, with the command that adds you to the kvm group.
Details and sources
Details
- The
.runscript computesdefault_qemu_accel_machine_propwhen it runs:,accel=kvm:tcgwhenQEMU -accel helplistskvm, else,accel=tcg(mk/010-qemu-x86_64.mk:63);QEMU_ACCEL_MACHINE_PROPfrom the environment wins (mk/010-qemu-x86_64.mk:72) and is appended to-machine(mk/010-qemu-x86_64.mk:96). The provisioning boot uses the same test (mk/010-qemu-x86_64.mk:17). Every architecture's file does the same. kvm:tcgtells QEMU to try KVM and fall back to TCG (derived from QEMU's documentation).image-garden makeandallocaterunkvm_check: when/dev/kvmexists but is not writable, it prints a banner, thensudo chmod 666 /dev/kvmas advice in CI (CIset) orsudo usermod -aG kvm $LOGNAMEelsewhere (image-garden:669-709).- The README and the man page document
QEMU_ACCEL_MACHINE_PROPwith the same default (README.md:943-945, image-garden.1:670-677).
Read more
04 Emulation works, slowly
Without KVM, as for every foreign architecture, QEMU emulates the CPU with TCG, many times slower. Builds and tests of such systems need longer timeouts.
KVM helps only when the guest and the host share an architecture. Build an aarch64 system on a PC and there's no hardware to lend: QEMU translates every guest instruction itself, with TCG, its tiny code generator. That's what lets one PC boot all six architectures. But it's many times slower, and the README says so: cross-architecture emulation works, but it's slow, and not for routine use. The same happens when KVM is missing altogether, as on hosted CI runners without /dev/kvm. So give emulated systems time, and run them where time is cheap.
Details and sources
Details
- The README on
make all: "Cross-architecture emulation works but is slow and not recommended for routine use" (README.md:529-534). - The README on CI: without
/dev/kvm, QEMU emulates the CPU (TCG), many times slower; GitLab.com's hosted runners do not give jobs/dev/kvm(README.md:778-785). - The README's host tiers: x86_64 tier 1, aarch64 tier 2, riscv64, s390x and ppc64le hosts tier 3, "Native where KVM is available" and "May timeout in CI"; guests of another architecture are emulated (README.md:384-396).
- A QEMU offers KVM only for its own architecture on a matching host, so a foreign architecture always runs under TCG (derived).
05 Four CPUs for your own architecture
The host's architecture gets -smp 4, every other one -smp 1, written into the .run script. QEMU_SMP_OPTION="-smp 2" overrides it for one run.
How many CPUs does a guest get? That depends on your host, not on the guest. Each architecture's makefile sets one, and then the common QEMU makefile raises your host's own architecture to four, because native cores are cheaper than emulated ones. So on a PC, ubuntu-cloud-24.04.x86_64 gets four CPUs and an aarch64 system gets one; on an Arm host it's the other way round. The number is written into the .run script when make creates it. To change it for one run, set QEMU_SMP_OPTION, for example to -smp 2.
Details and sources
Details
- Every per-architecture file sets
QEMU_SMP_<ARCH> = 1, such as mk/010-qemu-x86_64.mk:7;mk/010-qemu.mksetsQEMU_SMP_$(shell echo $(GARDEN_HOST_ARCH) | tr a-z- A-Z_) = 4, with the comment "Try to give the native arch some more cores as those are usually cheaper than tcg" (mk/010-qemu.mk:5-6). - The
.runscript recordsdefault_qemu_smp_option="-smp N"and usesQEMU_SMP_OPTIONfrom the environment when set (mk/010-qemu-x86_64.mk:65, :74). A.runscript made on one host keeps that host's count when copied elsewhere (derived). GARDEN_HOST_ARCHisuname -mwithppc64lenamedppc64, so a POWER host raisesQEMU_SMP_PPC64(mk/000-garden.mk:14-17).- The README documents all six,
QEMU_SMP_X86_64,_AARCH64,_RISCV64,_LOONGARCH64,_S390Xand_PPC64, as "4 native, 1 cross-arch" (README.md:890-895), and its host tier table gives every host 4 vCPUs for its own architecture (README.md:389-396). - allocate's memory check counts memory only, not CPUs; spread's backend can pass
QEMU_SMP_OPTION(the adhoc backend).
Read more
06 How the default is set for contributors
mk/010-qemu-ARCH.mk sets QEMU_SMP_ARCH = 1; mk/010-qemu.mk, concatenated after them, sets the host's to 4, named from uname -m with ppc64le as ppc64. .image-garden.mk and the command line can override it.
Here's how that works in make. The six per-architecture makefiles each set their CPU count to one, with a plain assignment, such as QEMU_SMP_X86_64 equals one. The common QEMU makefile sorts after them in image-garden.mk, because a dash sorts before a dot. It assigns four to the variable named after your host's architecture, upper-cased. The last assignment wins, and the value is read late, when the .run recipe runs. A project's .image-garden.mk comes later still, and the make command line beats them all. The host's architecture is uname -m, except that ppc64le counts as ppc64, so a POWER host gives its ppc64 guests four too.
Details and sources
Details
image-garden.mkisdata/000-systems.mk, then$(sort ...)ofmk/*.mk, then the generated files (Makefile:48-50). Byte order putsmk/010-qemu-aarch64.mkand its siblings beforemk/010-qemu.mk, as-(0x2d) sorts before.(0x2e).QEMU_SMP_X86_64 = 1(mk/010-qemu-x86_64.mk:7) is a recursive assignment; the later= 4replaces it (mk/010-qemu.mk:6). The.runrecipe expands$(QEMU_SMP_X86_64)when it runs (mk/010-qemu-x86_64.mk:65), after every file is read.mk/999-local.mkincludes the project's.image-garden.mk(mk/999-local.mk:4-5); a variable given on the make command line overrides assignments in makefiles (GNU make, derived).image-garden makepasses its arguments to make (image-garden:982-993).- The variable is named after
GARDEN_HOST_ARCH, which isuname -mwithppc64lenamedppc64(mk/000-garden.mk:14-17). On a POWER host the variable set to 4 isQEMU_SMP_PPC64, the one the ppc64.runrules read; the same name makesGARDEN_ARCHdefault toppc64, so those systems joinall::too.
07 Where to go from here
Next: changing a .run script's defaults, runners without KVM, the support matrix, boot types, firmware and loongarch64.
So each architecture is a machine, a CPU and a CPU count, with KVM where the host allows it. From here, the run script deck shows every variable you can change before a boot, memory and CPUs included. The container in CI deck covers runners without KVM. The support matrix shows which systems each architecture has, the boot deck what efi, bios and platform change, and the firmware deck what UEFI each machine loads. loongarch64 has a deck of its own.
Details and sources
Read more
Basis of the statements on this page
Read in mk/010-qemu-*.mk, mk/010-qemu.mk and mk/050-amazonlinux-2.mk of image-garden at the tag in garden.commit ("code"). What the accelerator and CPU names do inside QEMU, and how the defaults play out on a given host, are derived from those rules and QEMU's documentation. Nothing was run for this page.
Per architecture
| Architecture | QEMU program | Machine | CPU | CPUs on this host / elsewhere | Boot types | Snap component |
|---|---|---|---|---|---|---|
| x86_64 | qemu-system-x86_64 | q35 | max | 4 / 1 | efi, bios | qemu-x86-64 |
| aarch64 | qemu-system-aarch64 | virt,gic-version=max | max (neoverse-n1 for Amazon Linux) | 4 / 1 | efi | qemu-aarch64 |
| riscv64 | qemu-system-riscv64 | virt,acpi=off | max | 4 / 1 | efi | qemu-riscv64 |
| loongarch64 | qemu-system-loongarch64 | virt | max | 4 / 1 | efi | qemu-loongarch64 |
| s390x | qemu-system-s390x | s390-ccw-virtio | qemu | 4 / 1 | platform | qemu-s390x |
| ppc64 | qemu-system-ppc64 | pseries | power9 | 4 / 1 (a ppc64le host counts as ppc64) | platform | qemu-ppc64 |
Every architecture: -m 2048, -nographic, a virtio-blk disk, -device virtio-rng-pci, and -netdev user with a virtio-net-pci card.
The knobs
| Knob | Kind | Effect |
|---|---|---|
QEMU_ACCEL_MACHINE_PROP | environment, at run time | replaces ,accel=kvm:tcg or ,accel=tcg |
QEMU_SMP_OPTION | environment, at run time | replaces -smp N, such as QEMU_SMP_OPTION="-smp 2" |
QEMU_SMP_X86_64, _AARCH64, _RISCV64, _S390X, _PPC64, _LOONGARCH64 | make variable | the N written into new .run scripts |
QEMU_CPU_X86_64 and siblings | make variable | the -cpu model |
Make variables go on the command line (image-garden make QEMU_SMP_X86_64=2 SYSTEM.run) or in .image-garden.mk; a .run script that exists is not remade for a new value until a makefile it depends on changes. See the .run script.
Roads from here
SYSTEMS · distributions
- The support matrix — which systems each architecture has
ARCHITECTURES · machines and firmware
- Boot types — what efi, bios and platform change in the rules
- UEFI firmware per system — the UEFI firmware each machine loads
- loongarch64 and OpenCloudOS — the loongarch64 machine and its one system
- x86_64 — everything about x86_64
- aarch64 — everything about aarch64
- riscv64 — everything about riscv64
- s390x — everything about s390x, big-endian
- ppc64 — everything about ppc64el and POWER hosts
RUNTIME · make rules and the CLI
- The .run script — change memory, CPUs and the accelerator before a boot
CONTAINER · OCI image
- Running the container in CI — running on CI runners, with or without /dev/kvm