aarch64
virt, EDK2, older firmware for users for contributors
aarch64 guests run on QEMU's virt board with gic-version=max and -cpu max (neoverse-n1 for Amazon Linux), emulated with one vCPU on a PC and under KVM with four on Arm hosts such as a Raspberry Pi 5; 46 cloud and 6 Ubuntu Core systems boot efi, with the host's EDK2 or, for twelve older systems, Fedora's or Ubuntu's 2022 build.
On the map: architectures
▶ Watch this deck — 8 scenes, ~6 min
Worth seeing first
- Six architectures, six QEMU machines — the machine, CPU and accelerator rules every architecture shares
01 Arm guests, on a PC or on Arm
Arm guests are emulated on a PC and run under KVM on an Arm host, which the README ranks tier 2. ubuntu-cloud-24.04.aarch64 uses the arm64 image from the same SHA256SUMS as the x86_64 example.
You meet aarch64 in one of two ways. Either you test the Arm build of your software on a PC, where QEMU emulates every instruction, or you run image-garden on an Arm machine, where Arm guests run at full speed and PC guests are the slow ones. The README ranks an aarch64 host tier two, a feature preview, tested on Raspberry Pi. Take ubuntu-cloud-24.04.aarch64: the same release as the map's running example, from the same checksum list, with the arm64 image instead of amd64. This deck covers its machine, its firmware, the systems, where its QEMU comes from, and the gotchas.
Details and sources
Details
- The README's host tiers: aarch64 is tier 2, "Feature-preview", "Native on ARM hosts", 4 default vCPUs, "Partial support, tested on Raspberry Pi"; guests of another architecture are emulated with 1 vCPU (README.md:384-396).
ubuntu-cloud-24.04.aarch64downloadsnoble-server-cloudimg-arm64.imgand checks it againsthttps://cloud-images.ubuntu.com/noble/current/SHA256SUMS, the list the x86\_64 example uses. Distributions name the architecture arm64 or aarch64; image-garden's system names always say aarch64.- What the architectures share (
-accel help, the vCPU rule) is in six architectures.
for contributors
In the code
- The section
[ubuntu-cloud-24.04.aarch64](data/ubuntu.ini:244-249), beside the x86\_64 one (:237-242).map_to_debian_archturns aarch64 into arm64 where a download name needs it (mk/000-garden.mk:27-30).
Read more
02 The virt board
qemu-system-aarch64 -machine virt,gic-version=max -cpu max, -cpu neoverse-n1 for Amazon Linux; -smp 1 under TCG on a PC, -smp 4 on an aarch64 host; -m 2048.
An Arm guest has no PC chipset to sit on, so QEMU gives it virt, a board made for virtual machines. image-garden asks for gic-version=max: the newest interrupt controller the accelerator supports, as Arm hosts differ there. The CPU is max too, with one family as the exception: Amazon Linux 2 and 2023 get a Neoverse N1, which they need to boot. On a PC, all of this is emulated with TCG, and the guest gets one CPU. On an Arm host with KVM, it runs on your own cores and gets four. Either way, the default memory is two gigabytes.
Details and sources
Details
- Machine
virt,gic-version=max, CPUmax, 1 vCPU unless the host is aarch64, then 4; memory 2048 MiB; networkvirtio-net-pciwith-smbios type=41, as on x86\_64. - Amazon Linux 2 and 2023 on aarch64 use
-cpu neoverse-n1; the change "fixes booting for both … It requires sufficiently new ARM CPU if used natively on the same architecture" (commit 3477e1a). - What
virt,gic-version=maxandmaxmean is QEMU's documentation (derived):gic-version=maxpicks the highest GIC version the accelerator offers, which under KVM depends on the host's interrupt controller. That interrupt controllers differ across Arm hardware is noted in image-garden's CI (.gitlab-ci.yml:635-643).
for contributors
In the code
QEMU_SYSTEM_AARCH64,QEMU_SYSTEM_AARCH64_MACHINE = virt,gic-version=max,QEMU_CPU_AARCH64 = max,QEMU_SMP_AARCH64 = 1and the network option (mk/010-qemu-aarch64.mk:3-10); the target-specificQEMU_CPU_AARCH64=neoverse-n1(mk/050-amazonlinux-2.mk:10-11); the native count and memory (mk/010-qemu.mk:5-12).
Read more
03 Three builds of EDK2
Most aarch64 systems boot the host's EDK2; ten older ones boot Fedora's build and Ubuntu 14.04 and 16.04 Ubuntu's from 22.04, both in the snap's component or downloaded by checksum.
aarch64 boots efi only, with EDK2 built for Arm. Most systems, ubuntu-cloud-24.04.aarch64 among them, boot the host's copy: from Debian and Ubuntu's qemu-efi-aarch64 package, from the snap's aarch64 component, or from a QEMU you built. Ten older systems, such as AlmaLinux 8 and Ubuntu Core 22, stop with a Synchronous Exception under new EDK2, so they boot Fedora's build instead. Ubuntu 14.04 and 16.04 need Ubuntu's EDK2 from 2022. The snap carries both in its component; elsewhere, make downloads them and checks them by checksum. The firmware deck tells the whole story.
Details and sources
Details
- The host's EDK2 is looked for at
/usr/share/AAVMF/AAVMF_CODE.fd(Debian and Ubuntu'sqemu-efi-aarch64, derived from the package) and/usr/share/qemu/aavmf-aarch64-code.bin, then in the snap's component, then under$GARDEN_AARCH64_PREFIX/share/qemu/. - Fedora's build is used by
almalinux-cloud-8,amazonlinux-cloud-2,amazonlinux-cloud-2023,fedora-cloud-38,opensuse-cloud-15.4,opensuse-cloud-15.5,oracle-cloud-8,rocky-cloud-8,ubuntu-core-22andubuntu-core-22+dangerous; Ubuntu's 2022.02 build byubuntu-cloud-14.04andubuntu-cloud-16.04, whose 4.4 kernel stops with "Failed to update FDT and exit boot services" under Fedora's. - Outside the snap, the older builds are downloaded once into the download cache from image-garden's GitLab packages and checked by SHA-256, so the first build of such a system needs the network.
- The full story, with the reasons, is in UEFI firmware per system.
for contributors
In the code
- Efi only: rulegen.py:302-309. Search lists and copies: mk/010-qemu-aarch64.mk:76-112. The Fedora build, its systems and its download: :114-167; Ubuntu's 2022 build: :169-213.
Read more
04 Forty-six systems, and what's missing
46 cloud systems from 12 families and 6 Ubuntu Core systems, all efi. Not on aarch64: archlinux-cloud, oracle-cloud-7, the end-of-life Ubuntu interims, ubuntu-core-16, -18, -20 and the +tpm systems.
Most families you know have an Arm build. image-garden has forty-six aarch64 cloud systems, from twelve families: AlmaLinux, Alpine, Amazon Linux, CentOS, Debian, Fedora, Gentoo, OpenCloudOS, openSUSE, Oracle Linux, Rocky and Ubuntu. Ubuntu Core adds six: 22, 24 and 26, each with a +dangerous twin. What's missing, compared with x86_64? Arch Linux, Oracle Linux 7, the Ubuntu interim releases past their end of life, and Ubuntu Core 16, 18 and 20, which have no Arm image. And the +tpm systems: Secure Boot with a TPM exists only on x86_64. Every aarch64 system boots efi.
Details and sources
Details
- 46 aarch64 cloud systems:
almalinux-cloud-8,-9,-10;alpine-cloud-3;amazonlinux-cloud-2,-2023;centos-cloud-9,-10;debian-cloud-10to-14and-sid;fedora-cloud-38to-45;gentoo-cloud;opencloudos-cloud-9;opensuse-cloud-15.4,-15.5,-15.6,-16.0,-16.1,-tumbleweed;oracle-cloud-8,-9,-10;rocky-cloud-8,-9,-10;ubuntu-cloud-14.04,16.04,18.04,20.04,22.04,24.04,25.04,25.10,26.04,26.10(counted at the pinned commit). - Ubuntu Core on aarch64: 22, 24, 26 and their +dangerous variants (README.md:699-707).
- Asking for a +tpm system on aarch64 stops make with "ubuntu-core with +tpm is only available for x86\_64".
for contributors
In the code
- The sections are in
data/*.ini; Ubuntu Core's aarch64 systems leave out 16, 18 and 20, "no aarch64 image", and the +tpm check is at mk/050-ubuntu-core.mk:4-21.
Read more
05 The snap, the container or your distribution
Snap: snap install image-garden+qemu-aarch64, automatic on aarch64 hosts. Container: images for x86_64 and aarch64 hosts. Ubuntu: qemu-system-arm and qemu-efi-aarch64.
The snap keeps aarch64 in a component called qemu-aarch64. On an Arm host it installs itself. On a PC you add it with snap install image-garden+qemu-aarch64, and until you do, the stand-in for QEMU shows a message with that command. The component carries QEMU, its EDK2 and both older builds. The container is published for aarch64 hosts as well as x86_64 ones, so an Arm machine can run it, with /dev/kvm passed in. On Ubuntu, install the packages qemu-system-arm and qemu-efi-aarch64.
Details and sources
Details
- The snap installs
qemu-aarch64itself on an aarch64 host, at install and refresh; a snap installed from a file installs no component (snap/hooks/install:6-13). The component holdsqemu-system-aarch64,edk2-aarch64-code.fd,edk2-arm-vars.fd,efi-virtio.romand theedk2-fedora/andedk2-jammy/builds (snap/snapcraft.yaml:67-73, :402-403). - Without it, QEMU's stand-in shows "It seems that the component with QEMU aarch64 is not installed" and
snap install image-garden+qemu-aarch64(qemu-stubs/qemu-system-aarch64:9-31). - The container is published for x86\_64 and aarch64 hosts and has QEMU's own EDK2 for aarch64 (.gitlab-ci.yml:1238-1276, Dockerfile:98-102); pass
--device /dev/kvmto use KVM. - On Ubuntu:
qemu-system-arm(which hasqemu-system-aarch64) andqemu-efi-aarch64. Distribution packages may be older than the upstream release (the maintainer's statement).
for contributors
In the code
- image-garden's CI installs those packages (.gitlab-ci.yml:292-311) and builds the container on an aarch64 runner (
docker-arch, .gitlab-ci.yml:1238-1259).
Read more
06 Inside: one makefile and two lists for contributors
mk/010-qemu-aarch64.mk sets QEMU_CPU_AARCH64 = max, overridden per target in mk/050-amazonlinux-2.mk, and the lists GARDEN_AARCH64_FEDORA_EDK2_SYSTEMS and GARDEN_AARCH64_JAMMY_EDK2_SYSTEMS. The aarch64 CI stage has allow_failure: true.
For maintainers, aarch64 is one makefile and two lists. The makefile names virt with the newest interrupt controller, and CPU max with a plain assignment, so the Amazon Linux makefile can set a Neoverse N1 for just its four targets. Two lists, one for Fedora's firmware and one for Ubuntu's, name the systems that boot older EDK2, and the pinned versions and checksums beside them are read by the snap's build and by the CI jobs that publish the firmware. rulegen.py allows efi only. In CI, the aarch64 stage runs under KVM on Raspberry Pi 5s and may fail, and Amazon Linux is emulated on a PC instead.
Details and sources
Details
- The makefile's head (mk/010-qemu-aarch64.mk:3-10). Unlike
QEMU_CPU_X86_64 ?= max,QEMU_CPU_AARCH64 = maxis a plain assignment, so the environment cannot change it; the command line and target-specific assignments can (mk/050-amazonlinux-2.mk:10-11). GARDEN_AARCH64_FEDORA_EDK2_SYSTEMSandGARDEN_AARCH64_JAMMY_EDK2_SYSTEMS(?=) and the pinnedEDK2_AARCH64_FEDORA,EDK2_AARCH64_JAMMYand their checksums, which "snap/snapcraft.yaml and that job read" (mk/010-qemu-aarch64.mk:114-213, snap/snapcraft.yaml:431-460).BOOT_BY_ARCH: aarch64 takesefionly (rulegen.py:302-309); Ubuntu Core's aarch64 rules (mk/050-ubuntu-core.mk:30-53).- CI:
.systems-aarch64runs on runners taggedaarch64,kvm, withallow_failure: true; its comment names the reasons: interrupt controller and page size differ across hardware and images, much hardware lacks nested virtualisation, and some images are untested (.gitlab-ci.yml:635-661). Amazon Linux runs in"aarch64: [el-emulated]"on x86\_64 (.gitlab-ci.yml:907-917), because Neoverse N1 needs "more recent ARM instruction set than is supported on Raspberry Pi 5" (commit 422add9).
Read more
07 Worth knowing on Arm
Arm hosts vary in GIC, page size and nested virtualisation. On a Raspberry Pi 5, run Amazon Linux with QEMU_ACCEL_MACHINE_PROP=,accel=tcg. +tpm on aarch64 stops make: ubuntu-core with +tpm is only available for x86_64.
Arm machines differ more than PCs do. Interrupt controllers and memory page sizes vary between hosts and in what images expect, and the README calls Arm hosts a feature preview. Many Arm machines, cloud ones included, can't run KVM inside a virtual machine, so check that /dev/kvm exists before you count on full speed. Amazon Linux's Neoverse N1 asks for more than a Raspberry Pi 5's cores offer: on a Pi, run those two emulated, with QEMU_ACCEL_MACHINE_PROP set to ,accel=tcg. Ask for a +tpm system on aarch64 and make stops at once. And Arm here is little-endian, like x86.
Details and sources
Details
- The README ranks aarch64 hosts "Feature-preview", "Partial support, tested on Raspberry Pi" (README.md:384-396). Interrupt controller and page size "differs across hardware … and across what particular images expect", and much hardware "lacks nested virtualisation" (.gitlab-ci.yml:635-646). That a guest needs nested virtualisation to have
/dev/kvminside a cloud VM is general knowledge (derived). - Amazon Linux's
neoverse-n1CPU needs a newer Arm core than a Raspberry Pi 5 has (commits 3477e1a and 422add9);QEMU_ACCEL_MACHINE_PROP=,accel=tcgmakes a.runscript use TCG (mk/010-qemu-aarch64.mk:34). What KVM does with that CPU model on an older core was not run (derived). - "ubuntu-core with +tpm is only available for x86\_64" (mk/050-ubuntu-core.mk:18-21).
- aarch64 Linux is little-endian, s390x big-endian (derived, general knowledge).
Read more
08 Where to go from here
Next: UEFI firmware per system, six architectures, the support matrix, Ubuntu Core, snap components, the container in CI and x86_64.
So aarch64 is QEMU's virt board with CPU max, emulated on a PC and native on Arm, booting one of three EDK2 builds. From here, the firmware deck tells why the older systems need older firmware. The architectures deck covers KVM, emulation and the CPU count for all six. The support matrix and the Ubuntu Core deck show which systems you get. The snap components deck explains qemu-aarch64, the container in CI deck covers runners with KVM, and the x86_64 deck shows the PC side.
Details and sources
Read more
Basis of the statements on this page
Read in the README, mk/010-qemu-aarch64.mk, mk/010-qemu.mk, mk/050-amazonlinux-2.mk, mk/050-ubuntu-core.mk, rulegen.py, data/*.ini, snap/snapcraft.yaml, snap/hooks/install, qemu-stubs/qemu-system-aarch64, the Dockerfile, .gitlab-ci.yml and the commit messages of 3477e1a and 422add9 of image-garden, at the commit in garden.commit ("code"). The system counts were taken from the INI files at that commit. What virt, gic-version=max and the CPU models do inside QEMU and KVM, Debian's package paths, and byte order are derived. Nothing was run for this page.
aarch64 at a glance
| Fact | Value |
|---|---|
| QEMU program | qemu-system-aarch64; in the snap the image-garden.qemu-system-aarch64 app |
| Machine | virt,gic-version=max |
| CPU | max; neoverse-n1 for amazonlinux-cloud-2 and -2023 |
| Accelerator | kvm:tcg when QEMU lists KVM (an aarch64 host), else tcg |
| vCPUs | 4 on an aarch64 host, 1 elsewhere |
| Memory | -m 2048 |
| Boot types | efi only |
| Firmware | the host's EDK2 (AAVMF_CODE.fd, AAVMF_VARS.fd), copied to efi-code.aarch64.img and efi-vars.aarch64.img; Fedora's QEMU_EFI-pflash.qcow2 and vars-template-pflash.qcow2 for ten systems; Ubuntu's 2022.02 AAVMF_CODE.fd and AAVMF_VARS.fd for two |
| Network | -netdev user, virtio-net-pci, -smbios type=41 |
| Snap component | qemu-aarch64, installed by the snap on aarch64 hosts; snap install image-garden+qemu-aarch64 elsewhere |
| Container | images for x86\_64 and aarch64 hosts |
| Ubuntu packages | qemu-system-arm, qemu-efi-aarch64 |
| Host tier | 2: feature preview, tested on Raspberry Pi |
| Byte order | little-endian (derived) |
Systems and their firmware
| Firmware | Systems |
|---|---|
| Fedora's EDK2 | almalinux-cloud-8, amazonlinux-cloud-2, amazonlinux-cloud-2023, fedora-cloud-38, opensuse-cloud-15.4, opensuse-cloud-15.5, oracle-cloud-8, rocky-cloud-8, ubuntu-core-22, ubuntu-core-22+dangerous |
| Ubuntu's EDK2 2022.02 | ubuntu-cloud-14.04, ubuntu-cloud-16.04 |
| the host's EDK2 | the other 36 cloud systems and ubuntu-core-24, -26, -24+dangerous, -26+dangerous |
Not on aarch64: archlinux-cloud, oracle-cloud-7, the end-of-life Ubuntu interims 16.10 to 24.10, ubuntu-core-16, -18, -20, and every +tpm system.
Run-time knobs
| Variable | Effect |
|---|---|
QEMU_ACCEL_MACHINE_PROP | ,accel=tcg forces emulation, for Amazon Linux on a Raspberry Pi 5 |
QEMU_SMP_OPTION, QEMU_MEM_OPTION | replace -smp N and -m 2048 |
QEMU_BOOT_FIRMWARE_OPTION | replaces the two pflash drives |
for contributors
Where it is set
| Fact | Source |
|---|---|
| Program, machine, CPU, SMP, network | mk/010-qemu-aarch64.mk:3-10 |
| Amazon Linux CPU | mk/050-amazonlinux-2.mk:10-11 |
| Native vCPU count, memory | mk/010-qemu.mk:5-12 |
| Efi only | rulegen.py:302-309 |
| Firmware lists, copies, older builds | mk/010-qemu-aarch64.mk:76-213 |
| Snap component | snap/snapcraft.yaml:67-73, :402-403, :431-460 |
| CI | .gitlab-ci.yml:635-661, :841-917 |
| Make variable | Effect |
|---|---|
QEMU_CPU_AARCH64 (=) | the -cpu model; set on the make command line, not from the environment |
QEMU_SMP_AARCH64 | the default -smp |
QEMU_EDK2_CODE_AARCH64, QEMU_EDK2_VARS_AARCH64 (?=) | the EDK2 search lists |
GARDEN_AARCH64_FEDORA_EDK2_SYSTEMS, GARDEN_AARCH64_JAMMY_EDK2_SYSTEMS (?=) | which systems boot the older builds |
GARDEN_AARCH64_PREFIX (environment) | appends a self-built QEMU's share/qemu firmware |
Roads from here
SYSTEMS · distributions
- The support matrix — which systems have an aarch64 build
UBUNTU CORE · all snaps
- Ubuntu Core images — Ubuntu Core 22, 24 and 26 on aarch64
ARCHITECTURES · machines and firmware
- UEFI firmware per system — why older Arm systems need older EDK2, and where it comes from
- Six architectures, six QEMU machines — KVM, TCG and the vCPU count, for every architecture
- x86_64 — the PC side, where every system has a build
SNAP · snap-only features
- Snap components — how qemu-aarch64 is installed, and by whom
CONTAINER · OCI image
- Running the container in CI — the container on runners with /dev/kvm, Arm runners included