UEFI firmware per system

code and vars per system for contributors

Each efi system boots from its own qcow2 copies of the host's EDK2 code and variables, found through per-architecture search lists that end at the snap's component and GARDEN_ARCH_PREFIX; ten older aarch64 systems use Fedora's EDK2, Ubuntu 14.04 and 16.04 a 2022 build, and the +tpm systems an OVMF Secure Boot pair.

On the map: architectures

▶ Watch this deck — 7 scenes, ~5 min

Worth seeing first

  • Boot types — which systems are efi and what efi adds to the rules

01 Firmware of its own

FIRMWARE OF ITS OWN a UEFI guest writes to its variable store system A system B one shared store their boot entries collide on the host OVMF_CODE_4M.fd OVMF_VARS_4M.fd once per architecture efi-code.x86_64.img efi-vars.x86_64.img one copy per system ubuntu-cloud-24.04.x86_64.efi-code.qcow2 ubuntu-cloud-24.04.x86_64.efi-vars.qcow2 -drive if=pflash,format=qcow2,unit=0,file=…efi-code.qcow2,readonly=on -drive if=pflash,format=qcow2,unit=1,file=…efi-vars.qcow2 read-only writable
Two systems sharing one UEFI variable store would collide, so image-garden copies the host's OVMF code and variables once per architecture to efi-code.x86_64.img and efi-vars.x86_64.img, then gives each efi system its own copies, such as ubuntu-cloud-24.04.x86_64.efi-code.qcow2 and .efi-vars.qcow2. The .run script attaches the code as pflash unit 0, read-only, and the variables as unit 1, writable

efi-code.x86_64.img and efi-vars.x86_64.img are copied once from the host; each efi system gets X.efi-code.qcow2, attached read-only, and X.efi-vars.qcow2, writable.

A UEFI guest doesn't only run its firmware; it writes to it. Boot entries and settings land in a store of UEFI variables, and two systems sharing one store would trip over each other's entries. So image-garden works in two steps. Once per architecture, it copies the host's EDK2 firmware, the code and a blank variable store, next to your systems. Then every efi system, such as ubuntu-cloud-24.04.x86_64, gets its own qcow2 copy of both, named after it. The .run script attaches the code read-only and the variables writable, so whatever the guest saves stays with that one system.

Details and sources

Details

  • efi-code.x86_64.img and efi-vars.x86_64.img are copied with cp --preserve=timestamps from the first file found (mk/010-qemu-x86_64.mk:316-322); %.x86_64.efi-code.qcow2 and %.x86_64.efi-vars.qcow2 convert them with $(QEMU_IMG) convert -f raw -O qcow2, the qemu-img that QEMU_IMG names (mk/010-qemu-x86_64.mk:324-329). Every UEFI architecture's file has the same rules.
  • The .run script's default firmware option: unit 0 readonly=on, unit 1 writable (mk/010-qemu-x86_64.mk:67).
  • The README's file tables: SYSTEM.ARCH.efi-code.qcow2 is a copy of efi-code.ARCH.img, SYSTEM.ARCH.efi-vars.qcow2 holds the current variables; ten shared firmware files (README.md:827-828, README.md:839-852).
  • The shared and per-system files are made in the directory make runs in, .image-garden/ or the project directory (see the cache deck).
  • The man page names them the same way: $SYSTEM.$ARCH.efi-code.qcow2, a copy of efi-code.$ARCH.img, and $SYSTEM.$ARCH.efi-vars.qcow2, made from efi-vars.$ARCH.img (image-garden.1:159-167).

02 Where the firmware comes from

SEARCH LISTS QEMU_EDK2_CODE_X86_64 ?= /usr/share/OVMF/OVMF_CODE.fd /usr/share/OVMF/OVMF_CODE_4M.fd /usr/share/ovmf/x64/OVMF_CODE.4m.fd /usr/share/qemu/ovmf-x86_64-4m-code.bin Debian, Ubuntu Arch Linux openSUSE make copies the first file that exists efi-code.x86_64.img …/components/REV/qemu-x86-64/share/qemu/edk2-x86_64-code.fd in the snap $GARDEN_X86_64_PREFIX/share/qemu/edk2-x86_64-code.fd with the prefix set QEMU_EDK2_VARS_X86_64: a list of its own, searched the same way riscv64: truncate --size 32MiB on both copies, the size its flash expects
QEMU_EDK2_CODE_X86_64 lists the Debian and Ubuntu OVMF paths, then Arch Linux's and openSUSE's, then in the snap the qemu-x86-64 component's edk2-x86_64-code.fd, then with GARDEN_X86_64_PREFIX set the edk2 file under that prefix. make copies the first file that exists to efi-code.x86_64.img; the variables have a list of their own. On riscv64 both copies are truncated to 32 MiB

QEMU_EDK2_CODE_ARCH and QEMU_EDK2_VARS_ARCH list distribution paths, then the snap component's file, then $GARDEN_ARCH_PREFIX/share/qemu. The first file that exists is copied; riscv64 copies are truncated to 32 MiB.

Where does the host's firmware come from? Each architecture's makefile lists the places where distributions install EDK2, and make takes the first file that exists. For x86_64 that's OVMF: Debian and Ubuntu's paths first, then Arch Linux's, then openSUSE's. In the snap, the list goes on to the firmware in the snap's own QEMU component, and with a prefix variable set, it ends at the firmware next to a QEMU you built yourself. The code and the variable store are each looked up in a list of their own. riscv64 adds one twist: both copies are stretched to the thirty-two megabytes its flash expects.

Details and sources

Details

  • x86_64: QEMU_EDK2_CODE_X86_64 and QEMU_EDK2_VARS_X86_64, "default location on most Debian/Ubuntu systems, Arch and openSUSE", then the snap's qemu-x86-64/share/qemu/edk2-x86_64-code.fd and edk2-i386-vars.fd, then $GARDEN_X86_64_PREFIX/share/qemu/… (mk/010-qemu-x86_64.mk:260-280). Which distribution uses which path is derived from that comment.
  • $(firstword $(wildcard LIST)) picks the first existing file for the code and, separately, for the variables (mk/010-qemu-x86_64.mk:318-322).
  • aarch64: /usr/share/AAVMF/AAVMF_CODE.fd, /usr/share/qemu/aavmf-aarch64-code.bin, then the snap's edk2-aarch64-code.fd and edk2-arm-vars.fd, then the prefix (mk/010-qemu-aarch64.mk:76-91).
  • riscv64: /usr/share/qemu-efi-riscv64/RISCV_VIRT_CODE.fd, /usr/share/edk2/riscv/RISCV_VIRT_CODE.fd, /usr/share/qemu/ovmf-riscv64-code.bin, the snap's, the prefix's; both copies get truncate --size 32MiB (mk/010-qemu-riscv64.mk:75-110).
  • loongarch64: see loongarch64 and OpenCloudOS.
  • Every list is set with ?=, so a project can replace it in .image-garden.mk or on the make command line (derived from GNU make).

Read more

  • The full lists are in this deck's reference.

03 Older Arm systems need older firmware

OLDER ARM SYSTEMS EDK2 from 2023 on: memory that boot loaders load into is not executable older shim or grub: Synchronous Exception before the kernel starts Fedora's EDK2 almalinux-cloud-8, amazonlinux-cloud-2, -2023, fedora-cloud-38, opensuse-cloud-15.4, -15.5, oracle-cloud-8, rocky-cloud-8, ubuntu-core-22, +dangerous Ubuntu's EDK2, 2022 ubuntu-cloud-14.04, ubuntu-cloud-16.04 the snap: its qemu-aarch64 component elsewhere: downloaded, SHA-256 checked GARDEN_AARCH64_FEDORA_EDK2_SYSTEMS ?= … from image-garden's GitLab packages
EDK2 from 2023 on makes memory that boot loaders load into non-executable, and older shims and grubs stop with Synchronous Exception. Ten aarch64 systems, from almalinux-cloud-8 to ubuntu-core-22+dangerous, boot Fedora's EDK2; ubuntu-cloud-14.04 and 16.04 boot Ubuntu's EDK2 from 2022. The snap's qemu-aarch64 component carries both; elsewhere they are downloaded and checked by SHA-256

Ten aarch64 systems, listed in GARDEN_AARCH64_FEDORA_EDK2_SYSTEMS, use Fedora's EDK2; ubuntu-cloud-14.04 and 16.04 use Ubuntu's EDK2 from 22.04. Both come from the snap's qemu-aarch64 component or a checksummed download.

On Arm, one firmware doesn't boot everything. EDK2 from 2023 on marks the memory that boot loaders load into as not executable, and some older shims and grubs trip over that: they stop with Synchronous Exception before the kernel starts. So ten aarch64 systems, among them AlmaLinux 8, Rocky 8 and Ubuntu Core 22, boot with Fedora's EDK2, which is built to tolerate them. Ubuntu 14.04 and 16.04 need something older still: Ubuntu's EDK2 from 2022. The snap carries both builds in its aarch64 component. Elsewhere, make downloads them by checksum from image-garden's own GitLab packages.

Details and sources

Details

  • The comment above EDK2_AARCH64_FEDORA: from 2023 EDK2 marks memory that boot loaders load data into as not executable and offers the EFI memory attribute protocol, which shim 15.6 and 15.7 misuse; with Debian's and Ubuntu's EDK2 those systems stop with "Synchronous Exception" (mk/010-qemu-aarch64.mk:114-127).
  • GARDEN_AARCH64_FEDORA_EDK2_SYSTEMS ?= almalinux-cloud-8, amazonlinux-cloud-2, amazonlinux-cloud-2023, fedora-cloud-38, opensuse-cloud-15.4, opensuse-cloud-15.5, oracle-cloud-8, rocky-cloud-8, ubuntu-core-22, ubuntu-core-22+dangerous; their .aarch64.efi-code.qcow2 and .efi-vars.qcow2 are copies of QEMU_EFI-pflash.qcow2 and vars-template-pflash.qcow2 (mk/010-qemu-aarch64.mk:134-167).
  • GARDEN_AARCH64_JAMMY_EDK2_SYSTEMS ?= ubuntu-cloud-14.04, ubuntu-cloud-16.04: their 4.4 kernel gives up when the memory map changes while it exits boot services ("Failed to update FDT and exit boot services" under Fedora's EDK2); Ubuntu's EDK2 2022.02 boots them (mk/010-qemu-aarch64.mk:169-213).
  • In the snap both builds are in the qemu-aarch64 component (share/qemu/edk2-fedora, share/qemu/edk2-jammy); elsewhere download-with-sha256sum fetches them into $(GARDEN_DL_DIR)/edk2-aarch64-fedora/VERSION and edk2-aarch64-jammy/VERSION from the project's GitLab generic packages (mk/010-qemu-aarch64.mk:128-161, :182-207). The pinned builds are EDK2_AARCH64_FEDORA = 20260812-10.fc45 and EDK2_AARCH64_JAMMY = 2022.02-3ubuntu0.22.04.7.
  • Because the lists use ?=, a project can add a system to either list (derived).

04 The Secure Boot pair

THE SECURE BOOT PAIR for the four +tpm systems only OVMF's Secure Boot build needs SMM: -machine q35,smm=on variables with Microsoft's keys so the signed shim may boot /usr/share/OVMF/OVMF_CODE_4M.secboot.fd /usr/share/OVMF/OVMF_VARS_4M.ms.fd …/qemu-x86-64/share/ovmf/OVMF_CODE_4M.secboot.fd …/qemu-x86-64/share/ovmf/OVMF_VARS_4M.ms.fd first complete pair: efi-secboot-code.x86_64.img, efi-secboot-vars.x86_64.img first row: Ubuntu's ovmf package; second: the snap's x86_64 component please install x86_64 edk2 firmware with Secure Boot and Microsoft keys (the ovmf package) - not in /usr/share/OVMF/OVMF_CODE_4M.secboot.fd …
For the four +tpm systems only: OVMF's Secure Boot build, which needs SMM, and variables with Microsoft's keys, so the signed shim boots. The code and variable lists are walked as pairs, Ubuntu's ovmf package first, then the snap's qemu-x86-64 component; the first complete pair becomes efi-secboot-code.x86_64.img and efi-secboot-vars.x86_64.img. Without one, make stops asking for edk2 firmware with Secure Boot and Microsoft keys

OVMF_CODE_4M.secboot.fd with OVMF_VARS_4M.ms.fd, as a pair, become efi-secboot-code.x86_64.img and efi-secboot-vars.x86_64.img for the +tpm systems only.

The four Ubuntu Core +tpm systems need firmware that plain efi systems don't: Secure Boot. That's OVMF's Secure Boot build, which needs System Management Mode, and a variable store with Microsoft's keys already enrolled, so the signed shim is allowed to boot. Code and variables have to come from the same build, so here the two lists are walked as pairs, and the first pair where both files exist wins. Ubuntu's ovmf package has them, and so does the snap's x86_64 component. If neither is there, make stops and asks for edk2 firmware with Secure Boot and Microsoft keys.

Details and sources

Details

  • QEMU_EDK2_SECBOOT_CODE_X86_64 ?= /usr/share/OVMF/OVMF_CODE_4M.secboot.fd and QEMU_EDK2_SECBOOT_VARS_X86_64 ?= /usr/share/OVMF/OVMF_VARS_4M.ms.fd, plus the snap component's share/ovmf copies; QEMU_EDK2_SECBOOT_PAIR_X86_64 joins the Nth entries and takes the first pair where both exist (mk/010-qemu-x86_64.mk:282-296).
  • efi-secboot-code.x86_64.img and efi-secboot-vars.x86_64.img copy the pair or print "please install x86_64 edk2 firmware with Secure Boot and Microsoft keys (the ovmf package) - not in … with …" and fail; the check runs only when such a system is made (mk/010-qemu-x86_64.mk:298-308).
  • The +tpm systems' .efi-code.qcow2 and .efi-vars.qcow2 are converted from them, and their .run and .qcow2 use QEMU_SYSTEM_X86_64_MACHINE = q35,smm=on (mk/050-ubuntu-core.mk:55-61); "Secure Boot firmware keeps its variables in flash that only SMM may write" (mk/010-qemu-x86_64.mk:134-135).
  • The snap's ovmf-x86-64 part stages the pair from Ubuntu's ovmf-generic package into the qemu-x86-64 component (snap/snapcraft.yaml:645-665).

05 Firmware next to your own QEMU

YOUR OWN QEMU a QEMU you built, in PREFIX PREFIX/share/qemu/edk2-x86_64-code.fd … GARDEN_X86_64_PREFIX GARDEN_AARCH64_PREFIX GARDEN_RISCV64_PREFIX GARDEN_LOONGARCH64_PREFIX 1. the distribution's paths 2. the snap's QEMU component 3. PREFIX/share/qemu/edk2-*, last Dockerfile ENV GARDEN_{X86_64,AARCH64,RISCV64,LOONGARCH64}_PREFIX=/usr/local s390x and ppc64: no prefix, their machines take no firmware file RUN apt install … ovmf swtpm … x86_64 finds Ubuntu's OVMF first
A QEMU you built has EDK2 builds under its prefix. GARDEN_X86_64_PREFIX, GARDEN_AARCH64_PREFIX, GARDEN_RISCV64_PREFIX and GARDEN_LOONGARCH64_PREFIX add PREFIX/share/qemu/edk2-* at the end of the search lists, after the distribution paths and the snap component. The container sets these four, and only these, to /usr/local; s390x and ppc64 need no prefix, as their machines take no firmware file. It installs Ubuntu's ovmf package, so x86_64 finds Ubuntu's OVMF first

GARDEN_X86_64_PREFIX, _AARCH64_, _RISCV64_ and _LOONGARCH64_PREFIX append PREFIX/share/qemu/edk2-* to the search lists. The container sets them to /usr/local.

If you build QEMU yourself, it installs EDK2 builds of its own, under its prefix. Four variables point image-garden there, one per UEFI architecture, such as GARDEN_X86_64_PREFIX. Each adds that prefix's code and variables to the end of its list, so a distribution's firmware still wins when it's installed. The container does exactly this: it builds QEMU into its own prefix and sets all four, and only those four. s390x and ppc64 need no prefix, as their machines take no firmware file. And because the container has Ubuntu's ovmf package, x86_64 systems there boot Ubuntu's OVMF.

Details and sources

Details

06 When the firmware is missing

WHEN IT IS MISSING $ image-garden make list-systems image-garden.mk:…: please install x86_64 edk2 firmware - not in /usr/share/OVMF/… on every make: the list is checked while make reads its rules $ image-garden make ubuntu-cloud-24.04.x86_64.run cp --preserve=timestamps efi-code.x86_64.img cp: missing destination file operand after 'efi-code.x86_64.img' in the snap, without the component: install qemu-x86-64 component to use x86_64 emulation the fix: install the firmware, or the snap's QEMU component
Without firmware, every make, even list-systems, warns please install x86_64 edk2 firmware - not in followed by the search list, because the list is checked while make reads its rules. Making an efi system then fails at the copy. In the snap, a missing QEMU component stops the copy with install qemu-x86-64 component to use x86_64 emulation. The fix is to install the firmware

A missing firmware gives please install x86_64 edk2 firmware - not in … on every make. In the snap a missing component gives install qemu-x86-64 component to use x86_64 emulation.

What if none of the files exists? Then every make prints a warning, please install x86_64 edk2 firmware, followed by the list of places it looked. You see it even when you build something else, a bios system or a list of systems, because make checks the list while it reads its rules. The build of an efi system then fails at the copy, as there's nothing to copy. In the snap, the copy first checks that the architecture's QEMU component is installed, and without it make stops and names the component to install. Either way, the fix is the same: install the firmware.

Details and sources

Details

  • The prerequisite of efi-code.x86_64.img is $(or $(firstword $(wildcard …)),$(warning please install x86_64 edk2 firmware - not in …)) (mk/010-qemu-x86_64.mk:318-322). GNU make expands prerequisites when it reads the rule, so the warning appears on every run that reads image-garden.mk, once for each architecture whose list finds nothing (derived from GNU make).
  • With no prerequisite, the recipe cp --preserve=timestamps $< $@ gets one argument; the cp message on the slide is GNU coreutils' (derived, not run).
  • In the snap, efi-code.ARCH.img has check-qemu-ARCH-component as an order-only prerequisite, which runs ENSURE_QEMU_SNAP_COMPONENT: "install qemu-x86-64 component to use x86_64 emulation" (mk/010-qemu-x86_64.mk:310-316, mk/010-qemu.mk:50-55).
  • The README lists the host tools for a source install but not the EDK2 packages by name (README.md:122-126).

07 Where to go from here

ROADS UEFI firmware Boot types who is efi at all Secure Boot and TPM what the pair is for Six architectures the machines that load it The target chain where the copies fit The container in CI its own QEMU and prefixes The snap firmware in its components
Roads from UEFI firmware: boot types, Secure Boot and the TPM, the six machines, the target chain, the container in CI and the snap

Next: boot types, Secure Boot and the TPM, the six machines, the target chain, the container in CI and the snap.

So each efi system boots from its own copy of firmware the host already has, found in a list, with older builds for older Arm systems and a Secure Boot pair for the TPM systems. From here, the boot deck shows which systems are efi at all. The Secure Boot deck shows what the pair is for. The architectures deck has the machines that load the firmware, and the target chain shows where the copies fit among the files make creates. The container and the snap each bring firmware of their own.

Details and sources

Read more

Basis of the statements on this page

Read in mk/010-qemu-x86_64.mk, mk/010-qemu-aarch64.mk, mk/010-qemu-riscv64.mk, mk/010-qemu-loongarch64.mk, mk/050-ubuntu-core.mk, snap/snapcraft.yaml, the Dockerfile and the man page of image-garden at the tag in garden.commit ("code"). Which distribution installs which path, and what make and cp print when nothing is found, are derived. Nothing was run for this page.

Search lists

The first existing file of each list is copied. Lists are tried in this order: the paths below, then the snap's component (in the snap), then $GARDEN_ARCH_PREFIX/share/qemu/ (when set).

ArchCodeVariablesSnap component filePrefix file
x86_64/usr/share/OVMF/OVMF_CODE.fd, /usr/share/OVMF/OVMF_CODE_4M.fd, /usr/share/ovmf/x64/OVMF_CODE.4m.fd, /usr/share/qemu/ovmf-x86_64-4m-code.binthe matching VARS pathsqemu-x86-64/share/qemu/edk2-x86_64-code.fd, edk2-i386-vars.fdsame names
aarch64/usr/share/AAVMF/AAVMF_CODE.fd, /usr/share/qemu/aavmf-aarch64-code.binthe matching VARS pathsqemu-aarch64/share/qemu/edk2-aarch64-code.fd, edk2-arm-vars.fdsame names
riscv64/usr/share/qemu-efi-riscv64/RISCV_VIRT_CODE.fd, /usr/share/edk2/riscv/RISCV_VIRT_CODE.fd, /usr/share/qemu/ovmf-riscv64-code.binthe matching VARS pathsqemu-riscv64/share/qemu/edk2-riscv-code.fd, edk2-riscv-vars.fdsame names
loongarch64/usr/share/qemu-efi-loongarch64/QEMU_EFI.fd, /usr/share/edk2/loongarch64/QEMU_EFI.fdQEMU_VARS.fd in the same directoriesqemu-loongarch64/share/qemu/edk2-loongarch64-code.fd, -vars.fdsame names
x86_64 Secure Boot/usr/share/OVMF/OVMF_CODE_4M.secboot.fd/usr/share/OVMF/OVMF_VARS_4M.ms.fdqemu-x86-64/share/ovmf/ with the same namesnone

Files

FileMade fromUsed by
efi-code.ARCH.img, efi-vars.ARCH.imgthe first file of each listevery efi system of that architecture
efi-secboot-code.x86_64.img, efi-secboot-vars.x86_64.imgthe first complete Secure Boot pairthe four +tpm systems
X.efi-code.qcow2$(QEMU_IMG) convert of the codepflash unit 0, read-only
X.efi-vars.qcow2$(QEMU_IMG) convert of the variablespflash unit 1, writable
$(GARDEN_DL_DIR)/edk2-aarch64-fedora/VERSION/download, SHA-256 checkedten aarch64 systems (outside the snap)
$(GARDEN_DL_DIR)/edk2-aarch64-jammy/VERSION/download, SHA-256 checkedubuntu-cloud-14.04 and 16.04 on aarch64 (outside the snap)

Knobs

VariableKindEffect
QEMU_EDK2_CODE_ARCH, QEMU_EDK2_VARS_ARCHmake, ?=replace a search list
QEMU_EDK2_SECBOOT_CODE_X86_64, QEMU_EDK2_SECBOOT_VARS_X86_64make, ?=replace the Secure Boot lists
GARDEN_AARCH64_FEDORA_EDK2_SYSTEMS, GARDEN_AARCH64_JAMMY_EDK2_SYSTEMSmake, ?=which aarch64 systems use the older builds
GARDEN_X86_64_PREFIX, GARDEN_AARCH64_PREFIX, GARDEN_RISCV64_PREFIX, GARDEN_LOONGARCH64_PREFIXenvironmentappend a self-built QEMU's EDK2 to the lists
QEMU_BOOT_FIRMWARE_OPTIONenvironment, at run timereplaces both pflash drives of a .run script
QEMU_IMGmakethe qemu-img that converts the firmware copies to qcow2

Roads from here

UBUNTU CORE · all snaps

ARCHITECTURES · machines and firmware

RUNTIME · make rules and the CLI

SNAP · snap-only features

CONTAINER · OCI image