The .ini files in data/

META, sections, keys for contributors

Each of the thirteen data/*.ini files has a META section with Group and Variant, then one self-contained section per system, named after it, with Arch, URL or URLProgram, a checksum, Boot and Release; rulegen.py refuses a section whose name, URL or keys do not fit.

On the map: build time

▶ Watch this deck — 7 scenes, ~5 min

Worth seeing first

01 One file per distribution

DATA, ONE FILE PER DISTRIBUTION data/ almalinux.ini alpine.ini amazonlinux.ini archlinux.ini centos.ini debian.ini fedora.ini gentoo.ini opencloudos.ini opensuse.ini oracle.ini rocky.ini ubuntu.ini thirteen; Ubuntu Core is not here [META] Group=ubuntu Variant=cloud … one section per system … ubuntu.ini [ubuntu-cloud-24.04.x86_64] URL=https://cloud-images.ubuntu.com/noble/current/… SHA256SUMS=https://cloud-images.ubuntu.com/noble/… Arch=x86_64 Boot=efi Release=24.04 the rest: rules that every system shares
The data directory holds thirteen .ini files, one per distribution; ubuntu.ini opens with a META section with Group=ubuntu and Variant=cloud, then has one section per system; the ubuntu-cloud-24.04.x86_64 section is five lines: URL, SHA256SUMS, Arch, Boot and Release; the rest comes from rules every system shares

data/*.ini: thirteen files, a [META] section and one section per system. [ubuntu-cloud-24.04.x86_64] has URL, SHA256SUMS, Arch, Boot and Release.

Every system image-garden knows, apart from Ubuntu Core, is written down in the data directory, in one INI file per distribution. There are thirteen of them. Each opens with a META section, then has one section per system, and the section's name is the system's name. Open ubuntu.ini and find ubuntu-cloud-24.04.x86_64. It's five lines: the image URL, the checksum list, the architecture, the boot type and the release. That's all image-garden needs to know about this system; the rest comes from rules every system shares. This deck reads such a file key by key, and shows what rulegen.py refuses.

Details and sources

Details

  • The thirteen files are almalinux, alpine, amazonlinux, archlinux, centos, debian, fedora, gentoo, opencloudos, opensuse, oracle, rocky and ubuntu; they hold 139 system sections (counted from the section headers at the tag). Ubuntu's has 57 sections besides [META].
  • [META] in ubuntu.ini: Group=ubuntu, Variant=cloud (data/ubuntu.ini:4-6). The running example: URL, SHA256SUMS, Arch=x86_64, Boot=efi, Release=24.04 (data/ubuntu.ini:237-242).
  • rulegen.py reads [META] and then every other section as one system named after the section (rulegen.py:336-351); the build runs it once per file (Makefile:57-58).
  • Ubuntu Core's systems are made by hand-written rules in mk/050-ubuntu-core.mk, not from an INI file; see rules written by hand.
  • Lines starting with ; are comments, as in the SPDX header of each file.

02 The META section

THE META SECTION [META] Group=ubuntu Variant=cloud ~/.cache/image-garden/dl/ubuntu/ downloads user ubuntu the everyday user UBUNTU_…CLOUD_INIT_USER_DATA_TEMPLATE hostname ubuntu-24-04 with the release ubuntu-cloud-24.04.x86_64 in every name read, kept, used by no rule yet data/ubuntu.ini: error: [META] has no Group= data/ubuntu.ini: error: [META] has no Variant= either key left out
The META section of ubuntu.ini: Group=ubuntu names the download directory, the everyday user, the template prefix and the first half of the host name; Variant=cloud is in every section name but no rule reads it; leaving either key out stops rulegen.py with a message naming the key

Group=ubuntu picks the download directory, the user, the template prefix and the host name. Variant=cloud is required but not used by any rule; without either, rulegen.py says [META] has no Group=.

The META section has two keys. Group names the family, here ubuntu, and it does a lot. It's the folder in the download cache where the images go, the name of the everyday user in the guest, the prefix of the cloud-init templates make looks for, and the first half of the guest's host name. Variant says what kind of image the file holds. Every file today says cloud, and every section name repeats it, but no rule reads it yet: it's there for kinds of image still to come. Leave either key out, and rulegen.py stops and tells you which one is missing.

Details and sources

Details

03 The keys of a system

THE KEYS OF A SYSTEM [ubuntu-cloud-24.04.x86_64] Arch=x86_64 name must contain Arch: else suspicious one of six architectures x86_64 · aarch64 · riscv64 · s390x · ppc64 · loongarch64 URL=https://…/noble-server-cloudimg-amd64.img names the arch; amd64 and arm64 count or URLProgram=…, never both SHA256SUMS=https://…/SHA256SUMS a checksum: at least one key Boot=efi the firmware, by architecture x86_64: efi or bios · aarch64, riscv64, loongarch64: efi · s390x, ppc64: none Release=24.04 host name and template names
The keys of a system section with the checks rulegen.py makes: the name must contain Arch, one of six architectures; the URL must name the architecture, amd64 and arm64 counting; URL or URLProgram, not both; a checksum; a Boot the architecture has rules for, efi or bios on x86_64, efi on aarch64, riscv64 and loongarch64, none on s390x and ppc64; Release

Name contains Arch; exactly one of URL and URLProgram; at least one checksum key; Boot that the architecture has rules for; Release.

Now a system's own section. Its name has to contain the architecture you give in Arch, or rulegen.py calls the image suspicious. Arch is one of six: x86_64, aarch64, riscv64, s390x, ppc64 and loongarch64. The URL must name the architecture too, though amd64 counts for x86_64, and arm64 for aarch64. Instead of a URL, a section may give URLProgram, a command that prints one, but never both. Then a checksum. Boot picks the firmware: efi or bios on x86_64, efi on aarch64, riscv64 and loongarch64, and on s390x and ppc64 you leave it out. Release goes into the host name and the template names.

Details and sources

Details

  • Arch is required and must be one of x86_64, aarch64, riscv64, s390x, ppc64, loongarch64 (rulegen.py:23-29, rulegen.py:396-401), and the section name must contain it, else "Suspicious image" (rulegen.py:402-405).
  • A URL must contain the architecture; amd64 is accepted for x86_64 and arm64 for aarch64, else "Suspicious URL" (rulegen.py:406-416).
  • Exactly one of URL and URLProgram (rulegen.py:357-362); LocalFilename goes only with URL, URLPattern and LocalFilenameTemplate only with URLProgram, as a pair (rulegen.py:364-393). Gentoo's sections show the URLProgram form (data/gentoo.ini:8-14); see URLProgram.
  • At least one of SHA256SUMS, SHA512SUMS, SHA256SUM (rulegen.py:420-431); see checksums.
  • Boot is checked against the architecture: efi or bios on x86_64, efi on aarch64, riscv64 and loongarch64; s390x and ppc64 boot with the machine's own firmware, platform, which is their default and their only value (rulegen.py:300-333, rulegen.py:440). The ubuntu.ini ppc64el and s390x sections have no Boot. Release defaults to empty, as in Gentoo and Arch Linux (rulegen.py:441). The release goes into the host name and the template names (rulegen.py:461-467).
  • data/README.md describes the [META] keys and Arch, Boot, Release, URL and URLProgram (data/README.md:15-45), then the checksum and file-name keys.

04 Every section stands alone

EVERY SECTION STANDS ALONE [ubuntu-cloud-24.04.x86_64] URL=…/noble-server-cloudimg-amd64.img SHA256SUMS=…/noble/current/SHA256SUMS Arch=x86_64 Boot=efi Release=24.04 [ubuntu-cloud-24.04.aarch64] URL=…/noble-server-cloudimg-arm64.img SHA256SUMS=…/noble/current/SHA256SUMS Arch=aarch64 Boot=efi Release=24.04 only the name, the URL and Arch differ no [DEFAULT] section: nothing is inherited to add a system, copy its neighbour and change what differs [ubuntu-cloud-24.04.ppc64el] Arch=ppc64 Ubuntu's name in the section, QEMU's in Arch: the name contains it
The ubuntu-cloud-24.04 sections for x86_64 and aarch64 side by side: the checksum list, Boot and Release are the same, only the name, URL and Arch differ; no DEFAULT section, so nothing is inherited; the ppc64el section has Arch=ppc64, which its name contains

No [DEFAULT], no inheritance: every section repeats every key. [ubuntu-cloud-24.04.ppc64el] has Arch=ppc64, which its name contains.

You'll notice the files repeat themselves. Ubuntu 24.04's aarch64 section has the same checksum list, boot type and release as the x86_64 one; only the name, the URL and Arch differ. That's on purpose. No file has a DEFAULT section, so nothing is inherited, and each section reads on its own, top to bottom. To add a system, copy its neighbour and change what differs. Watch the names, though. Ubuntu's sections for 64-bit POWER end in ppc64el, the name Ubuntu uses, while Arch says ppc64. That passes, because ppc64el contains ppc64, and the system is called ppc64el everywhere.

Details and sources

Details

  • Ubuntu 24.04 for x86_64 and aarch64 (data/ubuntu.ini:237-249): the same SHA256SUMS, Boot and Release; the name, the URL and Arch differ.
  • No file in data/ has a [DEFAULT] section (searched at the tag). rulegen.py walks the sections and reads each key from the section itself (rulegen.py:336-355). Python's INI reader would hand the keys of a [DEFAULT] section to every section, [META] included (derived, not tried).
  • [ubuntu-cloud-14.04.ppc64el] has Arch=ppc64 and no Boot (data/ubuntu.ini:24-27); the check is a substring test (rulegen.py:402-405). The system's name, its .run script and its files all say ppc64el.

05 When rulegen.py says no

WHEN RULEGEN SAYS NO ./rulegen.py data/example.ini >data/050-example.gen.mk data/example.ini: error: … no traceback Image 'example-cloud-1.x86_64' defines both URL and URLProgram Suspicious image 'example-cloud-1.x86_64' for architecture aarch64 Suspicious URL 'https://example.org/disk.qcow2' for architecture x86_64 Image '…' defines no checksum, use one of SHA256SUMS=, SHA512SUMS= or SHA256SUM= Image '…' has Arch='x86-64', use one of x86_64, aarch64, riscv64, s390x, ppc64, loongarch64 Image '…' has no Boot=, use efi or bios for x86_64 per-group local filename collision in group 'example': 'disk.qcow2' used by … make: *** [Makefile:58: data/050-example.gen.mk] Error 1 make: *** Deleting file 'data/050-example.gen.mk'
make runs rulegen.py on each .ini file; a bad section stops the build with a message that names the file and says what is wrong: both or neither of URL and URLProgram, a suspicious image or URL, no checksum, a missing or unknown Arch or Boot with the values to use, or two sections that save the same file; make then deletes the empty data/050-example.gen.mk

data/X.ini: error: Suspicious URL … for architecture x86_64, … defines no checksum …, … has no Boot=, use efi or bios for x86_64. make then deletes the unfinished data/050-X.gen.mk.

rulegen.py checks every section while the tree is built, and one bad section stops the build. Every message starts with the file's name and the word error, then says what's wrong, with no Python traceback. A section with both URL and URLProgram, or neither, says so. A name without its architecture is a suspicious image, a URL without it a suspicious URL. A missing checksum has its own message. A missing or unknown Arch or Boot names the key and the values you can use. Two sections that would save the same file collide. And make deletes the empty rules file the failed run leaves behind, so the next make starts again from your fixed file.

Details and sources

Details

  • Every message on the slide was produced by running rulegen.py at the pinned commit on a one-section file (Python 3.14). main turns a ValueError from reading a file into SystemExit(f"{filename}: error: {exc}"), so it prints one line and no traceback (rulegen.py:753-757). Unreadable files and INI syntax errors, such as a section or key given twice, are reported the same way (rulegen.py:338-342).
  • Both or neither of URL and URLProgram (rulegen.py:358-363); suspicious image and URL (rulegen.py:402-416); no checksum (rulegen.py:426-431).
  • A missing or empty Arch gives Image 'S' has no Arch=; an unknown one Image 'S' has Arch='x86-64', use one of x86_64, aarch64, riscv64, s390x, ppc64, loongarch64 (rulegen.py:280-297, rulegen.py:396-401).
  • Boot is checked against the architecture (rulegen.py:300-333): Image 'S' has no Boot=, use efi or bios for x86_64 (or use efi for aarch64), Image 'S' has Boot='uefi', use one of platform, efi, bios, and Image 'S' has Boot=bios, which aarch64 does not support, use efi. On s390x and ppc64 a missing Boot means platform. See when Boot is wrong or missing.
  • The file-name collision is checked after reading, for names known when the rules are written, and reported the same way: data/X.ini: error: per-group local filename collision in group 'G': 'F' used by 'S1' and 'S2' (rulegen.py:758-775).
  • The build runs ./rulegen.py data/X.ini >data/050-X.gen.mk (Makefile:57-58). The shell creates the output before rulegen.py fails; the top-level Makefile has .DELETE_ON_ERROR (Makefile:16-18), so make deletes it (run: make: *** Deleting file 'data/050-example.gen.mk'). The next make runs rulegen.py again.
  • tests/check-rulegen.sh checks these messages, and that none of them comes with a traceback (tests/check-rulegen.sh:65-72, tests/check-rulegen.sh:96-223).

06 Trying a new section

TRY IT FROM THE TREE $ ./rulegen.py data/ubuntu.ini | less download · overlay and provisioning · .run script, for each section $ make 050-ubuntu.gen.mk image-garden.mk used in place by ./image-garden $ ./image-garden make list-systems | grep ubuntu-cloud-24.04 $ ./image-garden make ubuntu-cloud-24.04.x86_64.run $ make check-rulegen PASS: url-placeholder-sha256sums … PASS: loongarch64-bad-url
Trying a new section from the source tree: ./rulegen.py data/X.ini prints the system's rules; make writes the file's rules and the system list and joins image-garden.mk; ./image-garden make list-systems shows the system and ./image-garden make SYSTEM.run builds it; make check-rulegen runs the tests

./rulegen.py data/X.ini, then make, ./image-garden make list-systems and ./image-garden make SYSTEM.run, all from the tree. make check-rulegen runs tests/check-rulegen.sh.

To try a new section, you don't need to install anything. Run rulegen.py on the file and read the rules it prints for your system: the download, the overlay and the .run script. Then run make at the top of the tree. It writes that file's rules and the list of systems again, and joins image-garden.mk. The image-garden script next to it uses that file as it is. So ./image-garden make list-systems shows your system, and the same command with its .run name builds it. Last, make check-rulegen runs the project's own tests of these checks.

Details and sources

Details

07 Where to go from here

ROADS The .ini files System names group-variant-release.arch Checksums three keys and @URL@ URLProgram when the URL moves Boot types efi, bios, platform What rulegen.py emits .ini in, make rules out image-garden.mk three parts, one file
Six roads from The .ini files: system names, checksums, URLProgram, boot types, what rulegen.py emits and how image-garden.mk is put together

Next: checksums, URLProgram, boot types, what rulegen.py emits and how image-garden.mk is put together.

That's the whole format: a META section, then one self-contained section per system, checked when the tree is built. The names deck shows how those section names are put together. The checksum deck explains the three checksum keys and the URL placeholder. The URLProgram deck covers images whose address changes with every build. The boot types deck shows what Boot picks. And to see what rulegen.py makes of a section, and how its output becomes image-garden.mk, take the generator and assembly decks.

Details and sources

Read more

Read more

Basis of the statements on this page

Read in rulegen.py, data/*.ini, data/README.md, the Makefile and tests/check-rulegen.sh at the pinned commit ("code"). The error messages were produced by running that rulegen.py on one-section files with Python 3.14, and make's on a failing build ("run"). What Python's INI reader would do with keys the files do not use is "derived".

The [META] section

KeyRequiredExampleUsed for
Groupyesubuntudownload directory $(GARDEN_DL_DIR)/GROUP, the second user and its password, the template prefix GROUP_…, the host name
Variantyescloudnothing yet; every section name repeats it

The keys of a system section

KeyRequiredRuleNotes
section namemust contain Archbecomes the system name, GROUP-VARIANT-RELEASE.ARCH by convention
Archyesx86_64, aarch64, riscv64, s390x, ppc64, loongarch64the name must contain it
URLone of the twomust contain the architecture (amd64, arm64 accepted)its basename is the cached file name
URLProgramone of the twoa shell command, run by make, that prints the URLsee URLProgram
LocalFilenamenoonly with URL; no /, not . or ..overrides the cached file name
URLPattern, LocalFilenameTemplatenoonly with URLProgram, together; one capture group, @1a stable cached file name
SHA256SUMS, SHA512SUMS, SHA256SUMat least onelists by URL, or one hash; @URL@ only at the start of the first twosee checksums
Bootyes, except on s390x and ppc64efi or bios on x86_64; efi on aarch64, riscv64, loongarch64; platform (the default) on s390x and ppc64see boot types
Releasenoany text, default emptyhost name and per-release templates

Sources: rulegen.py:17-29, rulegen.py:300-333, rulegen.py:336-445, data/README.md:15-88.

rulegen.py's messages

Every message is one line on standard error, data/X.ini: error: and then the text below; there is no traceback (rulegen.py:753-775). make then deletes the empty data/050-X.gen.mk (.DELETE_ON_ERROR, Makefile:16-18).

MistakeMessage after data/X.ini: error:
no [META] sectionno [META] section
no Group or Variant in [META][META] has no Group= or [META] has no Variant=
a section or key given twice, unreadable filethe INI reader's or the system's message, as in While reading from 'data/X.ini' [line 3]: option 'group' in section 'META' already exists
both URL and URLProgramImage 'S' defines both URL and URLProgram
neitherImage 'S' defines neither URL nor URLProgram
no ArchImage 'S' has no Arch=
unknown ArchImage 'S' has Arch='x86-64', use one of x86_64, aarch64, riscv64, s390x, ppc64, loongarch64
name without ArchSuspicious image 'S' for architecture A
URL without the architectureSuspicious URL 'U' for architecture A
no checksumImage 'S' defines no checksum, use one of SHA256SUMS=, SHA512SUMS= or SHA256SUM=
no Boot where one is neededImage 'S' has no Boot=, use efi or bios for x86_64 (use efi for aarch64, and so on)
unknown BootImage 'S' has Boot='uefi', use one of platform, efi, bios
a Boot the architecture lacksImage 'S' has Boot=bios, which aarch64 does not support, use efi
@URL@ in another keyImage 'S' uses @URL@ in 'sha256sum', it is only allowed in SHA256SUMS= and SHA512SUMS=
@URL@ not at the start, or twiceImage 'S' SHA256SUMS= must use @URL@ once, at the start of the value, e.g. @URL@.sha256
two sections, one cached file nameper-group local filename collision in group 'G': 'F' used by 'S1' and 'S2'

A new section, step by step

  • Copy the section next to it in the same file and change the name, URL and Arch (or the release).
  • ./rulegen.py data/X.ini and read the rules for the new name.
  • make, then ./image-garden make list-systems and ./image-garden make NAME.run from the tree.
  • make check-rulegen, and make check before sending the change.

Roads from here

START HERE

ARCHITECTURES · machines and firmware

  • Boot types — what the Boot key picks, and what a missing one costs

BUILD TIME · data and rules