What rulegen.py emits
.ini in, make rules out for contributors
At build time rulegen.py checks every section of one INI file, then prints a header, the group's download directory and four blocks of make rules per system - common, fetch, URL and boot - whose recipes call macros defined in mk/.
On the map: build time
▶ Watch this deck — 7 scenes, ~5 min
Worth seeing first
- From a system's name to a running VM — the path from an INI section to a running VM that these rules drive
01 Five lines in, ninety-three out
At build time the Makefile runs ./rulegen.py data/ubuntu.ini >data/050-ubuntu.gen.mk. The five-line section of ubuntu-cloud-24.04.x86_64 becomes 93 lines of rules; all 139 systems give about 13,000.
The INI files describe 139 systems, and each one needs about ninety lines of make: rules for its cloud-init files, its download, its disk and its launcher. Nobody wants to write, or review, thirteen thousand lines like that. So rulegen.py writes them. When image-garden itself is built, the Makefile runs it once per INI file and saves what it prints as that distribution's rule file. Our system's section is five lines long. Out of it come ninety-three lines of rules, for files whose names all start with ubuntu-cloud-24.04.x86_64. The rule files are build output: they aren't in git, and you change them by changing the INI file or rulegen.py.
Details and sources
Details
- The section is five keys under
[ubuntu-cloud-24.04.x86_64](ubuntu.ini:237-242). - The Makefile's pattern rule
data/050-%.gen.mk: rulegen.py data/%.iniruns./rulegen.py data/X.ini >data/050-X.gen.mk(Makefile:57-58). rulegen.py accepts several files and prints to standard output (rulegen.py:741-757). - Counted in a scratch build of the pinned commit: 93 lines from the system's
GARDEN_SYSTEMS +=line to the next system's; 13,135 lines in the thirteendata/050-*.gen.mkfiles for 139 systems; 5,224 of them in050-ubuntu.gen.mk. data/*.mkandimage-garden.mkare in.gitignore(.gitignore:16-17). A checkout's copies may be older than the INI files; read rulegen.py and the INI files, or build them again.
Read more
- From a system's name to a running VM: the whole path in one deck.
- The .ini files in data/: what each of the five keys means.
02 Check everything, then print
Every section is read and checked before anything is printed: data/X.ini: error: … names the file and the section. Then the cached file names are compared. A file with a mistake gives no rules at all.
rulegen.py takes a file in three steps. First it reads every section and checks it: URL or URLProgram but not both, an architecture it knows and that the section's name contains, a URL that names that architecture, a checksum, and a boot type that architecture has rules for. The first problem ends the run, with the file's name, the word error, and a message naming the section. Next, it compares the cache file names the sections will use, so that no two images land on one file. Only then does it print, every rule of the file in one go. So for a file with a mistake in it, rulegen.py prints no rules at all.
Details and sources
Details
load_imagesreads the file and checks each section in this order:URLagainstURLProgram, the file-name keys,Arch(an architecture rulegen.py knows, contained in the section name), the architecture in theURL, the@URL@placeholder, a checksum, thenBoot, which must be one the architecture has rules for (rulegen.py:336-445, rulegen.py:300-333). Missing[META],Group,VariantorArchare reported the same way, as are INI syntax errors (rulegen.py:280-297, rulegen.py:338-348). The architectures and boot types are the enums at rulegen.py:17-29.mainturns the generator into a list first, so every section is checked before any output; aValueErrorbecomesSystemExit("FILE: error: MESSAGE"), exit status 1 (rulegen.py:753-757). The message on the slide is the one for a section without a checksum, astests/check-rulegen.shexpects it.- The second step collects the local file names known at build time, per group, and stops on the first one used twice (rulegen.py:758-775). It is reported the same way, without a traceback:
data/X.ini: error: per-group local filename collision in group 'G': 'NAME' used by 'A' and 'B'(run on a two-section test file). - The third step prints the header, then the rules of every system in file order (rulegen.py:777-822).
- Given several files, rulegen.py handles them one after the other, so the rules of earlier files are already printed when a later one fails. The Makefile gives it one file at a time.
Read more
- The .ini files in data/: every message and the key behind it.
- When Boot is wrong or missing, checksums, URLProgram.
- When a part fails: make deletes the empty rule file after an error.
03 Four blocks for every system
Per file: a header and the group's download directory. Per system: the common block, the fetch block, the .url and .filename rules, and the boot block, chosen by Arch and Boot.
Here's what it prints, in order. A header comes first, once per file, and a rule that makes the group's directory in the download cache. Then four blocks for each system. The common block adds the name to the list of systems, and to all when the architecture matches your machine, and writes the rules for the meta-data and user-data files and for clean. The fetch block makes the system's link in the cache from a verified download, with the helper its checksum key asks for. Next come the rules for the .url and .filename files. Last is the boot block, the disk and the launcher, picked by Arch and Boot.
Details and sources
Details
mainprintsruleset_header, then for the first systemruleset_group, and for every systemruleset_common,ruleset_fetch_common, the.urlrule (ruleset_fetch_directorruleset_fetch_prog), the.filenamerule (ruleset_filename_static,_dynamic_basenameor_dynamic_pattern) and the boot ruleset;ruleset_footerends the file (rulegen.py:777-822).- Common (rulegen.py:452-478):
GARDEN_SYSTEMS += NAME, whichlist-systemsprints;all:: NAME.qcow2 NAME.runinsideifeq ($(GARDEN_ARCH),ARCH);NAME.meta-dataandNAME.user-datafrom the templates, with the group, the upper-case group and the release;clean::lines for the link, the disk, the launcher and the cloud-init files. The meta-data template gets the group and the release, so itslocal-hostnamematches the user-datahostname. - Fetch (rulegen.py:480-518):
fetch:for the host's architecture,distclean::(the download, the link,.urland.filename),.PRECIOUSfor the link, and the rule for$(GARDEN_DL_DIR)/GROUP/NAME.qcow2, which runsdownload-with-sha512sums,-sha256sumor-sha256sumsand makes the link. All three are written inside nestedifneqtests on the checksum keys, so make keeps the first one that is set. - URL and file name (rulegen.py:520-591): see URLProgram and local file names.
- Boot (rulegen.py:593-724):
NAME.runandNAME.qcow2; which ruleset a system gets is in Architecture first, then boot type. Every combination rulegen.py accepts has a ruleset. - The group directory rule is printed once per file, for the first section, so every section of a file is expected to share one group, the file's
[META]group (rulegen.py:778-780). - Every section on x86_64, aarch64, riscv64 and loongarch64 has a
Bootthat has a boot block: rulegen.py refuses a missing or unsupported one, so no system getsall::for a.qcow2without a rule to make it (rulegen.py:300-333).
Read more
- Boot types, the target chain: the files these rules make, in the order make builds them.
04 Thin rules, macros in mk/
Generated recipes call macros: EMIT_QEMU_RUN_SCRIPT_QUERY_PREAMBLE, EMIT_QEMU_SYSTEM_X86_64_EFI_RUN_COMMAND, QEMU_SYSTEM_X86_64_EFI_CMDLINE, PICK_CLOUD_INIT_USER_DATA_TEMPLATE_FOR, and image-garden download-with-sha256sums. How a system boots is changed in mk/.
Look inside those rules and you'll find very little QEMU. The launcher rule calls two macros: one writes the query preamble, the other the command for an x86 machine with EFI. The disk rule calls the macro with that machine's command line. The user-data rule asks the template picker which template applies. The download calls image-garden itself. All of those live in the hand-written files in mk, or in the script. So rulegen.py mostly fills in names. To change how every EFI system on x86 boots, you edit the x86 QEMU makefile, not the generator. You touch rulegen.py when sections need a new key, or a new architecture needs rules of its own.
Details and sources
Details
- The x86_64 EFI ruleset: the
.runrecipe callsEMIT_QEMU_RUN_SCRIPT_QUERY_PREAMBLEandEMIT_QEMU_SYSTEM_X86_64_EFI_RUN_COMMAND; the.qcow2recipe runs$(QEMU_IMG) createandQEMU_SYSTEM_X86_64_EFI_CMDLINEwith the seed attached (rulegen.py:593-610). - The macros are defined by hand:
EMIT_QEMU_RUN_SCRIPT_QUERY_PREAMBLE(010-qemu.mk:57-57),CLOUD_INIT_META_DATA_TEMPLATE(010-qemu.mk:97-106),PICK_CLOUD_INIT_USER_DATA_TEMPLATE_FOR(010-qemu.mk:293-298),QEMU_SYSTEM_X86_64_EFI_CMDLINE(010-qemu-x86_64.mk:13-13) andEMIT_QEMU_SYSTEM_X86_64_EFI_RUN_COMMAND(010-qemu-x86_64.mk:49-49). Each architecture has its ownmk/010-qemu-ARCH.mkwith the same set. - The user-data and meta-data rules pass the group, the upper-case group and the release to the templates (rulegen.py:460-471).
$(GARDEN_ITSELF)is the path of theimage-gardenscript, whichimage-garden makepasses to make;download-with-sha256sumsis one of its commands (rulegen.py:510-514, image-garden:460-473).- Because the macros are expanded when make runs, an edit in
mk/reaches every generated system without running rulegen.py again;image-garden.mkstill has to be rebuilt from the parts (how image-garden.mk is put together). - What rulegen.py decides itself: the target names, the prerequisites, which checksum helper and which boot ruleset, the
ifeqonGARDEN_ARCH, the.urland.filenamerecipes.
05 Writing a ruleset
Rulesets are str.format templates: {img.name} is a field, {{ and }} are braces, $$ is a dollar for the shell. Recipe lines start with + under .RECIPEPREFIX=+, which the footer resets.
If you change a block, you're editing a Python string that becomes a makefile, and three kinds of escaping meet in it. Fields in braces, such as the system's name, are filled in with str.format. So a brace you want in the output, like the ones awk needs, is written twice. A dollar sign meant for the shell is doubled as well, this time for make. And every recipe line starts with a plus sign, not a tab. The header sets .RECIPEPREFIX to plus, so the templates don't depend on tabs surviving in Python source. The footer sets it back, so whatever make reads next uses tabs again.
Details and sources
Details
- Each ruleset is a module-level string printed with
.format(img=img)or.format(group=…), so{img.name},{img.group},{img.arch}and the properties ofImageare fields (rulegen.py:777-822). - Literal braces are doubled: the awk program of the
.filenamerule is writtenawk 'BEGIN {{ … }}', and"$${{META_DATA}}"becomes"$${META_DATA}"in the rule file and"${META_DATA}"for the shell (rulegen.py:460-471, rulegen.py:546-591). $$is make's escape for one$;$@,$<,$^and$(…)are left single on purpose, for make. The row on the slide is from the.filenamerule for aURLProgramwithout a pattern (rulegen.py:538-544); the+mkdir -p $@row is the group directory rule (rulegen.py:447-450).- Values that come from the INI file and are quoted for the shell (
LocalFilename,URLPattern,LocalFilenameTemplate) go throughshlex.quoteand have every$doubled (rulegen.py:138-158).URLandURLProgramare pasted as they are, so they are written in make syntax in the INI file. - The header: "Set recipe prefix to the plus character. Rules are no longer sensitive to tabs."
.RECIPEPREFIX=+; the footer: "Reset rules to use tabs.".RECIPEPREFIX=(rulegen.py:726-738). A continuation line of a recipe, such as the-drivelines of the provisioning boot, starts with spaces. .RECIPEPREFIXneeds GNU make 3.82 or later (derived, from GNU make's documentation).
Read more
- URLProgram and local file names: the keys pasted into recipes as they are.
- How image-garden.mk is put together: what make reads before and after the rule files.
06 Testing a change to rulegen.py
make check-py runs mypy --strict; make check-rulegen runs tests/check-rulegen.sh, whose cases expect a fragment of the output or of the error. Comparing data/050-*.gen.mk before and after shows what the change did to each distribution.
Before you send a change to rulegen.py, run three checks from the top of the tree. make check-py runs mypy in strict mode over it, so every new field needs a type. make check-rulegen runs a shell script that writes tiny INI files, one test section each, runs rulegen.py on them, and expects either a piece of the rules or a piece of an error message. Each case prints PASS or FAIL. Add a case for every new key or message you introduce. And keep the rule files from before your change: build again and compare them. A distribution you didn't mean to touch should come out exactly the same.
Details and sources
Details
check-pyrunsmypy --strictonrulegen.pyand three Python test helpers;check-rulegenrunstests/check-rulegen.sh; both are part ofmake check(Makefile:66-67, Makefile:88-100). Run in a scratch copy of the pinned commit, both passed: "Success: no issues found in 4 source files" and 38PASSlines.- The script's helpers:
write_ini NAME FIELD…writes[META] Group=test Variant=cloudand one[test-cloud.x86_64]section withArch=x86_64,Boot=efiand the given fields;expect_ok NAME [PATTERN]wants rulegen.py to succeed with PATTERN, a fixed string, in its output;expect_error NAME PATTERNwants it to fail with PATTERN in the error and no Python traceback (tests/check-rulegen.sh:18-74). - The cases cover
@URL@, a missing checksum, loongarch64,LocalFilename,URLPatternandLocalFilenameTemplate, the file-name clash, the[META]keys,Arch,Bootper architecture, a duplicate section and a missing file (tests/check-rulegen.sh:76-223). - The comparison is advice, not a target of the Makefile: copy
data/050-*.gen.mkaside, rebuild withmake image-garden.mk, anddiffthe files. The file name on the slide is an example. - The other checks of
make checkneed more tools: shellcheck, the pkgproxy download tests, the allocate test, the rebase test and the unpack-lock test (Makefile:66-115).
Read more
- Try your section: the same tools for a change to an INI file.
- Every image has a checksum: the checksum cases one by one.
07 Where to go from here
Next: boot types, the target chain, how image-garden.mk is assembled, and the INI, checksum and URLProgram keys.
That's rulegen.py: it checks a whole file, then prints four blocks of thin rules for every system, and leaves the real work to macros in mk. The boot types deck shows which boot block a system gets. The target chain follows the files these rules make, in the order make builds them. To see how the rule files join the hand-written ones in image-garden.mk, take the assembly deck. And the INI, checksum and URLProgram decks explain the keys rulegen.py reads.
Details and sources
Details
- The order of a rule file: header, group directory, then per system the common, fetch, URL and boot blocks, then the footer (rulegen.py:777-822).
Read more
- Boot types, the target chain, how image-garden.mk is put together.
- The .ini files in data/, every image has a checksum, URLProgram and local file names.
- Rules written by hand: Ubuntu Core, which rulegen.py does not generate.
Read more
Basis of the statements on this page
Read in rulegen.py, the Makefile, tests/check-rulegen.sh, mk/010-qemu.mk and mk/010-qemu-x86_64.mk of image-garden at the pinned commit ("code"). The line counts, the error messages and the test results come from running make image-garden.mk, make check-py, make check-rulegen and rulegen.py on small INI files in a scratch copy of that commit ("run"). Advice on comparing rule files, and the GNU make version for .RECIPEPREFIX, are not in the source ("derived").
Command line
./rulegen.py [--systems-meta FILE] FILE.ini...
| Argument | Effect |
|---|---|
FILE.ini... | checked one after the other; the rules of each are printed to standard output |
--systems-meta FILE | also appends five variables per system to FILE; see how image-garden.mk is put together |
The Makefile runs ./rulegen.py data/X.ini >data/050-X.gen.mk for each INI file, and ./rulegen.py --systems-meta data/000-systems.mk data/X.ini >/dev/null for each to build the metadata (Makefile:57-64).
What a rule file holds
| Block | Ruleset | When | Main targets |
|---|---|---|---|
| header | ruleset_header | once per file | .RECIPEPREFIX=+ |
| group | ruleset_group | once per file, from the first section | $(GARDEN_DL_DIR)/GROUP |
| common | ruleset_common | every system | GARDEN_SYSTEMS, all::, NAME.meta-data, NAME.user-data, clean:: |
| fetch | ruleset_fetch_common | every system | fetch, distclean::, .PRECIOUS, $(GARDEN_DL_DIR)/GROUP/NAME.qcow2 |
| URL | ruleset_fetch_direct or ruleset_fetch_prog | URL or URLProgram | $(GARDEN_DL_DIR)/GROUP/NAME.url |
| file name | ruleset_filename_static, _dynamic_basename or _dynamic_pattern | by the keys | $(GARDEN_DL_DIR)/GROUP/NAME.filename |
| boot | ruleset_x86_64_efi, _x86_64_bios, _aarch64_efi, _riscv64_efi, _loongarch64_efi, ruleset_s390x, ruleset_ppc64 | by Arch and Boot | NAME.run, NAME.qcow2 |
| footer | ruleset_footer | once per file | .RECIPEPREFIX= |
Sources: rulegen.py:447-738, rulegen.py:777-822. Which boot ruleset each combination gets is tabled in boot types.
Sizes
| Lines | |
|---|---|
rules for ubuntu-cloud-24.04.x86_64 | 93 |
data/050-ubuntu.gen.mk (56 systems) | 5,224 |
all thirteen data/050-*.gen.mk (139 systems) | 13,135 |
data/000-systems.mk | 847 |
image-garden.mk | 15,889 |
How errors are reported
| Problem | Output |
|---|---|
a mistake in the data: a missing or wrong key, an unknown Arch or Boot, a Boot the architecture lacks, two sections with one static cache file name, an INI syntax error, a missing file | data/X.ini: error: MESSAGE, exit status 1, no traceback |
a lone % in a value | a Python traceback ending in configparser.InterpolationSyntaxError |
When the Makefile runs it, make then deletes the empty data/050-X.gen.mk (.DELETE_ON_ERROR, Makefile:16-18) and the next make runs rulegen.py again.
The messages for each key are listed in the .ini files in data/ and URLProgram and local file names.
The macros the rules call
| Macro or command | Defined in | Called from |
|---|---|---|
CLOUD_INIT_META_DATA_TEMPLATE | mk/010-qemu.mk | NAME.meta-data |
PICK_CLOUD_INIT_USER_DATA_TEMPLATE_FOR | mk/010-qemu.mk | NAME.user-data |
GARDEN_LOG_SECTION, GARDEN_LOG_SUBSECTION, GARDEN_SYMLINK | mk/000-garden.mk | most recipes |
image-garden download-with-… | the image-garden script | the download rule |
EMIT_QEMU_RUN_SCRIPT_QUERY_PREAMBLE | mk/010-qemu.mk | NAME.run |
EMIT_QEMU_SYSTEM_ARCH_…_RUN_COMMAND, QEMU_SYSTEM_ARCH_…_CMDLINE | mk/010-qemu-ARCH.mk | NAME.run, NAME.qcow2 |
QEMU_IMG, QEMU_IMG_SIZE | mk/010-qemu.mk | NAME.qcow2 |
Roads from here
ARCHITECTURES · machines and firmware
- Boot types — which boot block a system gets for its Arch and Boot
RUNTIME · make rules and the CLI
- From a target name to a VM — the files these rules make, in the order make builds them
BUILD TIME · data and rules
- How image-garden.mk is put together — how the rule files join the hand-written makefiles in image-garden.mk
- The .ini files in data/ — every key rulegen.py reads from a section
- Every image has a checksum — the checksum keys and the helper each one picks
- URLProgram and local file names — the URL and file-name rules for images that move
- Rules written by hand — Ubuntu Core and the other rules rulegen.py does not write