Rules written by hand
mk/050-*, Ubuntu Core for contributors
Four makefiles in mk/ name particular systems. mk/050-ubuntu-core.mk replaces the generator for nineteen Ubuntu Core systems, whose .img.xz images, user, seed, template and modifiers no INI key can express; the Amazon Linux 2, Gentoo and old-Ubuntu files adjust generated systems through a target-specific variable, template names and a hook.
On the map: build time
▶ Watch this deck — 6 scenes, ~5 min
Worth seeing first
- What rulegen.py emits — the pieces the generator writes, which hand-written rules must match
01 Four makefiles written by hand
mk/050-ubuntu-core.mk defines nineteen Ubuntu Core systems on its own. mk/050-amazonlinux-2.mk, mk/050.gentoo.mk and mk/050-ubuntu.mk adjust generated systems.
Not every system fits in an INI file. Four makefiles in mk are written by hand for particular systems, and they come in two kinds. One replaces the generator for a whole family: the Ubuntu Core file, over three hundred lines, holds every rule of nineteen systems. The other three are small, and adjust systems the generator already made. Amazon Linux 2 gets its own first-boot template and, on Arm, a different CPU. Gentoo gets a template that turns on SSH. And Ubuntu's old interim releases get an apt mirror that still serves them. This deck shows how each kind plugs in, and what you take on when you write rules by hand.
Details and sources
Details
mk/050-ubuntu-core.mk(324 lines at the tag) addsubuntu-core-R.x86_64for 16, 18, 20, 22, 22+dangerous, 24, 24+dangerous, 26 and 26+dangerous, the same from 22 on for aarch64, and four+tpmsystems on x86_64: nineteen in all (050-ubuntu-core.mk:4-16).mk/050-amazonlinux-2.mk:AMAZONLINUX_2_CLOUD_INIT_USER_DATA_TEMPLATE, which addsshutdown --poweroff now, andQEMU_CPU_AARCH64=neoverse-n1for Amazon Linux's aarch64 systems (050-amazonlinux-2.mk:4-11). The CPU line names the 2023 systems too.mk/050.gentoo.mk:GENTOO_CLOUD_INIT_USER_DATA_TEMPLATE, which addssystemctl enable --now sshd.service(050.gentoo.mk:4-8).mk/050-ubuntu.mk:UBUNTU_OLD_RELEASES_VERSIONS, 16.10 to 24.10, and the apt block for old-releases.ubuntu.com throughEXTRA_CLOUD_INIT_APT_CONFIG(050-ubuntu.mk:4-28).- The other
mk/files are machinery shared by every system (QEMU, templates, instances); this deck is about the four that name particular systems.
02 What a hand-written system must provide
GARDEN_SYSTEMS, all::, fetch, meta-data, user-data, the verified download, the cache link, .qcow2, .run and clean::: Ubuntu Core writes them as static pattern rules over UBUNTU_CORE_RELEASES.
To stand in for the generator, a hand-written family has to provide every piece a generated system gets. Its names go into the list of systems, so list-systems shows them. Its disks and launchers join all, and its downloads join fetch, for your machine's architecture. Then each system needs rules for its meta-data and user-data, a download that's verified, a link in the cache named after the system, the disk, the launcher, and its lines in clean. Ubuntu Core writes most of these once, as pattern rules over a list of releases. Adding a release is one more word in that list, plus a download rule and a link.
Details and sources
Details
- What the generator writes for every system:
GARDEN_SYSTEMS,all::, meta-data, user-data andclean::(rulegen.py:452-478);fetch,distclean::and the verified download with its link (rulegen.py:480-518);.runand.qcow2(rulegen.py:593-610). - Ubuntu Core's counterparts: the release lists and
GARDEN_SYSTEMS(050-ubuntu-core.mk:4-16);.runand.qcow2as static pattern rules,$(foreach R,…): ubuntu-core-%.x86_64.run: …(050-ubuntu-core.mk:23-53); meta-data and user-data (050-ubuntu-core.mk:111-129);all::andfetch, on x86_64 and aarch64 only (050-ubuntu-core.mk:186-197); one download rule per release and channel, and one link per system (050-ubuntu-core.mk:199-308);.PRECIOUS,clean::anddistclean::(050-ubuntu-core.mk:310-328). - Between the download and the link sits a conversion that the generator never needs:
%.qcow2: %.img.xzin 010-qemu.mk:340-347 turnsubuntu-core-24-amd64.img.xzintoubuntu-core-24-amd64.qcow2, which the link points at. - The download rules use the same helper as the generated ones,
$(GARDEN_ITSELF) download-with-sha256sums, withSHA256SUMSfrom the same directory on cdimage.ubuntu.com. ubuntu-core-16,18and20have no aarch64 rules; the+tpmsystems have rules of their own, covered in Secure Boot and a TPM.
Read more
- Download and convert: what these rules do for Ubuntu Core, step by step.
- Four blocks for every system: the generated pieces.
03 Why Ubuntu Core is written by hand
Unpacking and converting .img.xz, a group that is not the user, a SCSI CD-ROM seed, its own template, and modifiers that share a download: none of these is a key in data/*.ini.
Why not just give Ubuntu Core an INI file? Because it needs things the INI schema can't say. Its images come compressed, as .img.xz, and must be unpacked and converted before anything can start from them. Its group, which names the cache directory, is ubuntu-core, but its user is ubuntu, and the generator always makes the group the user. Its seed goes in on a SCSI CD-ROM, not a virtio disk. Its first boot needs a template of its own. And its modifiers share downloads: +dangerous comes from another channel, and +tpm reuses an image through one more link, with swtpm running beside QEMU.
Details and sources
Details
- Compressed images. The downloads are
ubuntu-core-R-ARCH.img.xz;%.qcow2: %.img.xzunpacks and converts them under a lock (010-qemu.mk:340-347). The generated download rule links the downloaded file itself (rulegen.py:510-514). - Group and user. The generated rules pass the group as the template's user and as
--query's (rulegen.py:467-467, rulegen.py:596-596); Ubuntu Core passesubuntu-coreas the group andubuntuas the user, andonly:ubuntuto--query(050-ubuntu-core.mk:25-25, 050-ubuntu-core.mk:116-116). - Seed. Generated:
-device virtio-blk(rulegen.py:604-609); Ubuntu Core:-device virtio-scsi-pci,id=scsi1 -device scsi-cd,drive=drive1,bus=scsi1.0(050-ubuntu-core.mk:37-53). - Template.
BASE_UBUNTU_CORE_CLOUD_INIT_USER_DATA_TEMPLATE, picked through the keyUBUNTU_CORE(050-ubuntu-core.mk:131-179). A generated system could get a group template too; the other four points are what rule it out. - Modifiers.
+dangerousdownloads fromdangerous-stable(050-ubuntu-core.mk:223-251); a+tpmsystem links to the image of the system without it, aftercheck-swtpm-component(050-ubuntu-core.mk:280-290), and its first boot starts swtpm (050-ubuntu-core.mk:78-106).
Read more
04 Three ways into generated rules
A target-specific QEMU_CPU_AARCH64=neoverse-n1; templates named AMAZONLINUX_2_CLOUD_INIT_USER_DATA_TEMPLATE and GENTOO_CLOUD_INIT_USER_DATA_TEMPLATE; a redefined EXTRA_CLOUD_INIT_APT_CONFIG. The same works in .image-garden.mk.
The three small files never touch the generated rules. They reach them in three ways make already allows. One, a target-specific variable: Amazon Linux's Arm disks and launchers get QEMU_CPU_AARCH64 set to neoverse-n1, for those targets only. Two, a template under a name the picker looks for: AMAZONLINUX_2 matches the group and release, GENTOO the group alone. Three, a hook redefined: the common template calls the apt hook for every system, and the Ubuntu file fills it in for the releases on its list. The generated rules ask by name, and these files answer. Your project's own makefile can use the same three ways.
Details and sources
Details
- Target-specific variable.
amazonlinux-cloud-2.aarch64.qcow2 amazonlinux-cloud-2.aarch64.run amazonlinux-cloud-2023.aarch64.qcow2 amazonlinux-cloud-2023.aarch64.run : QEMU_CPU_AARCH64=neoverse-n1(050-amazonlinux-2.mk:10-11); the default ismaxand the command line and the launcher read it (010-qemu-aarch64.mk:5-5, :14-14, :62-62). A target-specific value applies to that target's recipe and its prerequisites. - Template names. The generated user-data rule calls the picker with the upper-case group and the release (rulegen.py:467-467); the picker tries
GROUP_RELEASE@INSTANCE,GROUP@INSTANCE,GROUP_RELEASE,GROUP, then the common template (010-qemu.mk:293-298).AMAZONLINUX_2_…is a group-and-release name; Gentoo has no release, soGENTOO_…is the group name (050-amazonlinux-2.mk:4-8, 050.gentoo.mk:4-8). - Hook.
EXTRA_CLOUD_INIT_APT_CONFIG =is empty in 010-qemu.mk:47-47 and called with the group and release by the base template (010-qemu.mk:198-198); 050-ubuntu.mk:12-28 redefines it to emit an apt block forubuntureleases inUBUNTU_OLD_RELEASES_VERSIONS. - A project that defines the same template name or hook in
.image-garden.mkreplaces these, because its file is read later (what differs per distribution shows what to keep). - Known discrepancy: Quirk templates start from the base template. The Amazon Linux 2 and Gentoo templates build on
BASE_CLOUD_INIT_USER_DATA_TEMPLATE, notCLOUD_INIT_USER_DATA_TEMPLATE, so a project that redefines the common template does not change these two systems. This may be intended.
05 What you take on
Hand-written rules get no rulegen.py checks, keep their own lists, repeat what the generator writes (set -o pipefail, all::, fetch, distclean::), so a fix has to be made twice, and have no entry in data/000-systems.mk.
Writing rules by hand buys freedom, and it costs you in four ways. No checks: rulegen.py refuses a section without a checksum, or with a suspicious URL, but nothing looks over a hand-written download rule. Lists to keep in step: the old Ubuntu releases are a list whose comment asks you to keep it in sync with the INI file, and Ubuntu Core names each release again in its download rules and links. Drift: a change to the generated rules doesn't reach these. Ubuntu Core repeats the pipefail before QEMU, all, fetch and distclean, so a fix to them is made twice. And no metadata, so Ubuntu Core can't be the base of a named instance.
Details and sources
Details
- Checks. rulegen.py's checks run on INI sections only (rulegen.py:336-445). Ubuntu Core's download rules are written out in full, one per release and channel (050-ubuntu-core.mk:199-251); the file checks only the order of modifiers and
+tpmon other architectures (050-ubuntu-core.mk:20-21). - Lists.
UBUNTU_OLD_RELEASES_VERSIONSand its comment, "Keep this list in sync with EOL ubuntu-cloud entries in data/ubuntu.ini." (050-ubuntu.mk:4-12). Ubuntu Core's releases are listed inUBUNTU_CORE_RELEASESand again in each download and link rule (050-ubuntu-core.mk:4-7, :199-308). - Drift. The generated provisioning boots run QEMU as
set -o pipefail && … | tee(rulegen.py:606); Ubuntu Core's.qcow2rules write the same again, withSHELL = bash(050-ubuntu-core.mk:37-53). Itsall::andfetchlines, limited to x86_64 and aarch64 (050-ubuntu-core.mk:186-197), and itsdistclean::, which removes the.img.xzdownloads and the converted.qcow2(050-ubuntu-core.mk:326-328), stand in for the generatedfetchanddistclean::(rulegen.py:480-492). A change to one does not reach the other (derived). - Metadata.
data/000-systems.mkis written from the INI files only (rulegen.py:824-833), anddefine-instancereads the base's group, release, boot type and architecture from it (020-instances.mk:29-56). Its comment and the README say so: instances are made of the systems indata/*.ini, "which excludes Ubuntu Core";define-instancewith an Ubuntu Core system lists the instance inlist-systems, but make has no rule for it (020-instances.mk:4-10, README.md:976-979). Run in a scratch tree with$(eval $(call define-instance,ubuntu-core-24.x86_64,web)), make said "No rule to make target" for its.run. - The same costs apply to systems a project defines by hand in
.image-garden.mk(derived).
Read more
- What it inherits: named instances and the metadata.
- Why the metadata comes first.
06 Where to go from here
Next: Ubuntu Core, the per-distribution quirks, how image-garden.mk is assembled, rulegen.py, Secure Boot and the TPM, and named instances.
So there are two ways to write rules by hand: a whole family, as Ubuntu Core does, or a small file that answers when the generated rules ask by name. The Ubuntu Core deck shows what its rules do, step by step, and the quirks deck what the three small files change at first boot. The assembly deck explains where these files sit in image-garden.mk, and the rulegen deck what the generator writes instead. Secure Boot and the TPM have rules of their own, and named instances need the metadata Ubuntu Core lacks.
Details and sources
Details
- The four hand-written system files at the tag:
mk/050-ubuntu-core.mk,mk/050-amazonlinux-2.mk,mk/050.gentoo.mkandmk/050-ubuntu.mk.
Roads from here
UBUNTU CORE · all snaps
- Ubuntu Core images — what the Ubuntu Core rules do, step by step
- Ubuntu Core with Secure Boot and a TPM — the +tpm rules, with Secure Boot and swtpm
RUNTIME · make rules and the CLI
- Named instances — named instances, which need the metadata Ubuntu Core lacks
FIRST BOOT · cloud-init changes
- What differs per distribution — what the three small files change at first boot
BUILD TIME · data and rules
- How image-garden.mk is put together — where the hand-written files sit in image-garden.mk
- What rulegen.py emits — what the generator writes instead