Rules written by hand

mk/050-*, Ubuntu Core for contributors

Four makefiles in mk/ name particular systems. mk/050-ubuntu-core.mk replaces the generator for nineteen Ubuntu Core systems, whose .img.xz images, user, seed, template and modifiers no INI key can express; the Amazon Linux 2, Gentoo and old-Ubuntu files adjust generated systems through a target-specific variable, template names and a hook.

On the map: build time

▶ Watch this deck — 6 scenes, ~5 min

Worth seeing first

01 Four makefiles written by hand

TWO KINDS replaces the generator adjusts generated systems mk/050-ubuntu-core.mk, 324 lines ubuntu-core-16.x86_64 ubuntu-core-18.x86_64 … ubuntu-core-24.aarch64 ubuntu-core-24+dangerous.x86_64 ubuntu-core-24+tpm.x86_64 … 19 systems, no INI file mk/050-amazonlinux-2.mk, 11 lines amazonlinux-cloud-2: template, Arm CPU mk/050.gentoo.mk, 8 lines gentoo-cloud: a template that starts sshd mk/050-ubuntu.mk, 28 lines ubuntu-cloud 16.10 to 24.10: old-releases how each kind plugs in, and what writing by hand costs
Four hand-written makefiles. mk/050-ubuntu-core.mk, 324 lines, replaces the generator for nineteen Ubuntu Core systems. Three small files adjust generated systems: mk/050-amazonlinux-2.mk the amazonlinux-cloud-2 systems, mk/050.gentoo.mk the gentoo-cloud systems, and mk/050-ubuntu.mk the end-of-life Ubuntu interim releases

mk/050-ubuntu-core.mk defines nineteen Ubuntu Core systems on its own. mk/050-amazonlinux-2.mk, mk/050.gentoo.mk and mk/050-ubuntu.mk adjust generated systems.

Not every system fits in an INI file. Four makefiles in mk are written by hand for particular systems, and they come in two kinds. One replaces the generator for a whole family: the Ubuntu Core file, over three hundred lines, holds every rule of nineteen systems. The other three are small, and adjust systems the generator already made. Amazon Linux 2 gets its own first-boot template and, on Arm, a different CPU. Gentoo gets a template that turns on SSH. And Ubuntu's old interim releases get an apt mirror that still serves them. This deck shows how each kind plugs in, and what you take on when you write rules by hand.

Details and sources

Details

  • mk/050-ubuntu-core.mk (324 lines at the tag) adds ubuntu-core-R.x86_64 for 16, 18, 20, 22, 22+dangerous, 24, 24+dangerous, 26 and 26+dangerous, the same from 22 on for aarch64, and four +tpm systems on x86_64: nineteen in all (050-ubuntu-core.mk:4-16).
  • mk/050-amazonlinux-2.mk: AMAZONLINUX_2_CLOUD_INIT_USER_DATA_TEMPLATE, which adds shutdown --poweroff now, and QEMU_CPU_AARCH64=neoverse-n1 for Amazon Linux's aarch64 systems (050-amazonlinux-2.mk:4-11). The CPU line names the 2023 systems too.
  • mk/050.gentoo.mk: GENTOO_CLOUD_INIT_USER_DATA_TEMPLATE, which adds systemctl enable --now sshd.service (050.gentoo.mk:4-8).
  • mk/050-ubuntu.mk: UBUNTU_OLD_RELEASES_VERSIONS, 16.10 to 24.10, and the apt block for old-releases.ubuntu.com through EXTRA_CLOUD_INIT_APT_CONFIG (050-ubuntu.mk:4-28).
  • The other mk/ files are machinery shared by every system (QEMU, templates, instances); this deck is about the four that name particular systems.

02 What a hand-written system must provide

EVERY PIECE, BY HAND generated system mk/050-ubuntu-core.mk in the list of systems GARDEN_SYSTEMS += $(foreach r,$(UBUNTU_CORE_RELEASES),…) all and fetch all:: ….qcow2 ….run fetch: ….img.xz meta-data, user-data ubuntu-core-%.x86_64.user-data: export USER_DATA = … verified download …/ubuntu-core-24-%.img.xz: … download-with-sha256sums link in the cache …/ubuntu-core-24.x86_64.qcow2: … GARDEN_SYMLINK disk and launcher $(foreach R,…): ubuntu-core-%.x86_64.qcow2: … clean clean:: rm -f $(foreach R,$(UBUNTU_CORE_RELEASES),…) UBUNTU_CORE_RELEASES = 16 18 20 22 22+dangerous 24 24+dangerous 26 26+dangerous a new release: one more word, a download rule and a link
Each piece a generated system gets, and where mk/050-ubuntu-core.mk provides it: GARDEN_SYSTEMS through foreach over the releases, all:: and fetch, meta-data and user-data pattern rules, a download with download-with-sha256sums per release, a GARDEN_SYMLINK link per system, the .qcow2 and .run static pattern rules, and clean::. Adding a release means a word in UBUNTU_CORE_RELEASES, a download rule and a link

GARDEN_SYSTEMS, all::, fetch, meta-data, user-data, the verified download, the cache link, .qcow2, .run and clean::: Ubuntu Core writes them as static pattern rules over UBUNTU_CORE_RELEASES.

To stand in for the generator, a hand-written family has to provide every piece a generated system gets. Its names go into the list of systems, so list-systems shows them. Its disks and launchers join all, and its downloads join fetch, for your machine's architecture. Then each system needs rules for its meta-data and user-data, a download that's verified, a link in the cache named after the system, the disk, the launcher, and its lines in clean. Ubuntu Core writes most of these once, as pattern rules over a list of releases. Adding a release is one more word in that list, plus a download rule and a link.

Details and sources

Details

  • What the generator writes for every system: GARDEN_SYSTEMS, all::, meta-data, user-data and clean:: (rulegen.py:452-478); fetch, distclean:: and the verified download with its link (rulegen.py:480-518); .run and .qcow2 (rulegen.py:593-610).
  • Ubuntu Core's counterparts: the release lists and GARDEN_SYSTEMS (050-ubuntu-core.mk:4-16); .run and .qcow2 as static pattern rules, $(foreach R,…): ubuntu-core-%.x86_64.run: … (050-ubuntu-core.mk:23-53); meta-data and user-data (050-ubuntu-core.mk:111-129); all:: and fetch, on x86_64 and aarch64 only (050-ubuntu-core.mk:186-197); one download rule per release and channel, and one link per system (050-ubuntu-core.mk:199-308); .PRECIOUS, clean:: and distclean:: (050-ubuntu-core.mk:310-328).
  • Between the download and the link sits a conversion that the generator never needs: %.qcow2: %.img.xz in 010-qemu.mk:340-347 turns ubuntu-core-24-amd64.img.xz into ubuntu-core-24-amd64.qcow2, which the link points at.
  • The download rules use the same helper as the generated ones, $(GARDEN_ITSELF) download-with-sha256sums, with SHA256SUMS from the same directory on cdimage.ubuntu.com.
  • ubuntu-core-16, 18 and 20 have no aarch64 rules; the +tpm systems have rules of their own, covered in Secure Boot and a TPM.

Read more

03 Why Ubuntu Core is written by hand

WHAT AN INI FILE CANNOT SAY the generated rules Ubuntu Core needs start from the downloaded image .img.xz: unxz, then qemu-img convert, once the group is the user group ubuntu-core, user ubuntu seed on virtio-blk -device virtio-scsi-pci, -device scsi-cd the common user-data template UBUNTU_CORE_CLOUD_INIT_USER_DATA_TEMPLATE one image per section +dangerous: the dangerous-stable channel +tpm: the same image, a link, and swtpm
What the generated rules do and what Ubuntu Core needs instead: a downloaded qcow2 or raw image against an .img.xz to unpack and convert; the group as the user against group ubuntu-core with user ubuntu; the seed on virtio-blk against a SCSI CD-ROM; the common template against its own; one image per section against +dangerous from another channel and +tpm sharing a download, with swtpm

Unpacking and converting .img.xz, a group that is not the user, a SCSI CD-ROM seed, its own template, and modifiers that share a download: none of these is a key in data/*.ini.

Why not just give Ubuntu Core an INI file? Because it needs things the INI schema can't say. Its images come compressed, as .img.xz, and must be unpacked and converted before anything can start from them. Its group, which names the cache directory, is ubuntu-core, but its user is ubuntu, and the generator always makes the group the user. Its seed goes in on a SCSI CD-ROM, not a virtio disk. Its first boot needs a template of its own. And its modifiers share downloads: +dangerous comes from another channel, and +tpm reuses an image through one more link, with swtpm running beside QEMU.

Details and sources

Details

04 Three ways into generated rules

ASKED BY NAME hand-written, in mk/050-* where the generated rules read it 1 a target-specific variable amazonlinux-cloud-2.aarch64.qcow2 … : QEMU_CPU_AARCH64=neoverse-n1 -cpu $(QEMU_CPU_AARCH64) 2 a template the picker looks for define AMAZONLINUX_2_CLOUD_INIT_USER_… define GENTOO_CLOUD_INIT_USER_DATA_… PICK_…_FOR,AMAZONLINUX,2 3 a hook redefined define EXTRA_CLOUD_INIT_APT_CONFIG $(call UBUNTU_OLD_RELEASES_APT_CONFIG,…) $(call EXTRA_CLOUD_INIT_APT_CONFIG,…) the generated rules ask by name; these files answer and so can your project's .image-garden.mk
Three ways the small files reach generated rules. A target-specific variable: amazonlinux-cloud-2.aarch64.qcow2 and the other Amazon Linux aarch64 targets get QEMU_CPU_AARCH64=neoverse-n1, which the aarch64 command line reads. A template name: AMAZONLINUX_2_ and GENTOO_CLOUD_INIT_USER_DATA_TEMPLATE, which the picker finds for group and release or group. A redefined hook: EXTRA_CLOUD_INIT_APT_CONFIG, which the base template calls. The generated rules ask by name; .image-garden.mk can answer the same way

A target-specific QEMU_CPU_AARCH64=neoverse-n1; templates named AMAZONLINUX_2_CLOUD_INIT_USER_DATA_TEMPLATE and GENTOO_CLOUD_INIT_USER_DATA_TEMPLATE; a redefined EXTRA_CLOUD_INIT_APT_CONFIG. The same works in .image-garden.mk.

The three small files never touch the generated rules. They reach them in three ways make already allows. One, a target-specific variable: Amazon Linux's Arm disks and launchers get QEMU_CPU_AARCH64 set to neoverse-n1, for those targets only. Two, a template under a name the picker looks for: AMAZONLINUX_2 matches the group and release, GENTOO the group alone. Three, a hook redefined: the common template calls the apt hook for every system, and the Ubuntu file fills it in for the releases on its list. The generated rules ask by name, and these files answer. Your project's own makefile can use the same three ways.

Details and sources

Details

  • Target-specific variable. amazonlinux-cloud-2.aarch64.qcow2 amazonlinux-cloud-2.aarch64.run amazonlinux-cloud-2023.aarch64.qcow2 amazonlinux-cloud-2023.aarch64.run : QEMU_CPU_AARCH64=neoverse-n1 (050-amazonlinux-2.mk:10-11); the default is max and the command line and the launcher read it (010-qemu-aarch64.mk:5-5, :14-14, :62-62). A target-specific value applies to that target's recipe and its prerequisites.
  • Template names. The generated user-data rule calls the picker with the upper-case group and the release (rulegen.py:467-467); the picker tries GROUP_RELEASE@INSTANCE, GROUP@INSTANCE, GROUP_RELEASE, GROUP, then the common template (010-qemu.mk:293-298). AMAZONLINUX_2_… is a group-and-release name; Gentoo has no release, so GENTOO_… is the group name (050-amazonlinux-2.mk:4-8, 050.gentoo.mk:4-8).
  • Hook. EXTRA_CLOUD_INIT_APT_CONFIG = is empty in 010-qemu.mk:47-47 and called with the group and release by the base template (010-qemu.mk:198-198); 050-ubuntu.mk:12-28 redefines it to emit an apt block for ubuntu releases in UBUNTU_OLD_RELEASES_VERSIONS.
  • A project that defines the same template name or hook in .image-garden.mk replaces these, because its file is read later (what differs per distribution shows what to keep).
  • Known discrepancy: Quirk templates start from the base template. The Amazon Linux 2 and Gentoo templates build on BASE_CLOUD_INIT_USER_DATA_TEMPLATE, not CLOUD_INIT_USER_DATA_TEMPLATE, so a project that redefines the common template does not change these two systems. This may be intended.

05 What you take on

FOUR COSTS no checks …/ubuntu-core-24-%.img.xz: download-with-sha256sums \ https://cdimage.ubuntu.com/… rulegen.py never sees these lists to keep in step UBUNTU_OLD_RELEASES_VERSIONS = 16.10 17.04 … 24.10 # Keep this list in sync with EOL # ubuntu-cloud entries in data/ubuntu.ini drift generated, by rulegen.py: set -o pipefail && qemu … | tee Ubuntu Core, written again: set -o pipefail && qemu … | tee and all::, fetch, distclean: a fix, twice no metadata define-instance,ubuntu-core-24.x86_64,web No rule to make target 'ubuntu-core-24.x86_64@web.….run' no entry in data/000-systems.mk
Four costs of hand-written rules. No checks: Ubuntu Core's download rules are not checked by rulegen.py. Lists to keep in step: UBUNTU_OLD_RELEASES_VERSIONS, whose comment asks to keep it in sync with data/ubuntu.ini. Drift: Ubuntu Core writes set -o pipefail before QEMU | tee, all::, fetch and distclean again, so a fix to the generated rules has to be made twice. No metadata: define-instance on ubuntu-core-24.x86_64 leaves make with no rule for the instance's .run

Hand-written rules get no rulegen.py checks, keep their own lists, repeat what the generator writes (set -o pipefail, all::, fetch, distclean::), so a fix has to be made twice, and have no entry in data/000-systems.mk.

Writing rules by hand buys freedom, and it costs you in four ways. No checks: rulegen.py refuses a section without a checksum, or with a suspicious URL, but nothing looks over a hand-written download rule. Lists to keep in step: the old Ubuntu releases are a list whose comment asks you to keep it in sync with the INI file, and Ubuntu Core names each release again in its download rules and links. Drift: a change to the generated rules doesn't reach these. Ubuntu Core repeats the pipefail before QEMU, all, fetch and distclean, so a fix to them is made twice. And no metadata, so Ubuntu Core can't be the base of a named instance.

Details and sources

Details

  • Checks. rulegen.py's checks run on INI sections only (rulegen.py:336-445). Ubuntu Core's download rules are written out in full, one per release and channel (050-ubuntu-core.mk:199-251); the file checks only the order of modifiers and +tpm on other architectures (050-ubuntu-core.mk:20-21).
  • Lists. UBUNTU_OLD_RELEASES_VERSIONS and its comment, "Keep this list in sync with EOL ubuntu-cloud entries in data/ubuntu.ini." (050-ubuntu.mk:4-12). Ubuntu Core's releases are listed in UBUNTU_CORE_RELEASES and again in each download and link rule (050-ubuntu-core.mk:4-7, :199-308).
  • Drift. The generated provisioning boots run QEMU as set -o pipefail && … | tee (rulegen.py:606); Ubuntu Core's .qcow2 rules write the same again, with SHELL = bash (050-ubuntu-core.mk:37-53). Its all:: and fetch lines, limited to x86_64 and aarch64 (050-ubuntu-core.mk:186-197), and its distclean::, which removes the .img.xz downloads and the converted .qcow2 (050-ubuntu-core.mk:326-328), stand in for the generated fetch and distclean:: (rulegen.py:480-492). A change to one does not reach the other (derived).
  • Metadata. data/000-systems.mk is written from the INI files only (rulegen.py:824-833), and define-instance reads the base's group, release, boot type and architecture from it (020-instances.mk:29-56). Its comment and the README say so: instances are made of the systems in data/*.ini, "which excludes Ubuntu Core"; define-instance with an Ubuntu Core system lists the instance in list-systems, but make has no rule for it (020-instances.mk:4-10, README.md:976-979). Run in a scratch tree with $(eval $(call define-instance,ubuntu-core-24.x86_64,web)), make said "No rule to make target" for its .run.
  • The same costs apply to systems a project defines by hand in .image-garden.mk (derived).

Read more

06 Where to go from here

ROADS Rules by hand Ubuntu Core images download, convert, template Per-distribution quirks AL2, Gentoo, old Ubuntu image-garden.mk where these files sit What rulegen.py emits the generated rules Secure Boot and TPM the +tpm rules Named instances need the metadata
Six roads from this deck: Ubuntu Core images, the per-distribution quirks, how image-garden.mk is assembled, what rulegen.py emits, Secure Boot and the TPM, and named instances

Next: Ubuntu Core, the per-distribution quirks, how image-garden.mk is assembled, rulegen.py, Secure Boot and the TPM, and named instances.

So there are two ways to write rules by hand: a whole family, as Ubuntu Core does, or a small file that answers when the generated rules ask by name. The Ubuntu Core deck shows what its rules do, step by step, and the quirks deck what the three small files change at first boot. The assembly deck explains where these files sit in image-garden.mk, and the rulegen deck what the generator writes instead. Secure Boot and the TPM have rules of their own, and named instances need the metadata Ubuntu Core lacks.

Details and sources

Details

  • The four hand-written system files at the tag: mk/050-ubuntu-core.mk, mk/050-amazonlinux-2.mk, mk/050.gentoo.mk and mk/050-ubuntu.mk.

Roads from here

UBUNTU CORE · all snaps

RUNTIME · make rules and the CLI

  • Named instances — named instances, which need the metadata Ubuntu Core lacks

FIRST BOOT · cloud-init changes

BUILD TIME · data and rules