URLProgram and local file names
when the URL moves for contributors
For images without a fixed address, a section's URLProgram is a shell pipeline that make runs to write the image URL into the system's .url file; URLPattern and LocalFilenameTemplate turn that URL into the name the image is cached under. Amazon Linux and Gentoo use it.
On the map: build time
▶ Watch this deck — 6 scenes, ~5 min
Worth seeing first
- The .ini files in data/ — the keys of a section, of which URLProgram is one
01 When an image has no fixed URL
A section names its image with URL, a fixed address, or with URLProgram, a shell pipeline that prints the current one. Amazon Linux and Gentoo use URLProgram on x86_64 and aarch64.
Say you're adding a distribution whose cloud image has no fixed address. Ubuntu's is easy: the 24.04 image always sits at the same URL, so its section simply writes it down. Gentoo puts each new build in a directory named after the moment it was built. Amazon Linux puts a version number into every file name. Any URL you copied into the INI file today would point at an old build next month. So a section can say URLProgram instead of URL: a short shell pipeline that prints the current address. Six systems use it, Amazon Linux 2 and 2023 and Gentoo, each on two architectures. This deck follows gentoo-cloud.x86_64.
Details and sources
Details
[ubuntu-cloud-24.04.x86_64]writes its image address down withURL=https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img(ubuntu.ini:237-242).- The six sections with
URLProgramareamazonlinux-cloud-2andamazonlinux-cloud-2023on x86_64 and aarch64 (amazonlinux.ini:8-42) andgentoo-cloudon both (gentoo.ini:8-22). Gentoo has noRelease, so its names have none. - rulegen.py keeps the key as
url_program, "program for distributions without a stable URL" (rulegen.py:117-119), and requires exactly one ofURLandURLProgram(rulegen.py:352-363). - The two paths on the slide are what the programs printed on 21 November 2026 (run in a scratch build of the tag); the build stamps change as the distributions publish new images.
Read more
- The .ini files in data/: every other key of a section.
- Gentoo's and Amazon Linux 2's first-boot changes.
02 The program runs inside make
The pipeline becomes the recipe of $(GARDEN_DL_DIR)/gentoo/gentoo-cloud.x86_64.url. It is make syntax: $(CURL) is image-garden's curl, $$ is a dollar for the shell. The download, the @URL@ checksum address and the local file name all read the file it writes.
rulegen.py doesn't run your program. It copies it, word for word, into the recipe of a make rule whose target is the system's .url file in the download cache. So the line is written for make, not a plain shell: CURL in parentheses is image-garden's own curl with its options, and every dollar sign meant for the shell or awk is doubled. Gentoo's program fetches a short text file that names the latest build, and awk puts the mirror's address in front of its sixth line. make runs it the first time you build the system. Then three things read the .url file: the download, the checksum's address, and the local file name.
Details and sources
Details
- For a section with
URLProgram, rulegen.py writes the.urlrule with the program as its recipe,+{img.url_program} >$@, unchanged (rulegen.py:526-530, chosen at rulegen.py:793-795). A section withURLgets+echo {img.url} >$@instead (rulegen.py:520-524). - Gentoo's x86_64 program:
$(CURL) https://distfiles.gentoo.org/releases/amd64/autobuilds/latest-di-amd64-cloudinit.txt | awk 'NR==6 { print "https://distfiles.gentoo.org/releases/amd64/autobuilds/"$$1 }'(gentoo.ini:9). Amazon Linux's programs fetch the distribution'sSHA256SUMSand let awk pick the one qcow2 entry that matches (amazonlinux.ini:9). CURLiscurl --fail --silent --show-error --location, found on the PATH (000-garden.mk:21-21). make turns$$into$before the shell sees the line, so awk gets$1.- The text file and the
.urlcontent on the slide come from a run on 21 November 2026 in a scratch build of the tag; the sixth line of Gentoo's file is the build's path and size. - The
.urlfile is read by the download recipe as its first prerequisite (rulegen.py:499-514), by@URL@, which expands to$$(cat $(word 1,$^))(rulegen.py:82-88), and by the.filenamerule (rulegen.py:546-553). Gentoo's checksum key isSHA256SUMS=@URL@.sha256(gentoo.ini:12); Amazon Linux's is a fixed address.
Read more
- Every image has a checksum:
@URL@and its rules. - Four steps in the cache: the
.url,.filename, download and link of every system.
03 A name for the cache
The .filename rule names the cached image: the URL's last part by default, or LocalFilenameTemplate with @1 replaced by the one group of URLPattern. Gentoo's build is saved as gentoo-cloud-20261115T170104Z.x86_64.qcow2.
Next, make needs a name to save the image under, in the cache directory all Gentoo systems share: that's the .filename file. Ubuntu's is simply the last part of its URL. A section with a URLProgram can do better, with two more keys. URLPattern is a regular expression for the URL, with exactly one group in parentheses. LocalFilenameTemplate is the name to use, with @1 where the group's text goes. For Gentoo the group catches the build stamp, so the file is called gentoo-cloud, then the stamp, then the architecture. Each cached image says which system and build it is, and two URLs that end the same way can't overwrite each other.
Details and sources
Details
- Every system gets a rule for
$(GARDEN_DL_DIR)/GROUP/NAME.filename(rulegen.py:134-136). WithURL, it echoesLocalFilenameor the URL's basename; withURLProgram, it reads the.urlfile and takes the basename, or, when the section hasURLPatternandLocalFilenameTemplate, runs awk on it (rulegen.py:532-591, chosen at rulegen.py:782-799). - The awk program checks that the URL matches
URLPattern, cuts off the text before and after the capture group (rulegen.py works out both parts when it writes the rule), puts what is left in place of@1, and refuses an empty result,.,..or one with a/(rulegen.py:546-591, rulegen.py:216-273). - Gentoo:
URLPattern=https://distfiles.gentoo.org/releases/amd64/autobuilds/([0-9]{8}T[0-9]{6}Z)/di-amd64-cloudinit-[0-9]{8}T[0-9]{6}Z\.qcow2$,LocalFilenameTemplate=gentoo-cloud-@1.x86_64.qcow2(gentoo.ini:10-11). Amazon Linux 2023's group catches the version and leaves out the kernel (amazonlinux.ini:28-29). - The names on the slide were made by the
.urland.filenamerules of a scratch build of the tag on 21 November 2026;ubuntu-cloud-24.04.x86_64.filenameheldnoble-server-cloudimg-amd64.img. - The release notes give the reason: "to avoid cache filename clashes when different URLs share the same basename" (NEWS:402-410); data/README.md:71-88 documents the three keys.
- The checksum list still names the distribution's own file. The download helper looks the local name up in it first and falls back to the URL's basename (image-garden:508-515; NEWS:412-417).
- The placeholder is
@1, a character neither make nor the shell treats specially; a template with$1is rejected because it has no@1. rulegen.py also doubles every$in the quoted pattern and template, so an anchor such as the pattern's final$reaches awk intact (rulegen.py:138-158).
Read more
04 When upstream moves on
The .url file depends on the makefiles only, so the build found first is kept until image-garden.mk or .image-garden.mk changes. A newer build then gets its own file name and download, the system's link moves to it, and the old build stays in dl/gentoo/.
So what happens when Gentoo publishes its next build? Nothing, at first. The .url file is made again only when it's missing, or older than a makefile: image-garden.mk, or your project's .image-garden.mk. Until then, make sticks with the build it found. Upgrade image-garden, or edit your file, and the program runs again. A new build means a new URL and a new local name, so the helper downloads and verifies it as a separate file, and the system's link in the cache moves over to it. The next disk you prepare starts from the new build. The old one stays in the cache until you delete it yourself.
Details and sources
Details
- The
.urlrule's only prerequisite is$(MAKEFILE_LIST)(rulegen.py:526-530). When the generated rules are read, that list holdsimage-garden.mkand, if it exists, the project's.image-garden.mk, included by 999-local.mk:4-5. Nothing else makes make ask the distribution again. - The
.filenamerule depends on the.urlfile (rulegen.py:546-548), and the cached linkdl/gentoo/gentoo-cloud.x86_64.qcow2on both; its recipe downloads to the new local name and replaces the link in one rename withGARDEN_SYMLINK(rulegen.py:510-514, 000-garden.mk:58-64). - The prepared
gentoo-cloud.x86_64.qcow2is an overlay made withqemu-img create -bon that link (rulegen.py:601-604). That make prepares it again once the link points at a newer download is derived from the rule, not tried. distcleanremoves the file the link points at, the file.filenamenames, the link,.urland.filename(rulegen.py:486-492); earlier builds have no link and are left alone (derived).<next>on the slide stands for the newer build's stamp.- Known discrepancy: A cached download is checked again on every make after .url changes. When the
.urlfile is made again with the same address, the download rule runs again and verifies the cached image without fetching it; the documentation does not mention it.
Read more
- Where files live: what is safe to delete in the cache.
- The overlay and its provisioning boot, download and verify.
05 What rulegen.py rejects
At build time: both or neither of URL and URLProgram, LocalFilename with a program, half a pattern and template pair, a pattern without exactly one group, a template without @1. At make time: a URL that does not match. Not checked for programs: the architecture in the URL and clashing names.
rulegen.py checks these keys when image-garden itself is built, so a slip stops the build, not someone's test run. A section needs exactly one of URL and URLProgram. LocalFilename goes with URL. With a program, use the pattern and the template instead, and give both or neither. The pattern needs exactly one capture group, and the template needs @1. One mistake shows up only when make runs: if the distribution renames its files and the URL stops matching the pattern, the filename rule says so but leaves an empty file, and later runs stay stuck. And two checks skip programs entirely: that the URL names the architecture, and that no two systems share a cached file name.
Details and sources
Details
- Each message is printed as
data/FILE.ini: error: MESSAGE, one line with no traceback, and rulegen.py exits with status 1 (rulegen.py:753-757); make then deletes the emptydata/050-FILE.gen.mk. The messages on the slide were produced by running rulegen.py at the pinned commit on small INI files in a scratch directory. - Exactly one of
URLandURLProgram;URLPattern,LocalFilenameTemplateonly withURLProgram, and only as a pair;LocalFilenameonly withURL(rulegen.py:358-394). - The pattern must compile as a Python regular expression with exactly one capture group (
(?:…)groups do not count); the template must contain@1, no@2or higher, no/, and not be.or..(rulegen.py:183-213). A template written with$1fails the@1check. - At make time awk matches the URL against the pattern and stops with "URL for NAME does not match URLPattern", "Unresolved capture group in LocalFilenameTemplate for NAME" or "Invalid resolved local filename for NAME: …" (rulegen.py:561-587). awk's regular expressions are POSIX extended ones, not Python's: a pattern that compiles in Python may still behave differently in awk (derived).
- The URL must name the architecture (
amd64andarm64count for x86_64 and aarch64) only when the section hasURL(rulegen.py:406-416). - The clash check compares names known when rulegen.py runs:
LocalFilename, the URL's basename, or a template without@1(rulegen.py:160-169, rulegen.py:758-775); a clash is reported asdata/FILE.ini: error: per-group local filename collision in group 'G': 'F' used by 'S1' and 'S2'.tests/check-rulegen.shcovers the local file name keys and the clash (tests/check-rulegen.sh:111-170). A template with@1is skipped, so the architecture in Gentoo's and Amazon Linux's templates is what keeps their names apart.
Read more
- The .ini files in data/: the errors for the other keys.
- What rulegen.py emits: the order in which it checks a file.
- Known discrepancy: A failed URLProgram leaves empty .url and .filename files that make keeps. The
.urlrecipe ends in awk (rulegen.py:526-530) and the filename recipe in| tee $@(rulegen.py:546-591), so a failing program or a URL that no longer matches writes empty files, make exits 0, and later runs fail with "curl: (3) URL rejected: No host part in the URL" until a makefile changes or the files are deleted. Run at the commit in garden.commit withCURL=false.
06 Where to go from here
Next: the checksum placeholder, the target chain, downloads, the first-boot quirks of Amazon Linux 2 and Gentoo, the rest of rulegen.py's rules, and the cache.
That's URLProgram from start to finish: a pipeline in the section, a .url file in the cache, and a local name built from the pattern and the template. Gentoo's checksum sits next to its image, and the checksums deck explains the @URL@ placeholder that finds it. The target chain shows the same four cache steps for every system, and the download deck how the image is fetched, resumed and verified. Amazon Linux 2 and Gentoo also change the first boot. And to see the other rules rulegen writes for a section, or what's safe to delete from the cache, follow the last two roads.
Details and sources
Details
- The keys in one line:
URLProgram=a make recipe that prints the image URL;URLPattern=a regular expression with one capture group;LocalFilenameTemplate=a file name with@1.
Read more
Basis of the statements on this page
Read in rulegen.py, data/amazonlinux.ini, data/gentoo.ini, data/README.md, NEWS, mk/000-garden.mk and the image-garden script of image-garden at the pinned commit ("code"). The resolved URLs and file names were produced on 21 November 2026 by running the .url and .filename rules of a scratch build of the tag, and the error messages by running the pinned rulegen.py on small INI files ("run"); the build stamps change whenever the distributions publish. What happens to the cache and the prepared disk when a newer build appears is reasoned from the rules ("derived").
The keys
| Key | Goes with | Value | Checked by rulegen.py |
|---|---|---|---|
URL | a fixed address | the image URL | must contain the architecture (amd64, arm64 count) |
URLProgram | no URL | a make recipe that prints the image URL | exactly one of URL and URLProgram |
LocalFilename | URL only | the cached file name | no /, not . or .., no clash in the file |
URLPattern | URLProgram only, with the template | a regular expression with exactly one capture group | compiles; one capturing group; balanced brackets |
LocalFilenameTemplate | URLProgram only, with the pattern | a file name with @1 | contains @1, no @2 or higher, no / |
Sources: rulegen.py:183-214, rulegen.py:352-416, data/README.md:71-88.
The six systems with a URLProgram
| System | The program reads | Capture group | Cached as (21 November 2026) |
|---|---|---|---|
amazonlinux-cloud-2.x86_64 | os-images/latest/kvm/SHA256SUMS | version | amazonlinux-cloud-2-VERSION.x86_64.qcow2 |
amazonlinux-cloud-2.aarch64 | os-images/latest/kvm-arm64/SHA256SUMS | version | amazonlinux-cloud-2-VERSION.aarch64.qcow2 |
amazonlinux-cloud-2023.x86_64 | al2023/os-images/latest/kvm/SHA256SUMS | version | amazonlinux-cloud-2023-2023.12.20261120.0.x86_64.qcow2 |
amazonlinux-cloud-2023.aarch64 | al2023/os-images/latest/kvm-arm64/SHA256SUMS | version | amazonlinux-cloud-2023-VERSION.aarch64.qcow2 |
gentoo-cloud.x86_64 | latest-di-amd64-cloudinit.txt, line 6 | build stamp | gentoo-cloud-20261115T170104Z.x86_64.qcow2 |
gentoo-cloud.aarch64 | latest-di-arm64-cloudinit.txt, line 6 | build stamp | gentoo-cloud-STAMP.aarch64.qcow2 |
Amazon Linux checks its images against the same SHA256SUMS file its program reads; Gentoo uses SHA256SUMS=@URL@.sha256, the checksum file next to the image (amazonlinux.ini:8-42, gentoo.ini:8-22). Only the rows with a full name were run.
Writing a URLProgram
- It is a make recipe line:
$(CURL)iscurl --fail --silent --show-error --location(000-garden.mk:21-21); write$$for every$the shell or awk should see. - It must print one URL on standard output; make writes it to
$(GARDEN_DL_DIR)/GROUP/NAME.url(rulegen.py:526-530). - It runs when that file is missing or older than
image-garden.mkor.image-garden.mk, not on every make. - In the INI file a
%must be written%%: rulegen.py reads the files with Python's default interpolation, and a lone%stops it with anInterpolationSyntaxErrortraceback, the one data mistake still reported that way (run at the pinned commit; no file indata/has one).
The files, for gentoo-cloud.x86_64
In $(GARDEN_DL_DIR)/gentoo/, by default ~/.cache/image-garden/dl/gentoo/:
| File | Made from | Content |
|---|---|---|
gentoo-cloud.x86_64.url | the program | the image URL |
gentoo-cloud.x86_64.filename | the .url file, the pattern and the template | gentoo-cloud-STAMP.x86_64.qcow2 |
gentoo-cloud-STAMP.x86_64.qcow2 | image-garden download-with-sha256sums | the verified image |
gentoo-cloud.x86_64.qcow2 | GARDEN_SYMLINK | a link to the image |
Messages
| When | Message |
|---|---|
| build | Image 'NAME' defines both URL and URLProgram |
| build | Image 'NAME' defines neither URL nor URLProgram |
| build | Image 'NAME' defines URLPattern but uses URL |
| build | Image 'NAME' defines LocalFilenameTemplate but uses URL |
| build | Image 'NAME' defines LocalFilename but uses URLProgram; use URLPattern and LocalFilenameTemplate |
| build | Image 'NAME' must define both URLPattern and LocalFilenameTemplate together |
| build | Image 'NAME' has invalid URLPattern: … |
| build | Image 'NAME' URLPattern must contain exactly one capture group |
| build | Image 'NAME' LocalFilenameTemplate must contain '@1' placeholder |
| build | Image 'NAME' LocalFilenameTemplate supports only '@1' placeholder |
| make | URL for NAME does not match URLPattern |
| make | Unresolved capture group in LocalFilenameTemplate for NAME |
| make | Invalid resolved local filename for NAME: … |
Build-time messages are printed as data/FILE.ini: error: MESSAGE, without a traceback (rulegen.py:753-757).
Roads from here
RUNTIME · make rules and the CLI
- From a target name to a VM — the .url, .filename, download and link steps of every system
- Downloading and checking images — how the download helper fetches, resumes and verifies the image
- Where image-garden keeps its files — where the cached builds live and what is safe to delete
FIRST BOOT · cloud-init changes
- What differs per distribution — what Amazon Linux 2 and Gentoo change at first boot
BUILD TIME · data and rules
- Every image has a checksum — the @URL@ placeholder that builds a checksum address from the resolved URL
- What rulegen.py emits — the other rules rulegen.py writes for each section