URLProgram and local file names

when the URL moves for contributors

For images without a fixed address, a section's URLProgram is a shell pipeline that make runs to write the image URL into the system's .url file; URLPattern and LocalFilenameTemplate turn that URL into the name the image is cached under. Amazon Linux and Gentoo use it.

On the map: build time

▶ Watch this deck — 6 scenes, ~5 min

Worth seeing first

01 When an image has no fixed URL

NO FIXED URL [ubuntu-cloud-24.04.x86_64] URL=https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img Ubuntu: the same address for every build Gentoo, a directory per build …/autobuilds/20261115T170104Z/di-amd64-cloudinit-20261115T170104Z.qcow2 Amazon Linux 2023, a version in every name …/kvm/al2023-kvm-2023.12.20261120.0-kernel-6.1-x86_64.xfs.gpt.qcow2 [gentoo-cloud.x86_64] URLProgram=$(CURL) https://distfiles.gentoo.org/…/latest-di-amd64-cloudinit.txt | awk … amazonlinux-cloud-2.x86_64 amazonlinux-cloud-2023.x86_64 gentoo-cloud.x86_64 amazonlinux-cloud-2.aarch64 amazonlinux-cloud-2023.aarch64 gentoo-cloud.aarch64
Ubuntu 24.04's section writes down a fixed URL. Gentoo's image path carries the build's date and time, and Amazon Linux's file name a version, so neither can be written down once. Gentoo's section says URLProgram instead, a curl and awk pipeline. Six systems use it: Amazon Linux 2 and 2023 and Gentoo, on x86_64 and aarch64; gentoo-cloud.x86_64 is highlighted

A section names its image with URL, a fixed address, or with URLProgram, a shell pipeline that prints the current one. Amazon Linux and Gentoo use URLProgram on x86_64 and aarch64.

Say you're adding a distribution whose cloud image has no fixed address. Ubuntu's is easy: the 24.04 image always sits at the same URL, so its section simply writes it down. Gentoo puts each new build in a directory named after the moment it was built. Amazon Linux puts a version number into every file name. Any URL you copied into the INI file today would point at an old build next month. So a section can say URLProgram instead of URL: a short shell pipeline that prints the current address. Six systems use it, Amazon Linux 2 and 2023 and Gentoo, each on two architectures. This deck follows gentoo-cloud.x86_64.

Details and sources

Details

  • [ubuntu-cloud-24.04.x86_64] writes its image address down with URL=https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img (ubuntu.ini:237-242).
  • The six sections with URLProgram are amazonlinux-cloud-2 and amazonlinux-cloud-2023 on x86_64 and aarch64 (amazonlinux.ini:8-42) and gentoo-cloud on both (gentoo.ini:8-22). Gentoo has no Release, so its names have none.
  • rulegen.py keeps the key as url_program, "program for distributions without a stable URL" (rulegen.py:117-119), and requires exactly one of URL and URLProgram (rulegen.py:352-363).
  • The two paths on the slide are what the programs printed on 21 November 2026 (run in a scratch build of the tag); the build stamps change as the distributions publish new images.

02 The program runs inside make

URLPROGRAM BECOMES A RECIPE data/gentoo.ini URLProgram=$(CURL) https://…/latest-di-amd64-cloudinit.txt | awk 'NR==6 { … }' copied word for word by rulegen.py $(GARDEN_DL_DIR)/gentoo/gentoo-cloud.x86_64.url: $(MAKEFILE_LIST) | … +$(CURL) https://…/latest-di-amd64-cloudinit.txt | awk 'NR==6 { print "https://…/autobuilds/"$$1 }' >$@ make syntax: $(CURL) is image-garden's curl, $$ is a dollar for awk # Latest as of Sat, 21 Nov 2026 00:30:01 +0000 # ts=1795221001 20261115T170104Z/di-amd64-cloudinit-20261115T170104Z.qcow2 1205665792 lines 4 to 6 .url: https://distfiles.gentoo.org/…/20261115T170104Z/di-amd64-cloudinit-20261115T170104Z.qcow2 the download SHA256SUMS=@URL@.sha256 the local file name
Gentoo's URLProgram line is copied word for word into the recipe of the rule for gentoo-cloud.x86_64.url in the download cache, with $(CURL) and doubled dollar signs. The program reads a short text file whose sixth line names the latest build, and writes the full address into the .url file. Three things read that file: the download, the checksum address and the local file name

The pipeline becomes the recipe of $(GARDEN_DL_DIR)/gentoo/gentoo-cloud.x86_64.url. It is make syntax: $(CURL) is image-garden's curl, $$ is a dollar for the shell. The download, the @URL@ checksum address and the local file name all read the file it writes.

rulegen.py doesn't run your program. It copies it, word for word, into the recipe of a make rule whose target is the system's .url file in the download cache. So the line is written for make, not a plain shell: CURL in parentheses is image-garden's own curl with its options, and every dollar sign meant for the shell or awk is doubled. Gentoo's program fetches a short text file that names the latest build, and awk puts the mirror's address in front of its sixth line. make runs it the first time you build the system. Then three things read the .url file: the download, the checksum's address, and the local file name.

Details and sources

Details

  • For a section with URLProgram, rulegen.py writes the .url rule with the program as its recipe, +{img.url_program} >$@, unchanged (rulegen.py:526-530, chosen at rulegen.py:793-795). A section with URL gets +echo {img.url} >$@ instead (rulegen.py:520-524).
  • Gentoo's x86_64 program: $(CURL) https://distfiles.gentoo.org/releases/amd64/autobuilds/latest-di-amd64-cloudinit.txt | awk 'NR==6 { print "https://distfiles.gentoo.org/releases/amd64/autobuilds/"$$1 }' (gentoo.ini:9). Amazon Linux's programs fetch the distribution's SHA256SUMS and let awk pick the one qcow2 entry that matches (amazonlinux.ini:9).
  • CURL is curl --fail --silent --show-error --location, found on the PATH (000-garden.mk:21-21). make turns $$ into $ before the shell sees the line, so awk gets $1.
  • The text file and the .url content on the slide come from a run on 21 November 2026 in a scratch build of the tag; the sixth line of Gentoo's file is the build's path and size.
  • The .url file is read by the download recipe as its first prerequisite (rulegen.py:499-514), by @URL@, which expands to $$(cat $(word 1,$^)) (rulegen.py:82-88), and by the .filename rule (rulegen.py:546-553). Gentoo's checksum key is SHA256SUMS=@URL@.sha256 (gentoo.ini:12); Amazon Linux's is a fixed address.

Read more

03 A name for the cache

THE .FILENAME FILE gentoo-cloud.x86_64.url, in the cache directory dl/gentoo/ https://distfiles.gentoo.org/…/20261115T170104Z/di-amd64-cloudinit-20261115T170104Z.qcow2 Ubuntu: the URL's last part, noble-server-cloudimg-amd64.img URLPattern=https://…/autobuilds/([0-9]{8}T[0-9]{6}Z)/di-amd64-cloudinit-[0-9]{8}T[0-9]{6}Z\.qcow2$ LocalFilenameTemplate=gentoo-cloud-@1.x86_64.qcow2 awk, when make runs .filename: gentoo-cloud-20261115T170104Z.x86_64.qcow2 Amazon Linux 2023, the same way from: al2023-kvm-2023.12.20261120.0-kernel-6.1-x86_64.xfs.gpt.qcow2 to: amazonlinux-cloud-2023-2023.12.20261120.0.x86_64.qcow2
The .url file holds Gentoo's resolved address. Ubuntu's cached name is the last part of its URL. Gentoo's section adds URLPattern, a regular expression with one group around the build stamp, and LocalFilenameTemplate, gentoo-cloud-@1.x86_64.qcow2. The .filename file gets gentoo-cloud-20261115T170104Z.x86_64.qcow2; Amazon Linux 2023's image is saved as amazonlinux-cloud-2023-2023.12.20261120.0.x86_64.qcow2

The .filename rule names the cached image: the URL's last part by default, or LocalFilenameTemplate with @1 replaced by the one group of URLPattern. Gentoo's build is saved as gentoo-cloud-20261115T170104Z.x86_64.qcow2.

Next, make needs a name to save the image under, in the cache directory all Gentoo systems share: that's the .filename file. Ubuntu's is simply the last part of its URL. A section with a URLProgram can do better, with two more keys. URLPattern is a regular expression for the URL, with exactly one group in parentheses. LocalFilenameTemplate is the name to use, with @1 where the group's text goes. For Gentoo the group catches the build stamp, so the file is called gentoo-cloud, then the stamp, then the architecture. Each cached image says which system and build it is, and two URLs that end the same way can't overwrite each other.

Details and sources

Details

  • Every system gets a rule for $(GARDEN_DL_DIR)/GROUP/NAME.filename (rulegen.py:134-136). With URL, it echoes LocalFilename or the URL's basename; with URLProgram, it reads the .url file and takes the basename, or, when the section has URLPattern and LocalFilenameTemplate, runs awk on it (rulegen.py:532-591, chosen at rulegen.py:782-799).
  • The awk program checks that the URL matches URLPattern, cuts off the text before and after the capture group (rulegen.py works out both parts when it writes the rule), puts what is left in place of @1, and refuses an empty result, ., .. or one with a / (rulegen.py:546-591, rulegen.py:216-273).
  • Gentoo: URLPattern=https://distfiles.gentoo.org/releases/amd64/autobuilds/([0-9]{8}T[0-9]{6}Z)/di-amd64-cloudinit-[0-9]{8}T[0-9]{6}Z\.qcow2$, LocalFilenameTemplate=gentoo-cloud-@1.x86_64.qcow2 (gentoo.ini:10-11). Amazon Linux 2023's group catches the version and leaves out the kernel (amazonlinux.ini:28-29).
  • The names on the slide were made by the .url and .filename rules of a scratch build of the tag on 21 November 2026; ubuntu-cloud-24.04.x86_64.filename held noble-server-cloudimg-amd64.img.
  • The release notes give the reason: "to avoid cache filename clashes when different URLs share the same basename" (NEWS:402-410); data/README.md:71-88 documents the three keys.
  • The checksum list still names the distribution's own file. The download helper looks the local name up in it first and falls back to the URL's basename (image-garden:508-515; NEWS:412-417).
  • The placeholder is @1, a character neither make nor the shell treats specially; a template with $1 is rejected because it has no @1. rulegen.py also doubles every $ in the quoted pattern and template, so an anchor such as the pattern's final $ reaches awk intact (rulegen.py:138-158).

04 When upstream moves on

A NEWER BUILD build 20261115T170104Z Gentoo's next build gentoo-cloud.x86_64.url made again only when older than image-garden.mk .image-garden.mk URLProgram runs again a new .url a new .filename download and verify dl/gentoo/gentoo-cloud.x86_64.qcow2 -> gentoo-cloud-<next>.x86_64.qcow2 dl/gentoo/gentoo-cloud-<next>.x86_64.qcow2 dl/gentoo/gentoo-cloud-20261115T170104Z.x86_64.qcow2 the next disk starts here stays until you delete it
Gentoo publishes a newer build, but gentoo-cloud.x86_64.url is made again only when it is older than image-garden.mk or .image-garden.mk. When one of them changes, URLProgram runs again; the new URL gives a new .filename and a separate download and verification. In dl/gentoo/ the system's link moves to the new file, and the old build stays until you delete it

The .url file depends on the makefiles only, so the build found first is kept until image-garden.mk or .image-garden.mk changes. A newer build then gets its own file name and download, the system's link moves to it, and the old build stays in dl/gentoo/.

So what happens when Gentoo publishes its next build? Nothing, at first. The .url file is made again only when it's missing, or older than a makefile: image-garden.mk, or your project's .image-garden.mk. Until then, make sticks with the build it found. Upgrade image-garden, or edit your file, and the program runs again. A new build means a new URL and a new local name, so the helper downloads and verifies it as a separate file, and the system's link in the cache moves over to it. The next disk you prepare starts from the new build. The old one stays in the cache until you delete it yourself.

Details and sources

Details

  • The .url rule's only prerequisite is $(MAKEFILE_LIST) (rulegen.py:526-530). When the generated rules are read, that list holds image-garden.mk and, if it exists, the project's .image-garden.mk, included by 999-local.mk:4-5. Nothing else makes make ask the distribution again.
  • The .filename rule depends on the .url file (rulegen.py:546-548), and the cached link dl/gentoo/gentoo-cloud.x86_64.qcow2 on both; its recipe downloads to the new local name and replaces the link in one rename with GARDEN_SYMLINK (rulegen.py:510-514, 000-garden.mk:58-64).
  • The prepared gentoo-cloud.x86_64.qcow2 is an overlay made with qemu-img create -b on that link (rulegen.py:601-604). That make prepares it again once the link points at a newer download is derived from the rule, not tried.
  • distclean removes the file the link points at, the file .filename names, the link, .url and .filename (rulegen.py:486-492); earlier builds have no link and are left alone (derived). <next> on the slide stands for the newer build's stamp.
  • Known discrepancy: A cached download is checked again on every make after .url changes. When the .url file is made again with the same address, the download rule runs again and verifies the cached image without fetching it; the documentation does not mention it.

Read more

05 What rulegen.py rejects

CHECKED, AND WHEN rulegen.py, at build time Image 'gentoo-cloud.x86_64' defines both URL and URLProgram Image '…' defines LocalFilename but uses URLProgram; use URLPattern and … Image '…' must define both URLPattern and LocalFilenameTemplate together Image '…' URLPattern must contain exactly one capture group Image '…' LocalFilenameTemplate must contain '@1' placeholder make, at run time URL for gentoo-cloud.x86_64 does not match URLPattern not checked for URLProgram sections: the architecture in the URL two systems with one cached name
At build time rulegen.py stops with an error for a section with both URL and URLProgram, LocalFilename with URLProgram, URLPattern without LocalFilenameTemplate, a pattern without exactly one capture group, or a template without @1. At make time the .filename rule fails when the URL does not match URLPattern. For URLProgram sections the architecture in the URL and clashing cached names are not checked

At build time: both or neither of URL and URLProgram, LocalFilename with a program, half a pattern and template pair, a pattern without exactly one group, a template without @1. At make time: a URL that does not match. Not checked for programs: the architecture in the URL and clashing names.

rulegen.py checks these keys when image-garden itself is built, so a slip stops the build, not someone's test run. A section needs exactly one of URL and URLProgram. LocalFilename goes with URL. With a program, use the pattern and the template instead, and give both or neither. The pattern needs exactly one capture group, and the template needs @1. One mistake shows up only when make runs: if the distribution renames its files and the URL stops matching the pattern, the filename rule says so but leaves an empty file, and later runs stay stuck. And two checks skip programs entirely: that the URL names the architecture, and that no two systems share a cached file name.

Details and sources

Details

  • Each message is printed as data/FILE.ini: error: MESSAGE, one line with no traceback, and rulegen.py exits with status 1 (rulegen.py:753-757); make then deletes the empty data/050-FILE.gen.mk. The messages on the slide were produced by running rulegen.py at the pinned commit on small INI files in a scratch directory.
  • Exactly one of URL and URLProgram; URLPattern, LocalFilenameTemplate only with URLProgram, and only as a pair; LocalFilename only with URL (rulegen.py:358-394).
  • The pattern must compile as a Python regular expression with exactly one capture group ((?:…) groups do not count); the template must contain @1, no @2 or higher, no /, and not be . or .. (rulegen.py:183-213). A template written with $1 fails the @1 check.
  • At make time awk matches the URL against the pattern and stops with "URL for NAME does not match URLPattern", "Unresolved capture group in LocalFilenameTemplate for NAME" or "Invalid resolved local filename for NAME: …" (rulegen.py:561-587). awk's regular expressions are POSIX extended ones, not Python's: a pattern that compiles in Python may still behave differently in awk (derived).
  • The URL must name the architecture (amd64 and arm64 count for x86_64 and aarch64) only when the section has URL (rulegen.py:406-416).
  • The clash check compares names known when rulegen.py runs: LocalFilename, the URL's basename, or a template without @1 (rulegen.py:160-169, rulegen.py:758-775); a clash is reported as data/FILE.ini: error: per-group local filename collision in group 'G': 'F' used by 'S1' and 'S2'. tests/check-rulegen.sh covers the local file name keys and the clash (tests/check-rulegen.sh:111-170). A template with @1 is skipped, so the architecture in Gentoo's and Amazon Linux's templates is what keeps their names apart.

Read more

  • The .ini files in data/: the errors for the other keys.
  • What rulegen.py emits: the order in which it checks a file.
  • Known discrepancy: A failed URLProgram leaves empty .url and .filename files that make keeps. The .url recipe ends in awk (rulegen.py:526-530) and the filename recipe in | tee $@ (rulegen.py:546-591), so a failing program or a URL that no longer matches writes empty files, make exits 0, and later runs fail with "curl: (3) URL rejected: No host part in the URL" until a makefile changes or the files are deleted. Run at the commit in garden.commit with CURL=false.

06 Where to go from here

ROADS URLProgram Checksums @URL@ next to the image The target chain four steps in the cache Download and verify fetch, resume, check First-boot quirks Amazon Linux 2, Gentoo What rulegen.py emits every rule of a section Where files live what is safe to delete
Six roads from this deck: checksums and @URL@, the target chain, downloads, the first-boot quirks, what rulegen.py emits, and the cache

Next: the checksum placeholder, the target chain, downloads, the first-boot quirks of Amazon Linux 2 and Gentoo, the rest of rulegen.py's rules, and the cache.

That's URLProgram from start to finish: a pipeline in the section, a .url file in the cache, and a local name built from the pattern and the template. Gentoo's checksum sits next to its image, and the checksums deck explains the @URL@ placeholder that finds it. The target chain shows the same four cache steps for every system, and the download deck how the image is fetched, resumed and verified. Amazon Linux 2 and Gentoo also change the first boot. And to see the other rules rulegen writes for a section, or what's safe to delete from the cache, follow the last two roads.

Details and sources

Details

  • The keys in one line: URLProgram= a make recipe that prints the image URL; URLPattern= a regular expression with one capture group; LocalFilenameTemplate= a file name with @1.

Read more

Basis of the statements on this page

Read in rulegen.py, data/amazonlinux.ini, data/gentoo.ini, data/README.md, NEWS, mk/000-garden.mk and the image-garden script of image-garden at the pinned commit ("code"). The resolved URLs and file names were produced on 21 November 2026 by running the .url and .filename rules of a scratch build of the tag, and the error messages by running the pinned rulegen.py on small INI files ("run"); the build stamps change whenever the distributions publish. What happens to the cache and the prepared disk when a newer build appears is reasoned from the rules ("derived").

The keys

KeyGoes withValueChecked by rulegen.py
URLa fixed addressthe image URLmust contain the architecture (amd64, arm64 count)
URLProgramno URLa make recipe that prints the image URLexactly one of URL and URLProgram
LocalFilenameURL onlythe cached file nameno /, not . or .., no clash in the file
URLPatternURLProgram only, with the templatea regular expression with exactly one capture groupcompiles; one capturing group; balanced brackets
LocalFilenameTemplateURLProgram only, with the patterna file name with @1contains @1, no @2 or higher, no /

Sources: rulegen.py:183-214, rulegen.py:352-416, data/README.md:71-88.

The six systems with a URLProgram

SystemThe program readsCapture groupCached as (21 November 2026)
amazonlinux-cloud-2.x86_64os-images/latest/kvm/SHA256SUMSversionamazonlinux-cloud-2-VERSION.x86_64.qcow2
amazonlinux-cloud-2.aarch64os-images/latest/kvm-arm64/SHA256SUMSversionamazonlinux-cloud-2-VERSION.aarch64.qcow2
amazonlinux-cloud-2023.x86_64al2023/os-images/latest/kvm/SHA256SUMSversionamazonlinux-cloud-2023-2023.12.20261120.0.x86_64.qcow2
amazonlinux-cloud-2023.aarch64al2023/os-images/latest/kvm-arm64/SHA256SUMSversionamazonlinux-cloud-2023-VERSION.aarch64.qcow2
gentoo-cloud.x86_64latest-di-amd64-cloudinit.txt, line 6build stampgentoo-cloud-20261115T170104Z.x86_64.qcow2
gentoo-cloud.aarch64latest-di-arm64-cloudinit.txt, line 6build stampgentoo-cloud-STAMP.aarch64.qcow2

Amazon Linux checks its images against the same SHA256SUMS file its program reads; Gentoo uses SHA256SUMS=@URL@.sha256, the checksum file next to the image (amazonlinux.ini:8-42, gentoo.ini:8-22). Only the rows with a full name were run.

Writing a URLProgram

  • It is a make recipe line: $(CURL) is curl --fail --silent --show-error --location (000-garden.mk:21-21); write $$ for every $ the shell or awk should see.
  • It must print one URL on standard output; make writes it to $(GARDEN_DL_DIR)/GROUP/NAME.url (rulegen.py:526-530).
  • It runs when that file is missing or older than image-garden.mk or .image-garden.mk, not on every make.
  • In the INI file a % must be written %%: rulegen.py reads the files with Python's default interpolation, and a lone % stops it with an InterpolationSyntaxError traceback, the one data mistake still reported that way (run at the pinned commit; no file in data/ has one).

The files, for gentoo-cloud.x86_64

In $(GARDEN_DL_DIR)/gentoo/, by default ~/.cache/image-garden/dl/gentoo/:

FileMade fromContent
gentoo-cloud.x86_64.urlthe programthe image URL
gentoo-cloud.x86_64.filenamethe .url file, the pattern and the templategentoo-cloud-STAMP.x86_64.qcow2
gentoo-cloud-STAMP.x86_64.qcow2image-garden download-with-sha256sumsthe verified image
gentoo-cloud.x86_64.qcow2GARDEN_SYMLINKa link to the image

Messages

WhenMessage
buildImage 'NAME' defines both URL and URLProgram
buildImage 'NAME' defines neither URL nor URLProgram
buildImage 'NAME' defines URLPattern but uses URL
buildImage 'NAME' defines LocalFilenameTemplate but uses URL
buildImage 'NAME' defines LocalFilename but uses URLProgram; use URLPattern and LocalFilenameTemplate
buildImage 'NAME' must define both URLPattern and LocalFilenameTemplate together
buildImage 'NAME' has invalid URLPattern: …
buildImage 'NAME' URLPattern must contain exactly one capture group
buildImage 'NAME' LocalFilenameTemplate must contain '@1' placeholder
buildImage 'NAME' LocalFilenameTemplate supports only '@1' placeholder
makeURL for NAME does not match URLPattern
makeUnresolved capture group in LocalFilenameTemplate for NAME
makeInvalid resolved local filename for NAME: …

Build-time messages are printed as data/FILE.ini: error: MESSAGE, without a traceback (rulegen.py:753-757).

Roads from here

RUNTIME · make rules and the CLI

FIRST BOOT · cloud-init changes

BUILD TIME · data and rules