Every image has a checksum

SUMS files or one hash for contributors

rulegen.py refuses a section without SHA256SUMS, SHA512SUMS or a literal SHA256SUM. The two list keys are URLs fetched with the image and may start with @URL@, the image URL read from the .url file; with several keys SHA512SUMS wins, then SHA256SUM, and the rest is ignored.

On the map: build time

▶ Watch this deck — 6 scenes, ~4 min

Worth seeing first

01 No checksum, no system

NO CHECKSUM, NO SYSTEM every download is checked before anything boots from it [example-cloud-1.x86_64] URL=https://…/x86_64/disk.qcow2 Arch=x86_64 rulegen.py data/example.ini: error: Image 'example-cloud-1.x86_64' defines no checksum, use one of SHA256SUMS=, SHA512SUMS= or SHA256SUM= SHA256SUMS SHA512SUMS SHA256SUM [ubuntu-cloud-24.04.x86_64] SHA256SUMS=https://cloud-images.ubuntu.com/noble/current/SHA256SUMS which key, and the @URL@ placeholder · how the download uses them: the download deck
A section without a checksum key goes into rulegen.py and is refused with the message defines no checksum, use one of SHA256SUMS=, SHA512SUMS= or SHA256SUM=; ubuntu-cloud-24.04.x86_64 gives SHA256SUMS, the address of Ubuntu's list

A section without SHA256SUMS, SHA512SUMS or SHA256SUM stops the build: defines no checksum, use one of SHA256SUMS=, SHA512SUMS= or SHA256SUM=.

Every image image-garden downloads is checked before anything boots from it, and that starts in the data. Each system's section has to say how its image is checked, and rulegen.py refuses one that doesn't. The message names the section and the three keys you may use: SHA256SUMS, SHA512SUMS or SHA256SUM. There's no form of a section without one. Our example, ubuntu-cloud-24.04.x86_64, gives SHA256SUMS, the address of the list Ubuntu publishes next to its images. This deck is about choosing among the three keys, and the one placeholder they allow. How the download then uses them is the download deck's story.

Details and sources

Details

  • Checksums(SHA256SUMS=…, SHA256SUM=…, SHA512SUMS=…); when all three are empty, "Image {name} defines no checksum, use one of SHA256SUMS=, SHA512SUMS= or SHA256SUM=" (rulegen.py:420-431), prefixed with FILE: error: (rulegen.py:753-757). The message on the slide was produced by running rulegen.py at the pinned commit; it is one line, with no traceback.
  • data/README.md: "Each image section must use one of the following fields so that the downloaded image can be verified. rulegen.py fails on an image section without a checksum" (data/README.md:47-51).
  • The running example's list: SHA256SUMS=https://cloud-images.ubuntu.com/noble/current/SHA256SUMS (data/ubuntu.ini:237-242).
  • The script has a download helper without a checksum (image-garden:627-660), but no rule that rulegen.py writes uses it (rulegen.py:498-517).
  • tests/check-rulegen.sh holds this in place (tests/check-rulegen.sh:96-98).

02 A list, or a single hash

A LIST, OR A SINGLE HASH an address, fetched with the image SHA256SUMS SHA512SUMS ubuntu: …/noble/current/SHA256SUMS alpine: …-cloudinit-r0.qcow2.sha512 a new image its new list the list follows the image nothing in the data changes the hash itself, in the section SHA256SUM [oracle-cloud-7.x86_64] SHA256SUM=9e68a8fb682f… Oracle Linux, seven images pins exactly one file a replaced image fails the check until the data has its new hash
SHA256SUMS and SHA512SUMS are addresses of a list, or for Alpine of a file with one hash, fetched with the image, so they follow the image when the distribution replaces it; SHA256SUM is a literal hash, used by Oracle Linux for seven images, which pins one file and fails when the vendor replaces it

SHA256SUMS and SHA512SUMS are URLs fetched with the image, so they follow it. SHA256SUM is a literal hash that pins one file, as Oracle Linux does.

The keys come in two kinds. SHA256SUMS and SHA512SUMS are addresses: of a list the distribution publishes beside its images, or, for Alpine, of a file holding one hash. image-garden fetches it with the image. So the list follows the image: if Ubuntu replaces the image behind its current link, the list changes with it, and nothing in the data has to. SHA256SUM is the hash itself, written into the section. Oracle Linux uses it for its seven images. It pins exactly one file: if the vendor ever replaces it, the check fails until someone updates the hash in the data.

Details and sources

Details

  • In the data at the tag: SHA256SUMS in 118 sections, SHA512SUMS in 14 (Debian 12, Alpine 2), SHA256SUM in Oracle Linux's 7 (counted at the tag), for example data/oracle.ini:10-15, data/debian.ini:10-16 and data/alpine.ini:8-13.
  • Alpine's SHA512SUMS is the .sha512 file beside each image, holding a bare hash, a format the helper accepts when it is the only non-empty line (data/README.md:59-61, image-garden:75-92).
  • With a list, the helper fetches the list when it has no copy, then the image, and on a mismatch deletes both and fetches both again (image-garden:480-559). With SHA256SUM the hash comes from the data, and a mismatch deletes the image only and tries once more (image-garden:582-625).
  • Ubuntu's URLs point into …/noble/current/, which Ubuntu updates in place (derived from the path); Debian's into …/daily/latest/ (data/debian.ini:10-16). That a vendor may replace the file behind a fixed URL, so a literal hash stops matching, is derived.

03 A list next to the image

THE @URL@ PLACEHOLDER di-amd64-cloudinit-….qcow2 di-amd64-cloudinit-….qcow2.sha256 a checksum file next to each image, named after it URLProgram=$(CURL) …/latest-di-amd64-cloudinit.txt | awk … Gentoo: the URL changes with every build SHA256SUMS=@URL@.sha256 $(cat …/gentoo-cloud.x86_64.url).sha256 read when the download runs once, at the start, in SHA256SUMS or SHA512SUMS only: … uses @URL@ in 'sha256sum', it is only allowed in SHA256SUMS= and SHA512SUMS= … SHA256SUMS= must use @URL@ once, at the start of the value, e.g. @URL@.sha256
Gentoo's image URL comes from a URLProgram and changes with each build, with a checksum file named after the image beside it; SHA256SUMS=@URL@.sha256 is expanded when the download runs to the URL in the system's .url file plus .sha256; @URL@ is allowed once, at the start, in SHA256SUMS and SHA512SUMS only

SHA256SUMS=@URL@.sha256 becomes the resolved image URL plus .sha256, read from SYSTEM.url when the download runs. Only at the start, once, and only in SHA256SUMS or SHA512SUMS.

Some distributions publish a small checksum file next to each image, named after it. When the URL is fixed, you can write that address out. But Gentoo's URL comes from a program and changes with every build, so there's nothing to write. That's what the @URL@ placeholder is for. Put it at the start of SHA256SUMS or SHA512SUMS, as in @URL@.sha256, and the download rule replaces it with the URL the program printed, read from the system's .url file. It's allowed once, at the very start, and in those two keys only. rulegen.py says so if you try anything else.

Details and sources

Details

  • URL_PLACEHOLDER = "@URL@", allowed in SHA256SUMS and SHA512SUMS; expand_url_placeholder replaces it with $$(cat $(word 1,$^)), the .url file that is the download rule's first prerequisite (rulegen.py:51-88). The .url file holds the URL that URL gives or URLProgram prints (rulegen.py:520-530).
  • validate_url_placeholder: in any other key, "uses @URL@ in 'KEY', it is only allowed in SHA256SUMS= and SHA512SUMS="; not at the start, or more than once, "must use @URL@ once, at the start of the value, e.g. @URL@.sha256" (rulegen.py:91-111). The key in the first message is lower-case, as the INI reader stores it.
  • Gentoo: URLProgram and SHA256SUMS=@URL@.sha256 (data/gentoo.ini:8-14). OpenCloudOS uses the placeholder with a fixed URL (data/opencloudos.ini:13-16).
  • The helper looks up the saved file name in the list, then the URL's basename (image-garden:508-515); Gentoo's saved name comes from LocalFilenameTemplate, so its .sha256 file is matched by the basename (derived).
  • data/README.md describes the placeholder (data/README.md:65-69).

04 When a section gives more than one

ONE KEY WINS [example-cloud-1.x86_64] SHA256SUMS=https://…/SHA256SUMS SHA256SUM=9e68a8fb… SHA512SUMS=https://…/SHA512SUMS the download rule asks, in order 1. SHA512SUMS download-with-sha512sums 2. SHA256SUM download-with-sha256sum 3. SHA256SUMS download-with-sha256sums the others: ignored, no warning, never checked -SHA256SUM=9e68a8fb… +SHA512SUMS=https://…/SHA512SUMS moving to a list: take the old key out today every section in data/ gives exactly one
A section with all three checksum keys: the download rule uses SHA512SUMS if present, else SHA256SUM, else SHA256SUMS; the others are ignored without a warning and never checked; when changing kind, remove the old key; no section today has more than one

The download rule picks SHA512SUMS, else SHA256SUM, else SHA256SUMS. Any other key is ignored without a warning.

Nothing stops a section from giving two of these keys, or all three. rulegen.py doesn't complain; the download rule simply uses one. SHA512SUMS wins if it's there. Then SHA256SUM, the literal hash. SHA256SUMS comes last. The others are ignored without a word, so a second key isn't a fallback, and it isn't checked either. If you move a section from one kind of checksum to another, remove the old key, or the one you leave behind may be the one that counts. Today, no section in the data gives more than one.

Details and sources

Details

  • The download rule is ifneq (,SHA512SUMS) → download-with-sha512sums, else ifneq (,SHA256SUM) → download-with-sha256sum, else ifneq (,SHA256SUMS) → download-with-sha256sums (rulegen.py:498-517). rulegen.py checks only that at least one is set (rulegen.py:420-431).
  • Every section in data/*.ini at the tag has exactly one checksum key (counted).
  • The slide's three-key section is made up; the two-line change below it is an example edit.

05 What the tests hold in place

WHAT THE TESTS PIN $ make check-rulegen [META] Group=test Variant=cloud [test-cloud.x86_64] expects PASS: url-placeholder-sha256sums PASS: url-placeholder-sha512sums expands PASS: url-placeholder-in-sha256sum PASS: url-placeholder-not-at-start PASS: url-placeholder-twice refused PASS: no-checksum refused, its own message a new kind of checksum: extend tests/check-rulegen.sh
make check-rulegen writes one-section INI files and runs rulegen.py on each; it expects @URL@ to expand in SHA256SUMS and SHA512SUMS, errors for @URL@ in SHA256SUM, not at the start or twice, and an error for a section with no checksum

tests/check-rulegen.sh: @URL@ expands in both list keys, is refused in SHA256SUM, elsewhere in the value or twice, and a section without a checksum is refused.

These rules are pinned by tests, so a change to them shows up at once. make check-rulegen writes tiny INI files with a single section and runs rulegen.py on each. It expects @URL@ to become the contents of the .url file, in SHA256SUMS and in SHA512SUMS alike. It expects an error for @URL@ in SHA256SUM, for one that isn't at the start, and for one used twice. And it expects a section with no checksum at all to be refused, with its own message. If you add a kind of checksum, these are the tests to extend.

Details and sources

Details

  • write_ini NAME FIELD… writes [META] Group=test Variant=cloud and one [test-cloud.x86_64] section with Arch=x86_64 and Boot=efi (tests/check-rulegen.sh:18-28).
  • The checksum cases: url-placeholder-sha256sums and url-placeholder-sha512sums expect $$(cat $(word 1,$^)).sha256 and .sha512 in the download command; url-placeholder-in-sha256sum, url-placeholder-not-at-start, url-placeholder-twice expect the two placeholder errors; no-checksum expects "error: Image 'test-cloud.x86_64' defines no checksum" (tests/check-rulegen.sh:76-98). Every error case also fails if rulegen.py prints a Python traceback (:65-72). The same file also checks loongarch64 (:100-109), the local file name keys and clashes between file names (:111-170), and the [META], Arch and Boot keys (:172-223).
  • make check-rulegen runs it (Makefile:99-100); make check includes it, and the container build runs make check (Dockerfile:71).
  • The precedence of the three keys is not covered by a test (read in the test file).

06 Where to go from here

ROADS Checksums Download and check formats, resume, second try The .ini files every other key URLProgram an address that moves Downloads via pkgproxy images kept by hash
Four roads from Checksums: downloading and checking, the INI keys, URLProgram, and image downloads through pkgproxy

Next: downloading and checking, the other INI keys, URLProgram, and image downloads through pkgproxy.

So every section names a checksum: a list that follows the image, or one hash that pins it, with @URL@ for a list that sits next to an image whose address moves. The download deck shows what happens with them at run time: the formats a list may have, resuming, and the second try after a mismatch. The INI deck covers the other keys of a section, and the URLProgram deck the program behind a moving address. And the pkgproxy downloads deck shows how a cache keeps images by their hash.

Details and sources

Read more

Basis of the statements on this page

Read in rulegen.py, data/*.ini, data/README.md, tests/check-rulegen.sh and the image-garden script at the pinned commit ("code"); the counts were taken from the section headers and keys there. The error messages were produced by running that rulegen.py ("run").

The three keys

KeyValueHelperUsed by
SHA256SUMSURL of a SHA-256 list, or of one bare hash; may start with @URL@download-with-sha256sums118 sections: AlmaLinux, Amazon Linux, Arch Linux, CentOS, Fedora, Gentoo, OpenCloudOS, openSUSE, Rocky, Ubuntu
SHA512SUMSURL of a SHA-512 list, or of one bare hash; may start with @URL@download-with-sha512sums14 sections: Debian, Alpine
SHA256SUMthe SHA-256 hash itselfdownload-with-sha256sum7 sections: Oracle Linux

With more than one key, the first of SHA512SUMS, SHA256SUM, SHA256SUMS is used (rulegen.py:498-517).

List formats

FormatExample line
GNU, as sha256sum prints itHASH NAME or HASH *NAME
BSD, as sha256sum --tag prints itSHA256 (NAME) = HASH
one bare hashHASH, the only non-empty line, of the right length

The name looked up is the saved file name, else the URL's basename (image-garden:75-92, :508-515). More in downloading and checking images.

@URL@

ValueResult
SHA256SUMS=@URL@.sha256the URL in SYSTEM.url followed by .sha256, read when the download runs
SHA512SUMS=@URL@.sha512the same with .sha512
SHA256SUM=@URL@refused: only allowed in SHA256SUMS= and SHA512SUMS=
SHA256SUMS=http://example.org/@URL@refused: once, at the start
SHA256SUMS=@URL@@URL@refused: once, at the start

Sources: rulegen.py:51-111, tests/check-rulegen.sh:79-94.

Roads from here

RUNTIME · make rules and the CLI

PKGPROXY · caching for guests

BUILD TIME · data and rules