Every image has a checksum
SUMS files or one hash for contributors
rulegen.py refuses a section without SHA256SUMS, SHA512SUMS or a literal SHA256SUM. The two list keys are URLs fetched with the image and may start with @URL@, the image URL read from the .url file; with several keys SHA512SUMS wins, then SHA256SUM, and the rest is ignored.
On the map: build time
▶ Watch this deck — 6 scenes, ~4 min
Worth seeing first
- The .ini files in data/ — the other keys of a section and how rulegen.py reads them
01 No checksum, no system
A section without SHA256SUMS, SHA512SUMS or SHA256SUM stops the build: defines no checksum, use one of SHA256SUMS=, SHA512SUMS= or SHA256SUM=.
Every image image-garden downloads is checked before anything boots from it, and that starts in the data. Each system's section has to say how its image is checked, and rulegen.py refuses one that doesn't. The message names the section and the three keys you may use: SHA256SUMS, SHA512SUMS or SHA256SUM. There's no form of a section without one. Our example, ubuntu-cloud-24.04.x86_64, gives SHA256SUMS, the address of the list Ubuntu publishes next to its images. This deck is about choosing among the three keys, and the one placeholder they allow. How the download then uses them is the download deck's story.
Details and sources
Details
Checksums(SHA256SUMS=…, SHA256SUM=…, SHA512SUMS=…); when all three are empty, "Image {name} defines no checksum, use one of SHA256SUMS=, SHA512SUMS= or SHA256SUM=" (rulegen.py:420-431), prefixed withFILE: error:(rulegen.py:753-757). The message on the slide was produced by running rulegen.py at the pinned commit; it is one line, with no traceback.- data/README.md: "Each image section must use one of the following fields so that the downloaded image can be verified.
rulegen.pyfails on an image section without a checksum" (data/README.md:47-51). - The running example's list:
SHA256SUMS=https://cloud-images.ubuntu.com/noble/current/SHA256SUMS(data/ubuntu.ini:237-242). - The script has a
downloadhelper without a checksum (image-garden:627-660), but no rule that rulegen.py writes uses it (rulegen.py:498-517). tests/check-rulegen.shholds this in place (tests/check-rulegen.sh:96-98).
Read more
02 A list, or a single hash
SHA256SUMS and SHA512SUMS are URLs fetched with the image, so they follow it. SHA256SUM is a literal hash that pins one file, as Oracle Linux does.
The keys come in two kinds. SHA256SUMS and SHA512SUMS are addresses: of a list the distribution publishes beside its images, or, for Alpine, of a file holding one hash. image-garden fetches it with the image. So the list follows the image: if Ubuntu replaces the image behind its current link, the list changes with it, and nothing in the data has to. SHA256SUM is the hash itself, written into the section. Oracle Linux uses it for its seven images. It pins exactly one file: if the vendor ever replaces it, the check fails until someone updates the hash in the data.
Details and sources
Details
- In the data at the tag:
SHA256SUMSin 118 sections,SHA512SUMSin 14 (Debian 12, Alpine 2),SHA256SUMin Oracle Linux's 7 (counted at the tag), for example data/oracle.ini:10-15, data/debian.ini:10-16 and data/alpine.ini:8-13. - Alpine's
SHA512SUMSis the.sha512file beside each image, holding a bare hash, a format the helper accepts when it is the only non-empty line (data/README.md:59-61, image-garden:75-92). - With a list, the helper fetches the list when it has no copy, then the image, and on a mismatch deletes both and fetches both again (image-garden:480-559). With
SHA256SUMthe hash comes from the data, and a mismatch deletes the image only and tries once more (image-garden:582-625). - Ubuntu's URLs point into
…/noble/current/, which Ubuntu updates in place (derived from the path); Debian's into…/daily/latest/(data/debian.ini:10-16). That a vendor may replace the file behind a fixed URL, so a literal hash stops matching, is derived.
03 A list next to the image
SHA256SUMS=@URL@.sha256 becomes the resolved image URL plus .sha256, read from SYSTEM.url when the download runs. Only at the start, once, and only in SHA256SUMS or SHA512SUMS.
Some distributions publish a small checksum file next to each image, named after it. When the URL is fixed, you can write that address out. But Gentoo's URL comes from a program and changes with every build, so there's nothing to write. That's what the @URL@ placeholder is for. Put it at the start of SHA256SUMS or SHA512SUMS, as in @URL@.sha256, and the download rule replaces it with the URL the program printed, read from the system's .url file. It's allowed once, at the very start, and in those two keys only. rulegen.py says so if you try anything else.
Details and sources
Details
URL_PLACEHOLDER = "@URL@", allowed inSHA256SUMSandSHA512SUMS;expand_url_placeholderreplaces it with$$(cat $(word 1,$^)), the.urlfile that is the download rule's first prerequisite (rulegen.py:51-88). The.urlfile holds the URL thatURLgives orURLProgramprints (rulegen.py:520-530).validate_url_placeholder: in any other key, "uses @URL@ in 'KEY', it is only allowed in SHA256SUMS= and SHA512SUMS="; not at the start, or more than once, "must use @URL@ once, at the start of the value, e.g. @URL@.sha256" (rulegen.py:91-111). The key in the first message is lower-case, as the INI reader stores it.- Gentoo:
URLProgramandSHA256SUMS=@URL@.sha256(data/gentoo.ini:8-14). OpenCloudOS uses the placeholder with a fixedURL(data/opencloudos.ini:13-16). - The helper looks up the saved file name in the list, then the URL's basename (image-garden:508-515); Gentoo's saved name comes from
LocalFilenameTemplate, so its.sha256file is matched by the basename (derived). - data/README.md describes the placeholder (data/README.md:65-69).
Read more
04 When a section gives more than one
The download rule picks SHA512SUMS, else SHA256SUM, else SHA256SUMS. Any other key is ignored without a warning.
Nothing stops a section from giving two of these keys, or all three. rulegen.py doesn't complain; the download rule simply uses one. SHA512SUMS wins if it's there. Then SHA256SUM, the literal hash. SHA256SUMS comes last. The others are ignored without a word, so a second key isn't a fallback, and it isn't checked either. If you move a section from one kind of checksum to another, remove the old key, or the one you leave behind may be the one that counts. Today, no section in the data gives more than one.
Details and sources
Details
- The download rule is
ifneq (,SHA512SUMS)→download-with-sha512sums,else ifneq (,SHA256SUM)→download-with-sha256sum,else ifneq (,SHA256SUMS)→download-with-sha256sums(rulegen.py:498-517). rulegen.py checks only that at least one is set (rulegen.py:420-431). - Every section in
data/*.iniat the tag has exactly one checksum key (counted). - The slide's three-key section is made up; the two-line change below it is an example edit.
Read more
05 What the tests hold in place
tests/check-rulegen.sh: @URL@ expands in both list keys, is refused in SHA256SUM, elsewhere in the value or twice, and a section without a checksum is refused.
These rules are pinned by tests, so a change to them shows up at once. make check-rulegen writes tiny INI files with a single section and runs rulegen.py on each. It expects @URL@ to become the contents of the .url file, in SHA256SUMS and in SHA512SUMS alike. It expects an error for @URL@ in SHA256SUM, for one that isn't at the start, and for one used twice. And it expects a section with no checksum at all to be refused, with its own message. If you add a kind of checksum, these are the tests to extend.
Details and sources
Details
write_ini NAME FIELD…writes[META] Group=test Variant=cloudand one[test-cloud.x86_64]section withArch=x86_64andBoot=efi(tests/check-rulegen.sh:18-28).- The checksum cases:
url-placeholder-sha256sumsandurl-placeholder-sha512sumsexpect$$(cat $(word 1,$^)).sha256and.sha512in the download command;url-placeholder-in-sha256sum,url-placeholder-not-at-start,url-placeholder-twiceexpect the two placeholder errors;no-checksumexpects "error: Image 'test-cloud.x86_64' defines no checksum" (tests/check-rulegen.sh:76-98). Every error case also fails if rulegen.py prints a Python traceback (:65-72). The same file also checks loongarch64 (:100-109), the local file name keys and clashes between file names (:111-170), and the[META],ArchandBootkeys (:172-223). make check-rulegenruns it (Makefile:99-100);make checkincludes it, and the container build runsmake check(Dockerfile:71).- The precedence of the three keys is not covered by a test (read in the test file).
Read more
06 Where to go from here
Next: downloading and checking, the other INI keys, URLProgram, and image downloads through pkgproxy.
So every section names a checksum: a list that follows the image, or one hash that pins it, with @URL@ for a list that sits next to an image whose address moves. The download deck shows what happens with them at run time: the formats a list may have, resuming, and the second try after a mismatch. The INI deck covers the other keys of a section, and the URLProgram deck the program behind a moving address. And the pkgproxy downloads deck shows how a cache keeps images by their hash.
Details and sources
Read more
Basis of the statements on this page
Read in rulegen.py, data/*.ini, data/README.md, tests/check-rulegen.sh and the image-garden script at the pinned commit ("code"); the counts were taken from the section headers and keys there. The error messages were produced by running that rulegen.py ("run").
The three keys
| Key | Value | Helper | Used by |
|---|---|---|---|
SHA256SUMS | URL of a SHA-256 list, or of one bare hash; may start with @URL@ | download-with-sha256sums | 118 sections: AlmaLinux, Amazon Linux, Arch Linux, CentOS, Fedora, Gentoo, OpenCloudOS, openSUSE, Rocky, Ubuntu |
SHA512SUMS | URL of a SHA-512 list, or of one bare hash; may start with @URL@ | download-with-sha512sums | 14 sections: Debian, Alpine |
SHA256SUM | the SHA-256 hash itself | download-with-sha256sum | 7 sections: Oracle Linux |
With more than one key, the first of SHA512SUMS, SHA256SUM, SHA256SUMS is used (rulegen.py:498-517).
List formats
| Format | Example line |
|---|---|
GNU, as sha256sum prints it | HASH NAME or HASH *NAME |
BSD, as sha256sum --tag prints it | SHA256 (NAME) = HASH |
| one bare hash | HASH, the only non-empty line, of the right length |
The name looked up is the saved file name, else the URL's basename (image-garden:75-92, :508-515). More in downloading and checking images.
@URL@
| Value | Result |
|---|---|
SHA256SUMS=@URL@.sha256 | the URL in SYSTEM.url followed by .sha256, read when the download runs |
SHA512SUMS=@URL@.sha512 | the same with .sha512 |
SHA256SUM=@URL@ | refused: only allowed in SHA256SUMS= and SHA512SUMS= |
SHA256SUMS=http://example.org/@URL@ | refused: once, at the start |
SHA256SUMS=@URL@@URL@ | refused: once, at the start |
Sources: rulegen.py:51-111, tests/check-rulegen.sh:79-94.
Roads from here
RUNTIME · make rules and the CLI
- Downloading and checking images — what the download helpers do with each key at run time
PKGPROXY · caching for guests
- Cloud images through pkgproxy — how pkgproxy keeps images by their hash
BUILD TIME · data and rules
- The .ini files in data/ — every other key of a system section
- URLProgram and local file names — the program behind an image URL that moves
- What rulegen.py emits — the download rule among the other rules rulegen.py writes