Downloading and checking images

lock, resume, check, retry for users

make calls image-garden's download helpers with the image URL, its checksum list or hash, the cache directory and the file name. A helper takes a lock on the file, fetches the checksum list, then the image through pkgproxy or from the origin into a .partial file, resumes a transfer that ended short, checks the hash, and on a mismatch deletes both files and tries once more.

On the map: runtime

▶ Watch this deck — 7 scenes, ~5 min

Worth seeing first

01 Large files from someone else's server

DOWNLOADING AN IMAGE cloud-images.ubuntu.com a few hundred MB your cache connection drops stale checksum list Control C two makes, one file image-garden download-with-sha256sums URL SUMS-URL DIR FILE URL https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img SUMS-URL https://cloud-images.ubuntu.com/noble/current/SHA256SUMS DIR ~/.cache/image-garden/dl/ubuntu FILE noble-server-cloudimg-amd64.img
A cloud image of a few hundred megabytes travels from cloud-images.ubuntu.com to your cache; four things can go wrong: the connection drops, the checksum list is stale, you press Control C, two makes want one file; so make runs image-garden download-with-sha256sums with the URL, the SHA256SUMS URL, the cache directory and the file name

The download step runs image-garden download-with-sha256sums URL SUMS-URL DIR FILE, which copes with broken transfers, stale lists, interrupts and concurrent makes.

A cloud image is a few hundred megabytes on someone else's server, and a lot can go wrong while you fetch it. The connection can drop half-way. The distribution can publish a new image while your checksum list is old. You can press Control C. And two makes can want the same file at once. So make doesn't call curl directly. It calls image-garden download-with-sha256sums, one of the download helpers built into the image-garden command, with the URL, the checksum list, the directory and the file name. This deck follows that helper for Ubuntu 24.04 on x86.

Details and sources

Details

  • The download rule of a SHA256SUMS system runs $(GARDEN_ITSELF) download-with-sha256sums $(cat X.url) <SHA256SUMS> $(GARDEN_DL_DIR)/<group> $(cat X.filename) (rulegen.py:510-514); GARDEN_ITSELF is the image-garden script that started make (image-garden:982-993).
  • The helpers are download-with-sha512sums, download-with-sha256sums, download-with-sha256sum and download (no checksum, not used by the generated rules) (image-garden:358-668). The checksum list defaults to SHA256SUMS next to the image, the directory to $GARDEN_DL_DIR and the file name to the URL's last component (image-garden:460-473).
  • image-garden --help brackets the optional arguments, download-with-sha256sums URL [SHA256SUMS_URL [DIRECTORY_PREFIX [FILENAME]]], and says what they default to; only download-with-sha256sum needs its checksum (image-garden:898-904, image-garden:562-577). The man page agrees (image-garden.1:24-27).
  • The values on the slide are those of [ubuntu-cloud-24.04.x86_64] (ubuntu.ini:237-242) with the cache of a native or source install.
  • In CI, with CI set, curl runs without its progress meter (image-garden:107-119).

02 Three ways to name the checksum

THREE KINDS OF CHECKSUM Ubuntu …/SHA256SUMS a list Debian …/SHA512SUMS a list Oracle 9e68a8fb… the hash itself download-with-sha256sums download-with-sha512sums download-with-sha256sum 2c9a…e41f *noble-server-cloudimg-amd64.img SHA256 (Fedora-Cloud-Base-….qcow2) = 5b1f…90aa a91b…07cd hash and name BSD style one bare hash, as Alpine the entry is found by the saved file name, else by the end of the URL
Ubuntu's SHA256SUMS list goes to download-with-sha256sums, Debian's SHA512SUMS list to download-with-sha512sums, the single hash image-garden carries for Oracle Linux to download-with-sha256sum; a list line may be GNU style, BSD style or one bare hash, found by the saved file name or else by the URL's last part

Ubuntu's and Debian's images are checked against a published list of SHA-256 or SHA-512 hashes; Oracle's against a hash image-garden carries itself. Lists may be GNU style, BSD style or one bare hash.

Every image is checked against a checksum, which comes one of three ways. Ubuntu publishes SHA256SUMS, a list next to its images. Debian and Alpine publish SHA512SUMS lists, with longer hashes. For Oracle Linux, image-garden carries a single hash itself. Each way picks its own helper. Lists come in three shapes too: a hash and a file name per line, as Ubuntu's does, the BSD style with the name in brackets, and a file that holds one bare hash, as Alpine publishes. The helper looks for the saved file name first, then for the name at the end of the URL.

Details and sources

Details

  • The download rule uses download-with-sha512sums when SHA512SUMS is set, else download-with-sha256sum when SHA256SUM is set, else download-with-sha256sums (rulegen.py:498-517). A section without any of the three is refused when the rules are generated (rulegen.py:426-431).
  • In the data at the commit in garden.commit, Debian (12 sections) and Alpine (2) use SHA512SUMS, Oracle Linux (7) uses SHA256SUM, every other file SHA256SUMS. Fedora's SHA256SUMS points at its CHECKSUM file, openSUSE's at a .sha256 file; @URL@ in the value stands for the image URL (rulegen.py:82-88).
  • find_checksum_hash accepts HASH NAME or HASH *NAME lines, BSD lines KIND (NAME) = HASH, and a file whose only non-empty line is a bare hash of the right length (image-garden:75-92).
  • The name looked up is the saved file name when the list mentions it, otherwise the URL's basename (image-garden:508-515).
  • The hashes on the slide are shortened; the Fedora line shows the BSD format (that Fedora publishes this format is derived).

03 The list first, then the image

LIST, IMAGE, CHECK SHA256SUMS noble-server-cloudimg-amd64.img.sha256sums fetched only when this file is missing expected hash 2c9a…e41f pkgproxy, by hash the origin if pkgproxy can't help noble-server-cloudimg-amd64.img.partial noble-server-cloudimg-amd64.img renamed when complete check the hash: until it matches, make goes no further
The SHA256SUMS list is saved as noble-server-cloudimg-amd64.img.sha256sums, fetched only when missing, and gives the expected hash; the image comes from pkgproxy by that hash or from the origin into noble-server-cloudimg-amd64.img.partial, is renamed when complete, and is checked against the hash before make goes on

SHA256SUMS is saved as noble-server-cloudimg-amd64.img.sha256sums; the image arrives as .partial, through pkgproxy by hash or from the origin, is renamed when complete, then checked.

The helper works in a fixed order. First the checksum list. It's saved next to the image, under the image's name with .sha256sums on the end, and fetched only when that file is missing. From the list it takes the expected hash before the image arrives. That's what lets it ask pkgproxy, when there is one, for the image by its hash, and fall back to the origin if pkgproxy can't help. The image goes to a .partial file, and only a complete download is renamed to its real name. Then the hash is checked. Until it matches, make goes no further.

Details and sources

Details

  • The list is fetched only when DIR/FILE.sha256sums does not exist, to FILE.sha256sums.partial, and renamed when complete (image-garden:480-506). The hash is taken from it before the image is fetched, "so that the download can go through a pkgproxy" (image-garden:508-515).
  • With a pkgproxy (GARDEN_DL_PKGPROXY, by default PKGPROXY_URL or the snap's), the list is asked from /files/v1/volatile?url=… (image-garden:211-269) and the image from /files/v1/fetch?url=…&sha256=HASH, one try with a 10 second connect timeout, then from the URL directly (image-garden:281-308). The details are in image downloads through pkgproxy.
  • The image is fetched only when DIR/FILE does not exist, into FILE.partial, then mv -v to FILE (image-garden:517-540).
  • The check is printf '%s %s\n' HASH FILE | sha256sum --check, run in DIR (image-garden:542-545). It also runs when the image was already there.
  • The hash on the slide is shortened.
  • The README: "Images and their checksum lists are downloaded over HTTPS, through pkgproxy when one is available, and every image is checked against its checksum" (README.md:34-35); every image URL in the data is https.

04 A broken download picks up where it stopped

RESUME, DO NOT RESTART noble-server-cloudimg-amd64.img.partial the connection breaks curl exit 18 a response that ended short curl exit 56 a connection that broke resume from the end, up to five times any other error, such as a refused connection: curl's own five retries Control C noble-server-cloudimg-amd64.img.partial kept $ image-garden make ubuntu-cloud-24.04.x86_64.run continues from there noble-server-cloudimg-amd64.img.sha256sums.partial a partial checksum list is deleted: it is small, and it is what protects the image
A download of noble-server-cloudimg-amd64.img.partial breaks at sixty percent; curl exit 18 or 56 makes the helper resume from the end of the partial file, up to five times; other errors get curl's five retries; after Control C the partial image is kept and the next make continues it, while a partial checksum list is deleted

curl exit 18 (short response) or 56 (broken connection) is resumed with --continue-at -, up to five times. After Control C the image's .partial stays for the next run; a partial list is removed.

Say the connection breaks at sixty percent. curl's own retry would start over from zero, so the helper does something better. When curl reports a response that ended short, or a connection that broke, the helper resumes from the end of the .partial file, up to five times. Other errors, such as a refused connection, get curl's five retries. Press Control C, and the helper keeps the partial image and stops. Next time you run make, the download continues from there. A partial checksum list is the exception: it's deleted, because the list is small and it's what protects the image.

Details and sources

Details

  • "curl --retry starts a download over, it does not resume: a response that ends short (18) or a connection that breaks (56) is resumed here instead, a few times": the direct download loops up to five times on exit 18 or 56, each with curl --fail --location --retry 5 --retry-connrefused --continue-at - (image-garden:310-327, image-garden:107-119). It prints "Download of URL ended short, resuming".
  • On SIGINT a trap touches FILE.interrupted; the helper then removes that marker and returns 1. For the image, "Leave the partial file around, on interrupt or failure, so that we can resume the download later"; for the list, "Remove the partial file, on interrupt or failure, as it is our only protection against corruptions of the main content and the checksum files are small" (image-garden:480-540).
  • A download through pkgproxy is also written to .partial with --continue-at -, so the direct download that follows resumes what came through pkgproxy (image-garden:290-298).
  • The checksum list from the origin is fetched with --retry 3 (image-garden:269-269).
  • The download helper, which has no checksum, deletes its partial file on interrupt (image-garden:655-660).

05 A mismatch, then one more try

A MISMATCH Verifying sha256 checksum of …/noble-server-cloudimg-amd64.img noble-server-cloudimg-amd64.img: FAILED sha256sum: WARNING: 1 computed checksum did NOT match origin: a new image, same name your saved list: from before removed '…/noble-server-cloudimg-amd64.img' removed '…/noble-server-cloudimg-amd64.img.sha256sums' Fetch SHA256SUMS: noble-server-cloudimg-amd64.img once more, list first no entry for the file in the list: the same, after Cannot find checksum entry for … in ….sha256sums a second failure: the helper gives up and make stops with an error left in the cache: only noble-server-cloudimg-amd64.img.lock
The check prints FAILED for noble-server-cloudimg-amd64.img, often because the origin has a new image while the saved list is from before; the helper removes the image and its .sha256sums file and fetches the list and the image once more; a list without an entry is treated the same; a second failure stops make, leaving only the .lock file

On FAILED both the image and .sha256sums are removed and the download runs once more from the list; a second failure stops make. Only the .lock file is left.

Now the image is complete, but its hash doesn't match the list. Which one is wrong? Often the list: the distribution published a new image under the same name, and the list you saved is from before. So the helper deletes both, the image and the list, and runs the whole download once more, list first. The same happens when the list has no entry for the file. If the second try fails too, the helper gives up, and make stops with an error. Nothing that failed its check is kept, and the output names each file as it's removed, so you can see what happened.

Details and sources

Details

  • The helper loops for attempt in first retry. When the hash is empty or sha256sum --check fails, it runs rm -vf FILE FILE.sha256sums, "so that we re-download the checksum file. This can help if we have a stale checksum from the past that will never match the current download", continues on the first attempt and returns 1 on the second (image-garden:542-559). The SHA512 helper does the same (image-garden:440-457); the single-hash helper removes only the image (image-garden:613-623).
  • With no entry it prints "Cannot find checksum entry for NAME in LIST" first (image-garden:545-548).
  • The output on the slide was produced by running the v0.8.0 helper on a local file:// origin whose list held a wrong hash, with CI set; paths are shortened, and the rm -v lines are shown as an English locale prints them. After both attempts only FILE.lock was left in the directory.
  • A failing helper fails the download recipe, so make stops; the cache link is not made.

06 Two makes, one file

ONE FILE, TAKING TURNS make ubuntu-core-24.x86_64 make ubuntu-core-24+tpm.x86_64 ubuntu-core-24-amd64.img.xz ubuntu-core-24-amd64.img.xz.lock held across fetch, check and retry first fetch, check, release the lock second waits checks only ln -sfn FILE LINK.tmp.$$ && mv -fT LINK.tmp.$$ LINK one rename allocate: flock SYSTEM.lock image-garden make SYSTEM.run SYSTEM.qcow2
Two makes, for ubuntu-core-24.x86_64 and ubuntu-core-24+tpm.x86_64, need ubuntu-core-24-amd64.img.xz; each helper locks ubuntu-core-24-amd64.img.xz.lock; the first fetches and checks, the second waits and only checks; the system link is swapped in with one rename; allocate locks SYSTEM.lock around its make

Each helper holds flock on FILE.lock across fetch, check and retry. The link is replaced with one mv; allocate locks SYSTEM.lock around its make.

Last, sharing. Two systems can start from one image, such as Ubuntu Core 24 with and without a TPM, and in CI two jobs on one machine can make the same system at once. So each helper takes a lock on a .lock file next to the image before it does anything else. The second helper waits, then finds the image already there, and only checks it. The link named after the system is swapped in with a single rename, so two makes can write it at the same moment and neither fails. And when you allocate, a lock per system keeps two allocations from building the same disk at once.

Details and sources

Details

  • "Each download command parses its arguments, then runs its _locked body with fd 9 open on DIR/FILENAME.lock. The body takes an exclusive lock on fd 9 and holds it across the fetch, the checksum and the retry … Systems that share a base image download it once: the second waits, then reuses the verified file." (image-garden:351-356, image-garden:472-476). The second helper skips the fetch because the file exists, and runs the check again.
  • GARDEN_SYMLINK replaces the link with one rename, because "of two at once, one may find the other's link and fail, as uutils ln, the default ln in Ubuntu 26.04, does" (000-garden.mk:58-64); the link is .PRECIOUS (rulegen.py:494-496).
  • Ubuntu Core 24 and 24+tpm both start from ubuntu-core-24-amd64.img.xz; its conversion to qcow2 in the cache runs under a lock of its own (010-qemu.mk:304-347); see Ubuntu Core.
  • allocate runs flock <.image-garden/ or ./>SYSTEM.lock image-garden make SYSTEM.run SYSTEM.qcow2 unless --premade (image-garden:1058-1064).
  • The lock files stay in the cache after the download; they are empty.

07 Where to go from here

ROADS Download and verify Downloads via pkgproxy images kept by their hash Every image has a checksum for contributors Where files live the whole cache Overlay and first boot the disk on the image
Four roads from download and verify: image downloads through pkgproxy, checksums for contributors, where files live, and the overlay and the provisioning boot

Next: image downloads through pkgproxy, checksums for contributors, where files live and the overlay.

So a download is locked, resumed when it breaks, checked, and when the check fails, tried once more from a fresh list. To see the pkgproxy side, where images are kept by their hash for every machine on the network, go to image downloads through pkgproxy. How each system names its checksum has a deck of its own, for contributors. Where files live shows everything the downloads leave in the cache. And the overlay deck picks up with the verified image and builds the disk on top of it.

Details and sources

Read more

Basis of the statements on this page

The helpers were read in the image-garden script at the commit in garden.commit ("code"). The mismatch output was produced by running image-garden download-with-sha256sums from the v0.8.0 tree, whose helpers are the same, against a local file:// origin ("run"). Statements about formats published by distributions are marked "derived".

The download helpers

CommandChecksumUsed by
image-garden download-with-sha512sums URL [SUMS-URL] [DIR] [FILENAME]a SHA512SUMS list, by default SHA512SUMS next to URLsections with SHA512SUMS (Debian, Alpine)
image-garden download-with-sha256sums URL [SUMS-URL] [DIR] [FILENAME]a SHA256SUMS list, by default SHA256SUMS next to URLsections with SHA256SUMS (most), Ubuntu Core
image-garden download-with-sha256sum URL SHA256 [DIR] [FILENAME]one known hashsections with SHA256SUM (Oracle Linux)
image-garden download URL [DIR] [FILENAME]noneno generated rule

DIR defaults to $GARDEN_DL_DIR, FILENAME to the URL's last component (image-garden:358-668); image-garden --help shows the same brackets and defaults (image-garden:898-904). The generated rules always pass all four arguments.

Files in DIR

For FILE noble-server-cloudimg-amd64.img:

FileLifetime
FILE.lockmade by the first download, empty, left in place
FILE.sha256sums.partialwhile the list downloads; deleted on failure or interrupt
FILE.sha256sumsthe saved list; deleted with the image on a failed check
FILE.partialwhile the image downloads; kept on failure or interrupt, resumed next time
FILE.interruptedmarker made by the Control C trap, removed at once
FILEthe verified image

What the helper does, in order

1. Lock FILE.lock with flock. 2. Print the banner "Download and verify image: FILE (sha256sums)". 3. If FILE.sha256sums is missing: fetch the list through pkgproxy's volatile files, else from SUMS-URL with --retry 3, into .partial, then rename. 4. Find the hash of FILE in the list, else of the URL's basename. 5. If FILE is missing: fetch it through pkgproxy by hash, else from URL, into FILE.partial, resuming on curl exit 18 or 56 up to five times; rename. 6. Check the hash with sha256sum --check. On success, return. 7. On failure: remove FILE and FILE.sha256sums; on the first attempt go back to step 3, on the second fail.

Sources: image-garden:460-560, image-garden:211-327.

Messages

MessageMeaning
Downloading through pkgproxy at ADDRESSthe list or the image is asked from pkgproxy first
Cannot download through pkgproxy, downloading URL directlypkgproxy failed; the origin is used
Download of URL ended short, resumingcurl exit 18 or 56; the download continues
FILE: FAILEDthe hash does not match; both files are removed
Cannot find checksum entry for NAME in LISTthe list has no line for the file; both files are removed

The pkgproxy messages are explained in image downloads through pkgproxy and when pkgproxy is unreachable.

Checksum list formats

FormatExample line
GNUHASH NAME or HASH *NAME
BSDSHA256 (NAME) = HASH
barea file whose only non-empty line is a hash of the right length (Alpine)

(image-garden:75-92)

When the check runs again

The download recipe of a system runs when its cache link is older than its .url or .filename file. make follows the link to the image, so after an upgrade of image-garden or an edit of .image-garden.mk, which rewrite .url, the recipe runs on every make: the helper finds the list and the image, fetches nothing and checks the image again (run). This lasts until the image itself is downloaded again.

Roads from here

RUNTIME · make rules and the CLI

PKGPROXY · caching for guests

BUILD TIME · data and rules