Downloading and checking images
lock, resume, check, retry for users
make calls image-garden's download helpers with the image URL, its checksum list or hash, the cache directory and the file name. A helper takes a lock on the file, fetches the checksum list, then the image through pkgproxy or from the origin into a .partial file, resumes a transfer that ended short, checks the hash, and on a mismatch deletes both files and tries once more.
On the map: runtime
▶ Watch this deck — 7 scenes, ~5 min
Worth seeing first
- From a target name to a VM — where the download sits among the ten steps
01 Large files from someone else's server
The download step runs image-garden download-with-sha256sums URL SUMS-URL DIR FILE, which copes with broken transfers, stale lists, interrupts and concurrent makes.
A cloud image is a few hundred megabytes on someone else's server, and a lot can go wrong while you fetch it. The connection can drop half-way. The distribution can publish a new image while your checksum list is old. You can press Control C. And two makes can want the same file at once. So make doesn't call curl directly. It calls image-garden download-with-sha256sums, one of the download helpers built into the image-garden command, with the URL, the checksum list, the directory and the file name. This deck follows that helper for Ubuntu 24.04 on x86.
Details and sources
Details
- The download rule of a
SHA256SUMSsystem runs$(GARDEN_ITSELF) download-with-sha256sums $(cat X.url) <SHA256SUMS> $(GARDEN_DL_DIR)/<group> $(cat X.filename)(rulegen.py:510-514);GARDEN_ITSELFis theimage-gardenscript that started make (image-garden:982-993). - The helpers are
download-with-sha512sums,download-with-sha256sums,download-with-sha256sumanddownload(no checksum, not used by the generated rules) (image-garden:358-668). The checksum list defaults toSHA256SUMSnext to the image, the directory to$GARDEN_DL_DIRand the file name to the URL's last component (image-garden:460-473). image-garden --helpbrackets the optional arguments,download-with-sha256sums URL [SHA256SUMS_URL [DIRECTORY_PREFIX [FILENAME]]], and says what they default to; onlydownload-with-sha256sumneeds its checksum (image-garden:898-904, image-garden:562-577). The man page agrees (image-garden.1:24-27).- The values on the slide are those of
[ubuntu-cloud-24.04.x86_64](ubuntu.ini:237-242) with the cache of a native or source install. - In CI, with
CIset, curl runs without its progress meter (image-garden:107-119).
Read more
02 Three ways to name the checksum
Ubuntu's and Debian's images are checked against a published list of SHA-256 or SHA-512 hashes; Oracle's against a hash image-garden carries itself. Lists may be GNU style, BSD style or one bare hash.
Every image is checked against a checksum, which comes one of three ways. Ubuntu publishes SHA256SUMS, a list next to its images. Debian and Alpine publish SHA512SUMS lists, with longer hashes. For Oracle Linux, image-garden carries a single hash itself. Each way picks its own helper. Lists come in three shapes too: a hash and a file name per line, as Ubuntu's does, the BSD style with the name in brackets, and a file that holds one bare hash, as Alpine publishes. The helper looks for the saved file name first, then for the name at the end of the URL.
Details and sources
Details
- The download rule uses
download-with-sha512sumswhenSHA512SUMSis set, elsedownload-with-sha256sumwhenSHA256SUMis set, elsedownload-with-sha256sums(rulegen.py:498-517). A section without any of the three is refused when the rules are generated (rulegen.py:426-431). - In the data at the commit in garden.commit, Debian (12 sections) and Alpine (2) use
SHA512SUMS, Oracle Linux (7) usesSHA256SUM, every other fileSHA256SUMS. Fedora'sSHA256SUMSpoints at itsCHECKSUMfile, openSUSE's at a.sha256file;@URL@in the value stands for the image URL (rulegen.py:82-88). find_checksum_hashacceptsHASH NAMEorHASH *NAMElines, BSD linesKIND (NAME) = HASH, and a file whose only non-empty line is a bare hash of the right length (image-garden:75-92).- The name looked up is the saved file name when the list mentions it, otherwise the URL's basename (image-garden:508-515).
- The hashes on the slide are shortened; the Fedora line shows the BSD format (that Fedora publishes this format is derived).
Read more
03 The list first, then the image
SHA256SUMS is saved as noble-server-cloudimg-amd64.img.sha256sums; the image arrives as .partial, through pkgproxy by hash or from the origin, is renamed when complete, then checked.
The helper works in a fixed order. First the checksum list. It's saved next to the image, under the image's name with .sha256sums on the end, and fetched only when that file is missing. From the list it takes the expected hash before the image arrives. That's what lets it ask pkgproxy, when there is one, for the image by its hash, and fall back to the origin if pkgproxy can't help. The image goes to a .partial file, and only a complete download is renamed to its real name. Then the hash is checked. Until it matches, make goes no further.
Details and sources
Details
- The list is fetched only when
DIR/FILE.sha256sumsdoes not exist, toFILE.sha256sums.partial, and renamed when complete (image-garden:480-506). The hash is taken from it before the image is fetched, "so that the download can go through a pkgproxy" (image-garden:508-515). - With a pkgproxy (
GARDEN_DL_PKGPROXY, by defaultPKGPROXY_URLor the snap's), the list is asked from/files/v1/volatile?url=…(image-garden:211-269) and the image from/files/v1/fetch?url=…&sha256=HASH, one try with a 10 second connect timeout, then from the URL directly (image-garden:281-308). The details are in image downloads through pkgproxy. - The image is fetched only when
DIR/FILEdoes not exist, intoFILE.partial, thenmv -vtoFILE(image-garden:517-540). - The check is
printf '%s %s\n' HASH FILE | sha256sum --check, run in DIR (image-garden:542-545). It also runs when the image was already there. - The hash on the slide is shortened.
- The README: "Images and their checksum lists are downloaded over HTTPS, through pkgproxy when one is available, and every image is checked against its checksum" (README.md:34-35); every image URL in the data is https.
04 A broken download picks up where it stopped
curl exit 18 (short response) or 56 (broken connection) is resumed with --continue-at -, up to five times. After Control C the image's .partial stays for the next run; a partial list is removed.
Say the connection breaks at sixty percent. curl's own retry would start over from zero, so the helper does something better. When curl reports a response that ended short, or a connection that broke, the helper resumes from the end of the .partial file, up to five times. Other errors, such as a refused connection, get curl's five retries. Press Control C, and the helper keeps the partial image and stops. Next time you run make, the download continues from there. A partial checksum list is the exception: it's deleted, because the list is small and it's what protects the image.
Details and sources
Details
- "curl --retry starts a download over, it does not resume: a response that ends short (18) or a connection that breaks (56) is resumed here instead, a few times": the direct download loops up to five times on exit 18 or 56, each with
curl --fail --location --retry 5 --retry-connrefused --continue-at -(image-garden:310-327, image-garden:107-119). It prints "Download of URL ended short, resuming". - On SIGINT a trap touches
FILE.interrupted; the helper then removes that marker and returns 1. For the image, "Leave the partial file around, on interrupt or failure, so that we can resume the download later"; for the list, "Remove the partial file, on interrupt or failure, as it is our only protection against corruptions of the main content and the checksum files are small" (image-garden:480-540). - A download through pkgproxy is also written to
.partialwith--continue-at -, so the direct download that follows resumes what came through pkgproxy (image-garden:290-298). - The checksum list from the origin is fetched with
--retry 3(image-garden:269-269). - The
downloadhelper, which has no checksum, deletes its partial file on interrupt (image-garden:655-660).
Read more
05 A mismatch, then one more try
On FAILED both the image and .sha256sums are removed and the download runs once more from the list; a second failure stops make. Only the .lock file is left.
Now the image is complete, but its hash doesn't match the list. Which one is wrong? Often the list: the distribution published a new image under the same name, and the list you saved is from before. So the helper deletes both, the image and the list, and runs the whole download once more, list first. The same happens when the list has no entry for the file. If the second try fails too, the helper gives up, and make stops with an error. Nothing that failed its check is kept, and the output names each file as it's removed, so you can see what happened.
Details and sources
Details
- The helper loops
for attempt in first retry. When the hash is empty orsha256sum --checkfails, it runsrm -vf FILE FILE.sha256sums, "so that we re-download the checksum file. This can help if we have a stale checksum from the past that will never match the current download", continues on the first attempt and returns 1 on the second (image-garden:542-559). The SHA512 helper does the same (image-garden:440-457); the single-hash helper removes only the image (image-garden:613-623). - With no entry it prints "Cannot find checksum entry for NAME in LIST" first (image-garden:545-548).
- The output on the slide was produced by running the v0.8.0 helper on a local
file://origin whose list held a wrong hash, withCIset; paths are shortened, and therm -vlines are shown as an English locale prints them. After both attempts onlyFILE.lockwas left in the directory. - A failing helper fails the download recipe, so make stops; the cache link is not made.
Read more
06 Two makes, one file
Each helper holds flock on FILE.lock across fetch, check and retry. The link is replaced with one mv; allocate locks SYSTEM.lock around its make.
Last, sharing. Two systems can start from one image, such as Ubuntu Core 24 with and without a TPM, and in CI two jobs on one machine can make the same system at once. So each helper takes a lock on a .lock file next to the image before it does anything else. The second helper waits, then finds the image already there, and only checks it. The link named after the system is swapped in with a single rename, so two makes can write it at the same moment and neither fails. And when you allocate, a lock per system keeps two allocations from building the same disk at once.
Details and sources
Details
- "Each download command parses its arguments, then runs its _locked body with fd 9 open on DIR/FILENAME.lock. The body takes an exclusive lock on fd 9 and holds it across the fetch, the checksum and the retry … Systems that share a base image download it once: the second waits, then reuses the verified file." (image-garden:351-356, image-garden:472-476). The second helper skips the fetch because the file exists, and runs the check again.
GARDEN_SYMLINKreplaces the link with one rename, because "of two at once, one may find the other's link and fail, as uutils ln, the default ln in Ubuntu 26.04, does" (000-garden.mk:58-64); the link is.PRECIOUS(rulegen.py:494-496).- Ubuntu Core 24 and 24+tpm both start from
ubuntu-core-24-amd64.img.xz; its conversion to qcow2 in the cache runs under a lock of its own (010-qemu.mk:304-347); see Ubuntu Core. allocaterunsflock <.image-garden/ or ./>SYSTEM.lock image-garden make SYSTEM.run SYSTEM.qcow2unless--premade(image-garden:1058-1064).- The lock files stay in the cache after the download; they are empty.
Read more
07 Where to go from here
Next: image downloads through pkgproxy, checksums for contributors, where files live and the overlay.
So a download is locked, resumed when it breaks, checked, and when the check fails, tried once more from a fresh list. To see the pkgproxy side, where images are kept by their hash for every machine on the network, go to image downloads through pkgproxy. How each system names its checksum has a deck of its own, for contributors. Where files live shows everything the downloads leave in the cache. And the overlay deck picks up with the verified image and builds the disk on top of it.
Details and sources
Read more
Basis of the statements on this page
The helpers were read in the image-garden script at the commit in garden.commit ("code"). The mismatch output was produced by running image-garden download-with-sha256sums from the v0.8.0 tree, whose helpers are the same, against a local file:// origin ("run"). Statements about formats published by distributions are marked "derived".
The download helpers
| Command | Checksum | Used by |
|---|---|---|
image-garden download-with-sha512sums URL [SUMS-URL] [DIR] [FILENAME] | a SHA512SUMS list, by default SHA512SUMS next to URL | sections with SHA512SUMS (Debian, Alpine) |
image-garden download-with-sha256sums URL [SUMS-URL] [DIR] [FILENAME] | a SHA256SUMS list, by default SHA256SUMS next to URL | sections with SHA256SUMS (most), Ubuntu Core |
image-garden download-with-sha256sum URL SHA256 [DIR] [FILENAME] | one known hash | sections with SHA256SUM (Oracle Linux) |
image-garden download URL [DIR] [FILENAME] | none | no generated rule |
DIR defaults to $GARDEN_DL_DIR, FILENAME to the URL's last component (image-garden:358-668); image-garden --help shows the same brackets and defaults (image-garden:898-904). The generated rules always pass all four arguments.
Files in DIR
For FILE noble-server-cloudimg-amd64.img:
| File | Lifetime |
|---|---|
FILE.lock | made by the first download, empty, left in place |
FILE.sha256sums.partial | while the list downloads; deleted on failure or interrupt |
FILE.sha256sums | the saved list; deleted with the image on a failed check |
FILE.partial | while the image downloads; kept on failure or interrupt, resumed next time |
FILE.interrupted | marker made by the Control C trap, removed at once |
FILE | the verified image |
What the helper does, in order
1. Lock FILE.lock with flock. 2. Print the banner "Download and verify image: FILE (sha256sums)". 3. If FILE.sha256sums is missing: fetch the list through pkgproxy's volatile files, else from SUMS-URL with --retry 3, into .partial, then rename. 4. Find the hash of FILE in the list, else of the URL's basename. 5. If FILE is missing: fetch it through pkgproxy by hash, else from URL, into FILE.partial, resuming on curl exit 18 or 56 up to five times; rename. 6. Check the hash with sha256sum --check. On success, return. 7. On failure: remove FILE and FILE.sha256sums; on the first attempt go back to step 3, on the second fail.
Sources: image-garden:460-560, image-garden:211-327.
Messages
| Message | Meaning |
|---|---|
Downloading through pkgproxy at ADDRESS | the list or the image is asked from pkgproxy first |
Cannot download through pkgproxy, downloading URL directly | pkgproxy failed; the origin is used |
Download of URL ended short, resuming | curl exit 18 or 56; the download continues |
FILE: FAILED | the hash does not match; both files are removed |
Cannot find checksum entry for NAME in LIST | the list has no line for the file; both files are removed |
The pkgproxy messages are explained in image downloads through pkgproxy and when pkgproxy is unreachable.
Checksum list formats
| Format | Example line |
|---|---|
| GNU | HASH NAME or HASH *NAME |
| BSD | SHA256 (NAME) = HASH |
| bare | a file whose only non-empty line is a hash of the right length (Alpine) |
When the check runs again
The download recipe of a system runs when its cache link is older than its .url or .filename file. make follows the link to the image, so after an upgrade of image-garden or an edit of .image-garden.mk, which rewrite .url, the recipe runs on every make: the helper finds the list and the image, fetches nothing and checks the image again (run). This lasts until the image itself is downloaded again.
Roads from here
RUNTIME · make rules and the CLI
- Where image-garden keeps its files — every file the downloads leave in the cache
- The overlay and the provisioning boot — the disk that is built on the verified image
- From a target name to a VM — the other nine steps
PKGPROXY · caching for guests
- Cloud images through pkgproxy — how pkgproxy keeps images by their hash for every machine
- An unreachable pkgproxy — what happens when pkgproxy is down
BUILD TIME · data and rules
- Every image has a checksum — for contributors, how each system names its checksum