Cloud images through pkgproxy
files and volatile lists for CI operators
image-garden asks pkgproxy for each image by its URL and checksum and for each checksum list as a volatile file, falls back to the origin when pkgproxy cannot help, and checks the hash either way, so every machine downloads an image from the internet once and can fetch it while its origin is down.
On the map: pkgproxy
▶ Watch this deck — 6 scenes, ~5 min
Worth seeing first
- Downloading and checking images — the download helper this deck adds pkgproxy to
01 One image, many machines
image-garden asks a pkgproxy for each image by its checksum before going to the image's own URL. pkgproxy fetches it from the internet once, for every machine that uses it; the origin is the fallback, and the hash is checked either way.
Picture ten CI runners, each starting a job on ubuntu-cloud-24.04.x86_64. Every one of them fetches the same cloud image, hundreds of megabytes, from Ubuntu's servers, and every fresh runner does it again. A pkgproxy on your network can keep that image once. When image-garden knows where a pkgproxy is, it asks it for the image first, by the image's checksum, and pkgproxy goes to the internet only the first time. If pkgproxy can't help, image-garden goes to the origin itself, so a job doesn't fail because of the proxy. Either way, the checksum is verified. This deck is for anyone who runs image-garden on more than one machine.
Details and sources
Details
- "One pkgproxy on the local network then downloads each image from the internet once, for every machine and CI job that uses it, with no CI cache to upload or restore. When that pkgproxy cannot be reached, or cannot give an image, image-garden downloads it directly, so a CI job never fails because of it. The checksum of the image is verified either way." (README.md:250-254)
fetch_imageasks pkgproxy first when one is named and the checksum is known: "pkgproxy keeps a file by its checksum, so every machine that uses it downloads each image from the internet once" (image-garden:272-280).- The running example downloads
https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img, checked against theSHA256SUMSnext to it (ubuntu.ini:237-242). - Caching images by checksum needs pkgproxy 0.12 or later (README.md:234-240).
- "Hundreds of megabytes" is typical for a cloud image; the exact size depends on the release.
Read more
- Downloading and checking images: the helper that does the download and the check.
- What pkgproxy gives a garden: the guests' traffic through the same pkgproxy.
02 Which pkgproxy it asks
GARDEN_DL_PKGPROXY wins, and empty turns it off; then PKGPROXY_URL; then, in the snap, the pkgproxy its guests use. A socket that does not exist means none. A socket can be named as unix:/path/to/pkgproxy.sock.
Which pkgproxy does image-garden ask? It looks in three places, in order. First GARDEN_DL_PKGPROXY. When it's set, it wins, and when it's set to nothing at all, downloads skip pkgproxy entirely. Next PKGPROXY_URL, the variable that tells any software where a pkgproxy is, which a CI runner can set once for all its jobs. Inside the snap, with neither set, downloads follow the setting your guests use: the URL you gave, or the socket of the bundled pkgproxy, or the socket of a pkgproxy snap. A socket that isn't there means no pkgproxy, and nothing is said. You can name a socket yourself, too, as unix: and its path.
Details and sources
Details
GARDEN_DL_PKGPROXY, when set, is used as it is; set but empty, it turns pkgproxy off for downloads. Unset, it becomesPKGPROXY_URL; unset too, and in the snap, it follows$SNAP_COMMON/pkgproxy-mode:externalgives the URL in$SNAP_COMMON/pkgproxy-url,componentgivesunix:$SNAP_COMMON/pkgproxy.sock,snapgivesunix:$SNAP_COMMON/pkgproxy-host-snap/sockets/pkgproxy.sock. Aunix:path that is not a socket is dropped without a message (image-garden:31-59).- The snap only reads
pkgproxy-modewhen neither variable is set: in the snap,PKGPROXY_URLin the environment wins over thepkgproxysetting for downloads. unix:PATHis reached withcurl --unix-socket PATHathttp://pkgproxy; any other value is a base URL, its trailing slash dropped (image-garden:121-136).- "Guests built with
PKGPROXY_URLset use that pkgproxy too, also outside the snap" (README.md:246-248).GARDEN_DL_PKGPROXYaffects downloads only. - image-garden-action sets
GARDEN_DL_PKGPROXY=http://127.0.0.1:3142on the runner, whichever mode its guests use (action.yaml:375-393).
Read more
- The pkgproxy snap setting: the four modes and what the configure hook records.
- How a guest reaches pkgproxy: the same choice, seen from the guest.
03 The image, by its checksum
GET /files/v1/fetch?url=…&sha256=HASH, one try, a 10 second connect timeout. On failure, Cannot download through pkgproxy, downloading URL directly; the direct download resumes the .partial file. The hash is checked either way.
Now the image itself. image-garden already took the expected hash from the checksum list, so it asks pkgproxy at /files/v1/fetch, with the image's URL and that hash. pkgproxy keeps files by their checksum, so the next machine that asks for the same image gets the kept copy. image-garden gives pkgproxy one try, and waits ten seconds at most for the connection. If that fails, the log says Cannot download through pkgproxy, and the image comes from its URL directly. That download carries on from the partial file rather than starting over. However the bytes arrived, the hash is checked before make goes on.
Details and sources
Details
- The query is
files/v1/fetch?url=URL&KIND=HEX, with the URL percent-encoded andKINDsha256orsha512(image-garden:281-289, image-garden:94-105). - One try with
--connect-timeout 10 --continue-at -, without--retry: "a pkgproxy that is down, which answers 503 that --retry would retry, or a file that does not match its checksum, which pkgproxy ends short each time, are not worth waiting for" (image-garden:290-301). - On any failure but an interrupt it prints
Cannot download through pkgproxy, downloading URL directly, then downloads from the URL with--retry 5 --retry-connrefused --continue-at -into the same.partialfile, resuming a transfer that ends short up to five times (image-garden:302-327). - The hash comes from the list before the image is fetched "so that the download can go through a pkgproxy" (image-garden:508-515), and is checked with
sha256sum --checkafter (image-garden:542-545). - Only downloads with a checksum go through pkgproxy: the plain
downloadcommand passes none and goes to the URL (image-garden:287, image-garden:651). Every image and firmware rule in the tree uses adownload-with-…command. - The log on the slide was produced by running the helper against the test server
tests/dl-pkgproxy-server.pyin its 503 mode; URL and hash shortened.
Read more
- Downloading and checking images: resume, interrupt and retry in detail.
04 The checksum list, as a volatile file
Checksum lists come from /files/v1/volatile?url=…: pkgproxy asks the origin each time and keeps the last copy. While the origin is down it gives that copy, and image-garden logs when it was fetched.
The checksum list is trickier. Noble's SHA256SUMS changes every time Ubuntu publishes a new build, so pkgproxy can't keep it by a checksum. Instead, image-garden asks for it as a volatile file, at /files/v1/volatile. For those, pkgproxy goes to the origin every time, and keeps the last copy it got. When the origin is down, pkgproxy hands over that copy instead, and the log says so: it's using the copy that pkgproxy kept, and when it was fetched. With the list and the image both kept by pkgproxy, you can still make a VM while the origin is down.
Details and sources
Details
fetch_volatileasks/files/v1/volatile?url=URLwith one try and--connect-timeout 10, without--fail, keeping the response headers to tell what answered (image-garden:195-240). It is used forSHA512SUMSandSHA256SUMSlists (image-garden:386, image-garden:488); a system with a singleSHA256SUMhash has no list to fetch.- A 200 answer with
X-Pkgproxy-Cache: staleprintsUsing the copy of URL that pkgproxy kept, fetched at TIME, while the origin is down, the time fromX-Pkgproxy-Fetched(image-garden:249-254). The line on the slide was produced by running the helper againsttests/dl-pkgproxy-server.pyin itsstalemode, with the URL shortened. - README: "A VM then starts from an image that pkgproxy has, while its origin is down, and the log says that the checksum list is a kept copy, and when it was fetched" (README.md:260-266).
- The running example's list is
https://cloud-images.ubuntu.com/noble/current/SHA256SUMS(ubuntu.ini:237-242);currentfollows the newest build, which is why the list changes. - How pkgproxy stores volatile files and decides that the origin is down is pkgproxy's design: see the pkgproxy map.
Read more
- An unreachable pkgproxy: what happens when pkgproxy itself is down.
- Kinds of checksum: which systems have a list and which a single hash.
05 When pkgproxy can't give the list
Unreachable, no volatile files, or a refusal: the list comes from the origin. The origin's own error, passed on by pkgproxy, or a 502 or 503 with no copy kept: the download fails, as a direct one would.
And when pkgproxy can't give the list? That depends on why, and each case prints its own line. If pkgproxy can't be reached, or it's too old to have volatile files, the list comes straight from the origin. The same happens when pkgproxy refuses the file. Two answers end the download instead. If the origin itself answered with an error, such as not found, pkgproxy passes it on, and going direct would get the same answer. And if pkgproxy can't reach the origin and kept no copy, that's a 502 or 503. In both cases the partial list is removed, and make stops with the message.
Details and sources
Details
- The decision after one try at
/files/v1/volatile(image-garden:241-269):
| pkgproxy's answer | Message | Then |
|---|---|---|
| no connection, or a cut answer (curl fails) | Cannot reach pkgproxy, downloading URL directly | the origin; remembered when curl says 6, 7 or 28 |
| 200 | (the stale-copy line, when it is one) | the list is used |
no X-Pkgproxy-Engine: files-volatile header (a pkgproxy too old for volatile files) | pkgproxy has no volatile files, downloading URL directly | the origin |
X-Pkgproxy-Origin-URL present (the origin's own answer, such as 404) | Cannot download URL: the origin answered HTTP N through pkgproxy | fails, status 22 |
| 502 or 503 from files-volatile | Cannot download URL: pkgproxy cannot reach the origin and kept no copy (HTTP N) | fails, status 22 |
| any other code, such as 403 (a host it does not fetch from, a private address, a file too large) or 500 | pkgproxy does not give URL (HTTP N), downloading it directly | the origin |
- The direct download is
curl --fail --location --retry 3(image-garden:269). - On failure the helper removes
FILE.sha256sums.partialand returns the status (image-garden:496-503); make stops on that recipe. - Every line on the slide was produced by running the helper against
tests/dl-pkgproxy-server.pyin itsfail,old,refused,forbiddenand unreachable modes; the 404 line is the same code path as the 403 the server gives inforbiddenmode. - The README says the same: "The download fails when the origin answers with an error through pkgproxy, or when pkgproxy cannot reach the origin and has no copy, as a direct download would fail too. Any other error from pkgproxy, such as HTTP 403 or 500, is skipped like an unreachable one" (README.md:266-271). That a 500 goes to the origin was also confirmed by running the helper.
Read more
- An unreachable pkgproxy: how the unreachable case is remembered for five minutes.
- Downloading and checking images: the rest of the helper's messages.
06 Where to go from here
Next: the download helper, caching on GitHub, the GitLab components' shared pkgproxy, why pkgproxy, an unreachable pkgproxy and the snap setting.
That's how an image reaches you through pkgproxy: by its checksum, with the list as a volatile file, and the origin as the fallback. The download helper around it, with its lock, partial file and check, has its own deck. On GitHub, image-garden-action keeps what pkgproxy fetched in the GitHub cache, and on GitLab, the craft components lean on a shared pkgproxy instead. Everything else pkgproxy carries is in the deck on why it's there. What a download does when pkgproxy is down has a deck of its own. And in the snap, one setting picks the pkgproxy for guests and downloads alike.
Details and sources
Read more
- Downloading and checking images, caching on GitHub, the craft components and pkgproxy.
- What pkgproxy gives a garden, an unreachable pkgproxy, the pkgproxy snap setting.
- The pkgproxy map: how pkgproxy keeps files and volatile files.
Read more
Basis of the statements on this page
Read in the image-garden script and README.md of image-garden at the commit this map describes ("code"). The messages were produced by running the download-with-sha256sums helper of v0.8.0, whose download code that commit leaves as it was, against the project's own fake pkgproxy, tests/dl-pkgproxy-server.py, in each of its modes, and against a closed port ("run"). How pkgproxy itself stores files is pkgproxy's design and is not described here.
Which pkgproxy downloads use
| Set | Downloads go through |
|---|---|
GARDEN_DL_PKGPROXY=http://host:port | that pkgproxy |
GARDEN_DL_PKGPROXY=unix:/path/to/pkgproxy.sock | the pkgproxy listening on that socket |
GARDEN_DL_PKGPROXY= (set, empty) | no pkgproxy |
PKGPROXY_URL=http://host:port, GARDEN_DL_PKGPROXY unset | that pkgproxy |
neither, in the snap, pkgproxy=http://… | the URL from $SNAP_COMMON/pkgproxy-url |
neither, in the snap, pkgproxy=component (default) | unix:$SNAP_COMMON/pkgproxy.sock, if that socket exists |
neither, in the snap, pkgproxy=snap | unix:$SNAP_COMMON/pkgproxy-host-snap/sockets/pkgproxy.sock, if it exists |
| neither, outside the snap | no pkgproxy |
Source: image-garden:31-59.
The two requests
| What | Request | Tries | Falls back to the origin |
|---|---|---|---|
| an image with a known checksum | GET /files/v1/fetch?url=URL&sha256=HEX (or sha512=) | one, --connect-timeout 10, --continue-at - | on any failure but an interrupt |
a checksum list (SHA256SUMS, SHA512SUMS) | GET /files/v1/volatile?url=URL | one, --connect-timeout 10 | unless the origin's own error or a 502/503 came back through pkgproxy |
The URL is percent-encoded (image-garden:94-105). Images: image-garden:272-327. Lists: image-garden:195-269. A pkgproxy socket is reached with curl --unix-socket at http://pkgproxy (image-garden:121-136).
Messages
| Message | Meaning |
|---|---|
Downloading through pkgproxy at ADDRESS | the list or image is asked from pkgproxy first |
Using the copy of URL that pkgproxy kept, fetched at TIME, while the origin is down | the list is pkgproxy's last copy |
Cannot download through pkgproxy, downloading URL directly | the image did not come through pkgproxy; the origin is used, resuming the partial file |
Cannot reach pkgproxy, downloading URL directly | no connection, or a cut answer, for a list |
pkgproxy has no volatile files, downloading URL directly | the pkgproxy is too old for volatile files |
pkgproxy does not give URL (HTTP N), downloading it directly | pkgproxy refused the list (403) or failed otherwise |
Cannot download URL: the origin answered HTTP N through pkgproxy | the origin's own error; the download fails |
Cannot download URL: pkgproxy cannot reach the origin and kept no copy (HTTP N) | 502 or 503 with no copy; the download fails |
Not asking pkgproxy at ADDRESS again for 5 minutes | see an unreachable pkgproxy |
Skipping pkgproxy at ADDRESS, unreachable recently, downloading URL directly | see an unreachable pkgproxy |
What needs which pkgproxy
Images by checksum need pkgproxy 0.12 or later (README.md:234-240). A pkgproxy without volatile files is detected by the missing X-Pkgproxy-Engine: files-volatile header and skipped for lists (image-garden:255-256).
Roads from here
RUNTIME · make rules and the CLI
- Downloading and checking images — the lock, the partial file, the resume and the check around each download
BUILT ON IT · CI integrations
- How the action caches — how image-garden-action keeps what pkgproxy fetched between jobs
- Why the craft components keep no cache — the GitLab components, which rely on a shared pkgproxy instead of a CI cache
PKGPROXY · caching for guests
- What pkgproxy gives a garden — everything else that goes through pkgproxy
- An unreachable pkgproxy — what a download does when pkgproxy cannot be reached
SNAP · snap-only features
- The pkgproxy snap setting — the snap setting that also picks the pkgproxy for downloads
pkgproxy map ↗
- Files by checksum — on the pkgproxy map, how files are cached by their checksum