Cloud images through pkgproxy

files and volatile lists for CI operators

image-garden asks pkgproxy for each image by its URL and checksum and for each checksum list as a volatile file, falls back to the origin when pkgproxy cannot help, and checks the hash either way, so every machine downloads an image from the internet once and can fetch it while its origin is down.

On the map: pkgproxy

▶ Watch this deck — 6 scenes, ~5 min

Worth seeing first

01 One image, many machines

IMAGES THROUGH PKGPROXY runner 1 runner 2 … runner 10 Ubuntu's servers the same image, every time pkgproxy on your network by checksum the first time only if pkgproxy can't help: straight to the origin the checksum is verified either way
Several CI runners each download the same cloud image from Ubuntu's servers, every time. A pkgproxy on the network is asked first, by the image's checksum, and goes to the internet only the first time; when it cannot help, the runner goes to the origin itself. The checksum is verified either way.

image-garden asks a pkgproxy for each image by its checksum before going to the image's own URL. pkgproxy fetches it from the internet once, for every machine that uses it; the origin is the fallback, and the hash is checked either way.

Picture ten CI runners, each starting a job on ubuntu-cloud-24.04.x86_64. Every one of them fetches the same cloud image, hundreds of megabytes, from Ubuntu's servers, and every fresh runner does it again. A pkgproxy on your network can keep that image once. When image-garden knows where a pkgproxy is, it asks it for the image first, by the image's checksum, and pkgproxy goes to the internet only the first time. If pkgproxy can't help, image-garden goes to the origin itself, so a job doesn't fail because of the proxy. Either way, the checksum is verified. This deck is for anyone who runs image-garden on more than one machine.

Details and sources

Details

  • "One pkgproxy on the local network then downloads each image from the internet once, for every machine and CI job that uses it, with no CI cache to upload or restore. When that pkgproxy cannot be reached, or cannot give an image, image-garden downloads it directly, so a CI job never fails because of it. The checksum of the image is verified either way." (README.md:250-254)
  • fetch_image asks pkgproxy first when one is named and the checksum is known: "pkgproxy keeps a file by its checksum, so every machine that uses it downloads each image from the internet once" (image-garden:272-280).
  • The running example downloads https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img, checked against the SHA256SUMS next to it (ubuntu.ini:237-242).
  • Caching images by checksum needs pkgproxy 0.12 or later (README.md:234-240).
  • "Hundreds of megabytes" is typical for a cloud image; the exact size depends on the release.

Read more

02 Which pkgproxy it asks

WHICH PKGPROXY GARDEN_DL_PKGPROXY set? use it · set empty: no pkgproxy unset PKGPROXY_URL set? use that URL · a CI runner can set it unset in the snap: pkgproxy-mode external the URL you set component unix:$SNAP_COMMON/pkgproxy.sock snap unix:…/pkgproxy-host-snap/sockets/pkgproxy.sock a socket that is not there: no pkgproxy, no message GARDEN_DL_PKGPROXY=unix:/path/to/pkgproxy.sock
Three places in order: GARDEN_DL_PKGPROXY wins when set, and set empty means no pkgproxy; then PKGPROXY_URL; then, in the snap, the pkgproxy-mode its guests use: the URL, the bundled pkgproxy's socket or the pkgproxy snap's socket. A missing socket means no pkgproxy, silently. A socket can be named as unix: and a path.

GARDEN_DL_PKGPROXY wins, and empty turns it off; then PKGPROXY_URL; then, in the snap, the pkgproxy its guests use. A socket that does not exist means none. A socket can be named as unix:/path/to/pkgproxy.sock.

Which pkgproxy does image-garden ask? It looks in three places, in order. First GARDEN_DL_PKGPROXY. When it's set, it wins, and when it's set to nothing at all, downloads skip pkgproxy entirely. Next PKGPROXY_URL, the variable that tells any software where a pkgproxy is, which a CI runner can set once for all its jobs. Inside the snap, with neither set, downloads follow the setting your guests use: the URL you gave, or the socket of the bundled pkgproxy, or the socket of a pkgproxy snap. A socket that isn't there means no pkgproxy, and nothing is said. You can name a socket yourself, too, as unix: and its path.

Details and sources

Details

  • GARDEN_DL_PKGPROXY, when set, is used as it is; set but empty, it turns pkgproxy off for downloads. Unset, it becomes PKGPROXY_URL; unset too, and in the snap, it follows $SNAP_COMMON/pkgproxy-mode: external gives the URL in $SNAP_COMMON/pkgproxy-url, component gives unix:$SNAP_COMMON/pkgproxy.sock, snap gives unix:$SNAP_COMMON/pkgproxy-host-snap/sockets/pkgproxy.sock. A unix: path that is not a socket is dropped without a message (image-garden:31-59).
  • The snap only reads pkgproxy-mode when neither variable is set: in the snap, PKGPROXY_URL in the environment wins over the pkgproxy setting for downloads.
  • unix:PATH is reached with curl --unix-socket PATH at http://pkgproxy; any other value is a base URL, its trailing slash dropped (image-garden:121-136).
  • "Guests built with PKGPROXY_URL set use that pkgproxy too, also outside the snap" (README.md:246-248). GARDEN_DL_PKGPROXY affects downloads only.
  • image-garden-action sets GARDEN_DL_PKGPROXY=http://127.0.0.1:3142 on the runner, whichever mode its guests use (action.yaml:375-393).

Read more

03 The image, by its checksum

THE IMAGE, BY CHECKSUM image-garden sha256 2c9a…e41f GET /files/v1/fetch?url=https%3A%2F%2F…amd64.img&sha256=2c9a…e41f pkgproxy kept by its checksum the next machine gets this copy one try · 10 seconds to connect Cannot download through pkgproxy, downloading https://…/noble-server-cloudimg-amd64.img directly the image's URL noble-server-cloudimg-amd64.img.partial resumed, not restarted however it arrived, the hash is checked before make goes on
image-garden, holding the expected hash, asks pkgproxy for /files/v1/fetch with the image's URL and sha256; pkgproxy keeps the file by its checksum for the next machine. One try with a 10 second connect timeout; on failure the log says Cannot download through pkgproxy, downloading the URL directly, and the direct download resumes the .partial file. The hash is checked either way.

GET /files/v1/fetch?url=…&sha256=HASH, one try, a 10 second connect timeout. On failure, Cannot download through pkgproxy, downloading URL directly; the direct download resumes the .partial file. The hash is checked either way.

Now the image itself. image-garden already took the expected hash from the checksum list, so it asks pkgproxy at /files/v1/fetch, with the image's URL and that hash. pkgproxy keeps files by their checksum, so the next machine that asks for the same image gets the kept copy. image-garden gives pkgproxy one try, and waits ten seconds at most for the connection. If that fails, the log says Cannot download through pkgproxy, and the image comes from its URL directly. That download carries on from the partial file rather than starting over. However the bytes arrived, the hash is checked before make goes on.

Details and sources

Details

  • The query is files/v1/fetch?url=URL&KIND=HEX, with the URL percent-encoded and KIND sha256 or sha512 (image-garden:281-289, image-garden:94-105).
  • One try with --connect-timeout 10 --continue-at -, without --retry: "a pkgproxy that is down, which answers 503 that --retry would retry, or a file that does not match its checksum, which pkgproxy ends short each time, are not worth waiting for" (image-garden:290-301).
  • On any failure but an interrupt it prints Cannot download through pkgproxy, downloading URL directly, then downloads from the URL with --retry 5 --retry-connrefused --continue-at - into the same .partial file, resuming a transfer that ends short up to five times (image-garden:302-327).
  • The hash comes from the list before the image is fetched "so that the download can go through a pkgproxy" (image-garden:508-515), and is checked with sha256sum --check after (image-garden:542-545).
  • Only downloads with a checksum go through pkgproxy: the plain download command passes none and goes to the URL (image-garden:287, image-garden:651). Every image and firmware rule in the tree uses a download-with-… command.
  • The log on the slide was produced by running the helper against the test server tests/dl-pkgproxy-server.py in its 503 mode; URL and hash shortened.

Read more

04 The checksum list, as a volatile file

THE LIST, AS A VOLATILE FILE …/noble/current/SHA256SUMS changes with every new build: no fixed checksum image-garden GET /files/v1/volatile?url=https%3A%2F%2F…%2FSHA256SUMS pkgproxy every time the origin the last copy down Using the copy of https://…/noble/current/SHA256SUMS that pkgproxy kept, fetched at Mon, 02 Nov 2026 10:00:00 GMT, while the origin is down list and image kept: a VM can still be made
Noble's SHA256SUMS changes with every new build, so image-garden asks pkgproxy for it at /files/v1/volatile. pkgproxy asks the origin every time and keeps the last copy; when the origin is down it gives that copy, and the log says it uses the copy pkgproxy kept and when it was fetched. With the list and the image kept, a VM can still be made.

Checksum lists come from /files/v1/volatile?url=…: pkgproxy asks the origin each time and keeps the last copy. While the origin is down it gives that copy, and image-garden logs when it was fetched.

The checksum list is trickier. Noble's SHA256SUMS changes every time Ubuntu publishes a new build, so pkgproxy can't keep it by a checksum. Instead, image-garden asks for it as a volatile file, at /files/v1/volatile. For those, pkgproxy goes to the origin every time, and keeps the last copy it got. When the origin is down, pkgproxy hands over that copy instead, and the log says so: it's using the copy that pkgproxy kept, and when it was fetched. With the list and the image both kept by pkgproxy, you can still make a VM while the origin is down.

Details and sources

Details

  • fetch_volatile asks /files/v1/volatile?url=URL with one try and --connect-timeout 10, without --fail, keeping the response headers to tell what answered (image-garden:195-240). It is used for SHA512SUMS and SHA256SUMS lists (image-garden:386, image-garden:488); a system with a single SHA256SUM hash has no list to fetch.
  • A 200 answer with X-Pkgproxy-Cache: stale prints Using the copy of URL that pkgproxy kept, fetched at TIME, while the origin is down, the time from X-Pkgproxy-Fetched (image-garden:249-254). The line on the slide was produced by running the helper against tests/dl-pkgproxy-server.py in its stale mode, with the URL shortened.
  • README: "A VM then starts from an image that pkgproxy has, while its origin is down, and the log says that the checksum list is a kept copy, and when it was fetched" (README.md:260-266).
  • The running example's list is https://cloud-images.ubuntu.com/noble/current/SHA256SUMS (ubuntu.ini:237-242); current follows the newest build, which is why the list changes.
  • How pkgproxy stores volatile files and decides that the origin is down is pkgproxy's design: see the pkgproxy map.

Read more

05 When pkgproxy can't give the list

WHEN THE LIST CANNOT COME THROUGH Cannot reach pkgproxy, downloading URL directly pkgproxy has no volatile files, downloading URL directly pkgproxy does not give URL (HTTP 403), downloading it directly the origin two answers end the download Cannot download URL: the origin answered HTTP 404 through pkgproxy Cannot download URL: pkgproxy cannot reach the origin and kept no copy (HTTP 503) the partial list is removed, and make stops
When pkgproxy cannot be reached, has no volatile files, or refuses the file, image-garden prints why and fetches the checksum list from the origin. When the origin answered with an error through pkgproxy, or pkgproxy cannot reach the origin and kept no copy, image-garden prints that, removes the partial list and make stops.

Unreachable, no volatile files, or a refusal: the list comes from the origin. The origin's own error, passed on by pkgproxy, or a 502 or 503 with no copy kept: the download fails, as a direct one would.

And when pkgproxy can't give the list? That depends on why, and each case prints its own line. If pkgproxy can't be reached, or it's too old to have volatile files, the list comes straight from the origin. The same happens when pkgproxy refuses the file. Two answers end the download instead. If the origin itself answered with an error, such as not found, pkgproxy passes it on, and going direct would get the same answer. And if pkgproxy can't reach the origin and kept no copy, that's a 502 or 503. In both cases the partial list is removed, and make stops with the message.

Details and sources

Details

pkgproxy's answerMessageThen
no connection, or a cut answer (curl fails)Cannot reach pkgproxy, downloading URL directlythe origin; remembered when curl says 6, 7 or 28
200(the stale-copy line, when it is one)the list is used
no X-Pkgproxy-Engine: files-volatile header (a pkgproxy too old for volatile files)pkgproxy has no volatile files, downloading URL directlythe origin
X-Pkgproxy-Origin-URL present (the origin's own answer, such as 404)Cannot download URL: the origin answered HTTP N through pkgproxyfails, status 22
502 or 503 from files-volatileCannot download URL: pkgproxy cannot reach the origin and kept no copy (HTTP N)fails, status 22
any other code, such as 403 (a host it does not fetch from, a private address, a file too large) or 500pkgproxy does not give URL (HTTP N), downloading it directlythe origin
  • The direct download is curl --fail --location --retry 3 (image-garden:269).
  • On failure the helper removes FILE.sha256sums.partial and returns the status (image-garden:496-503); make stops on that recipe.
  • Every line on the slide was produced by running the helper against tests/dl-pkgproxy-server.py in its fail, old, refused, forbidden and unreachable modes; the 404 line is the same code path as the 403 the server gives in forbidden mode.
  • The README says the same: "The download fails when the origin answers with an error through pkgproxy, or when pkgproxy cannot reach the origin and has no copy, as a direct download would fail too. Any other error from pkgproxy, such as HTTP 403 or 500, is skipped like an unreachable one" (README.md:266-271). That a 500 goes to the origin was also confirmed by running the helper.

Read more

06 Where to go from here

ROADS Images via pkgproxy Download and check lock, partial, resume, check Caching on GitHub pkgproxy's cache between jobs GitLab components a shared pkgproxy, no cache Why pkgproxy apt, snaps, Go, images When it is down seconds, not minutes pkgproxy= guests and downloads alike
Six roads from Cloud images through pkgproxy: the download helper, caching on GitHub, the GitLab components' shared pkgproxy, why pkgproxy, an unreachable pkgproxy, and the snap's pkgproxy setting

Next: the download helper, caching on GitHub, the GitLab components' shared pkgproxy, why pkgproxy, an unreachable pkgproxy and the snap setting.

That's how an image reaches you through pkgproxy: by its checksum, with the list as a volatile file, and the origin as the fallback. The download helper around it, with its lock, partial file and check, has its own deck. On GitHub, image-garden-action keeps what pkgproxy fetched in the GitHub cache, and on GitLab, the craft components lean on a shared pkgproxy instead. Everything else pkgproxy carries is in the deck on why it's there. What a download does when pkgproxy is down has a deck of its own. And in the snap, one setting picks the pkgproxy for guests and downloads alike.

Details and sources

Read more

Basis of the statements on this page

Read in the image-garden script and README.md of image-garden at the commit this map describes ("code"). The messages were produced by running the download-with-sha256sums helper of v0.8.0, whose download code that commit leaves as it was, against the project's own fake pkgproxy, tests/dl-pkgproxy-server.py, in each of its modes, and against a closed port ("run"). How pkgproxy itself stores files is pkgproxy's design and is not described here.

Which pkgproxy downloads use

SetDownloads go through
GARDEN_DL_PKGPROXY=http://host:portthat pkgproxy
GARDEN_DL_PKGPROXY=unix:/path/to/pkgproxy.sockthe pkgproxy listening on that socket
GARDEN_DL_PKGPROXY= (set, empty)no pkgproxy
PKGPROXY_URL=http://host:port, GARDEN_DL_PKGPROXY unsetthat pkgproxy
neither, in the snap, pkgproxy=http://…the URL from $SNAP_COMMON/pkgproxy-url
neither, in the snap, pkgproxy=component (default)unix:$SNAP_COMMON/pkgproxy.sock, if that socket exists
neither, in the snap, pkgproxy=snapunix:$SNAP_COMMON/pkgproxy-host-snap/sockets/pkgproxy.sock, if it exists
neither, outside the snapno pkgproxy

Source: image-garden:31-59.

The two requests

WhatRequestTriesFalls back to the origin
an image with a known checksumGET /files/v1/fetch?url=URL&sha256=HEX (or sha512=)one, --connect-timeout 10, --continue-at -on any failure but an interrupt
a checksum list (SHA256SUMS, SHA512SUMS)GET /files/v1/volatile?url=URLone, --connect-timeout 10unless the origin's own error or a 502/503 came back through pkgproxy

The URL is percent-encoded (image-garden:94-105). Images: image-garden:272-327. Lists: image-garden:195-269. A pkgproxy socket is reached with curl --unix-socket at http://pkgproxy (image-garden:121-136).

Messages

MessageMeaning
Downloading through pkgproxy at ADDRESSthe list or image is asked from pkgproxy first
Using the copy of URL that pkgproxy kept, fetched at TIME, while the origin is downthe list is pkgproxy's last copy
Cannot download through pkgproxy, downloading URL directlythe image did not come through pkgproxy; the origin is used, resuming the partial file
Cannot reach pkgproxy, downloading URL directlyno connection, or a cut answer, for a list
pkgproxy has no volatile files, downloading URL directlythe pkgproxy is too old for volatile files
pkgproxy does not give URL (HTTP N), downloading it directlypkgproxy refused the list (403) or failed otherwise
Cannot download URL: the origin answered HTTP N through pkgproxythe origin's own error; the download fails
Cannot download URL: pkgproxy cannot reach the origin and kept no copy (HTTP N)502 or 503 with no copy; the download fails
Not asking pkgproxy at ADDRESS again for 5 minutessee an unreachable pkgproxy
Skipping pkgproxy at ADDRESS, unreachable recently, downloading URL directlysee an unreachable pkgproxy

What needs which pkgproxy

Images by checksum need pkgproxy 0.12 or later (README.md:234-240). A pkgproxy without volatile files is detected by the missing X-Pkgproxy-Engine: files-volatile header and skipped for lists (image-garden:255-256).

Roads from here

RUNTIME · make rules and the CLI

BUILT ON IT · CI integrations

PKGPROXY · caching for guests

SNAP · snap-only features

pkgproxy map ↗