An unreachable pkgproxy
seconds, not minutes for CI operators
A guest asks pkgproxy once at its first boot and, without an answer, skips its setup scripts and boots uncached; on the host, a download that cannot reach a pkgproxy URL remembers that for five minutes, so the other downloads go straight to the origin. A missing pkgproxy costs seconds instead of minutes.
On the map: pkgproxy
▶ Watch this deck — 6 scenes, ~5 min
Worth seeing first
- What pkgproxy gives a garden — what pkgproxy does for guests and downloads when it is there
01 Waiting for nobody
Without care, a missing pkgproxy makes every setup script in the guest and every download on the host wait for a timeout. Each side asks once, remembers, and skips the rest.
Now suppose the pkgproxy isn't there. The URL has a typo, or the machine that runs it is down. Two places would wait for it. In the guest, during its first boot, every setup script would try to connect, and GNU wget, left to itself, tries twenty times. On the host, every checksum list and every image would wait for a connection that never comes. Multiply that by each system in a spread run and a missing proxy costs minutes. image-garden makes it cost seconds instead: each side asks once, remembers the answer, and stops waiting. This deck follows both sides, for anyone who runs image-garden unattended.
Details and sources
Details
- The guest side: "A guest with neither curl nor wget skips them, as does one that cannot reach pkgproxy: it boots, uncached. bootcmd asks pkgproxy once, and when it does not answer, leaves PKGPROXY_UNREACHABLE_FILE in /run, so that neither batch waits for it again in that boot: an unreachable pkgproxy costs a guest seconds, not minutes" (010-qemu.mk:122-126).
- "GNU wget would try 20 times; busybox wget takes -t and ignores it" (010-qemu.mk:141-148).
- The host side: "A pkgproxy at a network address that cannot be reached costs each download its connect timeout", so image-garden remembers it for five minutes (image-garden:138-149, README.md:252-258).
- The GitLab craft components do not get this behaviour for their own setup step: their VM setup fails when the pkgproxy URL does not answer (see the craft components and pkgproxy).
02 The guest asks once
bootcmd adds the pkgproxy hosts line, then asks http://pkgproxy:3142/ once (curl: 5 s to connect, 20 s in all; wget: 5 s, one try). With no answer it prints why and touches /run/image-garden-pkgproxy-unreachable.
Start in the guest. Early in its first boot, cloud-init's bootcmd adds the pkgproxy line to /etc/hosts, then asks pkgproxy for its front page, just once. With curl, it waits five seconds for the connection and twenty for the whole answer. With wget, five seconds and a single try. If nothing answers, it prints image-garden: cannot reach pkgproxy, skipping its setup scripts in this boot, on the console. And it leaves a marker file, /run/image-garden-pkgproxy-unreachable. That empty file is the guest's memory of the failure for the rest of this boot.
Details and sources
Details
bootcmdrunsPKGPROXY_HOSTS_SHELL, thenPKGPROXY_PROBE_SHELL, then the apt setup on Debian and Ubuntu (010-qemu.mk:156-162).- The hosts line is added unless it is there:
grep -q '^IP pkgproxy$' /etc/hosts || { echo '# image-garden: pkgproxy, see URL/'; echo 'IP pkgproxy'; } >>/etc/hosts(010-qemu.mk:132-134).10.0.2.100is the address with the snap's socket; with a URL it is the URL's resolved address. - The probe:
curl -s -o /dev/null --connect-timeout 5 --max-time 20 URL/, orwget -q -T 5 -t 1 -O /dev/null URL/; on failure,image-garden: cannot reach pkgproxy at URL/, skipping its setup scripts in this booton standard error, andtouch /run/image-garden-pkgproxy-unreachable. A guest with neither tool asks nothing and leaves no file (010-qemu.mk:136-148). - The line was taken from the user-data that image-garden make generated for
ubuntu-cloud-24.04.x86_64withPKGPROXY_URLset, in a scratch directory. tests/check-pkgproxy-guest-shell.sh, part ofmake check, runs these lines under dash and busybox sh with stub curl and wget (proxy/README.md:255-259).
03 Skipped, and uncached for good
Both setup batches see the file and print image-garden: pkgproxy was unreachable, skipping setup scripts: apt (or snap). cloud-init is disabled after the provisioning boot, so the image stays uncached until SYSTEM.qcow2 is made again.
Both batches of setup scripts look for that file first. The apt script, in bootcmd, and the snap script, in runcmd, each print pkgproxy was unreachable, skipping setup scripts, and the script's name. So the guest boots without a cache, a few seconds late. Here's the catch. The provisioning boot is the only one that runs cloud-init: at its end, cloud-init is switched off. So the prepared image stays uncached, even after pkgproxy comes back. It does keep the hosts line, and GOPROXY in /etc/environment, which are written anyway. To get the cache, remove the prepared image and make it again.
Details and sources
Details
PKGPROXY_SETUP_SHELLchecks the marker first:image-garden: pkgproxy was unreachable, skipping setup scripts: TOOLSon standard error; otherwise it fetchesURL/setup/TOOLwithcurl -fsSL --connect-timeout 5, or wget, and pipes it tosh(010-qemu.mk:150-154). The apt batch runs frombootcmdon Debian and Ubuntu, the snap batch fromruncmdon all but Alpine (010-qemu.mk:127-130, 010-qemu.mk:156-168).- The hosts line is added before the probe, and again in
runcmd;GOPROXY="URL/go,direct"is appended to/etc/environmentbywrite_fileswhether or not pkgproxy answers (010-qemu.mk:132-134, 010-qemu.mk:170-180). - Derived from Go's documented behaviour, not run: with a comma-separated
GOPROXY, the go command moves on todirectonly when the proxy answers 404 or 410, so while pkgproxy stays unreachable, module downloads in such a guest fail to connect rather than going direct. - The last
runcmdcommand istouch /etc/cloud/cloud-init.disabled(010-qemu.mk:246); cloud-init does nothing on later boots (see provisioning runs once), sobootcmddoes not ask pkgproxy again. - proxy/README.md says the same: "All of this runs once, while the image is prepared: the last thing provisioning does is disable cloud-init, so later boots, including every boot of a
.runscript orallocate, neither ask pkgproxy again nor run the setup scripts. A guest prepared while pkgproxy could not be reached stays uncached until its image is made again: removeSYSTEM.qcow2and runimage-garden make SYSTEM.qcow2" (proxy/README.md:159-164). The comment on the marker file agrees: "Provisioning disables cloud-init at its end, so no later boot asks pkgproxy again" (010-qemu.mk:136-139). - Removing only
SYSTEM.qcow2is enough here: the user-data already holds the pkgproxy lines, so the new provisioning boot runs the setup scripts again. After a change of pkgproxy setting more files must go: see changing the proxy does not rebuild.
Read more
- Changing the proxy does not rebuild: which files to remove to make the system again.
- Provisioning runs once: how the first boot ends.
04 The host remembers for five minutes
curl exit 6, 7 or 28 writes $XDG_RUNTIME_DIR/image-garden/pkgproxy-unreachable.HASH and prints Not asking pkgproxy at URL again for 5 minutes. For 300 seconds every download prints Skipping pkgproxy at URL, unreachable recently, downloading URL directly.
On the host, each download gives pkgproxy one try, and waits ten seconds at most for the connection. When curl says it couldn't resolve the name, couldn't connect, or ran out of time connecting, image-garden writes a small file in your runtime directory, named after a hash of the pkgproxy's address. It prints Not asking pkgproxy at that address again for 5 minutes. Every download that starts in the next five minutes finds the file, still young, and goes straight to the origin, saying Skipping pkgproxy, unreachable recently. Five minutes covers a whole make or spread run, and still brings back a pkgproxy that was restarted soon after.
Details and sources
Details
- Both
fetch_volatileandfetch_imageskip pkgproxy whenpkgproxy_skipsucceeds (image-garden:216, image-garden:287), and callpkgproxy_unreachablewith curl's status after a failed try (image-garden:244-248, image-garden:302-307). - The file is
$XDG_RUNTIME_DIR/image-garden/pkgproxy-unreachable.followed by the first 16 hex digits of the sha256 of the address without its trailing slash; it holds the address (image-garden:152-161, image-garden:183-193). The hash on the slide is the one forhttp://pkgproxy.example.lan:3142. - Only curl's exit statuses 6 (could not resolve host), 7 (could not connect) and 28 (timed out; "the transfers through pkgproxy have no other time limit") are remembered (image-garden:178-187).
_pkgproxy_unreachable_ttl=300; a file 300 seconds old or older, or from the future, is ignored (image-garden:150, image-garden:163-176).- The log lines were produced by running the helper with
GARDEN_DL_PKGPROXYset to a closed port in a scratch directory; the address, the hash and the image URL on the slide are the example's. On that host the closed port timed out (28) rather than refusing (7). - Before the skip, the first download's own fallback prints
Cannot reach pkgproxy, downloading URL directlyfor a list, orCannot download through pkgproxy, downloading URL directlyfor an image.
Read more
- Where image-garden keeps its files: the runtime directory.
- Cloud images through pkgproxy: the downloads that skip it.
05 What is remembered, and what is not
Only a failure to resolve, connect or connect in time is remembered, per address. HTTP errors, cut answers and sockets are not. The file lives in $XDG_RUNTIME_DIR/image-garden, else /run/image-garden, and a reboot clears it.
Not every failure is remembered. An HTTP error, or an answer that's cut short, means pkgproxy is there, so the next download asks it again. A pkgproxy socket is never remembered: connecting to a socket that isn't there fails at once anyway. There's one file per address, so a dead pkgproxy doesn't make image-garden skip a different one. Downloads only check the file's age or write it, so parallel makes need no lock. A file from the future, after the clock was set back, counts as stale. Without a runtime directory it lives in /run, and where it can't be written, nothing is remembered. A reboot forgets it all.
Details and sources
Details
- "An HTTP error, or a response cut short, says that pkgproxy is there": only curl statuses 6, 7 and 28 write the file (image-garden:178-193).
- "A pkgproxy socket is not remembered: connecting to it fails at once";
pkgproxy_unreachable_fileprints nothing for aunix:address (image-garden:138-161). In the snap, a socket that does not exist is not used at all (image-garden:51-58). - "One file per pkgproxy address, named by a hash of it, so that another pkgproxy is not skipped. Every download only looks at the file's age, or touches it: concurrent ones need no lock" (image-garden:146-149).
- "A file from the future, as after the clock was set back, is stale" (image-garden:171-174).
XDG_RUNTIME_DIRdefaults to/run(image-garden:61-64); in the snap it is/run/user/UID/snap.image-garden, or/run/snap.image-gardenwhere/run/user/UIDdoes not exist (image-garden:66-73). When the directory cannot be made or the file written, nothing is remembered and no message is printed (image-garden:190-192)./runis a tmpfs on systemd systems, which is why a reboot forgets (derived).tests/check-dl-pkgproxy.shcovers each case: remembered, a file 290 and 301 seconds old, one dated an hour ahead, nowhere to remember, another pkgproxy, and HTTP errors not remembered (run at the tag: all pass).
Read more
- Where image-garden keeps its files: the other files in the runtime directory.
06 Where to go from here
Next: provisioning runs once, the craft components, the runtime directory, reaching pkgproxy, image downloads, starting a guest again and the user-data.
So a missing pkgproxy costs a guest a few seconds, and a download ten, once. Why the provisioning boot is the only one that sets a guest up is in the first boot deck. The GitLab craft components behave differently: their setup step fails when the proxy is down. The runtime directory, where the host keeps its note, has its own deck, and so does the way a guest reaches pkgproxy. Image downloads and their fallback come next. To make a guest again once pkgproxy is back, see why changing the proxy doesn't rebuild. And everything pkgproxy adds at first boot is in the user-data deck.
Details and sources
Read more
Basis of the statements on this page
Read in mk/010-qemu.mk, the image-garden script and proxy/README.md of image-garden at the commit this map describes ("code"). The guest's lines were taken from a user-data that v0.8.0's image-garden make generated with PKGPROXY_URL set; the host's messages were produced by running the download helper with GARDEN_DL_PKGPROXY at a closed port ("run"). That later boots do not probe again follows from cloud-init being disabled after the provisioning boot ("derived"), and proxy/README.md says so.
In the guest
| Step | Command | Limit | On failure |
|---|---|---|---|
probe, in bootcmd | curl -s -o /dev/null --connect-timeout 5 --max-time 20 http://pkgproxy:PORT/ | 5 s to connect, 20 s in all | image-garden: cannot reach pkgproxy at URL/, skipping its setup scripts in this boot, then touch /run/image-garden-pkgproxy-unreachable |
| probe without curl | wget -q -T 5 -t 1 -O /dev/null http://pkgproxy:PORT/ | 5 s, one try | the same |
apt setup, in bootcmd (Debian, Ubuntu) | curl -fsSL --connect-timeout 5 URL/setup/apt, piped to sh | 5 s to connect | skipped while the marker exists: image-garden: pkgproxy was unreachable, skipping setup scripts: apt |
snap setup, in runcmd (all but Alpine) | curl -fsSL --connect-timeout 5 URL/setup/snap, piped to sh | 5 s to connect | skipped: image-garden: pkgproxy was unreachable, skipping setup scripts: snap |
Sources: 010-qemu.mk:127-168. A guest with neither curl nor wget asks nothing and leaves no marker; its setup prints image-garden: no curl or wget to fetch URL/setup/TOOL.
Only the provisioning boot runs any of this: a guest prepared while pkgproxy could not be reached stays uncached until SYSTEM.qcow2 is removed and made again (proxy/README.md:159-164).
Written regardless of the probe: the pkgproxy line in /etc/hosts and GOPROXY="URL/go,direct" in /etc/environment (010-qemu.mk:132-134, 010-qemu.mk:170-180).
On the host
| Item | Value |
|---|---|
| tries per download | one, --connect-timeout 10 |
| remembered curl statuses | 6 (resolve), 7 (connect), 28 (timeout) |
| file | $XDG_RUNTIME_DIR/image-garden/pkgproxy-unreachable.HASH16, the address inside |
XDG_RUNTIME_DIR unset | /run |
| in the snap | /run/user/UID/snap.image-garden, or /run/snap.image-garden |
| lifetime | 300 seconds by the file's time; a file from the future is ignored |
sockets (unix:) | never remembered |
| directory not writable | nothing remembered, no message |
Sources: image-garden:61-73, image-garden:138-193.
Messages on the host
| Message | When |
|---|---|
Cannot reach pkgproxy, downloading URL directly | a list's try failed to connect |
Cannot download through pkgproxy, downloading URL directly | an image's try failed |
Not asking pkgproxy at ADDRESS again for 5 minutes | the failure was 6, 7 or 28 and the file was written |
Skipping pkgproxy at ADDRESS, unreachable recently, downloading URL directly | a later download within 300 seconds |
Roads from here
RUNTIME · make rules and the CLI
- Where image-garden keeps its files — the runtime directory where the host remembers
BUILT ON IT · CI integrations
- Why the craft components keep no cache — how the GitLab components fail instead when their pkgproxy is down
FIRST BOOT · cloud-init changes
- Provisioning runs once — why the provisioning boot is the only one that runs the setup
PKGPROXY · caching for guests
- How a guest reaches pkgproxy — why a guest may not reach a pkgproxy that is running
- Cloud images through pkgproxy — the downloads that fall back to the origin
- Changing the proxy does not rebuild — making a guest again once pkgproxy is back
- What pkgproxy adds to user-data — everything else pkgproxy adds to the guest's first boot