An unreachable pkgproxy

seconds, not minutes for CI operators

A guest asks pkgproxy once at its first boot and, without an answer, skips its setup scripts and boots uncached; on the host, a download that cannot reach a pkgproxy URL remembers that for five minutes, so the other downloads go straight to the origin. A missing pkgproxy costs seconds instead of minutes.

On the map: pkgproxy

▶ Watch this deck — 6 scenes, ~5 min

Worth seeing first

01 Waiting for nobody

AN UNREACHABLE PKGPROXY pkgproxy not there in the guest, first boot setup script: apt setup script: snap GNU wget: twenty tries on the host, downloads checksum list image × every system in a spread run: minutes of waiting ask once · remember · skip the rest: seconds
A pkgproxy that is not there. In the guest's first boot, the apt and snap setup scripts would wait for it, and GNU wget tries twenty times; on the host, every checksum list and image download would wait too. Across a spread run that costs minutes; image-garden asks once on each side, remembers, and skips, so it costs seconds.

Without care, a missing pkgproxy makes every setup script in the guest and every download on the host wait for a timeout. Each side asks once, remembers, and skips the rest.

Now suppose the pkgproxy isn't there. The URL has a typo, or the machine that runs it is down. Two places would wait for it. In the guest, during its first boot, every setup script would try to connect, and GNU wget, left to itself, tries twenty times. On the host, every checksum list and every image would wait for a connection that never comes. Multiply that by each system in a spread run and a missing proxy costs minutes. image-garden makes it cost seconds instead: each side asks once, remembers the answer, and stops waiting. This deck follows both sides, for anyone who runs image-garden unattended.

Details and sources

Details

  • The guest side: "A guest with neither curl nor wget skips them, as does one that cannot reach pkgproxy: it boots, uncached. bootcmd asks pkgproxy once, and when it does not answer, leaves PKGPROXY_UNREACHABLE_FILE in /run, so that neither batch waits for it again in that boot: an unreachable pkgproxy costs a guest seconds, not minutes" (010-qemu.mk:122-126).
  • "GNU wget would try 20 times; busybox wget takes -t and ignores it" (010-qemu.mk:141-148).
  • The host side: "A pkgproxy at a network address that cannot be reached costs each download its connect timeout", so image-garden remembers it for five minutes (image-garden:138-149, README.md:252-258).
  • The GitLab craft components do not get this behaviour for their own setup step: their VM setup fails when the pkgproxy URL does not answer (see the craft components and pkgproxy).

02 The guest asks once

THE GUEST ASKS ONCE bootcmd, early in the first boot /etc/hosts += 10.0.2.100 pkgproxy curl -s -o /dev/null --connect-timeout 5 --max-time 20 http://pkgproxy:3142/ wget -q -T 5 -t 1 -O /dev/null http://pkgproxy:3142/ without curl image-garden: cannot reach pkgproxy at http://pkgproxy:3142/, skipping its setup scripts in this boot on the console /run/image-garden-pkgproxy-unreachable an empty file: the guest's memory, for the rest of this boot
In the guest's first boot, bootcmd adds 10.0.2.100 pkgproxy to /etc/hosts, then asks http://pkgproxy:3142/ once, with curl's 5 second connect and 20 second total limits or wget's 5 seconds and one try. With no answer it prints that it cannot reach pkgproxy and skips its setup scripts in this boot, and touches /run/image-garden-pkgproxy-unreachable.

bootcmd adds the pkgproxy hosts line, then asks http://pkgproxy:3142/ once (curl: 5 s to connect, 20 s in all; wget: 5 s, one try). With no answer it prints why and touches /run/image-garden-pkgproxy-unreachable.

Start in the guest. Early in its first boot, cloud-init's bootcmd adds the pkgproxy line to /etc/hosts, then asks pkgproxy for its front page, just once. With curl, it waits five seconds for the connection and twenty for the whole answer. With wget, five seconds and a single try. If nothing answers, it prints image-garden: cannot reach pkgproxy, skipping its setup scripts in this boot, on the console. And it leaves a marker file, /run/image-garden-pkgproxy-unreachable. That empty file is the guest's memory of the failure for the rest of this boot.

Details and sources

Details

  • bootcmd runs PKGPROXY_HOSTS_SHELL, then PKGPROXY_PROBE_SHELL, then the apt setup on Debian and Ubuntu (010-qemu.mk:156-162).
  • The hosts line is added unless it is there: grep -q '^IP pkgproxy$' /etc/hosts || { echo '# image-garden: pkgproxy, see URL/'; echo 'IP pkgproxy'; } >>/etc/hosts (010-qemu.mk:132-134). 10.0.2.100 is the address with the snap's socket; with a URL it is the URL's resolved address.
  • The probe: curl -s -o /dev/null --connect-timeout 5 --max-time 20 URL/, or wget -q -T 5 -t 1 -O /dev/null URL/; on failure, image-garden: cannot reach pkgproxy at URL/, skipping its setup scripts in this boot on standard error, and touch /run/image-garden-pkgproxy-unreachable. A guest with neither tool asks nothing and leaves no file (010-qemu.mk:136-148).
  • The line was taken from the user-data that image-garden make generated for ubuntu-cloud-24.04.x86_64 with PKGPROXY_URL set, in a scratch directory.
  • tests/check-pkgproxy-guest-shell.sh, part of make check, runs these lines under dash and busybox sh with stub curl and wget (proxy/README.md:255-259).

03 Skipped, and uncached for good

SKIPPED, AND UNCACHED FOR GOOD /run/image-garden-pkgproxy-unreachable is there bootcmd: apt image-garden: pkgproxy was unreachable, skipping setup scripts: apt runcmd: snap image-garden: pkgproxy was unreachable, skipping setup scripts: snap the guest boots without a cache, a few seconds late - touch /etc/cloud/cloud-init.disabled prepared image: uncached for good kept anyway: the pkgproxy hosts line · GOPROXY in /etc/environment to get the cache: remove SYSTEM.qcow2 and make it again
With the marker file present, the apt setup in bootcmd and the snap setup in runcmd each print that pkgproxy was unreachable and skip; the guest boots uncached, a few seconds late. The last runcmd command disables cloud-init, so the prepared image stays uncached even after pkgproxy is back; the hosts line and GOPROXY are kept. Removing SYSTEM.qcow2 and making it again brings the cache.

Both setup batches see the file and print image-garden: pkgproxy was unreachable, skipping setup scripts: apt (or snap). cloud-init is disabled after the provisioning boot, so the image stays uncached until SYSTEM.qcow2 is made again.

Both batches of setup scripts look for that file first. The apt script, in bootcmd, and the snap script, in runcmd, each print pkgproxy was unreachable, skipping setup scripts, and the script's name. So the guest boots without a cache, a few seconds late. Here's the catch. The provisioning boot is the only one that runs cloud-init: at its end, cloud-init is switched off. So the prepared image stays uncached, even after pkgproxy comes back. It does keep the hosts line, and GOPROXY in /etc/environment, which are written anyway. To get the cache, remove the prepared image and make it again.

Details and sources

Details

  • PKGPROXY_SETUP_SHELL checks the marker first: image-garden: pkgproxy was unreachable, skipping setup scripts: TOOLS on standard error; otherwise it fetches URL/setup/TOOL with curl -fsSL --connect-timeout 5, or wget, and pipes it to sh (010-qemu.mk:150-154). The apt batch runs from bootcmd on Debian and Ubuntu, the snap batch from runcmd on all but Alpine (010-qemu.mk:127-130, 010-qemu.mk:156-168).
  • The hosts line is added before the probe, and again in runcmd; GOPROXY="URL/go,direct" is appended to /etc/environment by write_files whether or not pkgproxy answers (010-qemu.mk:132-134, 010-qemu.mk:170-180).
  • Derived from Go's documented behaviour, not run: with a comma-separated GOPROXY, the go command moves on to direct only when the proxy answers 404 or 410, so while pkgproxy stays unreachable, module downloads in such a guest fail to connect rather than going direct.
  • The last runcmd command is touch /etc/cloud/cloud-init.disabled (010-qemu.mk:246); cloud-init does nothing on later boots (see provisioning runs once), so bootcmd does not ask pkgproxy again.
  • proxy/README.md says the same: "All of this runs once, while the image is prepared: the last thing provisioning does is disable cloud-init, so later boots, including every boot of a .run script or allocate, neither ask pkgproxy again nor run the setup scripts. A guest prepared while pkgproxy could not be reached stays uncached until its image is made again: remove SYSTEM.qcow2 and run image-garden make SYSTEM.qcow2" (proxy/README.md:159-164). The comment on the marker file agrees: "Provisioning disables cloud-init at its end, so no later boot asks pkgproxy again" (010-qemu.mk:136-139).
  • Removing only SYSTEM.qcow2 is enough here: the user-data already holds the pkgproxy lines, so the new provisioning boot runs the setup scripts again. After a change of pkgproxy setting more files must go: see changing the proxy does not rebuild.

Read more

04 The host remembers for five minutes

THE HOST REMEMBERS download 1: SHA256SUMS pkgproxy.example.lan:3142 one try, 10 s curl: (28) Connection timed out after 10002 milliseconds exit 6, 7 or 28 $XDG_RUNTIME_DIR/image-garden/pkgproxy-unreachable.d7d8c41fe8945a2b Not asking pkgproxy at http://pkgproxy.example.lan:3142 again for 5 minutes download 2: the image Skipping pkgproxy at http://pkgproxy.example.lan:3142, unreachable recently, downloading https://…/noble-server-cloudimg-amd64.img directly for 300 seconds, then pkgproxy is asked again
The first download asks pkgproxy.example.lan once and curl times out connecting. image-garden writes $XDG_RUNTIME_DIR/image-garden/pkgproxy-unreachable followed by a hash of the address and prints that it will not ask that pkgproxy again for 5 minutes. The next download finds the young file, prints Skipping pkgproxy, unreachable recently, and goes to the origin. After 300 seconds pkgproxy is asked again.

curl exit 6, 7 or 28 writes $XDG_RUNTIME_DIR/image-garden/pkgproxy-unreachable.HASH and prints Not asking pkgproxy at URL again for 5 minutes. For 300 seconds every download prints Skipping pkgproxy at URL, unreachable recently, downloading URL directly.

On the host, each download gives pkgproxy one try, and waits ten seconds at most for the connection. When curl says it couldn't resolve the name, couldn't connect, or ran out of time connecting, image-garden writes a small file in your runtime directory, named after a hash of the pkgproxy's address. It prints Not asking pkgproxy at that address again for 5 minutes. Every download that starts in the next five minutes finds the file, still young, and goes straight to the origin, saying Skipping pkgproxy, unreachable recently. Five minutes covers a whole make or spread run, and still brings back a pkgproxy that was restarted soon after.

Details and sources

Details

  • Both fetch_volatile and fetch_image skip pkgproxy when pkgproxy_skip succeeds (image-garden:216, image-garden:287), and call pkgproxy_unreachable with curl's status after a failed try (image-garden:244-248, image-garden:302-307).
  • The file is $XDG_RUNTIME_DIR/image-garden/pkgproxy-unreachable. followed by the first 16 hex digits of the sha256 of the address without its trailing slash; it holds the address (image-garden:152-161, image-garden:183-193). The hash on the slide is the one for http://pkgproxy.example.lan:3142.
  • Only curl's exit statuses 6 (could not resolve host), 7 (could not connect) and 28 (timed out; "the transfers through pkgproxy have no other time limit") are remembered (image-garden:178-187).
  • _pkgproxy_unreachable_ttl=300; a file 300 seconds old or older, or from the future, is ignored (image-garden:150, image-garden:163-176).
  • The log lines were produced by running the helper with GARDEN_DL_PKGPROXY set to a closed port in a scratch directory; the address, the hash and the image URL on the slide are the example's. On that host the closed port timed out (28) rather than refusing (7).
  • Before the skip, the first download's own fallback prints Cannot reach pkgproxy, downloading URL directly for a list, or Cannot download through pkgproxy, downloading URL directly for an image.

Read more

05 What is remembered, and what is not

WHAT IS REMEMBERED a failed try curl exit 6, 7 or 28 remembered HTTP error · answer cut short asked again unix:…/pkgproxy.sock never remembered one file per address age only, no lock from the future: stale $XDG_RUNTIME_DIR/image-garden → /run/image-garden → nowhere a reboot forgets
After a failed try: curl exit 6, 7 or 28 is remembered for five minutes; an HTTP error or a cut answer is not, pkgproxy is asked again; a pkgproxy socket is never remembered. One file per address, only its age is checked so no lock is needed, a file from the future is stale. The file lives in $XDG_RUNTIME_DIR/image-garden, else /run/image-garden, else nowhere, and a reboot clears it.

Only a failure to resolve, connect or connect in time is remembered, per address. HTTP errors, cut answers and sockets are not. The file lives in $XDG_RUNTIME_DIR/image-garden, else /run/image-garden, and a reboot clears it.

Not every failure is remembered. An HTTP error, or an answer that's cut short, means pkgproxy is there, so the next download asks it again. A pkgproxy socket is never remembered: connecting to a socket that isn't there fails at once anyway. There's one file per address, so a dead pkgproxy doesn't make image-garden skip a different one. Downloads only check the file's age or write it, so parallel makes need no lock. A file from the future, after the clock was set back, counts as stale. Without a runtime directory it lives in /run, and where it can't be written, nothing is remembered. A reboot forgets it all.

Details and sources

Details

  • "An HTTP error, or a response cut short, says that pkgproxy is there": only curl statuses 6, 7 and 28 write the file (image-garden:178-193).
  • "A pkgproxy socket is not remembered: connecting to it fails at once"; pkgproxy_unreachable_file prints nothing for a unix: address (image-garden:138-161). In the snap, a socket that does not exist is not used at all (image-garden:51-58).
  • "One file per pkgproxy address, named by a hash of it, so that another pkgproxy is not skipped. Every download only looks at the file's age, or touches it: concurrent ones need no lock" (image-garden:146-149).
  • "A file from the future, as after the clock was set back, is stale" (image-garden:171-174).
  • XDG_RUNTIME_DIR defaults to /run (image-garden:61-64); in the snap it is /run/user/UID/snap.image-garden, or /run/snap.image-garden where /run/user/UID does not exist (image-garden:66-73). When the directory cannot be made or the file written, nothing is remembered and no message is printed (image-garden:190-192).
  • /run is a tmpfs on systemd systems, which is why a reboot forgets (derived).
  • tests/check-dl-pkgproxy.sh covers each case: remembered, a file 290 and 301 seconds old, one dated an hour ahead, nowhere to remember, another pkgproxy, and HTTP errors not remembered (run at the tag: all pass).

Read more

06 Where to go from here

ROADS When it is down Provisioning runs once cloud-init off after it The craft components their setup fails instead The runtime directory where the host keeps its note Reaching pkgproxy guestfwd or a URL Image downloads and their fallback No rebuild make it again In the user-data hosts, apt, snap, GOPROXY
Seven roads from An unreachable pkgproxy: provisioning runs once, the craft components, the runtime directory, reaching pkgproxy, image downloads, changing the proxy, and the user-data

Next: provisioning runs once, the craft components, the runtime directory, reaching pkgproxy, image downloads, starting a guest again and the user-data.

So a missing pkgproxy costs a guest a few seconds, and a download ten, once. Why the provisioning boot is the only one that sets a guest up is in the first boot deck. The GitLab craft components behave differently: their setup step fails when the proxy is down. The runtime directory, where the host keeps its note, has its own deck, and so does the way a guest reaches pkgproxy. Image downloads and their fallback come next. To make a guest again once pkgproxy is back, see why changing the proxy doesn't rebuild. And everything pkgproxy adds at first boot is in the user-data deck.

Details and sources

Read more

Basis of the statements on this page

Read in mk/010-qemu.mk, the image-garden script and proxy/README.md of image-garden at the commit this map describes ("code"). The guest's lines were taken from a user-data that v0.8.0's image-garden make generated with PKGPROXY_URL set; the host's messages were produced by running the download helper with GARDEN_DL_PKGPROXY at a closed port ("run"). That later boots do not probe again follows from cloud-init being disabled after the provisioning boot ("derived"), and proxy/README.md says so.

In the guest

StepCommandLimitOn failure
probe, in bootcmdcurl -s -o /dev/null --connect-timeout 5 --max-time 20 http://pkgproxy:PORT/5 s to connect, 20 s in allimage-garden: cannot reach pkgproxy at URL/, skipping its setup scripts in this boot, then touch /run/image-garden-pkgproxy-unreachable
probe without curlwget -q -T 5 -t 1 -O /dev/null http://pkgproxy:PORT/5 s, one trythe same
apt setup, in bootcmd (Debian, Ubuntu)curl -fsSL --connect-timeout 5 URL/setup/apt, piped to sh5 s to connectskipped while the marker exists: image-garden: pkgproxy was unreachable, skipping setup scripts: apt
snap setup, in runcmd (all but Alpine)curl -fsSL --connect-timeout 5 URL/setup/snap, piped to sh5 s to connectskipped: image-garden: pkgproxy was unreachable, skipping setup scripts: snap

Sources: 010-qemu.mk:127-168. A guest with neither curl nor wget asks nothing and leaves no marker; its setup prints image-garden: no curl or wget to fetch URL/setup/TOOL.

Only the provisioning boot runs any of this: a guest prepared while pkgproxy could not be reached stays uncached until SYSTEM.qcow2 is removed and made again (proxy/README.md:159-164).

Written regardless of the probe: the pkgproxy line in /etc/hosts and GOPROXY="URL/go,direct" in /etc/environment (010-qemu.mk:132-134, 010-qemu.mk:170-180).

On the host

ItemValue
tries per downloadone, --connect-timeout 10
remembered curl statuses6 (resolve), 7 (connect), 28 (timeout)
file$XDG_RUNTIME_DIR/image-garden/pkgproxy-unreachable.HASH16, the address inside
XDG_RUNTIME_DIR unset/run
in the snap/run/user/UID/snap.image-garden, or /run/snap.image-garden
lifetime300 seconds by the file's time; a file from the future is ignored
sockets (unix:)never remembered
directory not writablenothing remembered, no message

Sources: image-garden:61-73, image-garden:138-193.

Messages on the host

MessageWhen
Cannot reach pkgproxy, downloading URL directlya list's try failed to connect
Cannot download through pkgproxy, downloading URL directlyan image's try failed
Not asking pkgproxy at ADDRESS again for 5 minutesthe failure was 6, 7 or 28 and the file was written
Skipping pkgproxy at ADDRESS, unreachable recently, downloading URL directlya later download within 300 seconds

Roads from here

RUNTIME · make rules and the CLI

BUILT ON IT · CI integrations

FIRST BOOT · cloud-init changes

PKGPROXY · caching for guests