Changing the proxy does not rebuild

mode is baked in for CI operators

make writes the pkgproxy choice into a system's user-data and .run script when it makes them, and the provisioning boot writes it into the disk; neither file depends on the setting, so changing it rebuilds nothing. Remove the system's .user-data, .seed.iso and .run, or run image-garden make clean, to make it again with the new setting.

On the map: pkgproxy

▶ Watch this deck — 7 scenes, ~5 min

Worth seeing first

01 You change the setting, and nothing happens

NO REBUILD ubuntu-cloud-24.04.x86_64.qcow2 made with the bundled pkgproxy $ snap set image-garden pkgproxy=none $ image-garden make ubuntu-cloud-24.04.x86_64.qcow2 make: 'ubuntu-cloud-24.04.x86_64.qcow2' is up to date. apt snapd pkgproxy still, after pkgproxy=none the setting applies to systems made after the change
ubuntu-cloud-24.04.x86_64 was made with the bundled pkgproxy. snap set image-garden pkgproxy=none, then image-garden make answers that ubuntu-cloud-24.04.x86_64.qcow2 is up to date, and the guest's apt and snapd still point at pkgproxy: the setting applies only to systems made after the change.

snap set image-garden pkgproxy=none changes nothing that make can see: image-garden make keeps the guest built with the bundled pkgproxy. The setting applies to systems made after the change.

Say you built ubuntu-cloud-24.04.x86_64 with the snap's bundled pkgproxy, and now a test suite needs a guest without one. You run snap set image-garden pkgproxy=none, then image-garden make, and make tells you the image is up to date. As far as make can see, it is. But your guest still has apt and snapd pointed at pkgproxy, because the setting only applies to systems made after the change. This deck is for anyone who switches pkgproxy modes, in the snap or with PKGPROXY_URL: why nothing is rebuilt, what an old guest does, and which files to remove to start again.

Details and sources

Details

  • "The setting only affects instances built after the change; already provisioned VMs keep whatever cloud-init wrote at creation time" (README.md:223-226, proxy/README.md:132-133).
  • none exists mainly for test suites, such as snapd's spread suite, that do not yet expect the injected SNAPPY_FORCE_API_URL or the extra /etc/hosts entries (README.md:211-215, proxy/README.md:134-136).
  • Run at the tag, outside the snap, in a scratch directory: after the user-data was made without a pkgproxy, image-garden make ubuntu-cloud-24.04.x86_64.user-data with PKGPROXY_URL set answered make: 'ubuntu-cloud-24.04.x86_64.user-data' is up to date.; the prepared image and the .run script behaved the same way. The snap reads the mode from a file instead of the variable, with the same result (derived from 010-qemu.mk:20-23).
  • The same holds outside the snap for PKGPROXY_URL: setting, changing or unsetting it rebuilds nothing.

02 Read once, then written down

READ ONCE, WRITTEN DOWN pkgproxy-mode · PKGPROXY_URL make making a system's files ubuntu-cloud-24.04.x86_64.user-data 10.0.2.100 pkgproxy · setup/apt, setup/snap · GOPROXY=http://pkgproxy:3142/go,direct with a URL: its host is looked up now, and that address is written in ubuntu-cloud-24.04.x86_64.run guestfwd=tcp:10.0.2.100:3142-unix:…/pkgproxy.sock ubuntu-cloud-24.04.x86_64.qcow2, after the provisioning boot /etc/apt/apt.conf.d/99pkgproxy · /etc/systemd/system/snapd.service.d/pkgproxy.conf image downloads: the setting is read again on every download
make reads the pkgproxy mode or PKGPROXY_URL and, when it makes a system's files, copies the choice into the user-data (the hosts line, the setup scripts, GOPROXY, a URL's address looked up then) and into the .run script (QEMU's guestfwd to the socket). The provisioning boot writes apt's proxy and a snapd override into the disk. Image downloads read the setting each time.

The choice is copied into SYSTEM.user-data (hosts line, setup scripts, GOPROXY, a URL's address) and SYSTEM.run (the guestfwd), and the first boot writes it into the disk. Downloads read the setting each time.

Here's why. make reads the pkgproxy setting every time it runs, but uses it only when it makes a system's files, and then it writes it down. The user-data gets the hosts line, the setup scripts and GOPROXY. For a URL, the host's address is looked up right then and written in. The .run script gets QEMU's forward to the pkgproxy socket. Then, in the provisioning boot, the setup scripts write apt's proxy setting and a snapd override into the disk itself. From then on, those copies are what the guest uses. Image downloads are different: they look at the setting afresh every time.

Details and sources

Details

  • make reads $SNAP_COMMON/pkgproxy-mode, PKGPROXY_URL and whether the socket exists each time it runs (010-qemu.mk:20-44). With a URL, the host is resolved with getent ahostsv4 then, because "/etc/hosts maps names to addresses only" (010-qemu.mk:25-28).
  • The user-data gets the bootcmd and runcmd pkgproxy lines and the GOPROXY line of /etc/environment only when a pkgproxy was found (010-qemu.mk:127-185).
  • The .run script's default_qemu_net_option is -netdev user,id=netdev0 plus QEMU_DEFAULT_NETDEV_SYS_EXTRA, the guestfwd for the socket modes and nothing for a URL (010-qemu-x86_64.mk:8-11, 010-qemu-x86_64.mk:70); the other architectures' files do the same. With the bundled pkgproxy's LXD socket, a second guestfwd for port 3143 is added (010-qemu.mk:37-39).
  • In the guest, pkgproxy's apt script writes Acquire::http::Proxy to /etc/apt/apt.conf.d/99pkgproxy, and its snap script writes /etc/systemd/system/snapd.service.d/pkgproxy.conf, which sets SNAPPY_FORCE_API_URL (proxy/README.md:171-180).
  • Downloads choose their pkgproxy in the image-garden script, on each run (image-garden:31-59): see cloud images through pkgproxy.
  • The user-data line on the slide is condensed; the full lines are in what pkgproxy adds to user-data.

03 Why make doesn't notice

WHY MAKE DOES NOT NOTICE the makefiles .image-garden.mk ubuntu-cloud-24.04.x86_64.user-data ubuntu-cloud-24.04.x86_64.run $SNAP_COMMON/pkgproxy-mode PKGPROXY_URL not a prerequisite: nothing to compare bundled ⇄ pkgproxy snap: the same user-data, and in the .run script only the socket component unix:$SNAP_COMMON/pkgproxy.sock snap unix:$SNAP_COMMON/pkgproxy-host-snap/sockets/pkgproxy.sock
The user-data depends on the makefiles and .image-garden.mk, the .run script on the makefiles. The pkgproxy-mode file and PKGPROXY_URL are not prerequisites, so make has nothing to compare. Between the bundled pkgproxy and a pkgproxy snap the user-data is the same and the .run script differs only in the socket path.

SYSTEM.user-data depends on the makefiles and .image-garden.mk, SYSTEM.run on the makefiles. The mode file and PKGPROXY_URL are not prerequisites. Bundled and snap modes differ only in the .run script's socket path.

make decides what to rebuild by comparing file times. The user-data is made again when a makefile, or your .image-garden.mk, is newer than it. The .run script, when a makefile is. The pkgproxy setting is neither of those. In the snap it's a small file in the snap's common directory, and outside the snap it's the PKGPROXY_URL variable. make has nothing to compare, so a new setting looks just like the old one. Switching between the bundled pkgproxy and a pkgproxy snap is the mildest case: the user-data is the same for both, and only the forward in the .run script differs.

Details and sources

Details

  • SYSTEM.user-data: $(MAKEFILE_LIST) $(wildcard $(GARDEN_PROJECT_DIR)/.image-garden.mk) (rulegen.py:467-471; instances: 020-instances.mk:76). SYSTEM.run: $(MAKEFILE_LIST), with the prepared image and firmware copies as order-only prerequisites, which never make it older (rulegen.py:594-599; instances: 020-instances.mk:87).
  • The mode and URL are read with $(shell cat …) and from the environment (010-qemu.mk:20-23); neither is a prerequisite of any rule.
  • In both socket modes the guest's address is 10.0.2.100 and the port 3142; only PKGPROXY_SOCKET_DIR differs, $SNAP_COMMON or $SNAP_COMMON/pkgproxy-host-snap/sockets (010-qemu.mk:22-34). So moving between them needs only a new .run script.
  • The .run script is also made again whenever image-garden is upgraded, since the makefiles change; the user-data too, and with it the prepared image (see the overlay).

Read more

04 What an old guest does

WHAT AN OLD GUEST DOES built with the bundled pkgproxy, now pkgproxy=none apt, snapd old .run: guestfwd pkgproxy.sock removed by the snap downloads fail built without pkgproxy, now turned on apt, snapd the mirrors uncached, and nothing says so built with a URL whose name later moved /etc/hosts: 192.0.2.10 pkgproxy the old machine kept from build time
Three old guests. Built with the bundled pkgproxy, then set to none: the snap removes the socket, the old .run script still forwards to it, and apt and snapd in the guest still ask pkgproxy, so their downloads fail. Built without pkgproxy, then turned on: the guest goes to the mirrors, uncached, and nothing says so. Built with a URL whose name later moved: the guest's /etc/hosts keeps the old address.

Turned off: the old .run forwards to a removed socket and the guest's apt and snapd still use pkgproxy. Turned on: the old guest stays uncached. A URL: the guest keeps the address resolved when it was made.

So what does an old guest do with a new setting? Take the example. With pkgproxy=none, the snap stops its pkgproxy and removes the socket. The old .run script still forwards to it, and apt and snapd in the guest still ask pkgproxy, which isn't there any more, so their downloads fail. The other way round is quieter. A guest made without pkgproxy stays uncached after you turn it on, and nothing tells you. And with a URL, the guest keeps the address its host had when the guest was made, even after that name moves to another machine.

Details and sources

Details

  • Every mode but component stops and disables the bundled daemon and removes pkgproxy.sock, stats.sock and pkgproxy-lxd-tls.sock from $SNAP_COMMON (configure:6-9, configure:56-73).
  • Derived, not run: the patched guestfwd opens a new connection to the socket for each guest connection (guestfwd-unix-socket.patch), so the old .run script still boots, and each connection from the guest to 10.0.2.100:3142 fails. apt, with Acquire::http::Proxy set, and snapd, with SNAPPY_FORCE_API_URL set, then fail their downloads rather than going direct.
  • With a URL, make resolves the host when it makes the user-data and writes the address into the guest's /etc/hosts (010-qemu.mk:24-29, proxy/README.md:117-120). 192.0.2.10 is a documentation address.
  • A guest built while its pkgproxy could not be reached is the fourth case: it stays uncached, because only the provisioning boot runs the setup, and proxy/README.md says to remove SYSTEM.qcow2 and run image-garden make SYSTEM.qcow2 (proxy/README.md:159-164); see an unreachable pkgproxy.

05 Starting again

STARTING AGAIN $ rm .image-garden/ubuntu-cloud-24.04.x86_64.{user-data,seed.iso,run} $ image-garden make ubuntu-cloud-24.04.x86_64.qcow2 ubuntu-cloud-24.04.x86_64.run only the user-data removed user-data: remade seed.iso: kept disk: kept dated like the makefiles: older than the seed all three removed user-data: new seed.iso: new .run: new disk: fresh, provisioned again $ image-garden make clean the same, for every system
Remove the system's .user-data, .seed.iso and .run from .image-garden, then make the system. With only the user-data removed, it is remade with the makefiles' time and the seed and the disk stay. With all three removed, make writes a new user-data, seed and .run script and, as the seed is newer, a fresh disk with its provisioning boot. image-garden make clean does this for every system.

Remove SYSTEM.user-data, SYSTEM.seed.iso and SYSTEM.run, then make the system. Removing only the user-data is not enough. image-garden make clean removes the made files of every system.

To start again, remove three of the system's files: the user-data, the seed ISO and the .run script. They're in your project's .image-garden directory when there is one, otherwise where you run make. Then make the system again. Why not just the user-data? It's remade with the time of the makefiles, which is older than the seed, so the old seed and the old disk stay. With all three gone, make writes new user-data, a new seed and a new .run script, and because the seed is newer, a fresh disk with its own provisioning boot. image-garden make clean does this for every system at once.

Details and sources

Details

  • The user-data rule ends with touch --reference= the newest of its prerequisites, the makefiles and .image-garden.mk (rulegen.py:467-471); the seed depends on the user-data and meta-data (010-qemu.mk:268), the prepared image on the seed (rulegen.py:601-602).
  • Run at the tag in a scratch directory, with PKGPROXY_URL set after the files were made: removing only ubuntu-cloud-24.04.x86_64.user-data and making it again wrote the new user-data with the time of image-garden.mk, and make then said the .qcow2 and the .run were up to date. Removing the .user-data, the .seed.iso and the .run made make -n list the seed ISO, qemu-img create, the provisioning boot and the launcher script.
  • make -n alone is misleading here: it assumes a remade user-data is new, and lists the seed and the disk even when only the user-data was removed.
  • image-garden runs make in .image-garden/ when that directory exists, else in the current directory (image-garden:987-993).
  • image-garden make clean removes *.log, *.seed.iso, *.meta-data and *.user-data, and each system's .qcow2, .run and its link in the download cache (000-garden.mk:48-50, 010-qemu.mk:300-302, rulegen.py:473-477). The downloaded images stay; the next make checks them again and links them.
  • Moving between the bundled pkgproxy and a pkgproxy snap needs only the .run script removed (see the previous scenes).
  • The new disk starts empty: anything changed by booting the .run script without -snapshot is lost (see the overlay).

06 In CI, a cached image keeps its mode for CI operators

IN CI a job on an empty runner the guest is made with the setting of the day cached prepared images key: image-garden-img-SYSTEM-hashFiles(.image-garden.mk) no pkgproxy mode in the key port 3142 in use? no: snap mode, pkgproxy snap's socket yes: external, http://RUNNER-ADDRESS:3142 restored image: still the mode of the job that made it change .image-garden.mk: a new key, a fresh image
A CI job on an empty runner makes its guest with the setting of the day. image-garden-action's cache of prepared images is keyed by the system and a hash of .image-garden.mk, not by the pkgproxy mode. The action uses snap mode, or external mode at the runner's address when something already listens on port 3142. A restored image keeps the mode of the job that made it; editing .image-garden.mk gives a new key.

cache-prepared-images keys .image-garden/ by system and hashFiles('.image-garden.mk'). A restored image keeps the mode, snap or external, of the job that made it; changing .image-garden.mk changes the key.

In CI, most jobs start on an empty machine, so every guest is made with the setting of the day. The exception is a cache of prepared images. image-garden-action's cache-prepared-images keys that cache by the system and a hash of your .image-garden.mk, not by the pkgproxy mode. The action picks snap mode, unless something already listens on port 3142, like a pkgproxy you set up first. Then it uses that one, by the runner's address. Add or drop that step, and a restored image still has the old mode inside. A change to your .image-garden.mk gives a new key.

Details and sources

Details

  • With cache-prepared-images: true, the action caches SUBDIR/.image-garden without its logs under the key image-garden-img-GARDEN_SYSTEM-HASH, where HASH is hashFiles('SUBDIR/.image-garden.mk') (action.yaml:116-123). The .user-data, the .seed.iso, the .run script and the prepared .qcow2 are all in it.
  • The action uses external mode when ss finds a listener on port 3142, else snap mode (action.yaml:173-188). In snap mode it runs snap set image-garden pkgproxy=snap; in external mode pkgproxy=http://ADDRESS:3142, with the first address of hostname -I, "as 127.0.0.1 would be the guest itself" (action.yaml:375-393).
  • Derived, not run: a restored image made in snap mode has a .run script that forwards to the pkgproxy snap's socket, which an external-mode job does not have; one made in external mode has the first runner's address in its /etc/hosts. Even when make provisions the restored image again (see how the action caches), it does so from the restored seed, with the old mode.
  • Changing .image-garden.mk, even by a comment, gives a new key; cache-key-prefix names the pkgproxy caches only (action.yaml:30-31).

07 Where to go from here

ROADS No rebuild The overlay when the disk is made again Where files live .image-garden or here Caching on GitHub prepared images and keys When it is down made without a cache pkgproxy= four values, in the snap In the user-data every line pkgproxy adds
Six roads from Changing the proxy does not rebuild: when the disk is made again, where the files live, caching on GitHub, an unreachable pkgproxy, the snap's pkgproxy setting and the user-data

Next: when the disk is made again, where the files live, caching on GitHub, an unreachable pkgproxy, the snap setting and the user-data.

So the pkgproxy choice is copied into a system's files once, and stays there until you remove them. When make provisions a disk again for other reasons, and what you lose then, is in the overlay deck. Where the system's files live, in .image-garden or your current directory, has a deck of its own. On GitHub, the action's caches are explained in full. A guest made while pkgproxy was down is a close cousin of this problem. In the snap, one setting has four values. And the user-data deck shows every line pkgproxy adds.

Details and sources

Read more

Basis of the statements on this page

Read in rulegen.py, mk/010-qemu.mk, mk/020-instances.mk, the snap's configure hook and image-garden-action's action.yaml at the pinned commits ("code"). What make rebuilds was checked by running v0.8.0's image-garden make on ubuntu-cloud-24.04.x86_64 in a scratch directory, outside the snap, with PKGPROXY_URL changed after the files were made ("run"). What apt and snapd do in an old guest, and what a restored CI image holds, are reasoned from the code ("derived").

Where the choice is written

FileWhat it holdsRemade when
SYSTEM.user-datathe pkgproxy hosts line with its address, the apt and snap setup lines, GOPROXYa makefile or .image-garden.mk is newer, or it is missing
SYSTEM.seed.isothe user-data and meta-datathe user-data or meta-data is newer, or it is missing
SYSTEM.qcow2after the provisioning boot: /etc/apt/apt.conf.d/99pkgproxy, the snapd drop-in, /etc/hosts, /etc/environmentthe seed, the download or the firmware copies are newer, or it is missing
SYSTEM.runQEMU's guestfwd to the socket, in socket modesa makefile is newer, or it is missing

Sources: rulegen.py:461-471, rulegen.py:594-609, 010-qemu.mk:20-44, 010-qemu.mk:268.

What to remove after a change

ChangeRemove, then make the system
component ⇄ snapSYSTEM.run
any other change of mode or URLSYSTEM.user-data, SYSTEM.seed.iso, SYSTEM.run
no change, but pkgproxy could not be reached while the guest was preparedSYSTEM.qcow2 (proxy/README.md:159-164)
every system at onceimage-garden make clean

The files are in .image-garden/ when the project has that directory, otherwise in the directory you run image-garden from (image-garden:987-993). Removing only SYSTEM.user-data remakes it with the makefiles' time, older than the seed, so the seed and the disk are kept (run). A new disk starts empty: see the overlay.

Not affected

Image downloads choose their pkgproxy on every run, so GARDEN_DL_PKGPROXY, PKGPROXY_URL and the snap setting apply to the next download at once (image-garden:31-59).

Roads from here

RUNTIME · make rules and the CLI

BUILT ON IT · CI integrations

PKGPROXY · caching for guests

SNAP · snap-only features