Changing the proxy does not rebuild
mode is baked in for CI operators
make writes the pkgproxy choice into a system's user-data and .run script when it makes them, and the provisioning boot writes it into the disk; neither file depends on the setting, so changing it rebuilds nothing. Remove the system's .user-data, .seed.iso and .run, or run image-garden make clean, to make it again with the new setting.
On the map: pkgproxy
▶ Watch this deck — 7 scenes, ~5 min
Worth seeing first
- How a guest reaches pkgproxy — how the choice of pkgproxy reaches the guest
01 You change the setting, and nothing happens
snap set image-garden pkgproxy=none changes nothing that make can see: image-garden make keeps the guest built with the bundled pkgproxy. The setting applies to systems made after the change.
Say you built ubuntu-cloud-24.04.x86_64 with the snap's bundled pkgproxy, and now a test suite needs a guest without one. You run snap set image-garden pkgproxy=none, then image-garden make, and make tells you the image is up to date. As far as make can see, it is. But your guest still has apt and snapd pointed at pkgproxy, because the setting only applies to systems made after the change. This deck is for anyone who switches pkgproxy modes, in the snap or with PKGPROXY_URL: why nothing is rebuilt, what an old guest does, and which files to remove to start again.
Details and sources
Details
- "The setting only affects instances built after the change; already provisioned VMs keep whatever cloud-init wrote at creation time" (README.md:223-226, proxy/README.md:132-133).
noneexists mainly for test suites, such as snapd's spread suite, that do not yet expect the injectedSNAPPY_FORCE_API_URLor the extra/etc/hostsentries (README.md:211-215, proxy/README.md:134-136).- Run at the tag, outside the snap, in a scratch directory: after the user-data was made without a pkgproxy,
image-garden make ubuntu-cloud-24.04.x86_64.user-datawithPKGPROXY_URLset answeredmake: 'ubuntu-cloud-24.04.x86_64.user-data' is up to date.; the prepared image and the.runscript behaved the same way. The snap reads the mode from a file instead of the variable, with the same result (derived from 010-qemu.mk:20-23). - The same holds outside the snap for
PKGPROXY_URL: setting, changing or unsetting it rebuilds nothing.
02 Read once, then written down
The choice is copied into SYSTEM.user-data (hosts line, setup scripts, GOPROXY, a URL's address) and SYSTEM.run (the guestfwd), and the first boot writes it into the disk. Downloads read the setting each time.
Here's why. make reads the pkgproxy setting every time it runs, but uses it only when it makes a system's files, and then it writes it down. The user-data gets the hosts line, the setup scripts and GOPROXY. For a URL, the host's address is looked up right then and written in. The .run script gets QEMU's forward to the pkgproxy socket. Then, in the provisioning boot, the setup scripts write apt's proxy setting and a snapd override into the disk itself. From then on, those copies are what the guest uses. Image downloads are different: they look at the setting afresh every time.
Details and sources
Details
- make reads
$SNAP_COMMON/pkgproxy-mode,PKGPROXY_URLand whether the socket exists each time it runs (010-qemu.mk:20-44). With a URL, the host is resolved withgetent ahostsv4then, because "/etc/hosts maps names to addresses only" (010-qemu.mk:25-28). - The user-data gets the
bootcmdandruncmdpkgproxy lines and theGOPROXYline of/etc/environmentonly when a pkgproxy was found (010-qemu.mk:127-185). - The
.runscript'sdefault_qemu_net_optionis-netdev user,id=netdev0plusQEMU_DEFAULT_NETDEV_SYS_EXTRA, theguestfwdfor the socket modes and nothing for a URL (010-qemu-x86_64.mk:8-11, 010-qemu-x86_64.mk:70); the other architectures' files do the same. With the bundled pkgproxy's LXD socket, a secondguestfwdfor port 3143 is added (010-qemu.mk:37-39). - In the guest, pkgproxy's
aptscript writesAcquire::http::Proxyto/etc/apt/apt.conf.d/99pkgproxy, and itssnapscript writes/etc/systemd/system/snapd.service.d/pkgproxy.conf, which setsSNAPPY_FORCE_API_URL(proxy/README.md:171-180). - Downloads choose their pkgproxy in the image-garden script, on each run (image-garden:31-59): see cloud images through pkgproxy.
- The user-data line on the slide is condensed; the full lines are in what pkgproxy adds to user-data.
Read more
03 Why make doesn't notice
SYSTEM.user-data depends on the makefiles and .image-garden.mk, SYSTEM.run on the makefiles. The mode file and PKGPROXY_URL are not prerequisites. Bundled and snap modes differ only in the .run script's socket path.
make decides what to rebuild by comparing file times. The user-data is made again when a makefile, or your .image-garden.mk, is newer than it. The .run script, when a makefile is. The pkgproxy setting is neither of those. In the snap it's a small file in the snap's common directory, and outside the snap it's the PKGPROXY_URL variable. make has nothing to compare, so a new setting looks just like the old one. Switching between the bundled pkgproxy and a pkgproxy snap is the mildest case: the user-data is the same for both, and only the forward in the .run script differs.
Details and sources
Details
SYSTEM.user-data: $(MAKEFILE_LIST) $(wildcard $(GARDEN_PROJECT_DIR)/.image-garden.mk)(rulegen.py:467-471; instances: 020-instances.mk:76).SYSTEM.run: $(MAKEFILE_LIST), with the prepared image and firmware copies as order-only prerequisites, which never make it older (rulegen.py:594-599; instances: 020-instances.mk:87).- The mode and URL are read with
$(shell cat …)and from the environment (010-qemu.mk:20-23); neither is a prerequisite of any rule. - In both socket modes the guest's address is
10.0.2.100and the port3142; onlyPKGPROXY_SOCKET_DIRdiffers,$SNAP_COMMONor$SNAP_COMMON/pkgproxy-host-snap/sockets(010-qemu.mk:22-34). So moving between them needs only a new.runscript. - The
.runscript is also made again whenever image-garden is upgraded, since the makefiles change; the user-data too, and with it the prepared image (see the overlay).
Read more
- The target chain: every file a system is made from.
04 What an old guest does
Turned off: the old .run forwards to a removed socket and the guest's apt and snapd still use pkgproxy. Turned on: the old guest stays uncached. A URL: the guest keeps the address resolved when it was made.
So what does an old guest do with a new setting? Take the example. With pkgproxy=none, the snap stops its pkgproxy and removes the socket. The old .run script still forwards to it, and apt and snapd in the guest still ask pkgproxy, which isn't there any more, so their downloads fail. The other way round is quieter. A guest made without pkgproxy stays uncached after you turn it on, and nothing tells you. And with a URL, the guest keeps the address its host had when the guest was made, even after that name moves to another machine.
Details and sources
Details
- Every mode but
componentstops and disables the bundled daemon and removespkgproxy.sock,stats.sockandpkgproxy-lxd-tls.sockfrom$SNAP_COMMON(configure:6-9, configure:56-73). - Derived, not run: the patched
guestfwdopens a new connection to the socket for each guest connection (guestfwd-unix-socket.patch), so the old.runscript still boots, and each connection from the guest to10.0.2.100:3142fails. apt, withAcquire::http::Proxyset, and snapd, withSNAPPY_FORCE_API_URLset, then fail their downloads rather than going direct. - With a URL, make resolves the host when it makes the user-data and writes the address into the guest's
/etc/hosts(010-qemu.mk:24-29, proxy/README.md:117-120).192.0.2.10is a documentation address. - A guest built while its pkgproxy could not be reached is the fourth case: it stays uncached, because only the provisioning boot runs the setup, and proxy/README.md says to remove
SYSTEM.qcow2and runimage-garden make SYSTEM.qcow2(proxy/README.md:159-164); see an unreachable pkgproxy.
05 Starting again
Remove SYSTEM.user-data, SYSTEM.seed.iso and SYSTEM.run, then make the system. Removing only the user-data is not enough. image-garden make clean removes the made files of every system.
To start again, remove three of the system's files: the user-data, the seed ISO and the .run script. They're in your project's .image-garden directory when there is one, otherwise where you run make. Then make the system again. Why not just the user-data? It's remade with the time of the makefiles, which is older than the seed, so the old seed and the old disk stay. With all three gone, make writes new user-data, a new seed and a new .run script, and because the seed is newer, a fresh disk with its own provisioning boot. image-garden make clean does this for every system at once.
Details and sources
Details
- The user-data rule ends with
touch --reference=the newest of its prerequisites, the makefiles and.image-garden.mk(rulegen.py:467-471); the seed depends on the user-data and meta-data (010-qemu.mk:268), the prepared image on the seed (rulegen.py:601-602). - Run at the tag in a scratch directory, with
PKGPROXY_URLset after the files were made: removing onlyubuntu-cloud-24.04.x86_64.user-dataand making it again wrote the new user-data with the time ofimage-garden.mk, and make then said the.qcow2and the.runwere up to date. Removing the.user-data, the.seed.isoand the.runmademake -nlist the seed ISO,qemu-img create, the provisioning boot and the launcher script. make -nalone is misleading here: it assumes a remade user-data is new, and lists the seed and the disk even when only the user-data was removed.- image-garden runs make in
.image-garden/when that directory exists, else in the current directory (image-garden:987-993). image-garden make cleanremoves*.log,*.seed.iso,*.meta-dataand*.user-data, and each system's.qcow2,.runand its link in the download cache (000-garden.mk:48-50, 010-qemu.mk:300-302, rulegen.py:473-477). The downloaded images stay; the next make checks them again and links them.- Moving between the bundled pkgproxy and a pkgproxy snap needs only the
.runscript removed (see the previous scenes). - The new disk starts empty: anything changed by booting the
.runscript without-snapshotis lost (see the overlay).
06 In CI, a cached image keeps its mode for CI operators
cache-prepared-images keys .image-garden/ by system and hashFiles('.image-garden.mk'). A restored image keeps the mode, snap or external, of the job that made it; changing .image-garden.mk changes the key.
In CI, most jobs start on an empty machine, so every guest is made with the setting of the day. The exception is a cache of prepared images. image-garden-action's cache-prepared-images keys that cache by the system and a hash of your .image-garden.mk, not by the pkgproxy mode. The action picks snap mode, unless something already listens on port 3142, like a pkgproxy you set up first. Then it uses that one, by the runner's address. Add or drop that step, and a restored image still has the old mode inside. A change to your .image-garden.mk gives a new key.
Details and sources
Details
- With
cache-prepared-images: true, the action cachesSUBDIR/.image-gardenwithout its logs under the keyimage-garden-img-GARDEN_SYSTEM-HASH, whereHASHishashFiles('SUBDIR/.image-garden.mk')(action.yaml:116-123). The.user-data, the.seed.iso, the.runscript and the prepared.qcow2are all in it. - The action uses
externalmode whenssfinds a listener on port 3142, elsesnapmode (action.yaml:173-188). In snap mode it runssnap set image-garden pkgproxy=snap; in external modepkgproxy=http://ADDRESS:3142, with the first address ofhostname -I, "as 127.0.0.1 would be the guest itself" (action.yaml:375-393). - Derived, not run: a restored image made in snap mode has a
.runscript that forwards to the pkgproxy snap's socket, which an external-mode job does not have; one made in external mode has the first runner's address in its/etc/hosts. Even when make provisions the restored image again (see how the action caches), it does so from the restored seed, with the old mode. - Changing
.image-garden.mk, even by a comment, gives a new key;cache-key-prefixnames the pkgproxy caches only (action.yaml:30-31).
Read more
07 Where to go from here
Next: when the disk is made again, where the files live, caching on GitHub, an unreachable pkgproxy, the snap setting and the user-data.
So the pkgproxy choice is copied into a system's files once, and stays there until you remove them. When make provisions a disk again for other reasons, and what you lose then, is in the overlay deck. Where the system's files live, in .image-garden or your current directory, has a deck of its own. On GitHub, the action's caches are explained in full. A guest made while pkgproxy was down is a close cousin of this problem. In the snap, one setting has four values. And the user-data deck shows every line pkgproxy adds.
Details and sources
Read more
Basis of the statements on this page
Read in rulegen.py, mk/010-qemu.mk, mk/020-instances.mk, the snap's configure hook and image-garden-action's action.yaml at the pinned commits ("code"). What make rebuilds was checked by running v0.8.0's image-garden make on ubuntu-cloud-24.04.x86_64 in a scratch directory, outside the snap, with PKGPROXY_URL changed after the files were made ("run"). What apt and snapd do in an old guest, and what a restored CI image holds, are reasoned from the code ("derived").
Where the choice is written
| File | What it holds | Remade when |
|---|---|---|
SYSTEM.user-data | the pkgproxy hosts line with its address, the apt and snap setup lines, GOPROXY | a makefile or .image-garden.mk is newer, or it is missing |
SYSTEM.seed.iso | the user-data and meta-data | the user-data or meta-data is newer, or it is missing |
SYSTEM.qcow2 | after the provisioning boot: /etc/apt/apt.conf.d/99pkgproxy, the snapd drop-in, /etc/hosts, /etc/environment | the seed, the download or the firmware copies are newer, or it is missing |
SYSTEM.run | QEMU's guestfwd to the socket, in socket modes | a makefile is newer, or it is missing |
Sources: rulegen.py:461-471, rulegen.py:594-609, 010-qemu.mk:20-44, 010-qemu.mk:268.
What to remove after a change
| Change | Remove, then make the system |
|---|---|
component ⇄ snap | SYSTEM.run |
| any other change of mode or URL | SYSTEM.user-data, SYSTEM.seed.iso, SYSTEM.run |
| no change, but pkgproxy could not be reached while the guest was prepared | SYSTEM.qcow2 (proxy/README.md:159-164) |
| every system at once | image-garden make clean |
The files are in .image-garden/ when the project has that directory, otherwise in the directory you run image-garden from (image-garden:987-993). Removing only SYSTEM.user-data remakes it with the makefiles' time, older than the seed, so the seed and the disk are kept (run). A new disk starts empty: see the overlay.
Not affected
Image downloads choose their pkgproxy on every run, so GARDEN_DL_PKGPROXY, PKGPROXY_URL and the snap setting apply to the next download at once (image-garden:31-59).
Roads from here
RUNTIME · make rules and the CLI
- The overlay and the provisioning boot — when make provisions the disk again, and what is lost
- Where image-garden keeps its files — where the system's files are, in .image-garden or the current directory
BUILT ON IT · CI integrations
- How the action caches — the prepared-image cache that can bring an old setting back
PKGPROXY · caching for guests
- An unreachable pkgproxy — a guest made while pkgproxy was down
- What pkgproxy adds to user-data — exactly what pkgproxy adds to the user-data
SNAP · snap-only features
- The pkgproxy snap setting — the four values of the snap setting