What pkgproxy adds to user-data

hosts, apt, snap, GOPROXY for CI operators

With a pkgproxy in reach when make writes a system's user-data, cloud-init adds pkgproxy to /etc/hosts, runs pkgproxy's apt setup script from bootcmd on Debian and Ubuntu and its snap setup script from runcmd on every family but Alpine, and appends GOPROXY to /etc/environment; Ubuntu Core gets none of it.

On the map: pkgproxy

▶ Watch this deck — 7 scenes, ~5 min

Worth seeing first

01 Three places in one file

ONE FILE, THREE ADDITIONS #cloud-config timezone: Europe/Warsaw power_state: … hostname: ubuntu-24-04 ssh_pwauth: true users: … write_files: - path: /root/sshd_config.APPEND … runcmd: - cat /root/hosts.APPEND >> /etc/hosts … - touch /etc/cloud/cloud-init.disabled ubuntu-cloud-24.04.x86_64.user-data, in outline bootcmd: … bootcmd: hosts, a probe, apt - path: /etc/environment … GOPROXY for every login - | (hosts, snap setup) runcmd: hosts, snap without a pkgproxy, the three are empty; the rest is the same no apt or snapd settings of its own: http://pkgproxy:3142/setup/TOOL pkgproxy's own scripts
An outline of ubuntu-cloud-24.04.x86_64.user-data. With a pkgproxy it gains three things: a bootcmd block with the hosts line, a probe and the apt setup; an /etc/environment entry in write_files for GOPROXY; and a runcmd item with the hosts line and the snap setup. Without a pkgproxy the three are empty. image-garden writes no apt proxy file and no snapd setting; the guest runs pkgproxy's own setup scripts from http://pkgproxy:3142/setup/.

With a pkgproxy, ubuntu-cloud-24.04.x86_64.user-data gains a bootcmd block, an /etc/environment entry in write_files and a runcmd item. The apt and snapd settings come from pkgproxy's own setup scripts.

Your guest's apt, snapd and Go all need to be pointed at pkgproxy, and it has to happen in the very first boot. So when make prepares ubuntu-cloud-24.04.x86_64 with a pkgproxy in reach, the user-data it writes grows in three places: a bootcmd block near the top, a file to append in write_files, and one more item in runcmd. Without a pkgproxy, those spots are empty and the rest is the same. What you won't find is an apt proxy file or a snapd setting written by image-garden. The guest fetches pkgproxy's own setup scripts instead, so the setup always matches the pkgproxy that serves it.

Details and sources

Details

  • The common template puts $(call PKGPROXY_CLOUD_INIT_BOOTCMD,$1) after ssh_pwauth, $(INJECTED_CLOUD_INIT_FILES) at the end of write_files and $(call PKGPROXY_CLOUD_INIT_RUNCMD,$1) before the last runcmd item, which creates /etc/cloud/cloud-init.disabled (010-qemu.mk:187-247). Without a pkgproxy all three are empty (010-qemu.mk:181-185).
  • "Guests know pkgproxy as "pkgproxy" and set themselves up with the scripts it serves under /setup/, so that they always match the pkgproxy in use" (010-qemu.mk:108-109); "image-garden keeps no copies of them" (proxy/README.md:147-152). tests/check-pkgproxy-user-data.py checks that image-garden writes no apt or snapd configuration of its own.
  • Run for this deck: image-garden make ubuntu-cloud-24.04.x86_64.user-data with a fixture socket, and again with nothing set; the two files differ only in the three places.
  • The port is the URL's with PKGPROXY_URL, and 3142 with the snap's sockets (010-qemu.mk:128).
  • A project template that does not start from the common template loses these lines: see start from the common template.

02 A name for the host

THE HOSTS LINE bootcmd: - | grep -q '^10.0.2.100 pkgproxy$' /etc/hosts || { echo '# image-garden: pkgproxy, see http://pkgproxy:3142/'; echo '10.0.2.100 pkgproxy'; } >>/etc/hosts 10.0.2.100: the snap's socket or your URL's host, as an address only if it isn't there runcmd: - | grep -q … >>/etc/hosts again, in case cloud-init rewrote it http://pkgproxy:3142 curl -fsSL … http://pkgproxy:3142/setup/TOOL | sh or wget, where there is no curl
bootcmd's first line: grep -q '^10.0.2.100 pkgproxy$' /etc/hosts or append a comment and 10.0.2.100 pkgproxy. The address is 10.0.2.100 for the snap's socket or the URL host's address. runcmd adds the same line again, in case cloud-init rewrote /etc/hosts. The guest then uses http://pkgproxy:3142, and fetches setup scripts from http://pkgproxy:3142/setup/TOOL with curl, or wget.

10.0.2.100 pkgproxy (or the URL's address) goes into /etc/hosts in bootcmd and again in runcmd, unless it is there. Scripts come from http://pkgproxy:PORT/setup/TOOL.

Every piece starts with a name. The first line of the bootcmd block adds pkgproxy to /etc/hosts, with the address make chose: 10.0.2.100 for the snap's socket, or the address of your URL's host. It adds the line only if it isn't there yet, and the runcmd item adds it again, in case cloud-init rewrote the hosts file in between. From then on, everything in the guest just says pkgproxy and the port. The setup scripts come from there too, from pkgproxy's setup path, fetched with curl, or with wget where there's no curl.

Details and sources

Details

  • PKGPROXY_HOSTS_SHELL: "A shell line that adds pkgproxy to /etc/hosts unless it is there. It runs from runcmd as well, in case cloud-init rewrote /etc/hosts since bootcmd" (010-qemu.mk:132-134). It is the first line of the bootcmd block and of the runcmd item (010-qemu.mk:156-168).
  • The lines on the slide are from ubuntu-cloud-24.04.x86_64.user-data made with a fixture socket; with PKGPROXY_URL=http://192.0.2.9:3142 only the address changes.
  • PKGPROXY_GUEST_URL = http://pkgproxy:$(PKGPROXY_PORT) (010-qemu.mk:128).
  • PKGPROXY_SETUP_SHELL fetches each script with curl -fsSL --connect-timeout 5 http://pkgproxy:PORT/setup/"$tool" | sh, or wget -q -T 5 -O - … | sh, or prints image-garden: no curl or wget to fetch … (010-qemu.mk:150-154). "Only the connection has a short timeout: the scripts are small, but the network may be slow."
  • Every line of the blocks is POSIX sh without backslashes, "which the echo that writes the user-data could take as escapes" (010-qemu.mk:146-147).
  • The base template's own /root/hosts.APPEND, which maps the guest's host name to 127.0.0.1, is appended in runcmd too, before the pkgproxy item (010-qemu.mk:231-245).

Read more

03 apt, early, from bootcmd

APT, EARLY bootcmd packages: runcmd power off Debian and Ubuntu the first boot already through pkgproxy a probe: asks pkgproxy once no answer: no setup in this boot curl … /setup/apt | sh /etc/apt/apt.conf.d/99pkgproxy Acquire::http::Proxy … apt keeps its mirrors and fetches through pkgproxy
The first boot runs bootcmd, then cloud-init's packages, then runcmd, then powers off. On Debian and Ubuntu, bootcmd probes pkgproxy once and runs curl … /setup/apt | sh, which writes Acquire::http::Proxy to /etc/apt/apt.conf.d/99pkgproxy, so packages installed by cloud-init already come through pkgproxy. apt keeps its mirrors and fetches through the proxy.

On Debian and Ubuntu, bootcmd probes pkgproxy once, then runs /setup/apt, which writes Acquire::http::Proxy to /etc/apt/apt.conf.d/99pkgproxy before cloud-init installs packages.

On Debian and Ubuntu, apt is set up first, from bootcmd. That's very early in the first boot, before cloud-init installs any packages, and it matters when your template adds a packages list: those installs already go through pkgproxy. The block asks pkgproxy once whether it's there at all. Then it fetches the apt setup script and hands it to the shell. The script writes apt's HTTP proxy setting into a file under apt's configuration directory. apt keeps its usual mirrors and simply fetches through the proxy. On every other family this step is left out.

Details and sources

Details

  • PKGPROXY_BOOTCMD_SETUP = $(if $(filter debian ubuntu,$1),apt) (010-qemu.mk:129). It "runs from bootcmd, early in the first boot, so that cloud-init's own package installs (packages: in a custom template) already go through pkgproxy" (010-qemu.mk:114-116).
  • The bootcmd block is the hosts line, the probe and, where the group has one, the setup line (010-qemu.mk:156-162). The probe asks http://pkgproxy:PORT/ once and, without an answer, leaves /run/image-garden-pkgproxy-unreachable, so that neither batch waits again in that boot (010-qemu.mk:122-126): see an unreachable pkgproxy.
  • pkgproxy's apt script "writes Acquire::http::Proxy to /etc/apt/apt.conf.d/99pkgproxy" (proxy/README.md:171-174). The script itself is pkgproxy's, served by the running daemon; its exact content is in pkgproxy's tree, not image-garden's.
  • apt is "a real forward proxy. Clients keep their real sources.list mirror URLs and point Acquire::http::Proxy at the proxy" (proxy/README.md:40-41).
  • Separately, the common template's runcmd rewrites https:// to http:// in every *.list and deb822 *.sources file under /etc/apt, with or without a pkgproxy, so that what apt fetches is plain http that pkgproxy can cache (010-qemu.mk:244); see SSH logins and apt sources.

04 snapd, late, from runcmd

SNAPD, LATE bootcmd packages: runcmd power off every family but Alpine the first boot not from bootcmd: the restart waits for snapd, which may start after cloud-init's early stages curl … /setup/snap | sh /etc/systemd/system/snapd.service.d/pkgproxy.conf SNAPPY_FORCE_API_URL: pkgproxy as the store a snapd installed later reads the same drop-in the next item: touch /etc/cloud/cloud-init.disabled
On every family but Alpine, runcmd at the end of the first boot runs curl … /setup/snap | sh, not bootcmd, because the restart waits for snapd, which may start after cloud-init's early stages. The script writes /etc/systemd/system/snapd.service.d/pkgproxy.conf setting SNAPPY_FORCE_API_URL, so snapd uses pkgproxy as the store, also a snapd installed later. The next runcmd item switches cloud-init off.

On every family but Alpine, runcmd runs /setup/snap: a drop-in for snapd.service with SNAPPY_FORCE_API_URL, written even before snapd is installed, and a restart of snapd where it is.

snapd is set up last, from runcmd, at the end of the first boot, on every family but Alpine. It can't go earlier. The snap setup script restarts snapd, the restart command waits until snapd is back, and snapd may be ordered after cloud-init's early stages, so from bootcmd that wait could hold up the boot. The script writes a systemd drop-in that points snapd at pkgproxy as if it were the snap store. It writes the drop-in even where snapd isn't installed yet, so a snapd your tests install later uses pkgproxy too. Right after this item, cloud-init is switched off.

Details and sources

Details

  • PKGPROXY_RUNCMD_SETUP = $(if $(filter alpine,$1),,snap) (010-qemu.mk:130). "PKGPROXY_RUNCMD_SETUP runs from runcmd, at the end of the first boot. The snap script restarts snapd, which must not happen from bootcmd: systemctl waits for the restart, and snapd may be ordered after cloud-init's early stages" (010-qemu.mk:117-120). That the wait would hold up the boot is the slide's reading of that comment (derived).
  • The runcmd item is the hosts line and the setup line (010-qemu.mk:164-168), placed just before touch /etc/cloud/cloud-init.disabled (010-qemu.mk:245-246).
  • pkgproxy's snap script "writes a systemd drop-in for snapd (/etc/systemd/system/snapd.service.d/pkgproxy.conf) that sets SNAPPY_FORCE_API_URL, also where snapd is installed later, and restarts snapd where it is installed" (proxy/README.md:175-180). snapd is "pointed at the proxy directly, as if it were the store … not a traditional proxy setting" (proxy/README.md:42-44).
  • pkgproxy=none exists mainly for snapd's own spread suite, where some tests are not yet adjusted for the injected SNAPPY_FORCE_API_URL or the extra /etc/hosts entries (proxy/README.md:134-136).

05 Go, through /etc/environment

GO, THE ODD ONE OUT the go command: no system-wide config pkgproxy's go script: one user, go installed not run at first boot - path: /etc/environment permissions: '0644' append: true content: | GOPROXY="http://pkgproxy:3142/go,direct" ,direct: Go's way back to the module's own source every SSH session $ ssh … (a login shell) $ ssh … go build ./...
The go command has no system-wide configuration file, and pkgproxy's go script configures one user and needs Go installed, so it is not run at first boot. Instead write_files appends GOPROXY="http://pkgproxy:3142/go,direct" to /etc/environment, which every SSH session reads: a login shell and a one-off go build alike.

write_files appends GOPROXY="http://pkgproxy:3142/go,direct" to /etc/environment, which PAM reads for every SSH session, before and after Go is installed.

Go is the odd one out. The go command has no system-wide configuration file, and pkgproxy's go script sets up only the user who runs it, and only where Go is installed, which at first boot it usually isn't. So image-garden doesn't run that script. It appends one line to /etc/environment instead: GOPROXY, pointing at pkgproxy's go path, then ,direct, Go's way back to the module's own source. That file is read for every SSH session, interactive or not. So a shell you log into, and a one-off go build over SSH, both get the proxy.

Details and sources

Details

  • "The go command has no system-wide configuration, and pkgproxy's setup script for it configures the user who runs it, where go is installed: at first boot it usually is not. /etc/environment reaches every user's SSH session, before and after go is installed" (010-qemu.mk:170-173). The write_files entry is INJECTED_CLOUD_INIT_FILES (010-qemu.mk:174-180).
  • go env -w, which pkgproxy's script uses, "only writes the invoking user's own $GOENV"; /etc/environment is read by PAM (pam_env, enabled by the template's UsePam yes sshd setting) for every SSH session (proxy/README.md:192-202, 010-qemu.mk:212-220).
  • The README says the trailing ,direct keeps modules pkgproxy can't serve reachable from their source "once GOPRIVATE excludes them" (proxy/README.md:200-202). In Go's GOPROXY list a comma moves to the next entry only when the proxy answers 404 or 410 (derived: Go's documented behaviour).
  • The port is the URL's with PKGPROXY_URL; 3142 with the snap's sockets.
  • The craft components run pkgproxy's go script themselves, as root, because snapcraft builds as root: see before the build, the setup scripts.

06 Family by family

WHO GETS WHAT hosts apt snap GOPROXY Debian, Ubuntu yes yes yes yes Fedora, Arch, openSUSE, … yes - yes yes Alpine yes - - yes Ubuntu Core none: a template of its own # .image-garden.mk, an example: no snap setup on Debian either PKGPROXY_RUNCMD_SETUP = $(if $(filter alpine debian,$1),,snap) # the default: PKGPROXY_BOOTCMD_SETUP = $(if $(filter debian ubuntu,$1),apt) $1 is the guest's group, such as debian or fedora; the result names the scripts
A table of what each family gets: Debian and Ubuntu the hosts line, apt, snap and GOPROXY; the other cloud families the hosts line, snap and GOPROXY; Alpine the hosts line and GOPROXY; Ubuntu Core nothing. Below, an example .image-garden.mk that redefines PKGPROXY_RUNCMD_SETUP to skip the snap setup on Debian too; both variables are called with the guest's group as $1.

Debian, Ubuntu: hosts, apt, snap, GOPROXY. Other families: hosts, snap, GOPROXY. Alpine: hosts, GOPROXY. Ubuntu Core: none. A project's .image-garden.mk may redefine PKGPROXY_BOOTCMD_SETUP and PKGPROXY_RUNCMD_SETUP.

Put together, each family gets its own mix. Debian and Ubuntu get all four: the hosts line, apt, snapd and GOPROXY. Fedora, Arch, openSUSE and the other cloud families get the hosts line, snapd and GOPROXY. Alpine gets the hosts line and GOPROXY. Ubuntu Core gets none of it, because its user-data comes from a template of its own. If your project needs a different mix, your .image-garden.mk can redefine the two variables that pick the scripts, PKGPROXY_BOOTCMD_SETUP and PKGPROXY_RUNCMD_SETUP. Each is called with the guest's group, and returns the names of the scripts to run.

Details and sources

Details

  • PKGPROXY_BOOTCMD_SETUP = $(if $(filter debian ubuntu,$1),apt) and PKGPROXY_RUNCMD_SETUP = $(if $(filter alpine,$1),,snap) (010-qemu.mk:129-130). The scripts are picked per family because "pkgproxy's setup scripts do nothing where their tool is missing, but there is no one script for a whole guest yet (zygoon/pkgproxy#92)" (010-qemu.mk:110-112).
  • The groups in data/*.ini are almalinux, alpine, amazonlinux, archlinux, centos, debian, fedora, gentoo, opencloudos, opensuse, oracle, rocky and ubuntu; all but alpine, debian and ubuntu get the "other families" row.
  • "A .image-garden.mk can override either variable; $1 is the guest's group, such as debian or fedora" (proxy/README.md:182-184). The example on the slide is illustrative, not taken from the tree; an empty result leaves the setup line out of that block.
  • "Ubuntu Core guests are not set up for pkgproxy" (proxy/README.md:204): their user-data comes from BASE_UBUNTU_CORE_CLOUD_INIT_USER_DATA_TEMPLATE, which has no pkgproxy lines (050-ubuntu-core.mk:131-179).
  • A release or group template that starts from the common template keeps these lines; one written from scratch does not: see your template replaces these.
  • tests/check-pkgproxy-config.sh checks one system of each of debian, ubuntu, fedora, alpine, amazonlinux and opensuse (check-pkgproxy-config.sh:24-25).

07 Where to go from here

ROADS In the user-data Reaching pkgproxy where the address comes from When it is down the probe and its marker The common template the file these blocks live in .image-garden.mk changing it from your project No rebuild made images keep their setup
Five roads from In the user-data: reaching pkgproxy, an unreachable pkgproxy, the common template, your .image-garden.mk, and no rebuild after a change

Next: reaching pkgproxy, an unreachable pkgproxy, the common template, your .image-garden.mk and why made images keep their setup.

That's everything pkgproxy adds: a name in the hosts file, apt set up early, snapd set up late, and GOPROXY for every login. To see where the name's address comes from, take the reaching deck. If you wonder what the probe does when nothing answers, the unreachable deck has it. The template decks show the file these blocks live in, and how your own .image-garden.mk changes it. And the rebuild deck explains why images made before you change the setting keep what they had.

Details and sources

Read more

Basis of the statements on this page

Read in mk/010-qemu.mk, mk/050-ubuntu-core.mk and proxy/README.md of image-garden at the v0.8.0 tag ("code"). The excerpts below were produced by running image-garden make ubuntu-cloud-24.04.x86_64.user-data from an export of that tag, once with a fixture $SNAP_COMMON holding pkgproxy.sock and once with nothing set ("run"). What pkgproxy's setup scripts write is from proxy/README.md: the scripts are served by pkgproxy and are not in image-garden's tree.

The three additions, as made

With the snap's socket (10.0.2.100, port 3142). With PKGPROXY_URL=http://192.0.2.9:3142, only the address on the two hosts lines changes.

bootcmd:
- |
  grep -q '^10.0.2.100 pkgproxy$' /etc/hosts || { echo '# image-garden: pkgproxy, see http://pkgproxy:3142/'; echo '10.0.2.100 pkgproxy'; } >>/etc/hosts
  if command -v curl >/dev/null 2>&1; then curl -s -o /dev/null --connect-timeout 5 --max-time 20 http://pkgproxy:3142/; elif command -v wget >/dev/null 2>&1; then wget -q -T 5 -t 1 -O /dev/null http://pkgproxy:3142/; fi || { echo "image-garden: cannot reach pkgproxy at http://pkgproxy:3142/, skipping its setup scripts in this boot" >&2; touch /run/image-garden-pkgproxy-unreachable; }
  if [ -e /run/image-garden-pkgproxy-unreachable ]; then echo "image-garden: pkgproxy was unreachable, skipping setup scripts: apt" >&2; else for tool in apt; do if command -v curl >/dev/null 2>&1; then curl -fsSL --connect-timeout 5 http://pkgproxy:3142/setup/"$tool" | sh; elif command -v wget >/dev/null 2>&1; then wget -q -T 5 -O - http://pkgproxy:3142/setup/"$tool" | sh; else echo "image-garden: no curl or wget to fetch http://pkgproxy:3142/setup/$tool" >&2; fi; done; fi
write_files:
...
- path: /etc/environment
  permissions: '0644'
  append: true
  content: |
    GOPROXY="http://pkgproxy:3142/go,direct"
runcmd:
...
- |
  grep -q '^10.0.2.100 pkgproxy$' /etc/hosts || { echo '# image-garden: pkgproxy, see http://pkgproxy:3142/'; echo '10.0.2.100 pkgproxy'; } >>/etc/hosts
  if [ -e /run/image-garden-pkgproxy-unreachable ]; then echo "image-garden: pkgproxy was unreachable, skipping setup scripts: snap" >&2; else for tool in snap; do if command -v curl >/dev/null 2>&1; then curl -fsSL --connect-timeout 5 http://pkgproxy:3142/setup/"$tool" | sh; elif command -v wget >/dev/null 2>&1; then wget -q -T 5 -O - http://pkgproxy:3142/setup/"$tool" | sh; else echo "image-garden: no curl or wget to fetch http://pkgproxy:3142/setup/$tool" >&2; fi; done; fi
- touch /etc/cloud/cloud-init.disabled

What each setup changes in the guest

PieceRuns fromFamiliesWhat it leaves in the guest
hosts linebootcmd and runcmdall but Ubuntu CoreADDRESS pkgproxy and a comment in /etc/hosts
probebootcmdall but Ubuntu Core/run/image-garden-pkgproxy-unreachable when pkgproxy does not answer
/setup/aptbootcmddebian, ubuntuAcquire::http::Proxy in /etc/apt/apt.conf.d/99pkgproxy
/setup/snapruncmdall but alpine and Ubuntu Core/etc/systemd/system/snapd.service.d/pkgproxy.conf with SNAPPY_FORCE_API_URL; snapd restarted where installed
GOPROXYwrite_filesall but Ubuntu CoreGOPROXY="http://pkgproxy:PORT/go,direct" appended to /etc/environment

Sources: 010-qemu.mk:127-185, proxy/README.md:166-204.

The make variables

VariableWhat it isOverride in .image-garden.mk
PKGPROXY_GUEST_URLhttp://pkgproxy:PORTno need
PKGPROXY_BOOTCMD_SETUPscripts for bootcmd, called with the group as $1; apt for debian and ubuntuyes
PKGPROXY_RUNCMD_SETUPscripts for runcmd, called with the group; snap for all but alpineyes
PKGPROXY_HOSTS_SHELLthe hosts linenot documented
PKGPROXY_PROBE_SHELLthe probenot documented
PKGPROXY_UNREACHABLE_FILE/run/image-garden-pkgproxy-unreachablenot documented
PKGPROXY_SETUP_SHELLthe loop that fetches and runs scriptsnot documented
INJECTED_CLOUD_INIT_FILESthe /etc/environment entrynot documented

All of them are defined only when make found a pkgproxy; otherwise the three blocks are empty (010-qemu.mk:181-185).

Roads from here

BUILT ON IT · CI integrations

FIRST BOOT · cloud-init changes

PKGPROXY · caching for guests

pkgproxy map ↗