pkgproxy for CI

operations for CI operators for administrators

Putting CI runners, job containers, VMs and build containers behind one pkgproxy, over the network or a UNIX socket, with the Docker mirror, files by checksum, offline mode for repeatable runs, and PKGPROXY_URL to tell jobs where pkgproxy is.

On the map: operations

▶ Watch this deck — 9 scenes, ~6 min

Part of journeys

01 What this topic covers

ABOUT THIS TOPIC pkgproxy for CI runners, VMs and build containers sharing one pkgproxy 1 Many machines, one cache 2 Jobs in containers 3 A VM with no route to the host 4 Sharing the snap's cache with another snap 5 Docker pulls and cloud images 6 Offline for a repeatable run 7 Telling jobs where pkgproxy is
The topic title, pkgproxy for CI, and its seven scenes in order: many machines, jobs in containers, a VM over a UNIX socket, sharing the snap's sockets, Docker pulls and cloud images, offline runs, telling jobs where pkgproxy is

CI runners, VMs and build containers share one pkgproxy, over the network or a UNIX socket, and can run against only what is cached.

A CI farm of runners, virtual machines and build containers can share one pkgproxy, so that each run gets its packages from the cache. First you see many machines pointed at one cache, then jobs that run in containers, then a virtual machine with no route to the host, and the snap's cache shared with another snap. Then come Docker pulls and cloud images, offline mode for a run that must be repeatable, and how a job learns where pkgproxy is.

02 Many machines, one cache

ONE CACHE runner 1 runner 2 test VM build container pkgproxy cache.lan:3142 $ curl -fsSL http://cache.lan:3142/setup/apt | sudo sh once per machine /setup/apt /setup/snap /setup/go /setup/docker without the tool: nothing to do, success first download upstream the others: from disk
Several runners, a VM and a container on a local network all use one pkgproxy, set up with a script fetched from its setup page; the first download goes upstream and the rest come from disk

curl -fsSL http://host:3142/setup/TOOL | sudo sh configures a machine; ?undo=1 reverses it. public-url sets the address the page names, for example behind NAT.

Imagine a CI farm: runners, virtual machines and containers, all downloading the same packages. You can point every one of them at a single pkgproxy, once, with a setup script from the setup page: fetch the apt script from port 3142 and pipe it to sudo sh. The scripts for apt, snap, Go and Docker do nothing on a machine without that tool, so one list can run on every guest of a test matrix. After the first machine downloads a package, the others get it from disk.

03 Jobs in containers

JOBS IN CONTAINERS the runner machine pkgproxy :3142 job on the machine the machine's settings, made once job container sees none of the machine's settings pkgproxy:host-gateway firewall: allow port 3142 # GitLab: config.toml [runners.docker], plus a mounted apt.conf.d file extra_hosts = ["pkgproxy:host-gateway"] # GitHub: container: options: --add-host=pkgproxy:host-gateway $ curl -fsSL http://pkgproxy:3142/setup/apt | sh
On a runner machine that runs pkgproxy, a job on the machine uses the machine's settings, made once; job containers reach the machine as the host name pkgproxy through Docker's host-gateway on port 3142; GitLab's runner mounts an apt configuration into every job, and a GitHub job runs the setup script in its first step

GitLab: extra_hosts = ["pkgproxy:host-gateway"] in config.toml, plus a mounted apt.conf.d file and GOPROXY. GitHub: options: --add-host=pkgproxy:host-gateway, then curl -fsSL http://pkgproxy:3142/setup/apt | sh.

A job that runs on the runner machine itself uses whatever that machine is set up for, so you run the setup scripts there once. A job in a container, as GitLab's docker executor runs every job, sees none of that. Give every job container a name for the machine that runs pkgproxy, through Docker's host-gateway, and let the containers' networks reach port 3142. The GitLab runner can then mount an apt configuration into every job, and a GitHub job can run the setup page's script in its first step.

04 A VM with no route to the host

UNIX SOCKET virtual machine apt no route to the host pkgproxy -socket-path PATH PATH no TCP port on the host :3142 guestfwd -addr still names the host:port the setup instructions print: give the one the guests use -lxd-tls-socket-path the same, for the HTTPS port of LXD remotes
Inside a VM, apt talks to an address that QEMU's guestfwd forwards to a UNIX socket on the host, where pkgproxy listens instead of on TCP; -addr still names what the instructions print

-socket-path replaces the TCP listener with a UNIX socket (mode 0777), for QEMU's guestfwd; -addr still names the host:port the instructions print. -lxd-tls-socket-path does the same for -lxd-tls-addr.

A virtual machine does not need a network route to the host. With the -socket-path flag, pkgproxy listens on a UNIX socket instead of TCP, and QEMU's guest forwarding can connect an address inside the guest to that socket. The guest uses that address like any other. The -addr flag still decides what the setup instructions print, so set it to the address the guests use. The -lxd-tls-socket-path does the same for the HTTPS port.

05 Sharing the snap's cache with another snap

CONTENT SOCKETS pkgproxy snap :3142 and :3143 for machines on the network $SNAP_COMMON/sockets/ pkgproxy.sock pkgproxy-lxd-tls.sock plain HTTPS :sockets content slot image-garden VM VM no second pkgproxy, no second cache $ sudo snap connect OTHER-SNAP:PLUG pkgproxy:sockets from the store, same publisher: connected automatically sideloaded only
The pkgproxy snap keeps its TCP ports and also listens on pkgproxy.sock and pkgproxy-lxd-tls.sock in its sockets directory, which its sockets content slot shares with another snap such as image-garden, which bridges them into its VMs

The snap runs with -content-socket-path and -lxd-tls-content-socket-path under $SNAP_COMMON/sockets, shared by the sockets slot (content pkgproxy-sockets). The TCP listeners stay as they are. snapd auto-connects a plug with the same content from the same publisher; sideloaded (--dangerous) snaps need snap connect.

The pkgproxy snap can share its cache with another snap. Besides its TCP ports, it always listens on two UNIX sockets in a sockets directory, one for the plain port and one for the HTTPS port, and offers that directory through a content interface. A snap that plugs it, such as image-garden, can bridge those sockets into its own virtual machines instead of running a second pkgproxy with a second cache. When both snaps come from the store from the same publisher, snapd connects them on its own: the base declaration allows it when the publishers and the content names match. A sideloaded snap has no publisher, so it is connected by hand.

06 Docker pulls and cloud images

DOCKER AND FILES docker pull "registry-mirrors": ["http://…:3142"] pkgproxy public Docker Hub images only registry-1.docker.io if pkgproxy is down VM test GET /files/v1/fetch?url=https%3A%2F%2Fcloud-images.ubuntu.com%2F…%2Fnoble-server-cloudimg-amd64.img &sha256=HEX cache key: the checksum, not the URL files-allow-host · files-allow-network · files-max-size 8G
Docker's registry-mirrors names pkgproxy first, with Docker Hub as the fallback; a test VM's cloud image is fetched through /files/v1/fetch with a URL and a SHA-256 that becomes the cache key

setup docker edits registry-mirrors in daemon.json (Docker Hub, anonymous pulls). /files/v1/fetch?url=…&sha256=… caches by checksum, over http or https; files-allow-host, files-allow-network and files-max-size (8G) limit it.

Two proxy engines matter most to CI. Setup docker puts pkgproxy first in the Docker daemon's registry-mirrors, so pulls of public Docker Hub images are cached, and if pkgproxy is down Docker goes to Docker Hub directly. Large files with no package format, such as the cloud images a VM test boots, are fetched by checksum: the client names the URL and its SHA-256, and the checksum is the cache key. The files-allow-host setting limits where pkgproxy may fetch from.

07 Offline for a repeatable run

OFFLINE RUNS $ pkgproxyctl offline on or: sudo snap set pkgproxy offline-mode=offline pkgproxy offline upstream cached .deb hit not cached 503 at once store-overloaded online, but an upstream in trouble is left alone for a while auto-offline=off on a network that reaches only a local mirror
In offline mode pkgproxy answers cached requests as hits and refuses uncached ones with 503 at once, contacting no upstream; store-overloaded leaves a troubled upstream alone; auto-offline=off for an isolated network

snap set pkgproxy offline-mode=offline or pkgproxyctl offline on; neither restarts pkgproxy. store-overloaded backs off from a troubled upstream. auto-offline=off on an isolated network.

For a run that must see exactly what was cached, switch pkgproxy to offline mode. It then serves only from the cache and contacts no upstream. Package lists are not refreshed, and anything not cached is refused at once with a 503. store-overloaded is the softer mode: an upstream that fails or asks to slow down is left alone for a while. On a network that only reaches a local mirror, turn auto-offline off, or pkgproxy takes the missing internet route for no network.

08 Telling jobs where pkgproxy is

PKGPROXY_URL image-garden · snapcraft and rockcraft CI components · pkgproxy's spread suite PKGPROXY_URL=http://cache.example:3142 CI runner sets it for every job VM container pkgproxy reachable from the VMs and containers: not localhost $ curl -fsSL $PKGPROXY_URL/setup/apt | sh # or /files/v1/fetch $ sudo snap set pkgproxy upstream-pkgproxy="$PKGPROXY_URL" # its own pkgproxy unset or empty: there is no pkgproxy, and the software works without one
A CI runner sets PKGPROXY_URL for its jobs to the address of the pkgproxy next to it, reachable from the job's VMs and containers; the job sets up its tools, downloads files, or runs its own pkgproxy with it as upstream

PKGPROXY_URL=http://HOST[:PORT], no path, plain HTTP; then curl -fsSL $PKGPROXY_URL/setup/<tool> | sh, /files/v1/fetch, or upstream-pkgproxy. The daemon itself never reads it.

Software that can use a pkgproxy when there is one, such as image-garden, the snapcraft and rockcraft CI components, or pkgproxy's own test suite, learns of it from one environment variable: PKGPROXY_URL. A runner sets it for its jobs, to the address of the pkgproxy next to it. That must be an address the job's virtual machines and containers can reach too, not localhost. A job then sets up its tools, downloads files through pkgproxy, or runs a pkgproxy of its own with that one as its upstream. Unset, there is no pkgproxy.

09 Where to go next general version

WHERE NEXT your runners, VMs and build containers share one cache; a package one run downloads is on disk for the next you are here pkgproxy for CI The setup page the scripts every machine fetches Hosted CI runners jobs on GitHub's and GitLab.com's own runners Two tiers each job's pkgproxy behind the one next to the runners
You are here, pkgproxy for CI; next: The setup page, Hosted CI runners, Two tiers

Next: The setup page, for the scripts every machine fetches; Hosted CI runners, for jobs on the forge's runners; Two tiers, for each job's pkgproxy behind the one next to the runners.

Your runners, virtual machines and build containers now share one cache, so a package that one run downloads is on disk for the next. To see the scripts every machine fetches, watch The setup page. To run pkgproxy inside jobs on GitHub's and GitLab.com's own runners, watch Hosted CI runners. To put each job's own pkgproxy behind the one next to the runners, watch Two tiers.

Read more

Jobs on the runner machine

A job that runs on the machine itself, as a GitHub self-hosted runner does by default and GitLab's shell executor does, uses whatever the machine is set up to use. Set each tool up once:

pkgproxyctl setup apt | sudo sh
pkgproxyctl setup snap | sudo sh
pkgproxyctl setup docker | sudo sh
sudo -u <runner user> sh -c 'pkgproxyctl setup go | sh'

setup go sets up the user who runs it, so run it as the user the runner runs its jobs as. On a machine that does not run pkgproxy itself, take the scripts from the one that does: curl -fsSL 'http://<host>:3142/setup/apt' | sudo sh.

Jobs in containers

A job container sees none of the runner machine's settings. Give it the host name pkgproxy for the machine, through Docker's host-gateway, and let the containers' networks reach port 3142 through the machine's firewall, for example sudo ufw allow from 172.16.0.0/12 to any port 3142 proto tcp.

GitLab's docker executor, in the runner's section of /etc/gitlab-runner/config.toml:

[[runners]]
  environment = ["GOPROXY=http://pkgproxy:3142/go,direct"]
  [runners.docker]
    extra_hosts = ["pkgproxy:host-gateway"]
    volumes = ["/cache", "/etc/gitlab-runner/pkgproxy-apt.conf:/etc/apt/apt.conf.d/99pkgproxy:ro"]

with the apt configuration every job container mounts:

echo 'Acquire::http::Proxy "http://pkgproxy:3142";' | sudo tee /etc/gitlab-runner/pkgproxy-apt.conf
echo 'Acquire::Retries::Delay "false";' | sudo tee -a /etc/gitlab-runner/pkgproxy-apt.conf

The second line is for job images with apt older than 3.1.3, such as Ubuntu 24.04's. A GitHub job in a container:

jobs:
  build:
    runs-on: self-hosted
    container:
      image: ubuntu:24.04
      options: --add-host=pkgproxy:host-gateway
    steps:
      - run: |
          apt-get update && apt-get install -y curl
          curl -fsSL http://pkgproxy:3142/setup/apt | sh

A tool that fetches large files by checksum gets http://pkgproxy:3142 in the job's environment; image-garden reads it from GARDEN_DL_PKGPROXY.

Telling jobs where pkgproxy is

Software that can use a pkgproxy when there is one, such as image-garden, the snapcraft and rockcraft CI components and pkgproxy's own spread suite, reads PKGPROXY_URL:

PKGPROXY_URL=http://cache.example:3142
  • The value is http://HOST[:PORT], with no path; a trailing slash is allowed.
  • HOST must be reachable from where the variable is set and from the VMs and containers started there: the machine's address on the network, not localhost and not a container runtime's name for the host.
  • Unset or empty, there is no pkgproxy, and the software works without one.
  • A runner sets it for its jobs, a shell profile for a person's commands. A job uses it with curl -fsSL $PKGPROXY_URL/setup/<tool> | sh, /files/v1/fetch, or as the upstream-pkgproxy of a pkgproxy of its own (see Two tiers).

The daemon itself never reads PKGPROXY_URL.

A pkgproxy in each job

A job that runs its own pkgproxy, as the snap in a fresh VM, starts with an empty cache. Set its upstream-pkgproxy to the pkgproxy next to the runners, which runs with offline-mode=store-overloaded, and what that one has cached is a LAN away for every job, also while an origin is down. See Two tiers.

Checking that jobs use the cache

Run the same job twice, then run pkgproxyctl stats: every download of the second run counts as a hit. The access log says the same per request (outcome=hit), and so does the cache page. A job that does not show up there does not use pkgproxy.

Roads from here

CORE · the path every proxy engine shares

DOCKER

FILES

SETUP AND THE COMMAND LINE

RUNNING PKGPROXY

OPERATIONS

image-garden map ↗