pkgproxy for CI

operations for CI operators for administrators

Putting CI runners, job containers, VMs and build containers behind one pkgproxy, over the network or a UNIX socket, with the Docker mirror, files by checksum and offline mode for repeatable runs.

On the map: operations

▶ Watch this deck — 8 scenes, ~5 min

01 What this topic covers

ABOUT THIS TOPIC pkgproxy for CI runners, VMs and build containers sharing one pkgproxy 1 Many machines, one cache 2 Jobs in containers 3 A VM with no route to the host 4 Sharing the snap's cache with another snap 5 Docker pulls and cloud images 6 Offline for a repeatable run
The topic title, pkgproxy for CI, and its six scenes in order: many machines, jobs in containers, a VM over a UNIX socket, sharing the snap's sockets, Docker pulls and cloud images, offline runs

CI runners, VMs and build containers share one pkgproxy, over the network or a UNIX socket, and can run against only what is cached.

A CI farm of runners, virtual machines and build containers can share one pkgproxy, so that each run gets its packages from the cache. First you see many machines pointed at one cache, then jobs that run in containers, then a virtual machine with no route to the host, and the snap's cache shared with another snap. Then come Docker pulls and cloud images, and offline mode for a run that must be repeatable.

02 Many machines, one cache

ONE CACHE runner 1 runner 2 test VM build container pkgproxy cache.lan:3142 $ curl -fsSL http://cache.lan:3142/setup/apt | sudo sh once per machine /setup/apt /setup/snap /setup/go /setup/docker without the tool: nothing to do, success first download upstream the others: from disk
Several runners, a VM and a container on a local network all use one pkgproxy, set up with a script fetched from its setup page; the first download goes upstream and the rest come from disk

curl -fsSL http://host:3142/setup/TOOL | sudo sh configures a machine; ?undo=1 reverses it. public-url sets the address the page names, for example behind NAT.

Imagine a CI farm: runners, virtual machines and containers, all downloading the same packages. You can point every one of them at a single pkgproxy, once, with a setup script from the setup page: fetch the apt script from port 3142 and pipe it to sudo sh. The scripts for apt, snap, Go and Docker do nothing on a machine without that tool, so one list can run on every guest of a test matrix. After the first machine downloads a package, the others get it from disk.

03 Jobs in containers

JOBS IN CONTAINERS the runner machine pkgproxy :3142 job on the machine the machine's settings, made once job container sees none of the machine's settings pkgproxy:host-gateway firewall: allow port 3142 # GitLab: config.toml [runners.docker], plus a mounted apt.conf.d file extra_hosts = ["pkgproxy:host-gateway"] # GitHub: container: options: --add-host=pkgproxy:host-gateway $ curl -fsSL http://pkgproxy:3142/setup/apt | sh
On a runner machine that runs pkgproxy, a job on the machine uses the machine's settings, made once; job containers reach the machine as the host name pkgproxy through Docker's host-gateway on port 3142; GitLab's runner mounts an apt configuration into every job, and a GitHub job runs the setup script in its first step

GitLab: extra_hosts = ["pkgproxy:host-gateway"] in config.toml, plus a mounted apt.conf.d file and GOPROXY. GitHub: options: --add-host=pkgproxy:host-gateway, then curl -fsSL http://pkgproxy:3142/setup/apt | sh.

A job that runs on the runner machine itself uses whatever that machine is set up for, so you run the setup scripts there once. A job in a container, as GitLab's docker executor runs every job, sees none of that. Give every job container a name for the machine that runs pkgproxy, through Docker's host-gateway, and let the containers' networks reach port 3142. The GitLab runner can then mount an apt configuration into every job, and a GitHub job can run the setup page's script in its first step.

04 A VM with no route to the host

UNIX SOCKET virtual machine apt no route to the host pkgproxy -socket-path PATH PATH no TCP port on the host :3142 guestfwd -addr still names the host:port the setup instructions print: give the one the guests use -lxd-tls-socket-path the same, for the HTTPS port of LXD remotes
Inside a VM, apt talks to an address that QEMU's guestfwd forwards to a UNIX socket on the host, where pkgproxy listens instead of on TCP; -addr still names what the instructions print

-socket-path replaces the TCP listener with a UNIX socket (mode 0777), for QEMU's guestfwd; -addr still names the host:port the instructions print. -lxd-tls-socket-path does the same for -lxd-tls-addr.

A virtual machine does not need a network route to the host. With the -socket-path flag, pkgproxy listens on a UNIX socket instead of TCP, and QEMU's guest forwarding can connect an address inside the guest to that socket. The guest uses that address like any other. The -addr flag still decides what the setup instructions print, so set it to the address the guests use. The -lxd-tls-socket-path does the same for the HTTPS port.

05 Sharing the snap's cache with another snap

CONTENT SOCKETS pkgproxy snap :3142 and :3143 for machines on the network $SNAP_COMMON/sockets/ pkgproxy.sock pkgproxy-lxd-tls.sock plain HTTPS :sockets content slot image-garden VM VM no second pkgproxy, no second cache $ sudo snap connect OTHER-SNAP:PLUG pkgproxy:sockets from the store, same publisher: connected automatically sideloaded only
The pkgproxy snap keeps its TCP ports and also listens on pkgproxy.sock and pkgproxy-lxd-tls.sock in its sockets directory, which its sockets content slot shares with another snap such as image-garden, which bridges them into its VMs

The snap runs with -content-socket-path and -lxd-tls-content-socket-path under $SNAP_COMMON/sockets, shared by the sockets slot (content pkgproxy-sockets). The TCP listeners stay as they are. snapd auto-connects a plug with the same content from the same publisher; sideloaded (--dangerous) snaps need snap connect.

The pkgproxy snap can share its cache with another snap. Besides its TCP ports, it always listens on two UNIX sockets in a sockets directory, one for the plain port and one for the HTTPS port, and offers that directory through a content interface. A snap that plugs it, such as image-garden, can bridge those sockets into its own virtual machines instead of running a second pkgproxy with a second cache. When both snaps come from the store from the same publisher, snapd connects them on its own: the base declaration allows it when the publishers and the content names match. A sideloaded snap has no publisher, so it is connected by hand.

06 Docker pulls and cloud images

DOCKER AND FILES docker pull "registry-mirrors": ["http://…:3142"] pkgproxy public Docker Hub images only registry-1.docker.io if pkgproxy is down VM test GET /files/v1/fetch?url=https%3A%2F%2Fcloud-images.ubuntu.com%2F…%2Fnoble-server-cloudimg-amd64.img &sha256=HEX cache key: the checksum, not the URL files-allow-host · files-allow-network · files-max-size 8G
Docker's registry-mirrors names pkgproxy first, with Docker Hub as the fallback; a test VM's cloud image is fetched through /files/v1/fetch with a URL and a SHA-256 that becomes the cache key

setup docker edits registry-mirrors in daemon.json (Docker Hub, anonymous pulls). /files/v1/fetch?url=…&sha256=… caches by checksum, over http or https; files-allow-host, files-allow-network and files-max-size (8G) limit it.

Two sides matter most to CI. Setup docker puts pkgproxy first in the Docker daemon's registry-mirrors, so pulls of public Docker Hub images are cached, and if pkgproxy is down Docker goes to Docker Hub directly. Large files with no package format, such as the cloud images a VM test boots, are fetched by checksum: the client names the URL and its SHA-256, and the checksum is the cache key. The files-allow-host setting limits where pkgproxy may fetch from.

07 Offline for a repeatable run

OFFLINE RUNS $ pkgproxyctl offline on or: sudo snap set pkgproxy offline-mode=offline pkgproxy offline upstream cached .deb hit not cached 503 at once store-overloaded online, but an upstream in trouble is left alone for a while auto-offline=off on a network that reaches only a local mirror
In offline mode pkgproxy answers cached requests as hits and refuses uncached ones with 503 at once, contacting no upstream; store-overloaded leaves a troubled upstream alone; auto-offline=off for an isolated network

snap set pkgproxy offline-mode=offline or pkgproxyctl offline on; neither restarts pkgproxy. store-overloaded backs off from a troubled upstream. auto-offline=off on an isolated network.

For a run that must see exactly what was cached, switch pkgproxy to offline mode. It then serves only from the cache and contacts no upstream. Package lists are not refreshed, and anything not cached is refused at once with a 503. store-overloaded is the softer mode: an upstream that fails or asks to slow down is left alone for a while. On a network that only reaches a local mirror, turn auto-offline off, or pkgproxy takes the missing internet route for no network.

08 Where to go next general version

WHERE NEXT your runners, VMs and build containers share one cache; a package one run downloads is on disk for the next you are here pkgproxy for CI The setup page the scripts every machine fetches Hosted CI runners jobs on GitHub's and GitLab.com's own runners Offline mode runs that see only what is cached
You are here, pkgproxy for CI; next: The setup page, Hosted CI runners, Offline mode

Next: The setup page, for the scripts every machine fetches; Hosted CI runners, for jobs on the forge's runners; Offline mode, for runs that see only what is cached.

Your runners, virtual machines and build containers now share one cache, so a package that one run downloads is on disk for the next. To see the scripts every machine fetches, watch The setup page. To run pkgproxy inside jobs on GitHub's and GitLab.com's own runners, watch Hosted CI runners. To make runs that see only what is cached, watch Offline mode.

Read more

Jobs on the runner machine

A job that runs on the machine itself, as a GitHub self-hosted runner does by default and GitLab's shell executor does, uses whatever the machine is set up to use. Set each tool up once:

pkgproxyctl setup apt | sudo sh
pkgproxyctl setup snap | sudo sh
pkgproxyctl setup docker | sudo sh
sudo -u <runner user> sh -c 'pkgproxyctl setup go | sh'

setup go sets up the user who runs it, so run it as the user the runner runs its jobs as. On a machine that does not run pkgproxy itself, take the scripts from the one that does: curl -fsSL 'http://<host>:3142/setup/apt' | sudo sh.

Jobs in containers

A job container sees none of the runner machine's settings. Give it the host name pkgproxy for the machine, through Docker's host-gateway, and let the containers' networks reach port 3142 through the machine's firewall, for example sudo ufw allow from 172.16.0.0/12 to any port 3142 proto tcp.

GitLab's docker executor, in the runner's section of /etc/gitlab-runner/config.toml:

[[runners]]
  environment = ["GOPROXY=http://pkgproxy:3142/go,direct"]
  [runners.docker]
    extra_hosts = ["pkgproxy:host-gateway"]
    volumes = ["/cache", "/etc/gitlab-runner/pkgproxy-apt.conf:/etc/apt/apt.conf.d/99pkgproxy:ro"]

with the apt configuration every job container mounts:

echo 'Acquire::http::Proxy "http://pkgproxy:3142";' | sudo tee /etc/gitlab-runner/pkgproxy-apt.conf
echo 'Acquire::Retries::Delay "false";' | sudo tee -a /etc/gitlab-runner/pkgproxy-apt.conf

The second line is for job images with apt older than 3.1.3, such as Ubuntu 24.04's. A GitHub job in a container:

jobs:
  build:
    runs-on: self-hosted
    container:
      image: ubuntu:24.04
      options: --add-host=pkgproxy:host-gateway
    steps:
      - run: |
          apt-get update && apt-get install -y curl
          curl -fsSL http://pkgproxy:3142/setup/apt | sh

A tool that fetches large files by checksum gets http://pkgproxy:3142 in the job's environment; image-garden reads it from GARDEN_DL_PKGPROXY.

Checking that jobs use the cache

Run the same job twice, then run pkgproxyctl stats: every download of the second run counts as a hit. The access log says the same per request (outcome=hit), and so does the cache page. A job that does not show up there does not use pkgproxy.

Roads from here

CORE · the path every side shares

DOCKER

FILES

SETUP AND THE COMMAND LINE

OPERATIONS