pkgproxy for CI
operations for CI operators for administrators
Putting CI runners, job containers, VMs and build containers behind one pkgproxy, over the network or a UNIX socket, with the Docker mirror, files by checksum and offline mode for repeatable runs.
On the map: operations
01 What this topic covers
CI runners, VMs and build containers share one pkgproxy, over the network or a UNIX socket, and can run against only what is cached.
A CI farm of runners, virtual machines and build containers can share one pkgproxy, so that each run gets its packages from the cache. First you see many machines pointed at one cache, then jobs that run in containers, then a virtual machine with no route to the host, and the snap's cache shared with another snap. Then come Docker pulls and cloud images, and offline mode for a run that must be repeatable.
02 Many machines, one cache
curl -fsSL http://host:3142/setup/TOOL | sudo sh configures
a machine; ?undo=1 reverses it. public-url sets
the address the page names, for example behind NAT.
Imagine a CI farm: runners, virtual machines and containers, all downloading the same packages. You can point every one of them at a single pkgproxy, once, with a setup script from the setup page: fetch the apt script from port 3142 and pipe it to sudo sh. The scripts for apt, snap, Go and Docker do nothing on a machine without that tool, so one list can run on every guest of a test matrix. After the first machine downloads a package, the others get it from disk.
03 Jobs in containers
GitLab: extra_hosts = ["pkgproxy:host-gateway"] in
config.toml, plus a mounted apt.conf.d file and
GOPROXY. GitHub: options: --add-host=pkgproxy:host-gateway,
then curl -fsSL http://pkgproxy:3142/setup/apt | sh.
A job that runs on the runner machine itself uses whatever that machine is set up for, so you run the setup scripts there once. A job in a container, as GitLab's docker executor runs every job, sees none of that. Give every job container a name for the machine that runs pkgproxy, through Docker's host-gateway, and let the containers' networks reach port 3142. The GitLab runner can then mount an apt configuration into every job, and a GitHub job can run the setup page's script in its first step.
04 A VM with no route to the host
-socket-path replaces the TCP listener with a UNIX socket
(mode 0777), for QEMU's guestfwd; -addr still
names the host:port the instructions print.
-lxd-tls-socket-path does the same for -lxd-tls-addr.
A virtual machine does not need a network route to the host. With the -socket-path flag, pkgproxy listens on a UNIX socket instead of TCP, and QEMU's guest forwarding can connect an address inside the guest to that socket. The guest uses that address like any other. The -addr flag still decides what the setup instructions print, so set it to the address the guests use. The -lxd-tls-socket-path does the same for the HTTPS port.
05 Sharing the snap's cache with another snap
The snap runs with -content-socket-path and
-lxd-tls-content-socket-path under
$SNAP_COMMON/sockets, shared by the sockets slot
(content pkgproxy-sockets). The TCP listeners stay as they are. snapd auto-connects a plug with the same content from the same publisher; sideloaded (--dangerous) snaps need snap connect.
The pkgproxy snap can share its cache with another snap. Besides its TCP ports, it always listens on two UNIX sockets in a sockets directory, one for the plain port and one for the HTTPS port, and offers that directory through a content interface. A snap that plugs it, such as image-garden, can bridge those sockets into its own virtual machines instead of running a second pkgproxy with a second cache. When both snaps come from the store from the same publisher, snapd connects them on its own: the base declaration allows it when the publishers and the content names match. A sideloaded snap has no publisher, so it is connected by hand.
06 Docker pulls and cloud images
setup docker edits registry-mirrors in
daemon.json (Docker Hub, anonymous pulls).
/files/v1/fetch?url=…&sha256=… caches by checksum, over
http or https; files-allow-host,
files-allow-network and files-max-size (8G)
limit it.
Two sides matter most to CI. Setup docker puts pkgproxy first in the Docker daemon's registry-mirrors, so pulls of public Docker Hub images are cached, and if pkgproxy is down Docker goes to Docker Hub directly. Large files with no package format, such as the cloud images a VM test boots, are fetched by checksum: the client names the URL and its SHA-256, and the checksum is the cache key. The files-allow-host setting limits where pkgproxy may fetch from.
07 Offline for a repeatable run
snap set pkgproxy offline-mode=offline or
pkgproxyctl offline on; neither restarts pkgproxy.
store-overloaded backs off from a troubled upstream.
auto-offline=off on an isolated network.
For a run that must see exactly what was cached, switch pkgproxy to offline mode. It then serves only from the cache and contacts no upstream. Package lists are not refreshed, and anything not cached is refused at once with a 503. store-overloaded is the softer mode: an upstream that fails or asks to slow down is left alone for a while. On a network that only reaches a local mirror, turn auto-offline off, or pkgproxy takes the missing internet route for no network.
08 Where to go next general version
Next: The setup page, for the scripts every machine fetches; Hosted CI runners, for jobs on the forge's runners; Offline mode, for runs that see only what is cached.
Your runners, virtual machines and build containers now share one cache, so a package that one run downloads is on disk for the next. To see the scripts every machine fetches, watch The setup page. To run pkgproxy inside jobs on GitHub's and GitLab.com's own runners, watch Hosted CI runners. To make runs that see only what is cached, watch Offline mode.
Read more
Jobs on the runner machine
A job that runs on the machine itself, as a GitHub self-hosted runner does by default and GitLab's shell executor does, uses whatever the machine is set up to use. Set each tool up once:
pkgproxyctl setup apt | sudo sh pkgproxyctl setup snap | sudo sh pkgproxyctl setup docker | sudo sh sudo -u <runner user> sh -c 'pkgproxyctl setup go | sh'
setup go sets up the user who runs it, so run it as the user the runner runs its jobs as. On a machine that does not run pkgproxy itself, take the scripts from the one that does: curl -fsSL 'http://<host>:3142/setup/apt' | sudo sh.
Jobs in containers
A job container sees none of the runner machine's settings. Give it the host name pkgproxy for the machine, through Docker's host-gateway, and let the containers' networks reach port 3142 through the machine's firewall, for example sudo ufw allow from 172.16.0.0/12 to any port 3142 proto tcp.
GitLab's docker executor, in the runner's section of /etc/gitlab-runner/config.toml:
[[runners]]
environment = ["GOPROXY=http://pkgproxy:3142/go,direct"]
[runners.docker]
extra_hosts = ["pkgproxy:host-gateway"]
volumes = ["/cache", "/etc/gitlab-runner/pkgproxy-apt.conf:/etc/apt/apt.conf.d/99pkgproxy:ro"]
with the apt configuration every job container mounts:
echo 'Acquire::http::Proxy "http://pkgproxy:3142";' | sudo tee /etc/gitlab-runner/pkgproxy-apt.conf echo 'Acquire::Retries::Delay "false";' | sudo tee -a /etc/gitlab-runner/pkgproxy-apt.conf
The second line is for job images with apt older than 3.1.3, such as Ubuntu 24.04's. A GitHub job in a container:
jobs:
build:
runs-on: self-hosted
container:
image: ubuntu:24.04
options: --add-host=pkgproxy:host-gateway
steps:
- run: |
apt-get update && apt-get install -y curl
curl -fsSL http://pkgproxy:3142/setup/apt | sh
A tool that fetches large files by checksum gets http://pkgproxy:3142 in the job's environment; image-garden reads it from GARDEN_DL_PKGPROXY.
Checking that jobs use the cache
Run the same job twice, then run pkgproxyctl stats: every download of the second run counts as a hit. The access log says the same per request (outcome=hit), and so does the cache page. A job that does not show up there does not use pkgproxy.
In the documentation
- Runners you host: in pkgproxy in CI
- Jobs in containers
- Checking that jobs use the cache
Roads from here
CORE · the path every side shares
- One listener — TCP or a UNIX socket
- Offline mode — runs that see only what is cached
DOCKER
- Docker Hub registry mirror — Docker Hub pulls in CI
FILES
- Files by checksum — cloud images and other large files
SETUP AND THE COMMAND LINE
- The setup page — the scripts every machine fetches
OPERATIONS
- Hosted CI runners — jobs on GitHub and GitLab.com runners