pkgproxy in CI

CI jobs download the same packages, modules, images and files on every run. pkgproxy serves every download after the first from its disk, so what a job spends on downloading depends on how close the job is to a pkgproxy that already has them. How you set that up depends on who runs the machines the jobs run on:

On runners you run yourself, set pkgproxy up once and every job after that uses it. On runners the forge runs, a job cannot reach a pkgproxy on your network, and pkgproxy should not be put on the internet for it to (see below); pkgproxy runs in the job instead.

The pkgproxy map shows both in short narrated slides: pkgproxy for CI for runners you host, and Hosted CI runners for the forge’s.

Runners you host

Install pkgproxy on the runner machine, as README.md shows, or use one that already runs on the machine’s network. pkgproxy keeps everything it downloads and never deletes anything on its own, so give its cache room to grow; see Disk space. On a busy runner, a disk of its own for the cache keeps its reads and writes from slowing the jobs’ builds down: see Where the cache lives for the directory to mount it at.

Jobs on the runner machine

A job that runs on the machine itself, as a GitHub self-hosted runner does by default and GitLab’s shell executor does, uses whatever the machine is set up to use. Set each tool up once, with the commands that set up any other machine:

pkgproxyctl setup apt | sudo sh
pkgproxyctl setup snap | sudo sh
pkgproxyctl setup docker | sudo sh

setup go sets up the user who runs it, so run it as the user the runner runs its jobs as:

sudo -u <runner user> sh -c 'pkgproxyctl setup go | sh'

On a machine that does not run pkgproxy itself, take the same scripts from the pkgproxy that does, as on any other machine:

curl -fsSL 'http://<host>:3142/setup/apt' | sudo sh

Jobs in containers

A job that runs in a container, as GitLab’s docker executor runs every job and GitHub runs a job with container:, sees none of the runner machine’s settings. Give every job container a name for the machine that runs pkgproxy, and set its tools up from there.

Docker’s host-gateway is the address at which a container reaches the machine it runs on. pkgproxy listens on every address of that machine by default, the Docker bridge included. If the machine has a firewall, let the containers’ networks reach port 3142, for example with ufw:

sudo ufw allow from 172.16.0.0/12 to any port 3142 proto tcp

For GitLab’s docker executor, add to the runner’s section of /etc/gitlab-runner/config.toml:

[[runners]]
  environment = ["GOPROXY=http://pkgproxy:3142/go,direct"]
  [runners.docker]
    extra_hosts = ["pkgproxy:host-gateway"]
    volumes = ["/cache", "/etc/gitlab-runner/pkgproxy-apt.conf:/etc/apt/apt.conf.d/99pkgproxy:ro"]

and write the apt configuration that pkgproxyctl setup apt would write, for every job container to mount:

echo 'Acquire::http::Proxy "http://pkgproxy:3142";' | sudo tee /etc/gitlab-runner/pkgproxy-apt.conf

Job images with apt older than 3.1.3, such as Ubuntu 24.04’s, also need the line setup apt adds for them, so that apt does not wait between retries while pkgproxy is offline:

echo 'Acquire::Retries::Delay "false";' | sudo tee -a /etc/gitlab-runner/pkgproxy-apt.conf

gitlab-runner reads config.toml again on its own. Every job container then has pkgproxy as a host name, apt and go download through it, and /cache, GitLab’s own cache, is not needed for anything pkgproxy keeps.

For a GitHub job in a container, give the container the same name, and set its tools up in the job’s first step from the setup page’s scripts. The job runs as root in its container, so no sudo is needed:

jobs:
  build:
    runs-on: self-hosted
    container:
      image: ubuntu:24.04
      options: --add-host=pkgproxy:host-gateway
    steps:
      - run: |
          apt-get update && apt-get install -y curl
          curl -fsSL http://pkgproxy:3142/setup/apt | sh

apt-get install curl itself already goes straight to the archive; the apt-get commands after the script go through pkgproxy. A GitLab job can use the same scripts, in before_script, instead of the mounted file.

A tool that fetches large files by checksum through pkgproxy is pointed at it the same way, with the address http://pkgproxy:3142 in the job’s environment; image-garden, for one, reads it from GARDEN_DL_PKGPROXY. See Caching large files.

Docker pulls on the runner

The images jobs run in are pulled by the runner machine’s Docker, not by the jobs. Make pkgproxy that Docker’s registry mirror, on the runner machine, to pull them from the cache too:

pkgproxyctl setup docker | sudo sh

It restarts Docker, so do it while no job runs. Only anonymous pulls of public Docker Hub images are cached.

Checking that jobs use the cache

Run the same job twice, then look at what pkgproxy served:

pkgproxyctl stats

Every download the second run made counts as a hit. The access log says the same per request (outcome=hit), and so does the cache page at http://localhost:3142/pkgproxy/cache on the machine that runs pkgproxy, or from your own machine once its network is trusted (see Showing the cache to your network). A job that does not show up there does not use pkgproxy: see docs/troubleshooting.md.

Runners the forge hosts

GitHub-hosted runners and GitLab.com’s hosted runners are new virtual machines for every job, on the forge’s network. They cannot reach a pkgproxy on yours, and nothing they download is there for the next job.

Why not a public pkgproxy

Putting a pkgproxy on the internet for hosted runners to reach makes it a proxy for anyone who finds it:

• pkgproxy has no client authentication: every client that reaches it may use it.
• As apt’s proxy it passes plain http:// requests to any host but its own machine, and tunnels CONNECT to port 443 of any of them.
• Files by checksum download any public URL anyone names, up to files-max-size each, and the cache never deletes anything: anyone can fill its disk.

Reach a pkgproxy on your network from hosted runners only over a VPN that lets in your jobs and no one else, or run pkgproxy in the job.

Running pkgproxy in the job

pkgproxy is one program that needs no configuration and no root, so a job can start it, point its tools at 127.0.0.1, and keep its cache in the forge’s CI cache between jobs: the next job restores the cache and starts with everything the last one downloaded.

Whether that pays depends on what the jobs download. The forge’s CI cache restores the whole cache at the start of every job and saves it again when it has changed, so it is faster only when that restore is faster than the downloads it replaces: many small requests, such as apt’s and Go’s, to slow or rate-limited servers. It also gives repeated runs the same inputs: with offline mode on, a job sees only what is in the restored cache.

pkgproxy keeps each cached file’s metadata in an extended attribute of the file (see Where the cache lives). A CI cache that archives the directory itself may not keep extended attributes, and a file that lost its attribute is a miss: everything is downloaded again, and the cache never helps. Archive the cache yourself with a tool that keeps them, such as tar --xattrs, and let the CI cache keep that one file. Stop pkgproxy first, so that nothing is half written. A job whose cache directory is on a file system without extended attributes fails to start pkgproxy, and its log says why.

GitHub-hosted runners

setup-pkgproxy is a GitHub Action that does this: it installs pkgproxy on the runner, starts it as a service, restores its cache with actions/cache, and sets up apt and snapd. Its README says how to set up the other tools. It builds pkgproxy 0.11 with go install, keeping the binary in the CI cache, and saves pkgproxy’s cache as an archive made with tar --xattrs, as above, so the cache already survives the move to 0.12. Two major versions are maintained:

• zyga/setup-pkgproxy@v2 saves the cache itself at the end of the job, whether the job passed, failed or was cancelled, so using it takes one step. By default only runs on the repository’s default branch save it (save-package-cache): every run can restore the default branch’s cache, but a pull request’s helps only that pull request, and takes room in the repository’s cache storage. A workflow that runs only on pull_request needs a push trigger for the default branch, or nothing is ever saved. A job that added nothing to the cache saves nothing.
• zyga/setup-pkgproxy@v1 leaves the save to the workflow: a step that stops pkgproxy, which writes the archive, and an actions/cache/save step. From v1.2.0 on it saves the same archive as v2, under the same key prefix, so a workflow that moves to v2 deletes those two steps and keeps its warm cache.

Installing a released binary instead of building pkgproxy on every cache miss is still to come (#127).

GitLab.com’s hosted runners

A GitLab job runs in a container, without a service manager: start pkgproxy in the background, with its cache in the project directory, and keep its archive in GitLab’s cache, which keeps paths under the project directory. pkgproxy’s sockets stay in /run/pkgproxy, which also keeps their paths short (a UNIX socket’s path may not be much longer than 100 bytes, and $CI_PROJECT_DIR can be long). after_script runs whether the job passed or failed, so a failed job still saves what it downloaded:

variables:
  PKGPROXY_CACHE_DIR: $CI_PROJECT_DIR/.pkgproxy
  PKGPROXY_ADDR: 127.0.0.1:3142
  PKGPROXY_LXD_TLS_ADDR: "off"
build:
  image: golang:1.25
  cache:
    key: pkgproxy
    paths: [pkgproxy-cache.tar]
    when: always
  before_script:
    - go install gitlab.com/zygoon/pkgproxy/cmd/pkgproxy@latest
    - ln -s pkgproxy "$(go env GOPATH)/bin/pkgproxyctl"
    - if [ -f pkgproxy-cache.tar ]; then tar --xattrs --xattrs-include='user.*' -xf pkgproxy-cache.tar; fi
    - pkgproxy > pkgproxy.log 2>&1 & echo $! > pkgproxy.pid
    - until curl -fs -o /dev/null http://127.0.0.1:3142/; do sleep 1; done
    - pkgproxyctl setup apt -addr 127.0.0.1:3142 | sh
    - pkgproxyctl setup go -addr 127.0.0.1:3142 | sh
  script:
    - ...
  after_script:
    - pid=$(cat pkgproxy.pid); kill "$pid"; while kill -0 "$pid" 2>/dev/null; do sleep 1; done
    - tar --xattrs --xattrs-include='user.*' -cf pkgproxy-cache.tar .pkgproxy

Every setting is read from its PKGPROXY_ variable, as Settings shows. go install takes a Go toolchain and the time to build pkgproxy. From 0.12.0 on, every release also publishes pkgproxy itself, which a job can download and check instead, as Running pkgproxy without the snap shows. Keep pkgproxy.log as an artifact to see what each request was served from.