Hosted CI runners
operations for CI operators
Running pkgproxy inside jobs on GitHub's and GitLab.com's hosted runners, its cache kept in the CI cache with its extended attributes, and the setup-pkgproxy action, v2 and v1, that does it on GitHub.
On the map: operations
01 What this topic covers
On runners the forge hosts, pkgproxy runs inside the job and its cache lives in the CI cache between jobs; setup-pkgproxy does that on GitHub.
Hosted CI runners are the machines GitHub and GitLab.com run your jobs on: a new virtual machine on GitHub, or a new container on GitLab.com, for every job. pkgproxy runs in the job, and its cache is kept in the CI cache between jobs. First comes the setup-pkgproxy action, which does all of that on GitHub, v2 and then v1. Then the same steps by hand: running pkgproxy in the job, saving its cache with its metadata, and doing it on GitLab.com. Last, why pkgproxy should not be put on the internet instead.
02 setup-pkgproxy v2 on GitHub
uses: zyga/setup-pkgproxy@v2. save-package-cache
is default-branch, always or never;
a job that added nothing saves nothing. Set up other tools with
pkgproxyctl setup TOOL -addr "${{ steps.pkgproxy.outputs.addr }}".
On GitHub's hosted runners, the setup-pkgproxy action runs pkgproxy in the job, and v2 takes one step. It installs pkgproxy, starts it as a service, restores the newest cache and points apt and snapd at it. At the end of the job, passed, failed or cancelled, it stops pkgproxy and saves the cache with its extended attributes. By default only runs on the default branch save the cache, because every run can restore that one. A workflow that runs only for pull requests never saves it, so run the workflow on pushes to the default branch too.
03 setup-pkgproxy v1, and moving to v2
v1 workflows end with a step that stops pkgproxy.service
and a step with actions/cache/save. Caches saved by v1.0
and v1.1 were a plain directory, so the first run after moving from them
starts cold.
v1 of setup-pkgproxy starts pkgproxy the same way, but leaves saving the cache to the workflow. The workflow adds two steps at the end: one stops pkgproxy, which writes the archive of the cache, and the other saves that archive to the actions cache, on whatever branch the workflow runs. From v1.2 on, the archive and its key are the same as in v2. So to move to v2, delete those two steps, and the cache stays warm.
04 pkgproxy in the job
The job starts pkgproxy on 127.0.0.1:3142 and runs
pkgproxyctl setup against it. The CI cache restores the
whole cache at the start of each job and saves it when it changed.
Without the action, a job does the same by hand. pkgproxy is one program that needs no configuration and no root, so a job can start it, point its tools at its own machine, and keep the cache in the forge's CI cache. The next job restores that cache and starts with everything the last one downloaded. This pays only when restoring the whole cache is faster than the downloads it replaces, as with many small requests to slow or rate limited servers, which apt and Go make. With store-overloaded mode on, a mirror or store that fails, times out or asks to slow down is left alone for a while, and pkgproxy serves the job what it has cached meanwhile.
05 Saving the cache with its metadata
Archive with tar --xattrs --xattrs-include='user.*' after
pkgproxy has stopped, and unpack the same way before it starts. A file
that lost its attribute reads its .meta.json file, kept for
going back to 0.11; one that lost both is a miss, and pkgproxy says so
once in its log.
pkgproxy keeps what it knows of each cached file in an extended attribute of the file, and a copy in a .meta.json file beside it. A CI cache that archives the directory itself may drop extended attributes. The files still read through their .meta.json files, but only the attribute is replaced together with its file. So stop pkgproxy first, so that nothing is half written, archive the cache with tar, keeping extended attributes, and let the CI cache keep that one file. On a file system without extended attributes, pkgproxy does not start.
06 GitLab.com's hosted runners
PKGPROXY_CACHE_DIR under $CI_PROJECT_DIR,
cache: when: always, and after_script to save
even a failed job's downloads. Keep pkgproxy.log as an
artifact.
On GitLab.com's hosted runners, a job runs in a container with no service manager, so it starts pkgproxy itself, in the background, with its cache in the project directory, where GitLab's cache can keep the archive between jobs. Before the script, the job unpacks the archive, starts pkgproxy, waits until it answers and sets up apt and Go. After the script, which runs whether the job passed or failed, it stops pkgproxy and archives the cache again.
07 Why not a public pkgproxy
A public pkgproxy is an open proxy: no client authentication, plain
http:// to any host but its own and CONNECT to
port 443 of any of them, and files by checksum up to files-max-size each,
in a cache that deletes nothing. Run pkgproxy in the job instead.
A hosted runner gives every job a new machine: a virtual machine on GitHub, a container on GitLab.com. Nothing a job downloads is there for the next one. You could put a pkgproxy on the internet for all of them, but then it is a proxy for anyone who finds it: it has no client authentication, its apt side passes requests on to any host, and anyone can name files to download by checksum until its disk is full. Run pkgproxy in the job instead.
08 Where to go next
Next: pkgproxy for CI, for runners you host; The store, for the cache on disk; Offline mode, for leaving a troubled mirror or store alone.
Your hosted jobs now start with what earlier jobs downloaded, instead of fetching all of it again. To compare this with runners you host yourself, watch pkgproxy for CI. To see where the cache and its metadata live on disk, watch The store. To see how store-overloaded mode leaves a troubled mirror alone, watch Offline mode.
Read more
setup-pkgproxy on GitHub-hosted runners
setup-pkgproxy is a GitHub Action that runs pkgproxy in the job. Both major versions:
- restore a cached pkgproxy binary, or build it with
go installon a miss, and install it as/usr/local/bin/pkgproxywith apkgproxyctllink; - restore the package cache to
/var/cache/pkgproxy, and startpkgproxy.service; - point the runner's apt and snapd at it, unless
configure-hostisfalse; - save the cache as an archive made with
tar --xattrs(from v1.2.0 on), which the unit writes to/var/cache/pkgproxy-archive/cache.tarwhen pkgproxy stops.
v2 (zyga/setup-pkgproxy@v2) | v1 (zyga/setup-pkgproxy@v1) | |
|---|---|---|
| saving the cache | the action's post step, at the end of the job, passed, failed or cancelled | the workflow's own steps: stop pkgproxy.service, hand the archive to the runner user, actions/cache/save |
| which runs save | save-package-cache: default-branch (default), always or never | wherever the workflow's save step runs, pull requests included |
| cache keys | a new key per saving job, <package-cache-key>-<arch>-<run>-…; the next job restores the newest | one key per CPU arch, <package-cache-key>-<arch>-success or -partial |
| a job that added nothing | saves nothing | the save step skips an exact hit (package-cache-hit) |
| pkgproxy | 0.12.0 | 0.12.0 |
v2
on:
push:
branches: [main] # only the default branch saves the cache
pull_request:
# ...
steps:
- uses: zyga/setup-pkgproxy@v2
id: pkgproxy
- run: pkgproxyctl setup go -addr "${{ steps.pkgproxy.outputs.addr }}" | sh
| Input | Default | |
|---|---|---|
version | v0.12.0 | the pkgproxy version to build |
go-version | 1.27.1 | the Go that builds it on a binary-cache miss |
addr | :3142 | the address pkgproxy listens on |
binary-cache-key | pkgproxy-bin | key prefix of the built binary |
package-cache-key | pkgproxy-cache-v4 | key prefix of the package cache |
save-package-cache | default-branch | which runs save the package cache |
configure-host | true | run pkgproxyctl setup apt and setup snap on the runner |
| Output | |
|---|---|
addr | the address to pass to pkgproxyctl setup … -addr |
cache-dir | /var/cache/pkgproxy |
package-cache-restored-key | the key the cache was restored from, empty on a cold start |
package-cache-key | the key the post step saves under, empty when this run may not save |
Every run can restore the caches of its own ref, of its pull request's base branch and of the default branch. A cache a pull request saved helps only that pull request, and takes room the default branch's needs, so v2 saves on the default branch only by default. A workflow that runs only for pull requests then never saves the cache, so run it on pushes to the default branch too, as in the example above. In a matrix with several jobs on one CPU arch, give all but one save-package-cache: never.
v1, and moving to v2
- uses: zyga/setup-pkgproxy@v1
id: pkgproxy
# ... the job's own steps
- if: always()
run: |
sudo systemctl stop pkgproxy.service || true
sudo chown -R "$USER:$USER" ${{ steps.pkgproxy.outputs.cache-dir }}
- if: steps.pkgproxy.outputs.package-cache-hit != 'true'
uses: actions/cache/save@v6
with:
path: ${{ steps.pkgproxy.outputs.cache-dir }}
key: ${{ steps.pkgproxy.outputs.package-cache-key-success }}
In v1, cache-dir is /var/cache/pkgproxy-archive, and the outputs package-cache-hit, package-cache-key-success and package-cache-key-partial drive the save. To move to v2, change @v1 to @v2, delete the two save steps, and read the outputs v2 has instead. v1 stays maintained, on the action's releases/v1 branch. v1.2 saves the same archive under the same pkgproxy-cache-v4 prefix, so its cache warms v2; caches saved by v1.0 and v1.1 were the plain directory, and the first run after them starts cold.
Keeping the cache between jobs
pkgproxy keeps each cached file's metadata in the user.pkgproxy.meta extended attribute of the file, which is replaced together with the file, and a copy in a .meta.json file beside it, for going back to 0.11. A CI cache that archives the directory itself may not keep extended attributes: a file that lost its attribute then reads its .meta.json file, and one that lost both is a miss, downloaded again, and pkgproxy logs once that it found such a file. Archive the cache yourself, keeping the attributes, and let the CI cache keep that one file:
# stop pkgproxy first, so that nothing is half written tar --xattrs --xattrs-include='user.*' -cf pkgproxy-cache.tar .pkgproxy # before the next start tar --xattrs --xattrs-include='user.*' -xf pkgproxy-cache.tar
A cache directory on a file system without extended attributes keeps pkgproxy from starting, and its log says why.
GitLab.com's hosted runners
A GitLab job runs in a container without a service manager: start pkgproxy in the background, with its cache in the project directory, and keep its archive in GitLab's cache. Its sockets stay in /run/pkgproxy, which keeps their paths short.
variables:
PKGPROXY_CACHE_DIR: $CI_PROJECT_DIR/.pkgproxy
PKGPROXY_ADDR: 127.0.0.1:3142
PKGPROXY_LXD_TLS_ADDR: "off"
build:
image: golang:1.25
cache:
key: pkgproxy
paths: [pkgproxy-cache.tar]
when: always
before_script:
- go install gitlab.com/zygoon/pkgproxy/cmd/pkgproxy@latest
- ln -s pkgproxy "$(go env GOPATH)/bin/pkgproxyctl"
- if [ -f pkgproxy-cache.tar ]; then tar --xattrs --xattrs-include='user.*' -xf pkgproxy-cache.tar; fi
- pkgproxy > pkgproxy.log 2>&1 & echo $! > pkgproxy.pid
- until curl -fs -o /dev/null http://127.0.0.1:3142/; do sleep 1; done
- pkgproxyctl setup apt -addr 127.0.0.1:3142 | sh
- pkgproxyctl setup go -addr 127.0.0.1:3142 | sh
script:
- ...
after_script:
- pid=$(cat pkgproxy.pid); kill "$pid"; while kill -0 "$pid" 2>/dev/null; do sleep 1; done
- tar --xattrs --xattrs-include='user.*' -cf pkgproxy-cache.tar .pkgproxy
Every setting is read from its PKGPROXY_ variable. From 0.12.0 on, a job can download a released pkgproxy and check it instead of building it with go install. Keep pkgproxy.log as an artifact to see what each request was served from.
Why not a public pkgproxy
| clients | no authentication: every client that reaches pkgproxy may use it |
| apt's side | passes plain http:// requests to any host but pkgproxy's own machine, and tunnels CONNECT to port 443 of any of them |
| files by checksum | downloads any public URL anyone names, up to files-max-size each |
| the cache | never deletes anything, so anyone can fill its disk |
Run pkgproxy in the job instead.
In the documentation
- Runners the forge hosts, in pkgproxy in CI
- Where the cache lives and Running pkgproxy without the snap
- setup-pkgproxy: its README, with every tool's setup step, and its releases
Roads from here
CORE · the path every side shares
- The store — where the cache and its metadata live
- Offline mode — leaving a troubled mirror or store alone
SETUP AND THE COMMAND LINE
- Setting up clients — pointing the job's other tools at it
OPERATIONS
- pkgproxy for CI — runners you host yourself