Hosted CI runners

operations for CI operators

Running pkgproxy inside jobs on GitHub's and GitLab.com's hosted runners, its cache kept in the CI cache with its extended attributes, and the setup-pkgproxy action, v2 and v1, that does it on GitHub.

On the map: operations

▶ Watch this deck — 8 scenes, ~6 min

01 What this topic covers

ABOUT THIS TOPIC Hosted CI runners a new VM or container for every job: pkgproxy runs in the job 1 setup-pkgproxy v2 on GitHub 2 setup-pkgproxy v1, and moving to v2 3 pkgproxy in the job 4 Saving the cache with its metadata 5 GitLab.com's hosted runners 6 Why not a public pkgproxy
The topic title, Hosted CI runners, and its six scenes in order: setup-pkgproxy v2 on GitHub, setup-pkgproxy v1 and moving to v2, pkgproxy in the job, saving the cache with its metadata, GitLab.com's hosted runners, why not a public pkgproxy

On runners the forge hosts, pkgproxy runs inside the job and its cache lives in the CI cache between jobs; setup-pkgproxy does that on GitHub.

Hosted CI runners are the machines GitHub and GitLab.com run your jobs on: a new virtual machine on GitHub, or a new container on GitLab.com, for every job. pkgproxy runs in the job, and its cache is kept in the CI cache between jobs. First comes the setup-pkgproxy action, which does all of that on GitHub, v2 and then v1. Then the same steps by hand: running pkgproxy in the job, saving its cache with its metadata, and doing it on GitLab.com. Last, why pkgproxy should not be put on the internet instead.

02 setup-pkgproxy v2 on GitHub

SETUP-PKGPROXY V2 on: push: branches: [main] pull_request: # … - uses: zyga/setup-pkgproxy@v2 id: pkgproxy at the start install pkgproxy, building it on a binary-cache miss restore the newest cache, start pkgproxy.service point apt and snapd at it at the end: passed, failed or cancelled stop pkgproxy, save the cache with tar --xattrs default branch: restores and saves pull request: restores, saves nothing a workflow that runs only for pull requests never saves the cache: also run it on pushes to the default branch, as above
A GitHub workflow that runs on pushes to main and on pull requests, with one step using zyga/setup-pkgproxy@v2; the main step installs pkgproxy, restores the newest cache, starts the service and sets up apt and snapd; the post step stops pkgproxy and saves the cache on the default branch only

uses: zyga/setup-pkgproxy@v2. save-package-cache is default-branch, always or never; a job that added nothing saves nothing. Set up other tools with pkgproxyctl setup TOOL -addr "${{ steps.pkgproxy.outputs.addr }}".

On GitHub's hosted runners, the setup-pkgproxy action runs pkgproxy in the job, and v2 takes one step. It installs pkgproxy, starts it as a service, restores the newest cache and points apt and snapd at it. At the end of the job, passed, failed or cancelled, it stops pkgproxy and saves the cache with its extended attributes. By default only runs on the default branch save the cache, because every run can restore that one. A workflow that runs only for pull requests never saves it, so run the workflow on pushes to the default branch too.

03 setup-pkgproxy v1, and moving to v2

SETUP-PKGPROXY V1 - uses: zyga/setup-pkgproxy@v1 # the job's own steps - run: sudo systemctl stop pkgproxy.service writes the archive - uses: actions/cache/save@v6 on whatever branch runs it from v1.2: the same archive and key prefix as v2, pkgproxy-cache-v4 to v2: change @v1 to @v2 and delete the two save steps; the cache stays warm
A v1 workflow: the setup-pkgproxy@v1 step, the job's own steps, then a step that stops pkgproxy, which writes the archive, and an actions/cache/save step; from v1.2 the archive and key match v2, so moving to v2 means deleting the last two steps

v1 workflows end with a step that stops pkgproxy.service and a step with actions/cache/save. Caches saved by v1.0 and v1.1 were a plain directory, so the first run after moving from them starts cold.

v1 of setup-pkgproxy starts pkgproxy the same way, but leaves saving the cache to the workflow. The workflow adds two steps at the end: one stops pkgproxy, which writes the archive of the cache, and the other saves that archive to the actions cache, on whatever branch the workflow runs. From v1.2 on, the archive and its key are the same as in v2. So to move to v2, delete those two steps, and the cache stays warm.

04 pkgproxy in the job

IN THE JOB one job's VM or container apt, go, snap pkgproxy 127.0.0.1:3142 its cache the CI cache restored at the start saved at the end pays when restoring the whole cache is faster than the downloads it replaces: many small requests to slow or rate-limited servers, as apt and Go make store-overloaded: a troubled mirror or store is left alone, the cache is served
Inside one job's VM or container, the job's tools use pkgproxy on 127.0.0.1:3142 and its cache directory; the forge's CI cache restores the cache at the start of the job and saves it at the end; it pays when restoring is faster than many small downloads; with the store-overloaded mode, a troubled mirror or store is left alone for a while and the cache is served meanwhile

The job starts pkgproxy on 127.0.0.1:3142 and runs pkgproxyctl setup against it. The CI cache restores the whole cache at the start of each job and saves it when it changed.

Without the action, a job does the same by hand. pkgproxy is one program that needs no configuration and no root, so a job can start it, point its tools at its own machine, and keep the cache in the forge's CI cache. The next job restores that cache and starts with everything the last one downloaded. This pays only when restoring the whole cache is faster than the downloads it replaces, as with many small requests to slow or rate limited servers, which apt and Go make. With store-overloaded mode on, a mirror or store that fails, times out or asks to slow down is left alone for a while, and pkgproxy serves the job what it has cached meanwhile.

05 Saving the cache with its metadata

KEEP THE METADATA objects/…/hello.deb xattr user.pkgproxy.meta hello.deb.meta.json CI cache archive hello.deb no attribute hello.deb.meta.json still read, from .meta.json not switched with the file $ kill "$pid"; while kill -0 "$pid" 2>/dev/null; do sleep 1; done $ tar --xattrs --xattrs-include='user.*' -cf pkgproxy-cache.tar .pkgproxy # the CI cache keeps pkgproxy-cache.tar, one file a cache directory on a file system without extended attributes: pkgproxy does not start, and its log says why
A cached file carries its metadata in the extended attribute user.pkgproxy.meta, with a copy in hello.deb.meta.json beside it; a CI cache archive that drops the attribute brings the file back read from its .meta.json, which is not switched together with the file; stopping pkgproxy and archiving with tar --xattrs into one file keeps it

Archive with tar --xattrs --xattrs-include='user.*' after pkgproxy has stopped, and unpack the same way before it starts. A file that lost its attribute reads its .meta.json file, kept for going back to 0.11; one that lost both is a miss, and pkgproxy says so once in its log.

pkgproxy keeps what it knows of each cached file in an extended attribute of the file, and a copy in a .meta.json file beside it. A CI cache that archives the directory itself may drop extended attributes. The files still read through their .meta.json files, but only the attribute is replaced together with its file. So stop pkgproxy first, so that nothing is half written, archive the cache with tar, keeping extended attributes, and let the CI cache keep that one file. On a file system without extended attributes, pkgproxy does not start.

06 GitLab.com's hosted runners

GITLAB.COM variables: { PKGPROXY_CACHE_DIR: $CI_PROJECT_DIR/.pkgproxy, PKGPROXY_ADDR: 127.0.0.1:3142 } cache: { key: pkgproxy, paths: [pkgproxy-cache.tar], when: always } # no service manager in the job's container: start pkgproxy yourself before_script: - tar --xattrs --xattrs-include='user.*' -xf pkgproxy-cache.tar - pkgproxy > pkgproxy.log 2>&1 & echo $! > pkgproxy.pid - until curl -fs -o /dev/null http://127.0.0.1:3142/; do sleep 1; done - pkgproxyctl setup apt -addr 127.0.0.1:3142 | sh # and go after_script: # runs whether the job passed or failed - pid=$(cat pkgproxy.pid); kill "$pid"; while kill -0 "$pid" 2>/dev/null; do sleep 1; done - tar --xattrs --xattrs-include='user.*' -cf pkgproxy-cache.tar .pkgproxy
A GitLab CI job for GitLab.com's hosted runners: the cache directory in the project, the archive in GitLab's cache saved always; before_script unpacks the archive, starts pkgproxy in the background, waits for it and sets up apt and Go; after_script stops pkgproxy and archives the cache again

PKGPROXY_CACHE_DIR under $CI_PROJECT_DIR, cache: when: always, and after_script to save even a failed job's downloads. Keep pkgproxy.log as an artifact.

On GitLab.com's hosted runners, a job runs in a container with no service manager, so it starts pkgproxy itself, in the background, with its cache in the project directory, where GitLab's cache can keep the archive between jobs. Before the script, the job unpacks the archive, starts pkgproxy, waits until it answers and sets up apt and Go. After the script, which runs whether the job passed or failed, it stops pkgproxy and archives the cache again.

07 Why not a public pkgproxy

NOT A PUBLIC PKGPROXY the forge's runners job 1 job 2 a new VM on GitHub, a new container on GitLab.com nothing a job downloads is kept for the next one pkgproxy on the internet a proxy for anyone who finds it no client authentication apt: any host, and CONNECT to port 443 of any host files by checksum: any public URL, in a cache that deletes nothing instead: pkgproxy in the job
Every hosted job starts on a new VM on GitHub, or a new container on GitLab.com, and keeps nothing for the next job; a pkgproxy put on the internet for them serves anyone, with no client authentication, apt requests to any host and files that fill its disk; run pkgproxy in the job instead

A public pkgproxy is an open proxy: no client authentication, plain http:// to any host but its own and CONNECT to port 443 of any of them, and files by checksum up to files-max-size each, in a cache that deletes nothing. Run pkgproxy in the job instead.

A hosted runner gives every job a new machine: a virtual machine on GitHub, a container on GitLab.com. Nothing a job downloads is there for the next one. You could put a pkgproxy on the internet for all of them, but then it is a proxy for anyone who finds it: it has no client authentication, its apt side passes requests on to any host, and anyone can name files to download by checksum until its disk is full. Run pkgproxy in the job instead.

08 Where to go next

WHERE NEXT your hosted jobs start with what earlier jobs downloaded, instead of fetching all of it again you are here Hosted CI runners pkgproxy for CI runners you host yourself The store the cache and its metadata on disk Offline mode jobs that see only the restored cache
You are here, Hosted CI runners; next: pkgproxy for CI, The store, Offline mode

Next: pkgproxy for CI, for runners you host; The store, for the cache on disk; Offline mode, for leaving a troubled mirror or store alone.

Your hosted jobs now start with what earlier jobs downloaded, instead of fetching all of it again. To compare this with runners you host yourself, watch pkgproxy for CI. To see where the cache and its metadata live on disk, watch The store. To see how store-overloaded mode leaves a troubled mirror alone, watch Offline mode.

Read more

setup-pkgproxy on GitHub-hosted runners

setup-pkgproxy is a GitHub Action that runs pkgproxy in the job. Both major versions:

  • restore a cached pkgproxy binary, or build it with go install on a miss, and install it as /usr/local/bin/pkgproxy with a pkgproxyctl link;
  • restore the package cache to /var/cache/pkgproxy, and start pkgproxy.service;
  • point the runner's apt and snapd at it, unless configure-host is false;
  • save the cache as an archive made with tar --xattrs (from v1.2.0 on), which the unit writes to /var/cache/pkgproxy-archive/cache.tar when pkgproxy stops.
v2 (zyga/setup-pkgproxy@v2)v1 (zyga/setup-pkgproxy@v1)
saving the cachethe action's post step, at the end of the job, passed, failed or cancelledthe workflow's own steps: stop pkgproxy.service, hand the archive to the runner user, actions/cache/save
which runs savesave-package-cache: default-branch (default), always or neverwherever the workflow's save step runs, pull requests included
cache keysa new key per saving job, <package-cache-key>-<arch>-<run>-…; the next job restores the newestone key per CPU arch, <package-cache-key>-<arch>-success or -partial
a job that added nothingsaves nothingthe save step skips an exact hit (package-cache-hit)
pkgproxy0.12.00.12.0

v2

on:
  push:
    branches: [main]   # only the default branch saves the cache
  pull_request:
# ...
    steps:
      - uses: zyga/setup-pkgproxy@v2
        id: pkgproxy
      - run: pkgproxyctl setup go -addr "${{ steps.pkgproxy.outputs.addr }}" | sh
InputDefault
versionv0.12.0the pkgproxy version to build
go-version1.27.1the Go that builds it on a binary-cache miss
addr:3142the address pkgproxy listens on
binary-cache-keypkgproxy-binkey prefix of the built binary
package-cache-keypkgproxy-cache-v4key prefix of the package cache
save-package-cachedefault-branchwhich runs save the package cache
configure-hosttruerun pkgproxyctl setup apt and setup snap on the runner
Output
addrthe address to pass to pkgproxyctl setup … -addr
cache-dir/var/cache/pkgproxy
package-cache-restored-keythe key the cache was restored from, empty on a cold start
package-cache-keythe key the post step saves under, empty when this run may not save

Every run can restore the caches of its own ref, of its pull request's base branch and of the default branch. A cache a pull request saved helps only that pull request, and takes room the default branch's needs, so v2 saves on the default branch only by default. A workflow that runs only for pull requests then never saves the cache, so run it on pushes to the default branch too, as in the example above. In a matrix with several jobs on one CPU arch, give all but one save-package-cache: never.

v1, and moving to v2

- uses: zyga/setup-pkgproxy@v1
  id: pkgproxy
# ... the job's own steps
- if: always()
  run: |
    sudo systemctl stop pkgproxy.service || true
    sudo chown -R "$USER:$USER" ${{ steps.pkgproxy.outputs.cache-dir }}
- if: steps.pkgproxy.outputs.package-cache-hit != 'true'
  uses: actions/cache/save@v6
  with:
    path: ${{ steps.pkgproxy.outputs.cache-dir }}
    key: ${{ steps.pkgproxy.outputs.package-cache-key-success }}

In v1, cache-dir is /var/cache/pkgproxy-archive, and the outputs package-cache-hit, package-cache-key-success and package-cache-key-partial drive the save. To move to v2, change @v1 to @v2, delete the two save steps, and read the outputs v2 has instead. v1 stays maintained, on the action's releases/v1 branch. v1.2 saves the same archive under the same pkgproxy-cache-v4 prefix, so its cache warms v2; caches saved by v1.0 and v1.1 were the plain directory, and the first run after them starts cold.

Keeping the cache between jobs

pkgproxy keeps each cached file's metadata in the user.pkgproxy.meta extended attribute of the file, which is replaced together with the file, and a copy in a .meta.json file beside it, for going back to 0.11. A CI cache that archives the directory itself may not keep extended attributes: a file that lost its attribute then reads its .meta.json file, and one that lost both is a miss, downloaded again, and pkgproxy logs once that it found such a file. Archive the cache yourself, keeping the attributes, and let the CI cache keep that one file:

# stop pkgproxy first, so that nothing is half written
tar --xattrs --xattrs-include='user.*' -cf pkgproxy-cache.tar .pkgproxy
# before the next start
tar --xattrs --xattrs-include='user.*' -xf pkgproxy-cache.tar

A cache directory on a file system without extended attributes keeps pkgproxy from starting, and its log says why.

GitLab.com's hosted runners

A GitLab job runs in a container without a service manager: start pkgproxy in the background, with its cache in the project directory, and keep its archive in GitLab's cache. Its sockets stay in /run/pkgproxy, which keeps their paths short.

variables:
  PKGPROXY_CACHE_DIR: $CI_PROJECT_DIR/.pkgproxy
  PKGPROXY_ADDR: 127.0.0.1:3142
  PKGPROXY_LXD_TLS_ADDR: "off"
build:
  image: golang:1.25
  cache:
    key: pkgproxy
    paths: [pkgproxy-cache.tar]
    when: always
  before_script:
    - go install gitlab.com/zygoon/pkgproxy/cmd/pkgproxy@latest
    - ln -s pkgproxy "$(go env GOPATH)/bin/pkgproxyctl"
    - if [ -f pkgproxy-cache.tar ]; then tar --xattrs --xattrs-include='user.*' -xf pkgproxy-cache.tar; fi
    - pkgproxy > pkgproxy.log 2>&1 & echo $! > pkgproxy.pid
    - until curl -fs -o /dev/null http://127.0.0.1:3142/; do sleep 1; done
    - pkgproxyctl setup apt -addr 127.0.0.1:3142 | sh
    - pkgproxyctl setup go -addr 127.0.0.1:3142 | sh
  script:
    - ...
  after_script:
    - pid=$(cat pkgproxy.pid); kill "$pid"; while kill -0 "$pid" 2>/dev/null; do sleep 1; done
    - tar --xattrs --xattrs-include='user.*' -cf pkgproxy-cache.tar .pkgproxy

Every setting is read from its PKGPROXY_ variable. From 0.12.0 on, a job can download a released pkgproxy and check it instead of building it with go install. Keep pkgproxy.log as an artifact to see what each request was served from.

Why not a public pkgproxy

clientsno authentication: every client that reaches pkgproxy may use it
apt's sidepasses plain http:// requests to any host but pkgproxy's own machine, and tunnels CONNECT to port 443 of any of them
files by checksumdownloads any public URL anyone names, up to files-max-size each
the cachenever deletes anything, so anyone can fill its disk

Run pkgproxy in the job instead.

In the documentation

Roads from here

CORE · the path every side shares

  • The store — where the cache and its metadata live
  • Offline mode — leaving a troubled mirror or store alone

SETUP AND THE COMMAND LINE

OPERATIONS