How pkgproxy is tested
contributors for contributors
Unit tests of pure logic, command tests through the real command tree, synctest bubbles instead of sleeps, exploratory runs of real clients and the spread suite in VMs, all behind ./check.sh and a pre-commit hook.
On the map: inside pkgproxy
01 What this topic covers
Unit tests, command tests, synctest bubbles, exploratory runs, the spread suite, outages on purpose, the rock and the charms, and the checks before a commit.
pkgproxy is tested at several levels, from unit tests of pure logic to real clients in virtual machines. If you have not seen The source tree yet, start there. You will see unit tests of pure logic, and command tests that run the real command tree. Then come synctest bubbles that replace sleeps, and real clients tried by hand. Then come the spread suite in virtual machines, upstreams that fail on purpose, and the tests of the rock and the charms. Last, what runs before a commit and before a push.
02 Pure logic, unit tested
Index parsing, cache keys, hash checks and the Store are
plain unit tests. ./check.sh is dagger check:
fmt, vet, golint, build, staticcheck, goconst, errcheck, wsl, ineffassign,
go test -race -short on two Go versions, and reuse lint.
Most of pkgproxy is pure logic, and it is unit tested like any Go code: index parsing, cache key normalization, hash verification, and the store interface against the local file system. Those tests touch no network, so they give the same result every time. ./check.sh runs them with every other check as a dagger check: go test with the race detector, on the pinned Go and the latest one, next to the linters. GitLab CI runs the same Dagger module.
03 Commands as a user runs them
invoke(t, "pkgproxyctl", "setup", …) wraps go-cmdr's
cmdtest; assert the whole stdout, with
<IGNORE> where it varies. noDaemon(t)
points SNAP_COMMON at an empty directory;
unixServer fakes a daemon.
pkgproxy and pkgproxyctl are tested the way a user runs them, through the real command tree. The invoke helper wraps the cmdtest package: it runs the command with its arguments, and its output buffers make every setup command print its piped script. The test then compares the whole output. Calling noDaemon first keeps a pkgproxy running on the test machine out of the output, and unixServer serves a fake daemon on a socket when a test needs one.
04 No sleeps, a bubble
synctest.Test runs the test in a bubble;
synctest.Wait() returns when everything else is blocked.
Timeouts and TTLs pass on a fake clock. Use
httptest.NewTestServer and its Client(), not a
real socket.
A test never waits for a goroutine with a sleep or a polling loop. It runs in a synctest bubble instead. synctest.Wait returns once every other goroutine in the bubble is blocked, such as requests that joined a shared fetch. Time in the bubble is a fake clock, so a thirty second idle timeout passes at once, and exactly. Anything the test talks to over HTTP is on httptest's in memory network, because a goroutine blocked on a real socket keeps the clock from moving.
05 Real clients, by hand
Exploratory apt runs in a podman container against the
pkgproxy on the host; no CI job snapshots a mirror. A bug found this way
becomes a narrow unit test of the logic that failed.
Interoperability with apt is checked by hand, not by a test pinned to an archive. Run a Debian or Ubuntu container with podman, point its apt at the pkgproxy on the host, and run apt update, apt install and apt download through it. Archives change over time and their mirrors are not ours to pin, so this stays exploratory. When it finds a bug, the fix comes with a narrow unit test of the logic underneath, such as a Release stanza that parsed wrongly.
06 The spread suite asserts behaviour
tests/<client>/: setup,
setup-setup, undo, undo-undo,
undo-first, smoke.
tests/lib/invariants.sh runs after every task. CI runs the
LXD 6 system; LXD 5.21 is left to local runs.
The spread suite tests the snap with the real apt, snapd, Docker, Go and LXD, in Ubuntu virtual machines that image-garden makes, with LXD 5.21 and LXD 6. Every client suite has the same tasks: setup, setup twice, undo, undo twice, undo first, and a smoke test where the second download is a cache hit. The tests assert behaviour, never archive content. After each task, an invariants script checks that no proxy setting was left behind.
07 Upstreams that fail on purpose
pp_guard_on [all] [reject|drop];
pp_assert_served_locally accepts hit,
stale and synthesised.
tests/offline-build/pack is manual, about an hour, on
ubuntu-cloud-26.04-at-bare, with pkgproxy as
pkgproxy-offline.service.
Some tasks make the upstreams fail on purpose, the way real outages look to a client. A leak guard in the firewall stops pkgproxy's own traffic: it can reject each connection at once, as a server that is down, or drop the packets, so that a connection hangs, as a silent server does. The silent archive and silent store tasks check that apt and snapd keep working from the cache, and a helper asserts that every request was a hit, a stale copy or a placeholder. The offline build suite goes furthest: a fresh machine packs a snap with only pkgproxy to talk to.
08 The rock and the charms
tests/rock/: apt, options.
tests/charm/ (manual): integration on MicroK8s,
machine on LXD. The spread-charm job runs it
with PKGPROXY_JUJU_CHANNEL 3.6/stable and
4/stable. ./check.sh runs
charm-lint and charm-test with uv.
The rock and the charms have tests of their own. Spread runs the rock in docker beside the snap: apt downloads through it, and a new container finds the cache again on the same volume. The charm suite is manual, because its setup takes minutes: concierge installs Juju, a Kubernetes cluster, and a controller on the virtual machine's LXD. Then jubilant tests deploy each charm, download with apt twice, and change its options. CI runs that suite on Juju 3.6 and on Juju 4, on main, on tags, and on a merge request labelled spread-charm. ./check.sh adds mypy --strict and pytest for the charms' Python.
09 Before a commit, before a push
git config core.hooksPath githooks enables
githooks/pre-commit, which also runs
go mod tidy -diff in every module. Tests, golint and reuse lint are left to
./check.sh and CI. Commit with git commit -s
for the DCO check.
Enable the pre-commit hook in every checkout you commit from. It runs the static checks that take seconds, gofmt, vet, staticcheck, goconst, errcheck, wsl and ineffassign, through go tool, and checks that every module is tidy, with no container, so a lint failure stops the commit rather than a CI run. It does not run the tests: run ./check.sh, or at least go test of what you changed, before pushing. Every commit needs a Signed-off-by line, or CI fails it.
10 Where to go next general version
Next: internal/fill, for the code synctest tests drive; The source tree, for the conventions; pkgproxy for CI, for the spread suite as a farm.
You now know which kind of test a change needs, and what runs before you commit and push it. To see the fill code that many synctest tests drive, watch internal/fill. For the conventions a change follows beyond its tests, watch The source tree. To see the spread suite's setup as a small CI farm, watch pkgproxy for CI.
Read more
The kinds of test
| Kind | Where | How to run |
|---|---|---|
| Unit tests of pure logic | *_test.go beside the code under internal/; fixtures in testdata/apt/ | go test ./internal/..., or one package |
| Command tests | cmd/pkgproxy/*_test.go, one beside each cmd_*.go | go test ./cmd/pkgproxy |
| Synctest tests (shared fetches, timeouts, TTLs) | in the packages they test: fill, flight, idletimeout, proxy, snapproxy, lxdproxy, reqlog, cmd/pkgproxy and others | as their package |
| Documentation checks | docs/docs_test.go: TestTablesOfContents, TestLinks, TestNoRawTags, TestHeader | go test ./docs |
| Podman runs of real apt and go | internal/integration/ | go test ./internal/integration; skipped under -short, without podman, or when a container cannot reach the host |
| snapd's own store client against the snap proxy engine | internal/integration/snapd/, a nested module | cd internal/integration/snapd && go test ./... |
| Exploratory apt runs | by hand: a Debian or Ubuntu container under podman with its apt pointed at the host's pkgproxy | not automated; a bug found becomes a narrow unit test |
| Spread suite | spread.yaml, tests/<suite>/<task>/task.yaml, tests/lib/ | spread garden: (below) |
| Charm unit tests | charms/<name>/tests/unit/, with ops.testing, for pkgproxy-k8s and pkgproxy | uv run pytest -q in charms/<name>/, or dagger check charm-test |
| Charm integration tests | charms/<name>/tests/integration/, with jubilant | the manual spread suite tests/charm/ (below), or uv run pytest tests/integration against a Juju model of your own |
| Charms kept in step with the daemon | cmd/pkgproxy/charm_test.go | go test ./cmd/pkgproxy |
| A benchmark | internal/upstreams/history_test.go: BenchmarkHistory30Days, a series' 30 days of hours, the same cost whatever the files hold (§28.7) | go test -run '^$' -bench History30Days ./internal/upstreams; neither ./check.sh nor CI runs it |
| Everything CI runs but spread | .dagger/main.go | ./check.sh |
Command test helpers
| Helper | File | What it does |
|---|---|---|
TestMain | main_test.go | Started as pkgproxy or pkgproxyctl (a symlink), the test binary runs main, so tests can run the snap's shell scripts against the real commands |
invoke(t, argv...) | main_test.go | Runs the real command tree in English with go-cmdr's cmdtest; its buffers make every setup command print its piped script |
invokeIn(t, locale, argv...) | main_test.go | The same in another locale |
invokeOn(t, h, argv...), invokeWith | main_test.go | With a host of test doubles, or another command tree |
check(t, errs...) | main_test.go | Reports every failed expectation of one run |
noDaemon(t) | main_test.go | Points SNAP_COMMON, and with it the default stats and admin sockets, at an empty directory |
unixServer(t, handler) | cmd_stats_test.go | Serves a fake daemon on a UNIX socket |
charm_test.go keeps the charms in step with what they wrap:
| Test | Checks |
|---|---|
TestCharmOptionsMatchFlags | in both charms, every option is a daemon flag of the same name and default, or one of the charm's own; every flag is an option, or listed with a reason as not one |
TestMachineCharmOptionsAreSnapOptions | every machine charm option is a snap option pkgproxy-env maps (lxd-tls as lxd-tls-addr) |
TestCharmServiceMatchesRock | the Kubernetes charm's Pebble service runs the rock's command (without its empty [ ] arguments), with the rock's kill-delay |
TestCharmSetupScriptMatchesSetup | the setup-script action offers every tool pkgproxyctl setup sets up, in its order |
TestCharmsShareCommon | src/common.py is the same file in every charm |
Expectations compare the whole output: inv.ExpectStdout, ExpectStderr, ExpectStderrTemplate, with <IGNORE> or <VARIABLE name> where output varies (host names, temporary paths, time).
Synctest rules
- Run the test in
synctest.Test;synctest.Wait()returns once every other goroutine in the bubble is blocked. - Never
time.Sleepor poll: timeouts, TTLs and sleeps pass on the bubble's fake clock. - Talk HTTP over
httptest.NewTestServerand itsClient(): a goroutine blocked on a real socket keeps the clock from moving.
./check.sh: the Dagger checks
./check.sh is exec dagger check; GitLab CI's dagger-check job runs the same module and then dagger call coverage export --path=coverage. It needs the dagger CLI and docker or podman. The container is golang:1.27 with shared module and build caches.
| Check | Runs |
|---|---|
go-fmt | go fmt ./... prints nothing |
go-vet | go vet ./... |
go-lint | go tool golang.org/x/lint/golint ./... |
go-build | go build ./... |
static-check | go tool honnef.co/go/tools/cmd/staticcheck ./... |
go-const | goconst, ignoring immutable, revalidate, tcp, unix, json, limit |
err-check | go tool github.com/kisielk/errcheck ./... |
wsl | go tool github.com/bombsimon/wsl/v5/cmd/wsl ./... |
ineff-assign | go tool github.com/gordonklaus/ineffassign ./... |
test | go test -race -short -coverprofile=coverage.txt -covermode=atomic ./..., then go tool cover -func |
go-test-latest | go test -race -short ./... with GOTOOLCHAIN set to the latest stable Go from go.dev |
snapd-test | go vet and go test -race -count=1 ./... in internal/integration/snapd |
snapd-lint | gofmt, staticcheck, errcheck and ineffassign in internal/integration/snapd, built with -modfile=/src/go.mod |
reuse-lint | reuse lint in fsfe/reuse:latest |
charm-lint | uv sync --locked, then uv run mypy (--strict, from the charm's pyproject.toml) in charms/pkgproxy-k8s and charms/pkgproxy |
charm-test | uv sync --locked, then uv run pytest -q (tests/unit) in each charm |
The charm checks run in ghcr.io/astral-sh/uv:0.9.30-python3.14-trixie-slim, with a shared uv cache. --locked fails when uv.lock no longer matches pyproject.toml.
| Command | Does |
|---|---|
dagger check -l | List the checks |
dagger check go-vet static-check | Run some by name |
dagger check --failfast | Stop at the first failure |
dagger call coverage export --path=coverage | coverage.txt, coverage-func.txt, Cobertura coverage.xml |
The pre-commit hook
Enable once per clone (all its worktrees share it): git config core.hooksPath githooks. It needs no container and runs, with the tools go.mod pins:
| Check | Command |
|---|---|
| gofmt | gofmt -l on every tracked .go file outside .dagger/ |
| go-vet | go vet ./... |
| staticcheck | go tool honnef.co/go/tools/cmd/staticcheck ./... |
| goconst | on a copy of the files CI sees, with CI's ignore list |
| errcheck | go tool github.com/kisielk/errcheck ./... |
| wsl | go tool github.com/bombsimon/wsl/v5/cmd/wsl ./... |
| ineffassign | go tool github.com/gordonklaus/ineffassign ./... |
| go mod tidy | go mod tidy -diff in every module but .dagger, the nested ones included |
It also reminds you to regenerate .dagger/dagger.gen.go when a staged .dagger Go file changed without it. Tests, golint and REUSE lint are left to ./check.sh and CI.
Commits
Every commit needs Signed-off-by:: commit with git commit -s, repair with git commit --amend -s or git rebase --signoff. CI's dco job runs dco-check on merge requests and the default branch.
The spread suite
| System | LXD | Workers | Suites |
|---|---|---|---|
ubuntu-cloud-26.04-at-lxd6 | 6 | 5 | every suite but lxd-upgrade, with tests/rock/ and the manual tests/charm/; the only system CI runs |
ubuntu-cloud-26.04-at-lxd5 | 5.21 | 3 | lxd, lxd-remote, snapcraft, rockcraft, charmcraft, lxd-upgrade |
ubuntu-cloud-26.04-at-bare | none | 1 | the manual tests/offline-build/ alone; the project's prepare, prepare-each and restore-each do nothing on it |
| Suite | Tasks |
|---|---|
tests/apt/ | the six setup tasks, offline, store-overloaded, upstream-silent (a silent archive: apt works from the cache, served stale) |
tests/snap/ | the six setup tasks, offline, offline-refresh, offline-session (a snapd with a serial but no session installs from the cache), store-overloaded, store-down (a store that cannot be reached at all), store-silent (a store that hangs) |
tests/docker/ | the six setup tasks, offline |
tests/go/ | the six setup tasks, setup-before-go, offline, offline-build (go get, build and install offline after the module cache is wiped) |
tests/lxd-remote/ | the six setup tasks, offline, refuses-sudo |
tests/lxd/ | the six setup tasks |
tests/files/ | smoke, offline, volatile (SHA256SUMS online twice, then the kept copy offline, marked STALE; refused once invalidated) |
tests/install/ | daemon, snap-option (one variant per snap option), offline-switch, auto-offline, migrate-meta, proxy-options |
tests/snapcraft/, tests/rockcraft/, tests/charmcraft/ | pack, twice, the second from the cache |
tests/lxd-upgrade/ | setup-again, on -at-lxd5: LXD 5.21 to 6 |
tests/offline-build/ | manual, about an hour: pack, a fresh machine with no snapd that packs tests/snapcraft/fixture once online and again offline with the leak guard and no default routes, from pkgproxy alone; pkgproxy runs outside snapd as pkgproxy-offline.service, state in /var/lib/pkgproxy-offline; what was refused is in the artifact refused.log |
tests/rock/ | apt: downloads through the rock in docker from a named volume, and finds the cache again in a new container; options: a variable, a flag, an invalid option, and stopping it |
tests/charm/ | manual: integration (the Kubernetes charm, on MicroK8s) and machine (the machine charm, on the VM's LXD) |
| Task | Asserts |
|---|---|
setup | The setup command, piped to sh, configures the client |
setup-setup | Twice leaves the machine as once |
undo | Setup then -undo leaves the machine as it was |
undo-undo | A second -undo succeeds and changes nothing |
undo-first | -undo on a machine never set up succeeds and changes nothing |
smoke | The client downloads through pkgproxy; the second download is a cache hit |
tests/lib/invariants.sh runs after every task: no proxy setting left behind, offline mode never, auto-offline=off, the default routes. tests/lib/pkgproxy.sh has the pp_* helpers (pp_not for a negative, pp_user to run as ubuntu, pp_hits_delta, pp_content_re, pp_assert_no_upstream, which accepts a hit, a synthesised answer, a stale copy or a refusal, pp_assert_served_locally T [ERE], which requires hit, stale or synthesised of every access log line since T, pp_volatile_drop, pp_tool_snaps, …); pp_guard_on [all] [reject|drop] loads the leak guard, an nft rule on the daemon's cgroup: reject refuses each new connection, as an upstream that is down, drop discards the packets, as one that hangs, and all stops already open connections too. PP_UNIT, PP_COMMON, PP_STATS_SOCK and PP_ADMIN_SOCK point the helpers at a pkgproxy outside the snap; tests/lib/workload/<client>.sh has the wl_* functions the setup tasks share.
rm -f pkgproxy_*.snap && snapcraft pack # the one snap the suite installs, in the main checkout rm -f pkgproxy_*.rock && rockcraft pack # the one rock tests/rock/ and tests/charm/ run spread -list spread garden: spread garden:ubuntu-cloud-26.04-at-lxd6:tests/apt/ spread garden:ubuntu-cloud-26.04-at-lxd6:tests/apt/smoke spread -debug ... # a shell in the VM when a task fails spread -reuse ... # keep the VMs for the next run
It needs image-garden and spread (the image-garden snap has both), KVM and about 6 GiB per VM, and network access. Optionally, an infrastructure pkgproxy as PKGPROXY_UPSTREAM_URL (by default PKGPROXY_URL, which CI runners set), an address the VMs reach, never localhost: the pkgproxy under test then fetches everything through it (upstream-pkgproxy, §27), and the snapcraft, rockcraft and charmcraft suites install their tools from it with pp_tool_snaps, so that they do not warm the cache of the pkgproxy under test. tests/rock/ publishes the rock's port 3142 on 4142, beside the snap.
The charm suite
tests/charm/ is manual: spread runs it only when it is named, as the cluster and the controllers take minutes to set up.
spread garden:ubuntu-cloud-26.04-at-lxd6:tests/charm/ PKGPROXY_JUJU_CHANNEL=4/stable spread garden:ubuntu-cloud-26.04-at-lxd6:tests/charm/
Its prepare, once per VM, lets pod and container traffic through Docker's FORWARD policy (DOCKER-USER rules for cali* and lxdbr0), makes the network namespace pkgproxy-client (192.168.250.2) as a client off the cluster, and runs concierge with tests/charm/concierge.yaml: Juju from PKGPROXY_JUJU_CHANNEL (default 3.6/stable), a strictly confined MicroK8s with hostpath-storage, dns, registry and metallb (10.64.140.43 to 49), and the controllers concierge-microk8s and concierge-lxd, whose models run update-status every minute, as Juju 4 runs config-changed only after the next one. It copies the rock into localhost:32000, and takes the charms CI packed from the root of the tree, or packs them with charmcraft pack --destructive-mode.
| Task | Runs | Asserts |
|---|---|---|
integration | charms/pkgproxy-k8s/tests/integration | active with the cache on storage; apt twice through the Service, the second a hit; still a hit after the pod is deleted; an option reaches the service; a refused option blocks the unit while the old configuration serves; offline mode; the stats, invalidate and setup-script actions; the LoadBalancer keeps the client's address |
machine | charms/pkgproxy/tests/integration | deployed with the snap as pkgproxy-snap; apt twice, the second a hit; an option reaches snap get; a refused option blocks the unit and keeps the old value; a refresh installs the snap again and the package is still a hit |
A run takes 9 to 13 minutes, most of it concierge. pytest.xml is each task's artifact (spread -artifacts). The Kubernetes tests also run against a cluster of your own: PKGPROXY_CHARM, PKGPROXY_IMAGE, JUJU_MODEL, and for the LoadBalancer test PKGPROXY_CLIENT_NETNS and PKGPROXY_CLIENT_ADDR, then uv run pytest tests/integration in charms/pkgproxy-k8s/.
In CI
| Job | Does |
|---|---|
snap-ci | packs the snap under test with snapcraft-gitlab-ci's build component, snapcraft pack --destructive-mode in a throwaway 26.04 VM, from a copy of the working tree with .git; no needs, so nothing untracked stamps the binaries +dirty; keeps it in snaps/ |
snap-ci-check | the binaries are not +dirty |
snap-ci-arm64, snap-ci-arm64-check | the same, natively on an aarch64 runner; spread tests the amd64 snap only |
rock-ci, rock-ci-check | packs the rock with rockcraft-gitlab-ci into rocks/, and checks its binary |
charm-ci, charm-ci-check | packs charms/pkgproxy-k8s/ with charmcraft-gitlab-ci into pkgproxy-k8s_amd64.charm, and checks its metadata (the container's user, the cache storage) and that its virtualenv has no test packages |
charm-machine-24.04, charm-machine-26.04 | pack charms/pkgproxy/ once for each base |
spread | moves the snap and the rock into the root of the tree and runs the -at-lxd6 tasks in five VMs, with spread-gitlab-ci's spread@0.4 component, through spread-gitlab-report; the runner's PKGPROXY_URL is the upstream of the pkgproxy under test |
spread-charm | runs tests/charm/ in one VM, as a matrix of PKGPROXY_JUJU_CHANNEL 3.6/stable and 4/stable |
spread-offline-build | runs tests/offline-build/pack with the snap snap-ci packed, in a VM of its own: on a schedule of main that sets PKGPROXY_OFFLINE_BUILD=1, which the publishing jobs skip, and by hand in any other pipeline; retries: 1 |
rock | after spread, pushes the rock, edge from main and the version from a tag, and names it by digest as ROCK_IMAGE |
snap-edge, snap-stable, charm-edge, charm-stable | publish to the Snap Store and Charmhub |
A retry of a spread job, GitLab's or by hand, runs only the tasks that did not pass: the component keeps them in .spread-rerun/, cached under spread-rerun-$CI_PIPELINE_ID-$CI_JOB_NAME_SLUG.
spread-charm takes 9 to 13 minutes of a large KVM slot, so it runs only on a merge request labelled spread-charm (add the label, then push or run the pipeline again), on a pipeline with PKGPROXY_SPREAD_CHARM=1 (by hand or on a schedule, which then runs snap-ci, rock-ci, charm-ci and spread-charm alone), and always on main and version tags. Ask for it on every merge request that changes the charms, the rock or tests/charm/.
In the documentation
Roads from here
SETUP AND THE COMMAND LINE
- Setting up clients — setup twice and undo twice
OPERATIONS
- pkgproxy for CI — the spread suite as a small CI farm
INSIDE PKGPROXY · for contributors
- The source tree — the conventions every change follows
- internal/fill — the shared fetch the synctest tests drive
- A request in code — the transport whose timeout runs on a fake clock
- Translations — translated output under test