How pkgproxy is tested
contributors for contributors
Unit tests of pure logic, command tests through the real command tree, synctest bubbles instead of sleeps, exploratory runs of real clients and the spread suite in VMs, all behind ./check.sh and a pre-commit hook.
On the map: inside pkgproxy
01 What this topic covers
Unit tests, command tests, synctest bubbles, exploratory runs and the spread suite, and the checks before a commit.
pkgproxy is tested at several levels, from unit tests of pure logic to real clients in virtual machines. If you have not seen The source tree yet, start there. You will see unit tests of pure logic, and command tests that run the real command tree. Then come synctest bubbles that replace sleeps, and real clients tried by hand. Last come the spread suite in virtual machines, and what runs before a commit and before a push.
02 Pure logic, unit tested
Index parsing, cache keys, hash checks and the Store are
plain unit tests. ./check.sh is dagger check:
fmt, vet, golint, build, staticcheck, goconst, errcheck, wsl, ineffassign,
go test -race -short on two Go versions, and reuse lint.
Most of pkgproxy is pure logic, and it is unit tested like any Go code: index parsing, cache key normalization, hash verification, and the store interface against the local file system. Those tests touch no network, so they give the same result every time. ./check.sh runs them with every other check as a dagger check: go test with the race detector, on the pinned Go and the latest one, next to the linters. GitLab CI runs the same Dagger module.
03 Commands as a user runs them
invoke(t, "pkgproxyctl", "setup", …) wraps go-cmdr's
cmdtest; assert the whole stdout, with
<IGNORE> where it varies. noDaemon(t)
points SNAP_COMMON at an empty directory;
unixServer fakes a daemon.
pkgproxy and pkgproxyctl are tested the way a user runs them, through the real command tree. The invoke helper wraps the cmdtest package: it runs the command with its arguments, and its output buffers make every setup command print its piped script. The test then compares the whole output. Calling noDaemon first keeps a pkgproxy running on the test machine out of the output, and unixServer serves a fake daemon on a socket when a test needs one.
04 No sleeps, a bubble
synctest.Test runs the test in a bubble;
synctest.Wait() returns when everything else is blocked.
Timeouts and TTLs pass on a fake clock. Use
httptest.NewTestServer and its Client(), not a
real socket.
A test never waits for a goroutine with a sleep or a polling loop. It runs in a synctest bubble instead. synctest.Wait returns once every other goroutine in the bubble is blocked, such as requests that joined a shared fetch. Time in the bubble is a fake clock, so a thirty second idle timeout passes at once, and exactly. Anything the test talks to over HTTP is on httptest's in memory network, because a goroutine blocked on a real socket keeps the clock from moving.
05 Real clients, by hand
Exploratory apt runs in a podman container against the
pkgproxy on the host; no CI job snapshots a mirror. A bug found this way
becomes a narrow unit test of the logic that failed.
Interoperability with apt is checked by hand, not by a test pinned to an archive. Run a Debian or Ubuntu container with podman, point its apt at the pkgproxy on the host, and run apt update, apt install and apt download through it. Archives change over time and their mirrors are not ours to pin, so this stays exploratory. When it finds a bug, the fix comes with a narrow unit test of the logic underneath, such as a Release stanza that parsed wrongly.
06 The spread suite asserts behaviour
tests/<client>/: setup,
setup-setup, undo, undo-undo,
undo-first, smoke.
tests/lib/invariants.sh runs after every task. CI runs the
LXD 6 system; LXD 5.21 is left to local runs.
The spread suite tests the snap with the real apt, snapd, Docker, Go and LXD, in Ubuntu virtual machines that image-garden makes, with LXD 5.21 and LXD 6. Every client suite has the same tasks: setup, setup twice, undo, undo twice, undo first, and a smoke test where the second download is a cache hit. The tests assert behaviour, never archive content. After each task, an invariants script checks that no proxy setting was left behind.
07 Before a commit, before a push
git config core.hooksPath githooks enables
githooks/pre-commit, which also runs
go mod tidy -diff in every module. Tests, golint and reuse lint are left to
./check.sh and CI. Commit with git commit -s
for the DCO check.
Enable the pre-commit hook in every checkout you commit from. It runs the static checks that take seconds, gofmt, vet, staticcheck, goconst, errcheck, wsl and ineffassign, through go tool, and checks that every module is tidy, with no container, so a lint failure stops the commit rather than a CI run. It does not run the tests: run ./check.sh, or at least go test of what you changed, before pushing. Every commit needs a Signed-off-by line, or CI fails it.
08 Where to go next general version
Next: internal/fill, for the code synctest tests drive; The source tree, for the conventions; pkgproxy for CI, for the spread suite as a farm.
You now know which kind of test a change needs, and what runs before you commit and push it. To see the fill code that many synctest tests drive, watch internal/fill. For the conventions a change follows beyond its tests, watch The source tree. To see the spread suite's setup as a small CI farm, watch pkgproxy for CI.
Read more
The kinds of test
| Kind | Where | How to run |
|---|---|---|
| Unit tests of pure logic | *_test.go beside the code under internal/; fixtures in testdata/apt/ | go test ./internal/..., or one package |
| Command tests | cmd/pkgproxy/*_test.go, one beside each cmd_*.go | go test ./cmd/pkgproxy |
| Synctest tests (shared fetches, timeouts, TTLs) | in the packages they test: fill, flight, idletimeout, proxy, snapproxy, lxdproxy, reqlog, cmd/pkgproxy and others | as their package |
| Documentation checks | docs/docs_test.go: TestTablesOfContents, TestLinks, TestNoRawTags, TestHeader | go test ./docs |
| Podman runs of real apt and go | internal/integration/ | go test ./internal/integration; skipped under -short, without podman, or when a container cannot reach the host |
| snapd's own store client against the snap side | internal/integration/snapd/, a nested module | cd internal/integration/snapd && go test ./... |
| Exploratory apt runs | by hand: a Debian or Ubuntu container under podman with its apt pointed at the host's pkgproxy | not automated; a bug found becomes a narrow unit test |
| Spread suite | spread.yaml, tests/<suite>/<task>/task.yaml, tests/lib/ | spread garden: (below) |
| Everything CI runs but spread | .dagger/main.go | ./check.sh |
Command test helpers
| Helper | File | What it does |
|---|---|---|
TestMain | main_test.go | Started as pkgproxy or pkgproxyctl (a symlink), the test binary runs main, so tests can run the snap's shell scripts against the real commands |
invoke(t, argv...) | main_test.go | Runs the real command tree in English with go-cmdr's cmdtest; its buffers make every setup command print its piped script |
invokeIn(t, locale, argv...) | main_test.go | The same in another locale |
invokeOn(t, h, argv...), invokeWith | main_test.go | With a host of test doubles, or another command tree |
check(t, errs...) | main_test.go | Reports every failed expectation of one run |
noDaemon(t) | main_test.go | Points SNAP_COMMON, and with it the default stats and admin sockets, at an empty directory |
unixServer(t, handler) | cmd_stats_test.go | Serves a fake daemon on a UNIX socket |
Expectations compare the whole output: inv.ExpectStdout, ExpectStderr, ExpectStderrTemplate, with <IGNORE> or <VARIABLE name> where output varies (host names, temporary paths, time).
Synctest rules
- Run the test in
synctest.Test;synctest.Wait()returns once every other goroutine in the bubble is blocked. - Never
time.Sleepor poll: timeouts, TTLs and sleeps pass on the bubble's fake clock. - Talk HTTP over
httptest.NewTestServerand itsClient(): a goroutine blocked on a real socket keeps the clock from moving.
./check.sh: the Dagger checks
./check.sh is exec dagger check; GitLab CI's dagger-check job runs the same module and then dagger call coverage export --path=coverage. It needs the dagger CLI and docker or podman. The container is golang:1.27 with shared module and build caches.
| Check | Runs |
|---|---|
go-fmt | go fmt ./... prints nothing |
go-vet | go vet ./... |
go-lint | go tool golang.org/x/lint/golint ./... |
go-build | go build ./... |
static-check | go tool honnef.co/go/tools/cmd/staticcheck ./... |
go-const | goconst, ignoring immutable, revalidate, tcp, unix, json, limit |
err-check | go tool github.com/kisielk/errcheck ./... |
wsl | go tool github.com/bombsimon/wsl/v5/cmd/wsl ./... |
ineff-assign | go tool github.com/gordonklaus/ineffassign ./... |
test | go test -race -short -coverprofile=coverage.txt -covermode=atomic ./..., then go tool cover -func |
go-test-latest | go test -race -short ./... with GOTOOLCHAIN set to the latest stable Go from go.dev |
snapd-test | go vet and go test -race -count=1 ./... in internal/integration/snapd |
snapd-lint | gofmt, staticcheck, errcheck and ineffassign in internal/integration/snapd, built with -modfile=/src/go.mod |
reuse-lint | reuse lint in fsfe/reuse:latest |
| Command | Does |
|---|---|
dagger check -l | List the checks |
dagger check go-vet static-check | Run some by name |
dagger check --failfast | Stop at the first failure |
dagger call coverage export --path=coverage | coverage.txt, coverage-func.txt, Cobertura coverage.xml |
The pre-commit hook
Enable once per clone (all its worktrees share it): git config core.hooksPath githooks. It needs no container and runs, with the tools go.mod pins:
| Check | Command |
|---|---|
| gofmt | gofmt -l on every tracked .go file outside .dagger/ |
| go-vet | go vet ./... |
| staticcheck | go tool honnef.co/go/tools/cmd/staticcheck ./... |
| goconst | on a copy of the files CI sees, with CI's ignore list |
| errcheck | go tool github.com/kisielk/errcheck ./... |
| wsl | go tool github.com/bombsimon/wsl/v5/cmd/wsl ./... |
| ineffassign | go tool github.com/gordonklaus/ineffassign ./... |
| go mod tidy | go mod tidy -diff in every module but .dagger, the nested ones included |
It also reminds you to regenerate .dagger/dagger.gen.go when a staged .dagger Go file changed without it. Tests, golint and REUSE lint are left to ./check.sh and CI.
Commits
Every commit needs Signed-off-by:: commit with git commit -s, repair with git commit --amend -s or git rebase --signoff. CI's dco job runs dco-check on merge requests and the default branch.
The spread suite
| System | LXD | Workers | Suites |
|---|---|---|---|
ubuntu-cloud-26.04-at-lxd6 | 6 | 5 | every suite but lxd-upgrade; the only system CI runs |
ubuntu-cloud-26.04-at-lxd5 | 5.21 | 3 | lxd, lxd-remote, snapcraft, rockcraft, charmcraft, lxd-upgrade |
| Suite | Tasks |
|---|---|
tests/apt/, tests/snap/ | setup, setup-setup, undo, undo-undo, undo-first, smoke, offline, store-overloaded |
tests/docker/, tests/go/ | setup, setup-setup, undo, undo-undo, undo-first, smoke, offline |
tests/lxd-remote/ | the six setup tasks, offline, refuses-sudo |
tests/lxd/ | the six setup tasks |
tests/files/ | smoke, offline |
tests/install/ | daemon, snap-option (one variant per snap option), offline-switch, auto-offline |
tests/snapcraft/, tests/rockcraft/, tests/charmcraft/ | pack, twice, the second from the cache |
tests/lxd-upgrade/ | setup-again, on -at-lxd5: LXD 5.21 to 6 |
tests/build/ | snap: packs the snap under test; manual, with PKGPROXY_SPREAD_BUILD=1 |
| Task | Asserts |
|---|---|
setup | The setup command, piped to sh, configures the client |
setup-setup | Twice leaves the machine as once |
undo | Setup then -undo leaves the machine as it was |
undo-undo | A second -undo succeeds and changes nothing |
undo-first | -undo on a machine never set up succeeds and changes nothing |
smoke | The client downloads through pkgproxy; the second download is a cache hit |
tests/lib/invariants.sh runs after every task: no proxy setting left behind, offline mode never, auto-offline=off, the default routes. tests/lib/pkgproxy.sh has the pp_* helpers (pp_not for a negative, pp_user to run as ubuntu, pp_hits_delta, pp_content_re, pp_assert_no_upstream, …); tests/lib/workload/<client>.sh has the wl_* functions the setup tasks share.
rm -f pkgproxy_*.snap && snapcraft pack # the one snap the suite installs, in the main checkout spread -list spread garden: spread garden:ubuntu-cloud-26.04-at-lxd6:tests/apt/ spread garden:ubuntu-cloud-26.04-at-lxd6:tests/apt/smoke spread -debug ... # a shell in the VM when a task fails spread -reuse ... # keep the VMs for the next run
It needs image-garden and spread (the image-garden snap has both), KVM and about 6 GiB per VM, and network access. In CI, spread-snap packs the snap in a VM from a git bundle, and spread runs the -at-lxd6 tasks against it in five VMs.
In the documentation
Roads from here
SETUP AND THE COMMAND LINE
- Setting up clients — setup twice and undo twice
OPERATIONS
- pkgproxy for CI — the spread suite as a small CI farm
INSIDE PKGPROXY · for contributors
- The source tree — the conventions every change follows
- internal/fill — the shared fetch the synctest tests drive
- A request in code — the transport whose timeout runs on a fake clock
- Translations — translated output under test