How pkgproxy is tested

contributors for contributors

Unit tests of pure logic, command tests through the real command tree, synctest bubbles instead of sleeps, exploratory runs of real clients and the spread suite in VMs, all behind ./check.sh and a pre-commit hook.

On the map: inside pkgproxy

▶ Watch this deck — 8 scenes, ~5 min

01 What this topic covers

ABOUT THIS TOPIC How pkgproxy is tested from unit tests of pure logic to real clients in virtual machines 1 Pure logic, unit tested 2 Commands as a user runs them 3 No sleeps, a bubble 4 Real clients, by hand 5 The spread suite asserts behaviour 6 Before a commit, before a push
About this topic, How pkgproxy is tested: pure logic, command tests, synctest, real clients, the spread suite, before a commit

Unit tests, command tests, synctest bubbles, exploratory runs and the spread suite, and the checks before a commit.

pkgproxy is tested at several levels, from unit tests of pure logic to real clients in virtual machines. If you have not seen The source tree yet, start there. You will see unit tests of pure logic, and command tests that run the real command tree. Then come synctest bubbles that replace sleeps, and real clients tried by hand. Last come the spread suite in virtual machines, and what runs before a commit and before a push.

02 Pure logic, unit tested

UNIT TESTS index parsing cache keys hash checks store on disk no network: the same result every time $ ./check.sh dagger check go test -race -short ./... pinned + latest Go fmt vet golint build staticcheck goconst errcheck wsl ineffassign reuse GitLab CI: the same Dagger module
Four kinds of pure logic are unit tested without a network; ./check.sh runs dagger check, the same module GitLab CI runs, with the race-detected tests and the linters

Index parsing, cache keys, hash checks and the Store are plain unit tests. ./check.sh is dagger check: fmt, vet, golint, build, staticcheck, goconst, errcheck, wsl, ineffassign, go test -race -short on two Go versions, and reuse lint.

Most of pkgproxy is pure logic, and it is unit tested like any Go code: index parsing, cache key normalization, hash verification, and the store interface against the local file system. Those tests touch no network, so they give the same result every time. ./check.sh runs them with every other check as a dagger check: go test with the race detector, on the pinned Go and the latest one, next to the linters. GitLab CI runs the same Dagger module.

03 Commands as a user runs them

COMMAND TESTS func TestCtlSetupLxdRemote(t *testing.T) { noDaemon(t) // SNAP_COMMON: an empty directory inv := invoke(t, "pkgproxyctl", "setup", "lxd-remote", "-addr", testAddr, "-lxd-tls-addr", ":8443", …) check(t, inv.ExpectExitCode(0), inv.ExpectStdout(`<IGNORE> { …the whole piped script, line by line… } `), inv.ExpectStderr("")) } unixServer(t, handler) a fake daemon on a UNIX socket, for stats, info and admin calls
A Go test calls noDaemon, then invoke with pkgproxyctl setup lxd-remote and its flags, and checks the exit code and the whole printed script; unixServer can stand in for a daemon on a socket

invoke(t, "pkgproxyctl", "setup", …) wraps go-cmdr's cmdtest; assert the whole stdout, with <IGNORE> where it varies. noDaemon(t) points SNAP_COMMON at an empty directory; unixServer fakes a daemon.

pkgproxy and pkgproxyctl are tested the way a user runs them, through the real command tree. The invoke helper wraps the cmdtest package: it runs the command with its arguments, and its output buffers make every setup command print its piped script. The test then compares the whole output. Calling noDaemon first keeps a pkgproxy running on the test machine out of the output, and unixServer serves a fake daemon on a socket when a test needs one.

04 No sleeps, a bubble

SYNCTEST time.Sleep(100 * time.Millisecond) synctest.Test(t, func(t *testing.T) { … }) request 1 request 2 request 3 one fetch synctest.Wait() returns once all of them are blocked fake clock +30s at once, exactly httptest .NewTestServer in memory a real socket stops the clock a goroutine blocked on it is not "blocked" to synctest
Inside a synctest bubble, three requests join one fetch and synctest.Wait returns once they are all blocked; the fake clock jumps a timeout exactly; the upstream is an in-memory httptest server, while a real socket would stop the clock

synctest.Test runs the test in a bubble; synctest.Wait() returns when everything else is blocked. Timeouts and TTLs pass on a fake clock. Use httptest.NewTestServer and its Client(), not a real socket.

A test never waits for a goroutine with a sleep or a polling loop. It runs in a synctest bubble instead. synctest.Wait returns once every other goroutine in the bubble is blocked, such as requests that joined a shared fetch. Time in the bubble is a fake clock, so a thirty second idle timeout passes at once, and exactly. Anything the test talks to over HTTP is on httptest's in memory network, because a goroutine blocked on a real socket keeps the clock from moving.

05 Real clients, by hand

EXPLORATORY podman: debian or ubuntu $ apt update $ apt install hello $ apt download hello pkgproxy on the host deb.debian.org changes over time, not ours to pin a bug a narrow unit test: "this Release stanza parses wrong" not a frozen end-to-end scenario
A podman container runs apt update, install and download through pkgproxy on the host to a real mirror; a bug found that way turns into a narrow unit test, not a frozen end-to-end test

Exploratory apt runs in a podman container against the pkgproxy on the host; no CI job snapshots a mirror. A bug found this way becomes a narrow unit test of the logic that failed.

Interoperability with apt is checked by hand, not by a test pinned to an archive. Run a Debian or Ubuntu container with podman, point its apt at the pkgproxy on the host, and run apt update, apt install and apt download through it. Archives change over time and their mirrors are not ours to pin, so this stays exploratory. When it finds a bug, the fix comes with a narrow unit test of the logic underneath, such as a Release stanza that parsed wrongly.

06 The spread suite asserts behaviour

SPREAD ubuntu-cloud-26.04-at-lxd5 ubuntu-cloud-26.04-at-lxd6 image-garden VMs, LXD 5.21 and LXD 6 CI runs this one tests/apt/ · snap/ · docker/ · go/ · lxd/ · lxd-remote/ setup setup-setup twice is once undo undo-undo undo-first smoke download through pkgproxy, then again: hit behaviour, never archive content tests/lib/invariants.sh after every task: no proxy setting left on the machine
Each client suite of the spread suite has the same six tasks, run in image-garden VMs with LXD 5.21 and LXD 6, and invariants.sh checks after every task that no proxy setting is left

tests/<client>/: setup, setup-setup, undo, undo-undo, undo-first, smoke. tests/lib/invariants.sh runs after every task. CI runs the LXD 6 system; LXD 5.21 is left to local runs.

The spread suite tests the snap with the real apt, snapd, Docker, Go and LXD, in Ubuntu virtual machines that image-garden makes, with LXD 5.21 and LXD 6. Every client suite has the same tasks: setup, setup twice, undo, undo twice, undo first, and a smoke test where the second download is a cache hit. The tests assert behaviour, never archive content. After each task, an invariants script checks that no proxy setting was left behind.

07 Before a commit, before a push

BEFORE A COMMIT $ git config core.hooksPath githooks once per clone git commit -s gofmt vet staticcheck goconst errcheck wsl ineffassign go mod tidy -diff, every module seconds, no container ./check.sh the tests, golint, reuse lint and the rest before a push GitLab CI dco-check fails a commit with no Signed-off-by
git commit runs the pre-commit hook's fast static checks and checks every module is tidy; ./check.sh runs the tests and the rest before a push; CI runs the same checks and dco-check, which needs a Signed-off-by line

git config core.hooksPath githooks enables githooks/pre-commit, which also runs go mod tidy -diff in every module. Tests, golint and reuse lint are left to ./check.sh and CI. Commit with git commit -s for the DCO check.

Enable the pre-commit hook in every checkout you commit from. It runs the static checks that take seconds, gofmt, vet, staticcheck, goconst, errcheck, wsl and ineffassign, through go tool, and checks that every module is tidy, with no container, so a lint failure stops the commit rather than a CI run. It does not run the tests: run ./check.sh, or at least go test of what you changed, before pushing. Every commit needs a Signed-off-by line, or CI fails it.

08 Where to go next general version

WHERE NEXT you know which kind of test a change needs, and what runs before you commit and push it you are here How it is tested internal/fill the code most synctest tests drive The source tree the conventions beyond the tests pkgproxy for CI the spread suite as a small CI farm
Where next from How pkgproxy is tested: internal/fill, The source tree, pkgproxy for CI

Next: internal/fill, for the code synctest tests drive; The source tree, for the conventions; pkgproxy for CI, for the spread suite as a farm.

You now know which kind of test a change needs, and what runs before you commit and push it. To see the fill code that many synctest tests drive, watch internal/fill. For the conventions a change follows beyond its tests, watch The source tree. To see the spread suite's setup as a small CI farm, watch pkgproxy for CI.

Read more

The kinds of test

KindWhereHow to run
Unit tests of pure logic*_test.go beside the code under internal/; fixtures in testdata/apt/go test ./internal/..., or one package
Command testscmd/pkgproxy/*_test.go, one beside each cmd_*.gogo test ./cmd/pkgproxy
Synctest tests (shared fetches, timeouts, TTLs)in the packages they test: fill, flight, idletimeout, proxy, snapproxy, lxdproxy, reqlog, cmd/pkgproxy and othersas their package
Documentation checksdocs/docs_test.go: TestTablesOfContents, TestLinks, TestNoRawTags, TestHeadergo test ./docs
Podman runs of real apt and gointernal/integration/go test ./internal/integration; skipped under -short, without podman, or when a container cannot reach the host
snapd's own store client against the snap sideinternal/integration/snapd/, a nested modulecd internal/integration/snapd && go test ./...
Exploratory apt runsby hand: a Debian or Ubuntu container under podman with its apt pointed at the host's pkgproxynot automated; a bug found becomes a narrow unit test
Spread suitespread.yaml, tests/<suite>/<task>/task.yaml, tests/lib/spread garden: (below)
Everything CI runs but spread.dagger/main.go./check.sh

Command test helpers

HelperFileWhat it does
TestMainmain_test.goStarted as pkgproxy or pkgproxyctl (a symlink), the test binary runs main, so tests can run the snap's shell scripts against the real commands
invoke(t, argv...)main_test.goRuns the real command tree in English with go-cmdr's cmdtest; its buffers make every setup command print its piped script
invokeIn(t, locale, argv...)main_test.goThe same in another locale
invokeOn(t, h, argv...), invokeWithmain_test.goWith a host of test doubles, or another command tree
check(t, errs...)main_test.goReports every failed expectation of one run
noDaemon(t)main_test.goPoints SNAP_COMMON, and with it the default stats and admin sockets, at an empty directory
unixServer(t, handler)cmd_stats_test.goServes a fake daemon on a UNIX socket

Expectations compare the whole output: inv.ExpectStdout, ExpectStderr, ExpectStderrTemplate, with <IGNORE> or <VARIABLE name> where output varies (host names, temporary paths, time).

Synctest rules

  • Run the test in synctest.Test; synctest.Wait() returns once every other goroutine in the bubble is blocked.
  • Never time.Sleep or poll: timeouts, TTLs and sleeps pass on the bubble's fake clock.
  • Talk HTTP over httptest.NewTestServer and its Client(): a goroutine blocked on a real socket keeps the clock from moving.

./check.sh: the Dagger checks

./check.sh is exec dagger check; GitLab CI's dagger-check job runs the same module and then dagger call coverage export --path=coverage. It needs the dagger CLI and docker or podman. The container is golang:1.27 with shared module and build caches.

CheckRuns
go-fmtgo fmt ./... prints nothing
go-vetgo vet ./...
go-lintgo tool golang.org/x/lint/golint ./...
go-buildgo build ./...
static-checkgo tool honnef.co/go/tools/cmd/staticcheck ./...
go-constgoconst, ignoring immutable, revalidate, tcp, unix, json, limit
err-checkgo tool github.com/kisielk/errcheck ./...
wslgo tool github.com/bombsimon/wsl/v5/cmd/wsl ./...
ineff-assigngo tool github.com/gordonklaus/ineffassign ./...
testgo test -race -short -coverprofile=coverage.txt -covermode=atomic ./..., then go tool cover -func
go-test-latestgo test -race -short ./... with GOTOOLCHAIN set to the latest stable Go from go.dev
snapd-testgo vet and go test -race -count=1 ./... in internal/integration/snapd
snapd-lintgofmt, staticcheck, errcheck and ineffassign in internal/integration/snapd, built with -modfile=/src/go.mod
reuse-lintreuse lint in fsfe/reuse:latest
CommandDoes
dagger check -lList the checks
dagger check go-vet static-checkRun some by name
dagger check --failfastStop at the first failure
dagger call coverage export --path=coveragecoverage.txt, coverage-func.txt, Cobertura coverage.xml

The pre-commit hook

Enable once per clone (all its worktrees share it): git config core.hooksPath githooks. It needs no container and runs, with the tools go.mod pins:

CheckCommand
gofmtgofmt -l on every tracked .go file outside .dagger/
go-vetgo vet ./...
staticcheckgo tool honnef.co/go/tools/cmd/staticcheck ./...
goconston a copy of the files CI sees, with CI's ignore list
errcheckgo tool github.com/kisielk/errcheck ./...
wslgo tool github.com/bombsimon/wsl/v5/cmd/wsl ./...
ineffassigngo tool github.com/gordonklaus/ineffassign ./...
go mod tidygo mod tidy -diff in every module but .dagger, the nested ones included

It also reminds you to regenerate .dagger/dagger.gen.go when a staged .dagger Go file changed without it. Tests, golint and REUSE lint are left to ./check.sh and CI.

Commits

Every commit needs Signed-off-by:: commit with git commit -s, repair with git commit --amend -s or git rebase --signoff. CI's dco job runs dco-check on merge requests and the default branch.

The spread suite

SystemLXDWorkersSuites
ubuntu-cloud-26.04-at-lxd665every suite but lxd-upgrade; the only system CI runs
ubuntu-cloud-26.04-at-lxd55.213lxd, lxd-remote, snapcraft, rockcraft, charmcraft, lxd-upgrade
SuiteTasks
tests/apt/, tests/snap/setup, setup-setup, undo, undo-undo, undo-first, smoke, offline, store-overloaded
tests/docker/, tests/go/setup, setup-setup, undo, undo-undo, undo-first, smoke, offline
tests/lxd-remote/the six setup tasks, offline, refuses-sudo
tests/lxd/the six setup tasks
tests/files/smoke, offline
tests/install/daemon, snap-option (one variant per snap option), offline-switch, auto-offline
tests/snapcraft/, tests/rockcraft/, tests/charmcraft/pack, twice, the second from the cache
tests/lxd-upgrade/setup-again, on -at-lxd5: LXD 5.21 to 6
tests/build/snap: packs the snap under test; manual, with PKGPROXY_SPREAD_BUILD=1
TaskAsserts
setupThe setup command, piped to sh, configures the client
setup-setupTwice leaves the machine as once
undoSetup then -undo leaves the machine as it was
undo-undoA second -undo succeeds and changes nothing
undo-first-undo on a machine never set up succeeds and changes nothing
smokeThe client downloads through pkgproxy; the second download is a cache hit

tests/lib/invariants.sh runs after every task: no proxy setting left behind, offline mode never, auto-offline=off, the default routes. tests/lib/pkgproxy.sh has the pp_* helpers (pp_not for a negative, pp_user to run as ubuntu, pp_hits_delta, pp_content_re, pp_assert_no_upstream, …); tests/lib/workload/<client>.sh has the wl_* functions the setup tasks share.

rm -f pkgproxy_*.snap && snapcraft pack     # the one snap the suite installs, in the main checkout
spread -list
spread garden:
spread garden:ubuntu-cloud-26.04-at-lxd6:tests/apt/
spread garden:ubuntu-cloud-26.04-at-lxd6:tests/apt/smoke
spread -debug ...                           # a shell in the VM when a task fails
spread -reuse ...                           # keep the VMs for the next run

It needs image-garden and spread (the image-garden snap has both), KVM and about 6 GiB per VM, and network access. In CI, spread-snap packs the snap in a VM from a git bundle, and spread runs the -at-lxd6 tasks against it in five VMs.

In the documentation

Roads from here

SETUP AND THE COMMAND LINE

OPERATIONS

INSIDE PKGPROXY · for contributors