The source tree
contributors for contributors
Where the code lives (one multi-call binary in cmd/pkgproxy and one internal package per side or shared concern), where the design is written down, and the conventions every change follows.
On the map: inside pkgproxy
01 What this topic covers
Where pkgproxy's code and design live, and the conventions every change follows.
The pkgproxy source keeps its commands in one directory, the packages they use under internal, and the design in docs. If you do not know what pkgproxy does yet, watch A request first. You will see the two directories that hold the code, and how the command directory keeps one file per command. Then you will see the design split across documents that share section numbers, and the conventions every change follows.
02 Two directories hold the code
cmd/pkgproxy is the multi-call binary;
internal/ holds the sides (proxy is apt), the
shared path, the protocol parsers and the supporting packages. Nothing
under internal/ is importable from outside the module.
The code lives in two directories. The command directory builds one binary with two names: run as pkgproxy it is the daemon, and run as pkgproxyctl, a symlink to the same file, it is the command line an operator uses. Everything else is under internal, one package per concern. Each side that serves a kind of client has its own package. The path every side shares has packages of its own: the gateway, the request log, fills, the store and cache keys. Parsers, offline mode, statistics and translations sit beside them.
03 One file per command
main.go routes on argv[0] with go-cmdr's
router.Cmd; ctl.go builds
pkgproxyctl's tree. Every leaf prints its help with
printUsage; tests use invoke(t, "pkgproxyctl", …).
In the source, the entry point wraps both names in one router that dispatches on the name the binary was run as, and a second router builds pkgproxyctl's own tree. The command directory groups code by command, not by topic: one file per command, named after it, so setup lxd lives in cmd_setup_lxd.go, with its test beside it. Code several commands share lives in files without the cmd prefix, such as the guide builder and the usage printer. Tests run the real command tree, the way a user runs it.
04 One design, many files
docs/design.md holds §1, §12 and §18 and the index; each
area's file holds its own §, cited as docs/design/lxd.md §17.11;
go test ./docs guards the published site.
The design is written down in docs/design.md and one document per area under docs/design: apt, snap, Go modules, LXD, Docker, files, offline mode and the rest. The section numbers are shared by all the files, so apt.md section eight names one place. A new area gets its own file, the next free number and a line in the index. A change to the design updates its document in the same commit. Go test of the docs package checks links, anchors and tables of contents, because the documents are published as a website.
05 The conventions of a change
Issues and milestones live in GitLab (glab issue list), not
in a TODO file. git config core.hooksPath githooks, then
git commit -s; ./check.sh before pushing.
A change starts from a GitLab issue, which says why, not only what, and belongs to a milestone. Errors in the code are built with l10n.D.Errorf, and a failed action is worded cannot, then a verb. The pre-commit hook, enabled once per clone, runs the fast linters before each commit. Every commit carries a Signed-off-by line, or CI fails it. ./check.sh runs the same Dagger checks CI runs, tests included, before a push.
06 Where to go next general version
Next: A request in code, for one request through the packages; How it is tested, for what a change needs; Translations, for errors and messages that translate.
You now know where the code for a change lives, which design section it touches, and what conventions it has to follow. To follow one request through those packages, watch A request in code. To see how a change is tested before you commit it, watch How it is tested. To write errors and messages that can be translated, watch Translations.
Read more
Top-level directories and files
| Path | What it holds |
|---|---|
cmd/pkgproxy/ | The one multi-call binary: the daemon pkgproxy and the command line pkgproxyctl, a symlink to the same file |
internal/ | Every other package, one per concern; not importable from outside module gitlab.com/zygoon/pkgproxy |
internal/integration/ | Tests only: podman runs of real apt and go (skipped under -short or without podman), and snapd/, a nested Go module that drives snapd's own store client |
data/ | Assets embedded with //go:embed: the icon (also the snap's), the upstream logos of the cache page's live views, the Ubuntu mark |
docs/ | The design (design.md, design/*.md), statsapi.md, statsapi-internals.md, testing.md, troubleshooting.md, and docs_test.go, which guards the published site |
snap/ | snapcraft.yaml, the configure hook, local/bin/pkgproxy-env (snap options to PKGPROXY_* variables), the desktop file |
tests/ | The spread suite: tests/<client>/<task>/task.yaml, helpers in tests/lib/, fixtures for the snapcraft, rockcraft and charmcraft packs |
testdata/apt/ | Release, InRelease, Packages and Sources excerpts of Debian and Ubuntu, for the parsers' unit tests |
githooks/ | pre-commit, enabled with git config core.hooksPath githooks |
.dagger/ | The Dagger module whose checks ./check.sh and GitLab CI run |
spread.yaml, .image-garden.mk | The spread project and the VM images image-garden makes for it |
check.sh | exec dagger check |
DEPLOYMENT.md, README.md, NEWS | Published at https://zygoon.pl/pkgproxy/ with docs/ |
cmd/pkgproxy: one file per command
| Command | File |
|---|---|
pkgproxy (the daemon) | cmd_pkgproxy.go, with filesflags.go and envflags.go (PKGPROXY_* defaults) |
pkgproxyctl (the router) | ctl.go |
pkgproxyctl stats | cmd_stats.go |
pkgproxyctl setup | cmd_setup.go; one child each: cmd_setup_apt.go, _snap, _go, _docker, _lxd, _lxd_remote, _snapcraft, _rockcraft, _charmcraft, _status |
pkgproxyctl cache invalidate | cmd_cache.go |
pkgproxyctl offline | cmd_offline.go |
pkgproxyctl open | cmd_open.go |
pkgproxyctl version | cmd_version.go, with version.go |
Files without the cmd_ prefix hold what several commands share:
| File | What it holds |
|---|---|
main.go | The router.Cmd on argv[0], the locale and the translator |
usage.go | printUsage, the help layout of every leaf command (docs/design/cli.md §14.3) |
guide.go | The guide builder: translated paragraphs and verbatim code blocks |
instructions.go | Client configuration paths and the startup banner |
host.go | host, what commands take from outside, which tests replace |
setuppage.go, setuppage_cache.go | The setup page, its scripts under /setup/, and the cache page |
cachemap.go, cachelive.go, cachepage.js | The cache page's map, its activity and its live events |
lxdtls.go, lxdremote.go, lxdbridge.go, lxdhook.go | The LXD HTTPS listener and certificate, -lxd-remote, the bridge and profile of setup lxd |
statssocket.go, snapoption.go | Default socket and cache paths; reading snap options |
signals_*.go, umask_*.go | Platform parts: shutdown signals, the umask around socket creation |
internal: the sides
| Package | Purpose |
|---|---|
proxy | apt: the forward proxy, cache key, store, origin and index wired into one handler (§7.5, §8) |
snapproxy | snap: a reverse relay snapd talks to through SNAPPY_FORCE_API_URL, caching snaps and assertions (§11) |
gomodproxy | Go: a GOPROXY origin chained to an upstream module proxy, with checksum database checks (§16) |
registryproxy | Docker: the read-only OCI distribution API for anonymous pulls (§19) |
lxdproxy | LXD: simplestreams image servers under <base>/<name> (§17) |
lxdapi | The part of LXD's REST API that LXD 6.10 and later download images through (§17.14) |
filesproxy | Files by checksum: GET /files/v1/fetch?url=…&sha256=… (§24) |
internal: the path every request shares
| Package | Purpose |
|---|---|
gateway | Dispatches each request to one side on the one listener |
reqlog | Wraps a listener's handler: one access log line per request, and events for observers (§23) |
offline | The offline switch: modes, per-host breakers, the transport, dial and middleware wrappers (§21) |
fill | Serves a large blob while it downloads into the store (§8.1) |
flight | singleflight with typed results and a joined bit |
revalidate | One detached revalidation per key, bounded wait, recent-failure memory |
store, store/fslocal, store/memstore | The store interface, its disk implementation, its in-memory one for tests (§7.1) |
cachekey | apt request URL to cache key and policy (§4, §7.3) |
index | Expected hashes of apt pool and by-hash paths from parsed indexes (§4.2, §7.4) |
lxdindex | Expected SHA256 of LXD blob paths from parsed catalogs, checkpointed to lxd-index.json |
httprange | Range, If-Range and conditional requests for apt, snap and LXD answers |
idletimeout | Fails an upstream request that makes no progress |
dialguard | Keeps apt's forward proxy from connecting to this machine for a client |
origin | Fetches from apt archives over HTTP |
internal: parsers and support
| Package | Purpose |
|---|---|
apt | Release, InRelease, Packages and Sources: names, sizes and hashes only |
snap | The Snap Store API v2 knowledge the snap side acts on, and the assertion chain check |
gomod | The GOPROXY protocol's five endpoints: parsing, validation, cache keys |
simplestreams | LXD's stream index and product catalogs |
stats, statsapi | The counters and their read-only JSON API on the stats socket (§13) |
adminapi | The mutating API on the admin socket: cache invalidation, offline mode |
cacheusage | Measures the cache directory for the cache page (§20.7) |
treemap | Lays out the cache page's treemap |
hostnet | Follows whether the host is offline, for -auto-offline |
l10n | The translation catalog and the log handler that translates (§22) |
logkey, logmsg, httpheader, enumutil, relpath | Shared spellings: log attribute keys, log messages, header names, enum fallbacks, relative path validation |
The design documents
docs/design.md holds §1, §12 and §18 and the index; each area has its own file under docs/design/, and the § numbers are shared by all of them. Cite a section as docs/design/lxd.md §17.11.
| § | File | Area |
|---|---|---|
| §1 | design.md | Goal and non-goals |
| §2–§8 | design/apt.md | What apt asks for, the forward proxy, normalization, policy, data model, Go interfaces (§7.1–§7.5), the fetch path (§8, §8.1) |
| §11 | design/snap.md | Snap packages and assertions (§11–§11.7) |
| §12 | design.md | Listener: TCP or UNIX socket |
| §13, §15 | design/stats.md | The stats API (§13.1–§13.6) and the default socket path |
| §14 | design/cli.md | pkgproxyctl as a multi-call binary, piped scripts, help layout (§14–§14.3) |
| §16–§16.4 | design/gomod.md | The Go module proxy and checksum database |
| §16.6–§16.10 | design/build-containers.md | LXD build containers and the admin API socket |
| §17 | design/lxd.md | LXD simplestreams image mirrors (§17–§17.14) |
| §18 | design.md | Tuning knobs: flags, environment, snap set |
| §19 | design/docker.md | The Docker Hub registry mirror (§19–§19.6.1) |
| §20 | design/setup-page.md | The setup page and the cache page (§20–§20.9) |
| §21 | design/offline.md | Offline mode |
| §22 | design/l10n.md | Localization |
| §23 | design/access-log.md | The access log and its outcomes |
| §24 | design/files.md | Files by checksum (§24–§24.6) |
| §25 | design/upgrades.md | Upgrades, reverts and snap epochs: the .meta.json files kept for 0.11, the post-refresh migration (§25–§25.6) |
A new area gets its own file, the next free § number and a line in the index in docs/design.md; a change to the design updates its document in the same commit. go test ./docs checks links and anchors, raw tags outside code, one title per file and the tables of contents.
Conventions
| Rule | Where it is written |
|---|---|
Work starts from a GitLab issue in a milestone (glab issue list, glab issue create) | AGENTS.md |
Errors with l10n.D.Errorf, worded "cannot …"; log messages wrapped in i18n.MarkForTranslation | AGENTS.md, §22; see Translations |
Every commit signed off (git commit -s); CI runs dco-check | AGENTS.md, .gitlab-ci.yml |
Linters are tool dependencies in go.mod (go get -tool), run with go tool | AGENTS.md |
./check.sh before pushing; the pre-commit hook before each commit | AGENTS.md; see How it is tested |
In the documentation
- The design plan: the scope, and an index of one document per area
- Integration tests
Roads from here
SETUP AND THE COMMAND LINE
- pkgproxyctl — the commands the binary provides
INSIDE PKGPROXY · for contributors
- A request in code — follow one request through the packages
- The sides' packages — how each side's package is built
- Translations — errors and messages that translate
- How pkgproxy is tested — how a change is tested
- Cache page code — the code behind the cache page