The source tree

contributors for contributors

Where the code lives (one multi-call binary in cmd/pkgproxy and one internal package per side or shared concern), where the design is written down, and the conventions every change follows.

On the map: inside pkgproxy

▶ Watch this deck — 6 scenes, ~4 min

01 What this topic covers

ABOUT THIS TOPIC The source tree the commands in cmd/, the packages under internal/, the design in docs/ 1 Two directories hold the code 2 One file per command 3 One design, many files 4 The conventions of a change
About this topic, The source tree: two directories hold the code, one file per command, one design in many files, the conventions of a change

Where pkgproxy's code and design live, and the conventions every change follows.

The pkgproxy source keeps its commands in one directory, the packages they use under internal, and the design in docs. If you do not know what pkgproxy does yet, watch A request first. You will see the two directories that hold the code, and how the command directory keeps one file per command. Then you will see the design split across documents that share section numbers, and the conventions every change follows.

02 Two directories hold the code

THE SOURCE TREE cmd/pkgproxy/ one binary, two names pkgproxy · daemon pkgproxyctl · symlink internal/ one package per concern the sides proxy snapproxy gomodproxy registryproxy lxdproxy filesproxy the path every side shares gateway · reqlog · fill · flight · store · cachekey · revalidate · idletimeout parsers apt · snap · gomod · simplestreams beside them offline · stats · statsapi · l10n · dialguard
The source tree: cmd/pkgproxy builds one binary run as pkgproxy or pkgproxyctl; internal/ holds one package per side, the shared path, the protocol parsers and supporting packages

cmd/pkgproxy is the multi-call binary; internal/ holds the sides (proxy is apt), the shared path, the protocol parsers and the supporting packages. Nothing under internal/ is importable from outside the module.

The code lives in two directories. The command directory builds one binary with two names: run as pkgproxy it is the daemon, and run as pkgproxyctl, a symlink to the same file, it is the command line an operator uses. Everything else is under internal, one package per concern. Each side that serves a kind of client has its own package. The path every side shares has packages of its own: the gateway, the request log, fills, the store and cache keys. Parsers, offline mode, statistics and translations sit beside them.

03 One file per command

ONE FILE PER COMMAND cmd/pkgproxy/ what it is main.go router.Cmd{UseArgv0: true} ctl.go newCtlRouter: stats setup cache … cmd_stats.go pkgproxyctl stats cmd_setup_lxd.go pkgproxyctl setup lxd cmd_setup_lxd_test.go its test, beside it guide.go usage.go instructions.go shared: guide builder, help printer, URLs inv := invoke(t, "pkgproxyctl", "setup", "apt", "-addr", testAddr) the real tree
Files of cmd/pkgproxy mapped to commands: main.go routes on argv0, ctl.go builds the pkgproxyctl tree, one cmd_ file per command, shared helpers in files without the cmd_ prefix

main.go routes on argv[0] with go-cmdr's router.Cmd; ctl.go builds pkgproxyctl's tree. Every leaf prints its help with printUsage; tests use invoke(t, "pkgproxyctl", …).

In the source, the entry point wraps both names in one router that dispatches on the name the binary was run as, and a second router builds pkgproxyctl's own tree. The command directory groups code by command, not by topic: one file per command, named after it, so setup lxd lives in cmd_setup_lxd.go, with its test beside it. Code several commands share lives in files without the cmd prefix, such as the guide builder and the usage printer. Tests run the real command tree, the way a user runs it.

04 One design, many files

THE DESIGN DOCUMENTS docs/design.md §1 goal and non-goals §12 the listener §18 tuning knobs and the index of every other § docs/design/ apt.md§2–§8 snap.md§11 gomod.md§16 lxd.md§17 docker.md§19 offline.md§21 l10n.md§22 files.md§24 … and six more · one numbering for all a new area: its own file, the next free §, a line in the index a change to the design updates its document in the same commit $ go test ./docs links · anchors · tables of contents · one title
docs/design.md as an index pointing at one file per area under docs/design, each holding its own section numbers from one shared sequence; go test ./docs checks links, anchors and tables of contents

docs/design.md holds §1, §12 and §18 and the index; each area's file holds its own §, cited as docs/design/lxd.md §17.11; go test ./docs guards the published site.

The design is written down in docs/design.md and one document per area under docs/design: apt, snap, Go modules, LXD, Docker, files, offline mode and the rest. The section numbers are shared by all the files, so apt.md section eight names one place. A new area gets its own file, the next free number and a line in the index. A change to the design updates its document in the same commit. Go test of the docs package checks links, anchors and tables of contents, because the documents are published as a website.

05 The conventions of a change

A CHANGE, FROM ISSUE TO PUSH a GitLab issue why, not only what in a milestone, such as v1 glab issue list return nil, l10n.D.Errorf( "cannot fetch a file: %w", err) not fmt.Errorf · not "failed to" githooks/pre-commit gofmt vet staticcheck goconst errcheck wsl ineffassign · seconds $ git commit -s Signed-off-by: … CI's dco-check fails without it $ ./check.sh dagger check: the same module GitLab CI runs, tests included
A change from left to right: a GitLab issue in a milestone, code with errors from l10n.D.Errorf worded cannot, the pre-commit hook, a commit signed off, and ./check.sh running the same checks as CI

Issues and milestones live in GitLab (glab issue list), not in a TODO file. git config core.hooksPath githooks, then git commit -s; ./check.sh before pushing.

A change starts from a GitLab issue, which says why, not only what, and belongs to a milestone. Errors in the code are built with l10n.D.Errorf, and a failed action is worded cannot, then a verb. The pre-commit hook, enabled once per clone, runs the fast linters before each commit. Every commit carries a Signed-off-by line, or CI fails it. ./check.sh runs the same Dagger checks CI runs, tests included, before a push.

06 Where to go next general version

WHERE NEXT you know where the code for a change lives, and what conventions it has to follow you are here The source tree A request in code one request through the packages How it is tested what a change needs before a commit Translations errors and messages that translate
Where next from The source tree: A request in code, How it is tested, Translations

Next: A request in code, for one request through the packages; How it is tested, for what a change needs; Translations, for errors and messages that translate.

You now know where the code for a change lives, which design section it touches, and what conventions it has to follow. To follow one request through those packages, watch A request in code. To see how a change is tested before you commit it, watch How it is tested. To write errors and messages that can be translated, watch Translations.

Read more

Top-level directories and files

PathWhat it holds
cmd/pkgproxy/The one multi-call binary: the daemon pkgproxy and the command line pkgproxyctl, a symlink to the same file
internal/Every other package, one per concern; not importable from outside module gitlab.com/zygoon/pkgproxy
internal/integration/Tests only: podman runs of real apt and go (skipped under -short or without podman), and snapd/, a nested Go module that drives snapd's own store client
data/Assets embedded with //go:embed: the icon (also the snap's), the upstream logos of the cache page's live views, the Ubuntu mark
docs/The design (design.md, design/*.md), statsapi.md, statsapi-internals.md, testing.md, troubleshooting.md, and docs_test.go, which guards the published site
snap/snapcraft.yaml, the configure hook, local/bin/pkgproxy-env (snap options to PKGPROXY_* variables), the desktop file
tests/The spread suite: tests/<client>/<task>/task.yaml, helpers in tests/lib/, fixtures for the snapcraft, rockcraft and charmcraft packs
testdata/apt/Release, InRelease, Packages and Sources excerpts of Debian and Ubuntu, for the parsers' unit tests
githooks/pre-commit, enabled with git config core.hooksPath githooks
.dagger/The Dagger module whose checks ./check.sh and GitLab CI run
spread.yaml, .image-garden.mkThe spread project and the VM images image-garden makes for it
check.shexec dagger check
DEPLOYMENT.md, README.md, NEWSPublished at https://zygoon.pl/pkgproxy/ with docs/

cmd/pkgproxy: one file per command

CommandFile
pkgproxy (the daemon)cmd_pkgproxy.go, with filesflags.go and envflags.go (PKGPROXY_* defaults)
pkgproxyctl (the router)ctl.go
pkgproxyctl statscmd_stats.go
pkgproxyctl setupcmd_setup.go; one child each: cmd_setup_apt.go, _snap, _go, _docker, _lxd, _lxd_remote, _snapcraft, _rockcraft, _charmcraft, _status
pkgproxyctl cache invalidatecmd_cache.go
pkgproxyctl offlinecmd_offline.go
pkgproxyctl opencmd_open.go
pkgproxyctl versioncmd_version.go, with version.go

Files without the cmd_ prefix hold what several commands share:

FileWhat it holds
main.goThe router.Cmd on argv[0], the locale and the translator
usage.goprintUsage, the help layout of every leaf command (docs/design/cli.md §14.3)
guide.goThe guide builder: translated paragraphs and verbatim code blocks
instructions.goClient configuration paths and the startup banner
host.gohost, what commands take from outside, which tests replace
setuppage.go, setuppage_cache.goThe setup page, its scripts under /setup/, and the cache page
cachemap.go, cachelive.go, cachepage.jsThe cache page's map, its activity and its live events
lxdtls.go, lxdremote.go, lxdbridge.go, lxdhook.goThe LXD HTTPS listener and certificate, -lxd-remote, the bridge and profile of setup lxd
statssocket.go, snapoption.goDefault socket and cache paths; reading snap options
signals_*.go, umask_*.goPlatform parts: shutdown signals, the umask around socket creation

internal: the sides

PackagePurpose
proxyapt: the forward proxy, cache key, store, origin and index wired into one handler (§7.5, §8)
snapproxysnap: a reverse relay snapd talks to through SNAPPY_FORCE_API_URL, caching snaps and assertions (§11)
gomodproxyGo: a GOPROXY origin chained to an upstream module proxy, with checksum database checks (§16)
registryproxyDocker: the read-only OCI distribution API for anonymous pulls (§19)
lxdproxyLXD: simplestreams image servers under <base>/<name> (§17)
lxdapiThe part of LXD's REST API that LXD 6.10 and later download images through (§17.14)
filesproxyFiles by checksum: GET /files/v1/fetch?url=…&sha256=… (§24)

internal: the path every request shares

PackagePurpose
gatewayDispatches each request to one side on the one listener
reqlogWraps a listener's handler: one access log line per request, and events for observers (§23)
offlineThe offline switch: modes, per-host breakers, the transport, dial and middleware wrappers (§21)
fillServes a large blob while it downloads into the store (§8.1)
flightsingleflight with typed results and a joined bit
revalidateOne detached revalidation per key, bounded wait, recent-failure memory
store, store/fslocal, store/memstoreThe store interface, its disk implementation, its in-memory one for tests (§7.1)
cachekeyapt request URL to cache key and policy (§4, §7.3)
indexExpected hashes of apt pool and by-hash paths from parsed indexes (§4.2, §7.4)
lxdindexExpected SHA256 of LXD blob paths from parsed catalogs, checkpointed to lxd-index.json
httprangeRange, If-Range and conditional requests for apt, snap and LXD answers
idletimeoutFails an upstream request that makes no progress
dialguardKeeps apt's forward proxy from connecting to this machine for a client
originFetches from apt archives over HTTP

internal: parsers and support

PackagePurpose
aptRelease, InRelease, Packages and Sources: names, sizes and hashes only
snapThe Snap Store API v2 knowledge the snap side acts on, and the assertion chain check
gomodThe GOPROXY protocol's five endpoints: parsing, validation, cache keys
simplestreamsLXD's stream index and product catalogs
stats, statsapiThe counters and their read-only JSON API on the stats socket (§13)
adminapiThe mutating API on the admin socket: cache invalidation, offline mode
cacheusageMeasures the cache directory for the cache page (§20.7)
treemapLays out the cache page's treemap
hostnetFollows whether the host is offline, for -auto-offline
l10nThe translation catalog and the log handler that translates (§22)
logkey, logmsg, httpheader, enumutil, relpathShared spellings: log attribute keys, log messages, header names, enum fallbacks, relative path validation

The design documents

docs/design.md holds §1, §12 and §18 and the index; each area has its own file under docs/design/, and the § numbers are shared by all of them. Cite a section as docs/design/lxd.md §17.11.

§FileArea
§1design.mdGoal and non-goals
§2–§8design/apt.mdWhat apt asks for, the forward proxy, normalization, policy, data model, Go interfaces (§7.1–§7.5), the fetch path (§8, §8.1)
§11design/snap.mdSnap packages and assertions (§11–§11.7)
§12design.mdListener: TCP or UNIX socket
§13, §15design/stats.mdThe stats API (§13.1–§13.6) and the default socket path
§14design/cli.mdpkgproxyctl as a multi-call binary, piped scripts, help layout (§14–§14.3)
§16–§16.4design/gomod.mdThe Go module proxy and checksum database
§16.6–§16.10design/build-containers.mdLXD build containers and the admin API socket
§17design/lxd.mdLXD simplestreams image mirrors (§17–§17.14)
§18design.mdTuning knobs: flags, environment, snap set
§19design/docker.mdThe Docker Hub registry mirror (§19–§19.6.1)
§20design/setup-page.mdThe setup page and the cache page (§20–§20.9)
§21design/offline.mdOffline mode
§22design/l10n.mdLocalization
§23design/access-log.mdThe access log and its outcomes
§24design/files.mdFiles by checksum (§24–§24.6)
§25design/upgrades.mdUpgrades, reverts and snap epochs: the .meta.json files kept for 0.11, the post-refresh migration (§25–§25.6)

A new area gets its own file, the next free § number and a line in the index in docs/design.md; a change to the design updates its document in the same commit. go test ./docs checks links and anchors, raw tags outside code, one title per file and the tables of contents.

Conventions

RuleWhere it is written
Work starts from a GitLab issue in a milestone (glab issue list, glab issue create)AGENTS.md
Errors with l10n.D.Errorf, worded "cannot …"; log messages wrapped in i18n.MarkForTranslationAGENTS.md, §22; see Translations
Every commit signed off (git commit -s); CI runs dco-checkAGENTS.md, .gitlab-ci.yml
Linters are tool dependencies in go.mod (go get -tool), run with go toolAGENTS.md
./check.sh before pushing; the pre-commit hook before each commitAGENTS.md; see How it is tested

In the documentation

Roads from here

SETUP AND THE COMMAND LINE

INSIDE PKGPROXY · for contributors