The source tree

contributors for contributors

Where the code lives (one multi-call binary in cmd/pkgproxy and one internal package per proxy engine or shared concern), where the design is written down, and the conventions every change follows.

On the map: inside pkgproxy

▶ Watch this deck — 6 scenes, ~4 min

01 What this topic covers

ABOUT THIS TOPIC The source tree the commands in cmd/, the packages under internal/, the design in docs/ 1 Two directories hold the code 2 One file per command 3 One design, many files 4 The conventions of a change
About this topic, The source tree: two directories hold the code, one file per command, one design in many files, the conventions of a change

Where pkgproxy's code and design live, and the conventions every change follows.

The pkgproxy source keeps its commands in one directory, the packages they use under internal, and the design in docs. If you do not know what pkgproxy does yet, watch A request first. You will see the two directories that hold the code, and how the command directory keeps one file per command. Then you will see the design split across documents that share section numbers, and the conventions every change follows.

02 Two directories hold the code

THE SOURCE TREE cmd/pkgproxy/ one binary, two names pkgproxy · daemon pkgproxyctl · symlink internal/ one package per concern the proxy engines proxy snapproxy gomodproxy registryproxy lxdproxy filesproxy the path every proxy engine shares gateway · reqlog · fill · flight · store · cachekey · revalidate · idletimeout parsers apt · snap · gomod · simplestreams beside them offline · upstreams · chain · stats · l10n
The source tree: cmd/pkgproxy builds one binary run as pkgproxy or pkgproxyctl; internal/ holds one package per proxy engine, the shared path, the protocol parsers and supporting packages

cmd/pkgproxy is the multi-call binary; internal/ holds the proxy engines (proxy is apt), the shared path, the protocol parsers and the supporting packages. Nothing under internal/ is importable from outside the module.

The code lives in two directories. The command directory builds one binary with two names: run as pkgproxy it is the daemon, and run as pkgproxyctl, a symlink to the same file, it is the command line an operator uses. Everything else is under internal, one package per concern. Each proxy engine, the part that serves one kind of client, has its own package. The path every proxy engine shares has packages of its own: the gateway, the request log, fills, the store and cache keys. Parsers, offline mode, the upstream status, statistics and translations sit beside them.

03 One file per command

ONE FILE PER COMMAND cmd/pkgproxy/ what it is main.go router.Cmd{UseArgv0: true} ctl.go newCtlRouter: stats setup cache … cmd_stats.go pkgproxyctl stats cmd_setup_lxd.go pkgproxyctl setup lxd cmd_setup_lxd_test.go its test, beside it guide.go usage.go instructions.go shared: guide builder, help printer, URLs inv := invoke(t, "pkgproxyctl", "setup", "apt", "-addr", testAddr) the real tree
Files of cmd/pkgproxy mapped to commands: main.go routes on argv0, ctl.go builds the pkgproxyctl tree, one cmd_ file per command, shared helpers in files without the cmd_ prefix

main.go routes on argv[0] with go-cmdr's router.Cmd; ctl.go builds pkgproxyctl's tree. Every leaf prints its help with printUsage; tests use invoke(t, "pkgproxyctl", …).

In the source, the entry point wraps both names in one router that dispatches on the name the binary was run as, and a second router builds pkgproxyctl's own tree. The command directory groups code by command, not by topic: one file per command, named after it, so setup lxd lives in cmd_setup_lxd.go, with its test beside it. Code several commands share lives in files without the cmd prefix, such as the guide builder and the usage printer. Tests run the real command tree, the way a user runs it.

04 One design, many files

THE DESIGN DOCUMENTS docs/design.md §1 goal and non-goals §12 the listener §18 tuning knobs and the index of every other § docs/design/ apt.md§2–§8 snap.md§11 gomod.md§16 lxd.md§17 docker.md§19 offline.md§21 l10n.md§22 files.md§24 … and six more · one numbering for all a new area: its own file, the next free §, a line in the index a change to the design updates its document in the same commit $ go test ./docs links · anchors · tables of contents · one title
docs/design.md as an index pointing at one file per area under docs/design, each holding its own section numbers from one shared sequence; go test ./docs checks links, anchors and tables of contents

docs/design.md holds §1, §12 and §18 and the index; each area's file holds its own §, cited as docs/design/lxd.md §17.11; go test ./docs guards the published site.

The design is written down in docs/design.md and one document per area under docs/design: apt, snap, Go modules, LXD, Docker, files, offline mode and the rest. The section numbers are shared by all the files, so apt.md section eight names one place. A new area gets its own file, the next free number and a line in the index. A change to the design updates its document in the same commit. Go test of the docs package checks links, anchors and tables of contents, because the documents are published as a website.

05 The conventions of a change

A CHANGE, FROM ISSUE TO PUSH a GitLab issue why, not only what in a milestone, such as v1 glab issue list return nil, l10n.D.Errorf( "cannot fetch a file: %w", err) not fmt.Errorf · not "failed to" githooks/pre-commit gofmt vet staticcheck goconst errcheck wsl ineffassign · seconds $ git commit -s Signed-off-by: … CI's dco-check fails without it $ ./check.sh dagger check: the same module GitLab CI runs, tests included
A change from left to right: a GitLab issue in a milestone, code with errors from l10n.D.Errorf worded cannot, the pre-commit hook, a commit signed off, and ./check.sh running the same checks as CI

Issues and milestones live in GitLab (glab issue list), not in a TODO file. git config core.hooksPath githooks, then git commit -s; ./check.sh before pushing.

A change starts from a GitLab issue, which says why, not only what, and belongs to a milestone. Errors in the code are built with l10n.D.Errorf, and a failed action is worded cannot, then a verb. The pre-commit hook, enabled once per clone, runs the fast linters before each commit. Every commit carries a Signed-off-by line, or CI fails it. ./check.sh runs the same Dagger checks CI runs, tests included, before a push.

06 Where to go next general version

WHERE NEXT you know where the code for a change lives, and what conventions it has to follow you are here The source tree A request in code one request through the packages How it is tested what a change needs before a commit Translations errors and messages that translate
Where next from The source tree: A request in code, How it is tested, Translations

Next: A request in code, for one request through the packages; How it is tested, for what a change needs; Translations, for errors and messages that translate.

You now know where the code for a change lives, which design section it touches, and what conventions it has to follow. To follow one request through those packages, watch A request in code. To see how a change is tested before you commit it, watch How it is tested. To write errors and messages that can be translated, watch Translations.

Read more

Top-level directories and files

PathWhat it holds
cmd/pkgproxy/The one multi-call binary: the daemon pkgproxy and the command line pkgproxyctl, a symlink to the same file
internal/Every other package, one per concern; not importable from outside module gitlab.com/zygoon/pkgproxy
internal/integration/Tests only: podman runs of real apt and go (skipped under -short or without podman), and snapd/, a nested Go module that drives snapd's own store client
data/Assets embedded with //go:embed: the icon (also the snap's), the upstream logos of the cache page's live views, the Ubuntu mark
docs/The design (design.md, design/*.md), statsapi.md, statsapi-internals.md, testing.md, troubleshooting.md, and docs_test.go, which guards the published site
snap/snapcraft.yaml, the configure hook, local/bin/pkgproxy-env (snap options to PKGPROXY_* variables), the desktop file
rockcraft.yamlThe rock: an OCI image of the same binary under Pebble (§29)
charms/pkgproxy-k8s/The Kubernetes charm, in Python with ops, which runs the rock (§30): src/charm.py, src/pkgproxy.py (the Pebble layer), src/loadbalancer.py, src/common.py, unit and integration tests, its own pyproject.toml and uv.lock
charms/pkgproxy/The machine charm, which installs the snap (§31): src/charm.py, the same src/common.py, snap.toml (the snap revisions it pins) and pin-snap, which rewrites them at publish time
tests/The spread suite: tests/<client>/<task>/task.yaml, helpers in tests/lib/, fixtures for the snapcraft, rockcraft and charmcraft packs, tests/rock/ and the manual tests/charm/
testdata/apt/Release, InRelease, Packages and Sources excerpts of Debian and Ubuntu, for the parsers' unit tests
githooks/pre-commit, enabled with git config core.hooksPath githooks
.dagger/The Dagger module whose checks ./check.sh and GitLab CI run
spread.yaml, .image-garden.mkThe spread project and the VM images image-garden makes for it
check.shexec dagger check
DEPLOYMENT.md, README.md, NEWSPublished at https://zygoon.pl/pkgproxy/ with docs/

cmd/pkgproxy: one file per command

CommandFile
pkgproxy (the daemon)cmd_pkgproxy.go, with filesflags.go and envflags.go (PKGPROXY_* defaults)
pkgproxyctl (the router)ctl.go
pkgproxyctl statscmd_stats.go
pkgproxyctl upstreamscmd_upstreams.go (§28.9)
pkgproxyctl setupcmd_setup.go; one child each: cmd_setup_apt.go, _snap, _go, _docker, _lxd, _lxd_remote, _snapcraft, _rockcraft, _charmcraft, _status
pkgproxyctl cache invalidatecmd_cache.go
pkgproxyctl offlinecmd_offline.go
pkgproxyctl opencmd_open.go
pkgproxyctl versioncmd_version.go, with version.go

Files without the cmd_ prefix hold what several commands share:

FileWhat it holds
main.goThe router.Cmd on argv[0], the locale and the translator
usage.goprintUsage, the help layout of every leaf command (docs/design/cli.md §14.3)
guide.goThe guide builder: translated paragraphs and verbatim code blocks
instructions.goClient configuration paths and the startup banner
host.gohost, what commands take from outside, which tests replace
setuppage.go, setuppage_cache.go, setuppage.html.tmpl, setuptool.html.tmplThe setup page, a page per tool, its scripts under /setup/, and the cache page
pageshell.goThe bar every page opens with, its status element, /pkgproxy/status and the /setup/all script (§20.10)
banner.goThe degraded or offline banner of every page and of pkgproxyctl stats and upstreams (§28.4)
kindhint.go, lang.goWhat an error kind such as http-429 means, in the visitor's language; the page language from ?lang=, the pkgproxy-lang cookie or Accept-Language, and the footer's switcher (§20.3)
upstreams.goThe upstream status recorder in -cache-dir/upstreams, and the snap session placeholders' prefix file
upstreamcheck.go-upstream-check-interval and the checks of quiet upstreams (§28.8)
upstreamspage.go, upstreamdetail.go, upstreamhow.go, upstreamspage.html.tmpl, upstreamdetail.html.tmpl/pkgproxy/upstreams, a page per host, the setup page's 30-day summary, the cache page's status dots, and what each page says of how it knows (§28.3)
upstreampkgproxy.go-upstream-pkgproxy: the chain, upstream-pkgproxy.json, the self check (§27)
cachemap.go, cachelive.go, cachepage.jsThe cache page's map, its activity and its live events
lxdtls.go, lxdremote.go, lxdbridge.go, lxdhook.goThe LXD HTTPS listener and certificate, -lxd-remote, the bridge and profile of setup lxd
statssocket.go, snapoption.goDefault socket and cache paths; reading snap options
signals_*.go, umask_*.goPlatform parts: shutdown signals, the umask around socket creation

internal: the proxy engines

PackagePurpose
proxyapt: the forward proxy, cache key, store, origin and index wired into one handler (§7.5, §8)
snapproxysnap: a reverse relay snapd talks to through SNAPPY_FORCE_API_URL, caching snaps and assertions (§11)
gomodproxyGo: a GOPROXY origin chained to an upstream module proxy, with checksum database checks (§16)
registryproxyDocker: the read-only OCI distribution API for anonymous pulls (§19)
lxdproxyLXD: simplestreams image servers under <base>/<name> (§17)
lxdapiThe part of LXD's REST API that LXD 6.10 and later download images through (§17.14)
filesproxyFiles by checksum, GET /files/v1/fetch?url=…&sha256=… (§24), and volatile files, GET /files/v1/volatile?url=… (§24.8)

internal: the path every request shares

PackagePurpose
gatewayDispatches each request to one proxy engine on the one listener, and stamps its answers X-Pkgproxy-Engine
reqlogWraps a listener's handler: one access log line per request, and events for observers (§23)
offlineThe offline switch: modes, per-host breakers, the transport, dial and middleware wrappers; tells the upstream status recorder of every upstream request (§21)
fillServes a large blob while it downloads into the store (§8.1)
flightsingleflight with typed results and a joined bit
revalidateOne detached revalidation per key, bounded wait, recent-failure memory
store, store/fslocal, store/memstoreThe store interface, its disk implementation, its in-memory one for tests (§7.1)
cachekeyapt request URL to cache key and policy (§4, §7.3)
indexExpected hashes of apt pool and by-hash paths from parsed indexes (§4.2, §7.4)
lxdindexExpected SHA256 of LXD blob paths from parsed catalogs, checkpointed to lxd-index.json
httprangeRange, If-Range and conditional requests for apt, snap and LXD answers
idletimeoutFails an upstream request that makes no progress
upstreamretrySends a request an upstream answered 408 again
upstreamlogLogs who answered when an upstream answers with an error
chainSends what the proxy engines fetch through an upstream pkgproxy (§27)
proxyengines/pkgproxy/v1/proxy-engines, the description a downstream reads, and the loop guard (§27.1, §27.5)
dialguardKeeps apt's forward proxy from connecting to this machine for a client
originFetches from apt archives over HTTP

internal: parsers and support

PackagePurpose
aptRelease, InRelease, Packages and Sources: names, sizes and hashes only
snapThe Snap Store API v2 knowledge the snap proxy engine acts on, and the assertion chain check
gomodThe GOPROXY protocol's five endpoints: parsing, validation, cache keys
simplestreamsLXD's stream index and product catalogs
stats, statsapiThe counters and their read-only JSON API on the stats socket (§13)
upstreamsThe upstream status: buckets, states and the files under -cache-dir/upstreams (§28)
adminapiThe mutating API on the admin socket: cache invalidation, offline mode
cacheusageMeasures the cache directory for the cache page (§20.7)
treemapLays out the cache page's treemap
hostnetFollows whether the host is offline, for -auto-offline
l10nThe translation catalog and the log handler that translates (§22)
logkey, logmsg, httpheader, enumutil, relpathShared spellings: log attribute keys, log messages, header names, enum fallbacks, relative path validation

The design documents

docs/design.md holds §1, §12 and §18 and the index; each area has its own file under docs/design/, and the § numbers are shared by all of them. Cite a section as docs/design/lxd.md §17.11.

§FileArea
§1design.mdGoal and non-goals
§2–§8design/apt.mdWhat apt asks for, the forward proxy, normalization, policy, data model, Go interfaces (§7.1–§7.5), the fetch path (§8, §8.1)
§11design/snap.mdSnap packages and assertions (§11–§11.7)
§12design.mdListener: TCP or UNIX socket
§13, §15design/stats.mdThe stats API (§13.1–§13.6) and the default socket path
§14design/cli.mdpkgproxyctl as a multi-call binary, piped scripts, help layout (§14–§14.3)
§16–§16.4design/gomod.mdThe Go module proxy and checksum database
§16.6–§16.10design/build-containers.mdLXD build containers and the admin API socket
§17design/lxd.mdLXD simplestreams image mirrors (§17–§17.14)
§18design.mdTuning knobs: flags, environment, snap set
§19design/docker.mdThe Docker Hub registry mirror (§19–§19.6.1)
§20design/setup-page.mdThe setup page, the cache page and the bar every page shares (§20–§20.10)
§21design/offline.mdOffline mode
§22design/l10n.mdLocalization
§23design/access-log.mdThe access log and its outcomes
§24design/files.mdFiles by checksum and volatile files (§24–§24.8)
§25design/upgrades.mdUpgrades, reverts and snap epochs: the .meta.json files kept for 0.11, the post-refresh migration (§25–§25.6)
§27design/upstream.mdAn upstream pkgproxy: two tiers (§27–§27.6)
§28design/upstreams.mdUpstream status (§28–§28.13)
§29design/rock.mdThe rock
§30design/charm.mdThe Kubernetes charm
§31design/machine-charm.mdThe machine charm

A new area gets its own file, the next free § number and a line in the index in docs/design.md; a change to the design updates its document in the same commit. go test ./docs checks links and anchors, raw tags outside code, one title per file and the tables of contents.

Conventions

RuleWhere it is written
Work starts from a GitLab issue in a milestone (glab issue list, glab issue create)AGENTS.md
Errors with l10n.D.Errorf, worded "cannot …"; log messages wrapped in i18n.MarkForTranslationAGENTS.md, §22; see Translations
Every commit signed off (git commit -s); CI runs dco-checkAGENTS.md, .gitlab-ci.yml
Linters are tool dependencies in go.mod (go get -tool), run with go toolAGENTS.md
./check.sh before pushing; the pre-commit hook before each commitAGENTS.md; see How it is tested

In the documentation

Roads from here

SETUP AND THE COMMAND LINE

INSIDE PKGPROXY · for contributors