pkgproxySnap StoreSource
Contents · Design index

pkgproxy — Design Plan

This file holds the scope (§1), the listener (§12) and the tuning knobs (§18), which every side shares. Each area has its own document under design/, with its own table of contents. The § numbers are shared by all of them, so a reference such as §16.9 names one place; the map below says which file holds each section.

1. Goal

Build a Go proxy that sits between apt clients and upstream Debian/Ubuntu-style archives, reachable over plain HTTP only, and caches everything it fetches forever. Multiple mirror hosts that serve the same archive content should collapse onto one backing store entry instead of being cached once per mirror. Storage is behind an interface; the first implementation is local disk, a future one is S3-compatible object storage.

Non-goals (v1)

  • Caching/verifying HTTPS content. We never terminate TLS, MITM, cache, or hash-verify anything fetched over https://. CONNECT is supported, but only as a blind, uncached TCP tunnel to port 443 — see §3. It has to be: apt sends CONNECT to whatever Acquire::http::Proxy names for any https:// source, with no Acquire::https::Proxy needed, so a client with one https-only source among its http ones cannot keep that traffic away from us, and refusing it breaks its apt update.
  • PDiff (incremental index patches). We always serve full index files. Not caching patches is safe — it just costs a client an extra full download instead of a diff. Only Debian publishes *.diff/Index; Ubuntu’s archive does not.
  • Repository signing / GPG. We are not a Verify layer. apt verifies Release/InRelease signatures itself; our only obligation is to serve those files byte-identical to the origin so signatures still validate.
  • Garbage collection. “Cache indefinitely” is the explicit ask; nothing is ever evicted. The intended direction for later is a mark-and-sweep that keeps pool and by-hash objects still referenced from a cached Release/Packages/Sources of a tracked suite, which first needs a rule for how many historical suites stay reachable.

12. Listener: TCP or UNIX socket

Every handler operates purely on net/http’s request/response model, so the listener is a protocol-agnostic choice made once at startup and shared by all of them through internal/gateway.

-addr (default :3142) is the TCP bind address, and it always drives the printed setup instructions, whatever the transport: that text describes the http://host:port clients connect to, which is a property of how clients reach pkgproxy, not of the local net.Listen call.

-socket-path (default empty) replaces the TCP listener with a UNIX socket, for deployments that want none, such as a host proxy bridged into a VM guest over QEMU’s guestfwd unix: form. pkgproxy removes a stale socket file left by a previous run before binding (this is not singleton enforcement). -addr still drives the instructions in this mode, so pass the hostname:port clients actually reach.

-addr and -lxd-tls-addr (unless off) are checked before anything binds, and by -check-config (§18), with checkListenAddr: host:port as net.SplitHostPort splits it, the host empty, an IP address or a host name, and the port a decimal number from 0 to 65535. net.Listen also takes a service name (:http) or no port, but the instructions name the port and need its number. A host name is not resolved, so the result does not depend on the network; one that does not resolve fails at start.

The socket is created mode 0777: bind(2) applies the umask, a socket left 0755 by snapd’s umask 0022 refuses connect(2) from an unprivileged relay (connecting needs write permission), and this is a single-host trust boundary. listenAt (cmd/pkgproxy) clears the umask around the bind, for every UNIX socket the daemon opens, the stats (§13) and admin sockets included.

18. Tuning knobs: flags, environment, snap set

Every interval is a daemon flag; DEPLOYMENT.md’s settings table lists them with their defaults. -upstream-idle-timeout bounds inactivity, not total time: every side’s upstream client goes through one internal/idletimeout.Transport, which fails a request once the upstream has made no progress for that long, while headers are awaited or inside a body Read. Time the caller spends between reads does not count, so a slow consumer never cancels a healthy upstream, and a 2 GB layer at 1 MB/s may take its 33 minutes. A fill whose upstream stalls fails like on any upstream error and leaves the in-progress table, so the next request starts a fresh one instead of everyone joining a dead connection. pkgproxyctl’s dial timeout and the LXD certificate validity stay constants: neither is a per-site concern.

Environment. applyEnvDefaults (cmd/pkgproxy/envflags.go) gives every daemon flag a variable PKGPROXY_<FLAG> (upper-cased, dashes to underscores), applied before the command line is parsed, so precedence is flag > environment > default with no per-flag code. The rule is generic on purpose: a new flag is configurable by environment the moment it exists.

snap set. snapd configuration is only readable through snapctl, and the daemon’s command line is static, so the snap maps options onto the environment: pkgproxy-env, in the daemon’s command-chain:, exports PKGPROXY_* from snapctl get for an explicit key list and execs the daemon. Option names equal flag names. Only keys the snap’s own command line does not pin can be listed, since a flag beats the environment, so -cache-dir and the content sockets are not options. -offline is not one either: offline-mode (§21) is the one name for that setting. The configure hook runs pkgproxy -check-config (parse flags and environment, then exit) through the same script, so an invalid snap set, a listen address included, fails up front instead of putting the service in a restart loop. A test keeps pkgproxy-env’s key list, the daemon’s flags (each either mapped or listed as not a snap option, with a reason) and DEPLOYMENT.md’s settings table in step.

Restart or not. The configure hook cannot ask snapd which options changed, so it compares the PKGPROXY_* environment it would start the daemon with, minus PKGPROXY_OFFLINE_MODE, against the copy saved in $SNAP_DATA/configured.env by its previous run. Equal means at most the offline mode changed, and the hook switches the running daemon with pkgproxyctl offline mode <mode>. A difference, a missing copy, or a pkgproxyctl that fails means a restart; a stopped daemon is left alone. Rejecting unknown keys waits for the hooks to move to Go (gitlab.com/zygoon/pkgproxy#49): snapctl get cannot list the keys that are set. pkgproxyctl setup and pkgproxyctl open learn the configured ports from the running daemon’s /pkgproxy/v1/info (§13.5), not from snapctl get, which works only inside the snap; the daemon is the authority on what it actually listens on.