pkgproxy design: the Kubernetes charm
The Juju charm that runs the rock on Kubernetes. Part of the pkgproxy design, whose index says which file holds each § number.
30. The charm: the rock on Kubernetes, with Juju
charms/pkgproxy-k8s/ is pkgproxy-k8s, a Kubernetes sidecar charm
written with ops and packed with charmcraft’s uv plugin. Its workload
is the rock (§29), from the pkgproxy-image resource. It runs
one unit, as the shared package cache of the clients in and around a
Juju model.
What does not depend on Kubernetes is in src/common.py: the options
that are flags, turning the configuration into settings, the ports, the
status from the health route, the storage’s probe, and what the
setup-script action offers and the address it gives. Each charm under
charms/ has its own copy, as charmcraft packs only the charm’s own
directory, and TestCharmsShareCommon keeps the copies identical.
The charm and the rock it runs come from the same commit, and the rock is named by its digest, never by a tag that moves:
-
On Charmhub, a channel holds a charm revision released together with a revision of its resource. The pipeline that tests a commit is to upload that commit’s rock, by the digest the
rockjob pushed toregistry.gitlab.com/zygoon/pkgproxyand names in itsROCK_IMAGE, as the resource, and release the charm with exactly that revision:latest/edgefrom main,latest/stablefrom a version tag (§30.9). -
The integration tests deploy the packed charm with the rock of the commit under test.
-
charmcraft.yamlhas noupstream-source, so a local deploy of a packed charm names the rock (--resource pkgproxy-image=IMAGE) instead of picking up whatever:edgeis that day. -
base: ubuntu@26.04: the charm’s own container, the release of the rock and of the snap’s core26. charmcraft supports it as a base; only itscharmplugin, which this charm does not use, is experimental there. -
platforms: amd64: the registry has the rock for amd64 only. -
assumes: juju >= 3.6, foruidandgidon the container (§30.2).
30.1 Where it lives
In this repository, beside the snap and the rock, rather than in one of
its own. The charm is a thin layer over the rock’s contract: the
PKGPROXY_* names, -check-config, the health route, the cache’s path
and the rock’s user. TestCharmOptionsMatchFlags
(cmd/pkgproxy/charm_test.go) keeps its options in step with the
daemon’s flags as TestSnapOptionsMatchFlags does the snap’s, and one
pipeline builds and tests the snap, the rock and the charm of the same
commit.
30.2 Storage
The cache is the Juju storage cache, a filesystem of at least 16G,
mounted at /var/cache/pkgproxy, the daemon’s default -cache-dir.
On Kubernetes it is a PersistentVolumeClaim of the model’s storage class,
which outlives the pod: after a restart or a juju refresh the cache,
the statistics and the LXD port’s certificate are still there.
The container sets uid: 584792 and gid: 584792, the rock’s
_daemon_. Without them Juju runs the workload as root, whatever the
image’s user is. The volume is mounted over the directory the rock
prepared, so it does not have the rock’s ownership: the charm’s own
container, which runs as root and sees the same volume, gives its top
directory to 584792:584792 before pkgproxy starts.
pkgproxy refuses a cache without extended attributes (DEPLOYMENT.md,
“Where the cache lives”). The charm probes the volume by setting a
user. attribute on a temporary file, and is blocked with a message
that names the problem when it cannot.
30.3 Options
Each option is the daemon’s flag of the same name and reaches it as its
PKGPROXY_* variable, only when it is set to something other than its
default: the daemon’s default stays the only default, and an option set
back to it leaves the environment. The list is curated: what an operator
of a shared cache needs (public-url, setup-page, trusted-networks,
offline-mode, the timing knobs, docker-upstream, the files-*
settings). notCharmOptions in the test gives the reason for each flag
that is not an option. lxd-tls, the charm’s own, sets
PKGPROXY_LXD_TLS_ADDR=off when false.
The model’s proxy settings (juju-http-proxy and the others) reach the
service as HTTP_PROXY, HTTPS_PROXY and NO_PROXY, in both
spellings.
The charm has no validation of its own: it runs pkgproxy -check-config
in the workload container with the environment it built, which parses
the flags and variables exactly as the daemon will. A refused
configuration blocks the charm with the daemon’s message and is not
pushed, so pkgproxy keeps running with the last good one.
30.4 The Pebble layer
The charm adds a layer of its own over the rock’s:
- the
pkgproxyservice withoverride: replace, with the rock’s command, without its empty default arguments ([ ]), and itskill-delay, and with its ownenvironment, which beats the container’s (§29.3). So the rock’s--argsdo not apply under the charm, which configures pkgproxy through the environment alone. Notoverride: merge: Pebble combines a pushed layer into the one of the same label service by service, so a merged service keeps every variable it ever had, and an option set back to its default would never leave the environment; on-successandon-failure: restart, where the rock shuts the container down: under Juju the charm needs the Pebble it asks about the service, and an invalid option never reaches the layer;- a
healthcheck oflevel: readyon the health route, which Juju turns into the container’s readiness probe.
It pushes the layer with combine, so each push replaces the previous
one’s service and check whole, and replans, which restarts pkgproxy only when its plan
changed. One handler does it all, on pkgproxy-pebble-ready,
config-changed, cache-storage-attached, upgrade-charm and
update-status, so the order of events does not matter.
30.5 Ports and Services
The unit opens 3142, and 3143 unless lxd-tls is false. Juju puts them
on the application’s ClusterIP Service, so clients in the cluster use
http://pkgproxy-k8s.MODEL.svc:3142. The ports are fixed: a pod has its
own network namespace, so nothing competes for them.
Clients outside the cluster need an address of their own, and pkgproxy
needs to see each client’s own address, for the access log
(§23) and trusted-networks. The charm’s option
service-type chooses:
cluster, the default: Juju’s ClusterIP Service alone. A plain deploy needs nojuju trust.loadbalancer: also a ServiceAPP-lbof type LoadBalancer, with the unit’s open ports andexternalTrafficPolicy: Local: without it, kube-proxy would hand pkgproxy a node’s address instead of the client’s.
Juju does not know of APP-lb, so the leader keeps it with lightkube
(src/loadbalancer.py), with server-side apply, in every hook that
reconciles (§30.4): a port lxd-tls closes leaves it too. Back on
cluster, and on remove, the leader deletes it.
Managing a Service takes the cluster’s API, as the application’s service
account. Juju 3.6 lets that account manage the Services of the model’s
own namespace, so a plain juju config is enough there. Where the API
refuses it (403), the charm is blocked with juju trust APP --scope=cluster in its message, while pkgproxy keeps serving the
clients in the cluster. A refusal to delete is taken to mean there is
nothing to delete, as a charm the API refuses cannot have made the
Service.
30.6 Status
The first that applies:
- blocked: more than one unit (gitlab.com/zygoon/pkgproxy#111), a
service-typethe charm does not know or the cluster’s API refused (§30.5), a cache without extended attributes, options-check-configrefused; - waiting: no cache storage yet, no Pebble yet, pkgproxy not answering its health route yet;
- active, with a message when the health route reports pkgproxy offline:
offline: serving the cache only, oroffline by itself: no-network. Offline is active, not blocked: what the cache holds is still served.
30.7 Actions
The rock has no shell, so what pkgproxyctl does on a snap install is
one juju run away instead. Each action runs pkgproxyctl in the workload container with
Pebble’s exec, and fails with what it printed when it fails:
stats:pkgproxyctl stats, the report, asoutput.invalidate url=URL:pkgproxyctl cache invalidate URL, which throws away the cached copy of one apt file. Only apt files can be thrown away one by one, as the command says.setup-script client=CLIENT [undo=true]:pkgproxyctl setup CLIENT -addr ADDR, the script that points a client machine’s tool at this pkgproxy, asscript. CLIENT is any toolpkgproxyctl setupand the setup page offer (§20.1), in its order: apt, snap, go, docker, lxd, lxd-remote, snapcraft, rockcraft and charmcraft;TestCharmSetupScriptMatchesSetupkeeps the action’s list in step. As on the setup page, a tool whose side is off is refused, with the reason: docker whendocker-upstreamisoff, lxd-remote whenlxd-tlsis false. lxd and the craft tools need neither: their containers download through port 3142, so their scripts work from any machine. The scripts of lxd, lxd-remote and the craft tools run as the user who runs lxc, not as root, and each craft tool takes its base images through lxd-remote, which is set up too. ADDR ispublic-url’s host and port when it is set (port 80 when it names none, as for anyhttp://address), else the LoadBalancer Service’s address whenservice-typeisloadbalancerand it has one (§30.5), else the application’s Service in the cluster,APP.MODEL.svc:3142.lxd-remotegets-lxd-tls-addron the same host.
There is no action to turn offline mode on or off: offline-mode is an
option, and an action that changed the running daemon behind it would
leave juju config saying one thing while pkgproxy does another.
Changing the option restarts pkgproxy, in seconds.
30.8 Tests
charms/pkgproxy-k8s/tests/unit tests the charm’s events with
ops.testing, and the plain functions of src/pkgproxy.py and
src/common.py, which turn the configuration into the environment and
the layer. The CharmTest Dagger check runs them in
uv’s image, with the packages uv.lock pins.
charms/pkgproxy-k8s/tests/integration tests the packed charm on Kubernetes with
jubilant, whose juju CLI calls
it drives from pytest. uv.lock pins jubilant in the dev group, so
CharmLint type-checks these tests too, while pytest’s testpaths
leave them out of CharmTest. One deployment, made by a session fixture
with the rock of the same commit as pkgproxy-image, serves every test,
in order:
- the unit goes active, and
pkgproxyctl statsin the workload finds the daemon; - apt, with configuration of its own, downloads a package through the application’s Service twice, and the access log shows the second as a hit;
- after
kubectl delete podthe unit comes back active by itself, and the package is still a hit: the cache is on the claim; setup-page=falsereaches the Pebble plan asPKGPROXY_SETUP_PAGEand turns/into a 404; reset, the variable leaves the plan;revalidate-ttl=soonblocks the unit with the daemon’s message, which namesPKGPROXY_REVALIDATE_TTL, and the plan and the running pkgproxy keep the last good configuration;offline-mode=offlinegives the active status its offline message, while the cached package is served and another is refused with a 503;- the actions:
statsgives a report;invalidateof the package’s URL makes its next downloadfetchedagain, and of a URL that is no apt file fails;setup-scriptprints the scripts of apt, lxd and snapcraft for the Service’s cluster address; service-type=loadbalancermakesAPP-lb(or, where the API refuses the charm, blocks the unit with thejuju trustcommand until the application is trusted).APP-lbgets an address, withexternalTrafficPolicy: Local, and apt in a network namespace whose only route is through the machine, as a client off the cluster’s nodes, downloads the package through it as a hit, with the namespace’s own address in the access log, andsetup-scriptprints a script for that address. Back oncluster, the Service is gone.
The tests reach the workload, whose rock has no shell, through its
Pebble from the charm’s container (juju exec), and run apt on the
machine the tests run on, which must reach the Service’s cluster
address. The spread suite tests/charm/ (docs/testing.md) provides
that: MicroK8s, with MetalLB for the LoadBalancer, and Juju (3.6, the
oldest the charm supports, and 4: CI runs the suite with each) in the
test VM, the rock under test in MicroK8s’s registry, the charm under
test, and the network namespace of the client.
In CI, charm-ci packs the charm with
charmcraft-gitlab-ci’s
build component, with charmcraft pack --destructive-mode in a
throwaway VM of its base, and fails when the pack leaves a file git
does not ignore. charm-ci-check checks the metadata in the packed
charm and that its virtualenv has none of the tests’ packages, and
spread-charm runs tests/charm/ with that charm, so the charm that
is tested is the one a release would upload. As the suite takes many
minutes, spread-charm runs on main, on version tags, and where it is
asked for: a merge request’s label, a variable, a schedule of its own
(docs/testing.md).
30.9 Publishing
CI publishes the charm with
charmcraft-gitlab-ci’s
publish template, as a deployment to the environment
charmhub/<channel>, which alone gets CHARMCRAFT_AUTH:
charm-edgereleases tolatest/edge: by hand from a pipeline of main, and on the nightly schedule, when the charm’s version,git describeof the commit, is not whatlatest/edgehas already.charm-stablereleases tolatest/stablefrom a version tag.
Each releases the charm with the rock of its commit, by digest: the one
the rock job pushed, or, on the nightly schedule, which pushes none,
:edge when rock-edge finds it is of the same commit. The same jobs
publish the machine charm, pinned to the snap of its commit
(§31.1).