Contents · The Kubernetes charm

pkgproxy design: the Kubernetes charm

The Juju charm that runs the rock on Kubernetes. Part of the pkgproxy design, whose index says which file holds each § number.

30. The charm: the rock on Kubernetes, with Juju

charms/pkgproxy-k8s/ is pkgproxy-k8s, a Kubernetes sidecar charm written with ops and packed with charmcraft’s uv plugin. Its workload is the rock (§29), from the pkgproxy-image resource. It runs one unit, as the shared package cache of the clients in and around a Juju model.

What does not depend on Kubernetes is in src/common.py: the options that are flags, turning the configuration into settings, the ports, the status from the health route, the storage’s probe, and what the setup-script action offers and the address it gives. Each charm under charms/ has its own copy, as charmcraft packs only the charm’s own directory, and TestCharmsShareCommon keeps the copies identical.

The charm and the rock it runs come from the same commit, and the rock is named by its digest, never by a tag that moves:

  • On Charmhub, a channel holds a charm revision released together with a revision of its resource. The pipeline that tests a commit is to upload that commit’s rock, by the digest the rock job pushed to registry.gitlab.com/zygoon/pkgproxy and names in its ROCK_IMAGE, as the resource, and release the charm with exactly that revision: latest/edge from main, latest/stable from a version tag (§30.9).

  • The integration tests deploy the packed charm with the rock of the commit under test.

  • charmcraft.yaml has no upstream-source, so a local deploy of a packed charm names the rock (--resource pkgproxy-image=IMAGE) instead of picking up whatever :edge is that day.

  • base: ubuntu@26.04: the charm’s own container, the release of the rock and of the snap’s core26. charmcraft supports it as a base; only its charm plugin, which this charm does not use, is experimental there.

  • platforms: amd64: the registry has the rock for amd64 only.

  • assumes: juju >= 3.6, for uid and gid on the container (§30.2).

30.1 Where it lives

In this repository, beside the snap and the rock, rather than in one of its own. The charm is a thin layer over the rock’s contract: the PKGPROXY_* names, -check-config, the health route, the cache’s path and the rock’s user. TestCharmOptionsMatchFlags (cmd/pkgproxy/charm_test.go) keeps its options in step with the daemon’s flags as TestSnapOptionsMatchFlags does the snap’s, and one pipeline builds and tests the snap, the rock and the charm of the same commit.

30.2 Storage

The cache is the Juju storage cache, a filesystem of at least 16G, mounted at /var/cache/pkgproxy, the daemon’s default -cache-dir. On Kubernetes it is a PersistentVolumeClaim of the model’s storage class, which outlives the pod: after a restart or a juju refresh the cache, the statistics and the LXD port’s certificate are still there.

The container sets uid: 584792 and gid: 584792, the rock’s _daemon_. Without them Juju runs the workload as root, whatever the image’s user is. The volume is mounted over the directory the rock prepared, so it does not have the rock’s ownership: the charm’s own container, which runs as root and sees the same volume, gives its top directory to 584792:584792 before pkgproxy starts.

pkgproxy refuses a cache without extended attributes (DEPLOYMENT.md, “Where the cache lives”). The charm probes the volume by setting a user. attribute on a temporary file, and is blocked with a message that names the problem when it cannot.

30.3 Options

Each option is the daemon’s flag of the same name and reaches it as its PKGPROXY_* variable, only when it is set to something other than its default: the daemon’s default stays the only default, and an option set back to it leaves the environment. The list is curated: what an operator of a shared cache needs (public-url, setup-page, trusted-networks, offline-mode, the timing knobs, docker-upstream, the files-* settings). notCharmOptions in the test gives the reason for each flag that is not an option. lxd-tls, the charm’s own, sets PKGPROXY_LXD_TLS_ADDR=off when false.

The model’s proxy settings (juju-http-proxy and the others) reach the service as HTTP_PROXY, HTTPS_PROXY and NO_PROXY, in both spellings.

The charm has no validation of its own: it runs pkgproxy -check-config in the workload container with the environment it built, which parses the flags and variables exactly as the daemon will. A refused configuration blocks the charm with the daemon’s message and is not pushed, so pkgproxy keeps running with the last good one.

30.4 The Pebble layer

The charm adds a layer of its own over the rock’s:

  • the pkgproxy service with override: replace, with the rock’s command, without its empty default arguments ([ ]), and its kill-delay, and with its own environment, which beats the container’s (§29.3). So the rock’s --args do not apply under the charm, which configures pkgproxy through the environment alone. Not override: merge: Pebble combines a pushed layer into the one of the same label service by service, so a merged service keeps every variable it ever had, and an option set back to its default would never leave the environment;
  • on-success and on-failure: restart, where the rock shuts the container down: under Juju the charm needs the Pebble it asks about the service, and an invalid option never reaches the layer;
  • a health check of level: ready on the health route, which Juju turns into the container’s readiness probe.

It pushes the layer with combine, so each push replaces the previous one’s service and check whole, and replans, which restarts pkgproxy only when its plan changed. One handler does it all, on pkgproxy-pebble-ready, config-changed, cache-storage-attached, upgrade-charm and update-status, so the order of events does not matter.

30.5 Ports and Services

The unit opens 3142, and 3143 unless lxd-tls is false. Juju puts them on the application’s ClusterIP Service, so clients in the cluster use http://pkgproxy-k8s.MODEL.svc:3142. The ports are fixed: a pod has its own network namespace, so nothing competes for them.

Clients outside the cluster need an address of their own, and pkgproxy needs to see each client’s own address, for the access log (§23) and trusted-networks. The charm’s option service-type chooses:

  • cluster, the default: Juju’s ClusterIP Service alone. A plain deploy needs no juju trust.
  • loadbalancer: also a Service APP-lb of type LoadBalancer, with the unit’s open ports and externalTrafficPolicy: Local: without it, kube-proxy would hand pkgproxy a node’s address instead of the client’s.

Juju does not know of APP-lb, so the leader keeps it with lightkube (src/loadbalancer.py), with server-side apply, in every hook that reconciles (§30.4): a port lxd-tls closes leaves it too. Back on cluster, and on remove, the leader deletes it.

Managing a Service takes the cluster’s API, as the application’s service account. Juju 3.6 lets that account manage the Services of the model’s own namespace, so a plain juju config is enough there. Where the API refuses it (403), the charm is blocked with juju trust APP --scope=cluster in its message, while pkgproxy keeps serving the clients in the cluster. A refusal to delete is taken to mean there is nothing to delete, as a charm the API refuses cannot have made the Service.

30.6 Status

The first that applies:

  • blocked: more than one unit (gitlab.com/zygoon/pkgproxy#111), a service-type the charm does not know or the cluster’s API refused (§30.5), a cache without extended attributes, options -check-config refused;
  • waiting: no cache storage yet, no Pebble yet, pkgproxy not answering its health route yet;
  • active, with a message when the health route reports pkgproxy offline: offline: serving the cache only, or offline by itself: no-network. Offline is active, not blocked: what the cache holds is still served.

30.7 Actions

The rock has no shell, so what pkgproxyctl does on a snap install is one juju run away instead. Each action runs pkgproxyctl in the workload container with Pebble’s exec, and fails with what it printed when it fails:

  • stats: pkgproxyctl stats, the report, as output.
  • invalidate url=URL: pkgproxyctl cache invalidate URL, which throws away the cached copy of one apt file. Only apt files can be thrown away one by one, as the command says.
  • setup-script client=CLIENT [undo=true]: pkgproxyctl setup CLIENT -addr ADDR, the script that points a client machine’s tool at this pkgproxy, as script. CLIENT is any tool pkgproxyctl setup and the setup page offer (§20.1), in its order: apt, snap, go, docker, lxd, lxd-remote, snapcraft, rockcraft and charmcraft; TestCharmSetupScriptMatchesSetup keeps the action’s list in step. As on the setup page, a tool whose side is off is refused, with the reason: docker when docker-upstream is off, lxd-remote when lxd-tls is false. lxd and the craft tools need neither: their containers download through port 3142, so their scripts work from any machine. The scripts of lxd, lxd-remote and the craft tools run as the user who runs lxc, not as root, and each craft tool takes its base images through lxd-remote, which is set up too. ADDR is public-url’s host and port when it is set (port 80 when it names none, as for any http:// address), else the LoadBalancer Service’s address when service-type is loadbalancer and it has one (§30.5), else the application’s Service in the cluster, APP.MODEL.svc:3142. lxd-remote gets -lxd-tls-addr on the same host.

There is no action to turn offline mode on or off: offline-mode is an option, and an action that changed the running daemon behind it would leave juju config saying one thing while pkgproxy does another. Changing the option restarts pkgproxy, in seconds.

30.8 Tests

charms/pkgproxy-k8s/tests/unit tests the charm’s events with ops.testing, and the plain functions of src/pkgproxy.py and src/common.py, which turn the configuration into the environment and the layer. The CharmTest Dagger check runs them in uv’s image, with the packages uv.lock pins.

charms/pkgproxy-k8s/tests/integration tests the packed charm on Kubernetes with jubilant, whose juju CLI calls it drives from pytest. uv.lock pins jubilant in the dev group, so CharmLint type-checks these tests too, while pytest’s testpaths leave them out of CharmTest. One deployment, made by a session fixture with the rock of the same commit as pkgproxy-image, serves every test, in order:

  • the unit goes active, and pkgproxyctl stats in the workload finds the daemon;
  • apt, with configuration of its own, downloads a package through the application’s Service twice, and the access log shows the second as a hit;
  • after kubectl delete pod the unit comes back active by itself, and the package is still a hit: the cache is on the claim;
  • setup-page=false reaches the Pebble plan as PKGPROXY_SETUP_PAGE and turns / into a 404; reset, the variable leaves the plan;
  • revalidate-ttl=soon blocks the unit with the daemon’s message, which names PKGPROXY_REVALIDATE_TTL, and the plan and the running pkgproxy keep the last good configuration;
  • offline-mode=offline gives the active status its offline message, while the cached package is served and another is refused with a 503;
  • the actions: stats gives a report; invalidate of the package’s URL makes its next download fetched again, and of a URL that is no apt file fails; setup-script prints the scripts of apt, lxd and snapcraft for the Service’s cluster address;
  • service-type=loadbalancer makes APP-lb (or, where the API refuses the charm, blocks the unit with the juju trust command until the application is trusted). APP-lb gets an address, with externalTrafficPolicy: Local, and apt in a network namespace whose only route is through the machine, as a client off the cluster’s nodes, downloads the package through it as a hit, with the namespace’s own address in the access log, and setup-script prints a script for that address. Back on cluster, the Service is gone.

The tests reach the workload, whose rock has no shell, through its Pebble from the charm’s container (juju exec), and run apt on the machine the tests run on, which must reach the Service’s cluster address. The spread suite tests/charm/ (docs/testing.md) provides that: MicroK8s, with MetalLB for the LoadBalancer, and Juju (3.6, the oldest the charm supports, and 4: CI runs the suite with each) in the test VM, the rock under test in MicroK8s’s registry, the charm under test, and the network namespace of the client.

In CI, charm-ci packs the charm with charmcraft-gitlab-ci’s build component, with charmcraft pack --destructive-mode in a throwaway VM of its base, and fails when the pack leaves a file git does not ignore. charm-ci-check checks the metadata in the packed charm and that its virtualenv has none of the tests’ packages, and spread-charm runs tests/charm/ with that charm, so the charm that is tested is the one a release would upload. As the suite takes many minutes, spread-charm runs on main, on version tags, and where it is asked for: a merge request’s label, a variable, a schedule of its own (docs/testing.md).

30.9 Publishing

CI publishes the charm with charmcraft-gitlab-ci’s publish template, as a deployment to the environment charmhub/<channel>, which alone gets CHARMCRAFT_AUTH:

  • charm-edge releases to latest/edge: by hand from a pipeline of main, and on the nightly schedule, when the charm’s version, git describe of the commit, is not what latest/edge has already.
  • charm-stable releases to latest/stable from a version tag.

Each releases the charm with the rock of its commit, by digest: the one the rock job pushed, or, on the nightly schedule, which pushes none, :edge when rock-edge finds it is of the same commit. The same jobs publish the machine charm, pinned to the snap of its commit (§31.1).