Setting up clients

command line for users for administrators for CI operators

One pkgproxyctl setup command per client prints a guide on a terminal and a shell script in a pipe; running it twice is running it once and -undo puts the machine back.

On the map: setup and the command line

▶ Watch this deck — 9 scenes, ~5 min

01 What this topic covers

ABOUT THIS TOPIC pkgproxyctl setup points each client on this machine at pkgproxy 1 guide or script 2 guarded 3 per client 4 the address 5 twice, undo 6 status
About this topic: pkgproxyctl setup, for users, administrators and CI operators; its scenes: guide or script, guarded, per client, the address, twice, undo, status

One pkgproxyctl setup command per client, printed as a guide or piped as a script, safe to run again and to undo.

pkgproxyctl setup is the command that points a client on this machine at pkgproxy. If you do not know what pkgproxy does yet, watch A request first. You see the guide it prints and the script you pipe to the shell, how that script is guarded, the command for each client, and the address it names. Then you see how to run it twice or undo it, and how to check what is set up.

02 A guide on a terminal, a script in a pipe

PRINT, THEN PIPE on a terminal: a guide $ pkgproxyctl setup apt To cache apt packages with pkgproxy, add this to /etc/apt/apt.conf.d/99pkgproxy on each client: Acquire::http::Proxy "http://192.0.2.1:3142"; This makes apt download plain HTTP sources through pkgproxy, which caches them. … piped: a script for sh $ pkgproxyctl setup apt | sudo sh { if command -v apt-get …; then mkdir -p /etc/apt/apt.conf.d && echo '…' > …/99pkgproxy && if dpkg … lt 3.1.3; then …; fi else echo '… nothing to do' >&2 fi } run it without | sh first to read what it will do /etc/apt/apt.conf.d/99pkgproxy Acquire::http::Proxy "http://192.0.2.1:3142";
Left, pkgproxyctl setup apt on a terminal prints a guide in prose with the configuration line; right, the same command piped to sudo sh prints a shell script; below, the one file it writes

On a terminal pkgproxyctl setup apt prints a guide; piped, it prints a script. Either way apt ends up with an Acquire::http::Proxy line in /etc/apt/apt.conf.d/99pkgproxy.

Every client is set up with one command. Run pkgproxyctl setup apt on a terminal and it explains what to change and why, with the configuration line to add. Pipe the same command to sudo sh and it prints only the commands, as a shell script, and the shell runs them. So you can read what a setup does before you apply it. For apt the result is one small file, naming the proxy's address on port 3142.

03 One block, guarded and chained

ONE BLOCK, GUARDED { # Only where there is apt: elsewhere there is nothing to set up. if command -v apt-get >/dev/null 2>&1; then [ "$(id -u)" = 0 ] || echo '…: run this as root, …' >&2 # Make apt download packages through pkgproxy. mkdir -p /etc/apt/apt.conf.d && echo 'Acquire::http::Proxy "…";' > …/99pkgproxy && if dpkg --compare-versions … lt 3.1.3; then …; fi else echo '…: there is no apt on this machine, nothing to do' >&2 fi } one block: read whole before the first step runs runs only where apt is elsewhere: nothing to do, and the script succeeds not root? it says so chained with &&: the first failure fails the script the same script is safe on any machine: it sets up what is there and says so
The piped apt script as one block, annotated: the braces, the guard that runs it only where apt is, the root check, the steps chained with &&, and the else branch that says there is nothing to do

Each piped script is one { … } block. It runs only where its tool is, warns unless it runs as root, and chains its steps with && so that a failed step fails the script.

The script is one braced block, so the shell has read all of it before the first step runs, and no command inside can swallow the rest. It starts with a guard: on a machine without apt it says there is nothing to do and succeeds, so the same script is safe on any machine in a fleet. Run without root, it says it must be. The steps are chained, so the first one that fails stops the rest and fails the script, rather than reporting success with nothing done.

04 One command per client

ONE COMMAND PER CLIENT client pkgproxyctl … what it changes apt setup apt | sudo sh /etc/apt/apt.conf.d/99pkgproxy snapd setup snap | sudo sh …/snapd.service.d/pkgproxy.conf Docker setup docker | sudo sh registry-mirrors in daemon.json go setup go | sh go env -w GOPROXY=…/go,direct containers setup lxd | sh the LXD profile pkgproxy image remotes setup lxd-remote | sh lxc remotes pkgproxy-ubuntu, … snapcraft setup snapcraft | sh profile of the snapcraft project root you sudo for system files · as yourself for your own go, lxc and LXD settings
A table of clients: apt, snapd and Docker are set up with sudo and change a file each; go, LXD containers, LXD image remotes and snapcraft are set up as yourself and change your own settings

apt, snap and docker change system files and need root. go, lxd, lxd-remote, snapcraft, rockcraft and charmcraft change your own settings and run as you.

There is one setup command per client, and each changes one thing. Apt gets a file in its configuration directory, snapd a drop in for its service, which is then restarted, and Docker its list of registry-mirrors; those three are piped to sudo sh. The Go command gets GOPROXY in your own go environment. For LXD, setup lxd writes a profile for containers, setup lxd-remote adds image servers, and the craft tools' setups configure their build project. These run as yourself, as lxc does.

05 The address a setup names

WHICH ADDRESS pkgproxyctl setup apt GET /pkgproxy/v1/info over the stats socket pkgproxy {"addr":":3142", "lxd_tls_addr":":3143", …} :3142 http://<host name>:3142 a wildcard host becomes this machine's host name $ pkgproxyctl setup apt -addr 192.0.2.1:3142 | sudo sh the flag wins $ pkgproxyctl setup apt -addr nonsense | sudo sh cannot use -addr "nonsense": it must be host:port or :port, … sh reads: exit 1
pkgproxyctl setup asks the daemon over the stats socket for its info, which names the listen address; a wildcard host becomes the host name; an explicit -addr wins; a bad -addr fails with exit 1 in the pipe

-addr defaults to what /pkgproxy/v1/info reports, else :3142. A wildcard host becomes the host name. A bad address prints the error on stderr and exit 1 to the pipe.

A setup has to name the address the proxy listens on. It asks the running daemon over the stats socket, whose info answer names the listen addresses, so a daemon moved to another port is set up correctly without flags. A wildcard host, as in the default, becomes this machine's host name. The -addr flag wins when it is given, and with no daemon running the defaults are used. An address the daemon would refuse fails the setup, and the pipe gets exit 1, so the shell fails too.

06 Twice is once, and undo puts it back

TWICE, UNDO, UNDO FIRST setup twice is setup once as it was setup set up setup set up, the same undo restores the machine, and a second undo changes nothing set up -undo as it was -undo as it was, the same undo before any setup succeeds never set up -undo nothing changed $ pkgproxyctl setup apt -undo | sudo sh safe to run again
Three rows of machine states: setup then setup again gives the same state; setup -undo returns the machine to how it was, and a second -undo changes nothing; -undo on a machine never set up changes nothing

pkgproxyctl setup X -undo | sh reverses setup X (with sudo sh for apt, snap and docker). Both can be run again safely.

Setup can be run again, after a port changes or just to be sure: running it a second time changes nothing more. The -undo flag prints the commands that reverse a setup, and after them the machine is as it was before. A second undo succeeds and changes nothing more, and so does an undo on a machine that was never set up. So a machine can run a setup or an undo again without checking its state first.

07 What is set up here

SETUP STATUS $ pkgproxyctl setup status | sh aptset up /etc/apt/apt.conf.d/99pkgproxy snapset up /etc/systemd/system/snapd.service.d/… gonot set up GOPROXY=https://proxy.golang.org,direct dockernot installed lxdset up profile pkgproxy of project default, … snapcraftnot set up rockcraftnot set up charmcraftnot set up lxd-remoteneeds setup again the states set up not set up set up for another address needs setup again not installed cannot check needs setup again: there, but cannot work as it is; run that setup command again only reads · runs as yourself, without sudo · does not try the connection
The output of pkgproxyctl setup status piped to sh: one line per setup command with its state and a note under it, and the six possible states listed beside it

pkgproxyctl setup status | sh reports configuration only, one line per setup command. It runs as you, without sudo.

To see what a machine is set up for, pipe pkgproxyctl setup status to sh. It prints one line per setup command with a state: set up, not set up, set up for another address, needs setup again, not installed, or cannot check, with notes under it saying where or why. Needs setup again means the setup is there but cannot work as it is, such as an LXD image remote at an old address; running that setup again fixes it. The script only reads, runs as yourself, and does not try the connection.

08 Where to go next for CI operators in place of the general version

WHERE NEXT your clients download through pkgproxy you are here pkgproxyctl setup Build containers what setup lxd and the craft tools set up LXD images the image servers setup lxd-remote adds Statistics check that the builds use the cache
Where next from pkgproxyctl setup: Build containers, what setup lxd and the craft tools set up; LXD images, the image servers setup lxd-remote adds; Statistics, check that the builds use the cache

Next: Build containers, for what setup lxd and the craft tools set up; LXD images, for the remotes setup lxd-remote adds; Statistics, to check the builds use the cache.

Each client you set up now downloads through pkgproxy, and you can read, rerun or undo every change setup made. To see what setup lxd and the craft tools put into a build container, watch Build containers. For the image servers that setup lxd-remote adds, watch LXD images. To check that the builds use the cache, watch Statistics.

09 Where to go next general version

WHERE NEXT your clients download through pkgproxy you are here pkgproxyctl setup The setup page a machine without pkgproxy installed Statistics check that a client uses the cache pkgproxyctl the other commands
Where next from pkgproxyctl setup: The setup page, a machine without pkgproxy installed; Statistics, check that a client uses the cache; pkgproxyctl, the other commands

Next: The setup page, for machines without pkgproxy; Statistics, to check that a client uses the cache; pkgproxyctl, for the other commands.

Each client you set up now downloads through pkgproxy, and you can read, rerun or undo every change setup made. To set up a machine without pkgproxy installed, watch The setup page. To check that a client really uses the cache, watch Statistics. For the other commands, watch pkgproxyctl.

Read more

What each setup changes

Every script but Go's runs only where its tool is; elsewhere it says there is nothing to do and succeeds. Go's needs no go command and sets up Go before it is installed. <host> is the address the setup names, as in pkgproxy:3142.

CommandRuns asWhat it changes
setup aptroot (warns otherwise)Writes /etc/apt/apt.conf.d/99pkgproxy with Acquire::http::Proxy "http://<host>";. Where apt is older than 3.1.3 it adds Acquire::Retries::Delay "false";, so apt does not hang while pkgproxy is offline.
setup snaproot (warns otherwise)Writes /etc/systemd/system/snapd.service.d/pkgproxy.conf setting SNAPPY_FORCE_API_URL=http://<host>, then systemctl daemon-reload and, where snapd is installed, restarts it. Needs systemd.
setup dockerroot (fails otherwise)Puts http://<host> first in registry-mirrors in /var/snap/docker/current/config/daemon.json (Docker snap) or /etc/docker/daemon.json, creating the file if needed and keeping other settings and mirrors, then restarts Docker. A file it cannot understand is left alone, with an error.
setup goyouWrites GOPROXY=http://<host>/go,direct to your own Go environment file, where go env -w would, so it works before Go is installed.
setup lxdyou, in the lxd groupCreates or replaces the LXD profile pkgproxy: environment.GOPROXY, environment.SNAPPY_FORCE_API_URL and two container hooks (below). Add the profile to containers yourself.
setup lxd -project Pyou, in the lxd groupThe same settings in the default profile of LXD project P, as the craft setups do.
setup snapcraft, setup rockcraft, setup charmcraftyou, in the lxd groupsetup lxd for the LXD project of the same name: creates the project if missing (copying the default profile of the default project), deletes its base-instance containers so the next build starts from a fresh base, and sets the three settings on the project's default profile.
setup lxd-remoteyou; refuses to run under sudoFor each image remote, lxc remote add <name> https://<host-name>:3143/lxd/<name> --protocol=simplestreams, with pkgproxy's certificate from http://<host>/pkgproxy-lxd-cert.pem saved as servercerts/<name>.crt in your lxc configuration directory (~/snap/lxd/common/config or ~/.config/lxc). A remote of the same name at another address is replaced. On an LXD with image registries it also adds a cluster link named pkgproxy-… and an image registry per remote.
setup statusyouChanges nothing: prints a report (see below).

Inside an LXD container

The profile's hooks run on the LXD host, around each container that uses it, so nothing needs installing in the image.

HookWhat it does in the container's root file system
before startWrites /etc/apt/apt.conf.d/99pkgproxy-lxd (Acquire::http::Proxy and Acquire::Retries::Delay "false") and /etc/systemd/system/snapd.service.d/pkgproxy-lxd.conf (SNAPPY_FORCE_API_URL). Rewrites https:// to http:// for the archive, security, ports and old-releases hosts of ubuntu.com in /etc/apt/sources.list and /etc/apt/sources.list.d/*.sources.
after stopRemoves the two files it wrote.

Containers reach pkgproxy at the address the setup names, or, when that address is this machine itself (a loopback address), at the address of the LXD bridge the project's default profile uses.

What -undo removes

Command-undo
setup aptRemoves /etc/apt/apt.conf.d/99pkgproxy.
setup snapRemoves the snapd drop-in, reloads systemd and restarts snapd.
setup dockerRemoves only pkgproxy's entry from registry-mirrors, and the key when no mirror is left, then restarts Docker.
setup goRemoves the GOPROXY line from your Go environment file, as go env -u GOPROXY does, if it has one.
setup lxdDeletes the pkgproxy profile.
setup lxd -project P and the craft setupsUnsets environment.GOPROXY, environment.SNAPPY_FORCE_API_URL and raw.lxc on the project's default profile. The project is kept.
setup lxd-remoteRemoves each remote that points at pkgproxy and its certificate, and puts craft-com.ubuntu.cloud-buildd and craft-com.ubuntu.cloud-buildd-daily back at https://cloud-images.ubuntu.com/buildd/releases and …/buildd/daily. Removes pkgproxy's image registries, the images LXD downloaded through them, and its cluster links.

Options

OptionCommandsMeaning
-addrallpkgproxy's listen address. Taken from the running daemon when not given, else :3142. A wildcard host becomes this machine's host name.
-undoall but statusPrint the commands that reverse the setup.
-go-base-path, -base-pathgo, status (-go-base-path only)pkgproxy's Go base path, /go by default; not taken from the daemon.
-projectlxdLXD project whose default profile to change, instead of a pkgproxy profile.
-lxd-base-path, -base-pathlxd-remote, status (-lxd-base-path only)pkgproxy's LXD base path, /lxd by default; not taken from the daemon.
-lxd-remotelxd-remote, statusThe image remotes, as given to the daemon; not taken from it. By default craft-com.ubuntu.cloud-buildd, craft-com.ubuntu.cloud-buildd-daily, pkgproxy-images, pkgproxy-ubuntu and pkgproxy-ubuntu-daily.
-lxd-tls-addrlxd-remote, statuspkgproxy's HTTPS address for LXD images. Taken from the daemon when not given, else :3143.

setup status

pkgproxyctl setup status | sh prints one line per setup command with one of these states, and notes under it.

StateMeaning
set upSet up for the pkgproxy at -addr.
not set upNothing of this setup is there.
set up for another addressSet up, but for another pkgproxy address.
needs setup againThere, but it cannot work as it is, such as an LXD remote at an old address; run that setup again.
not installedThe tool is not on this machine.
cannot checkThe script could not tell, with the reason in a note.

for users for administrators

Getting pkgproxyctl

pkgproxyctl is the pkgproxy binary under another name. The snap installs both. Without the snap, every release from 0.12.0 on publishes the binary for Linux on amd64 and arm64, with a SHA256SUMS file to check it against, on the releases page. To install the newest release, with arch set to the machine's architecture:

arch=amd64
url=https://gitlab.com/zygoon/pkgproxy/-/releases/permalink/latest/downloads
curl -fL -O "$url/pkgproxy-linux-$arch" -O "$url/SHA256SUMS"
sha256sum --check --ignore-missing SHA256SUMS
sudo install -m 755 "pkgproxy-linux-$arch" /usr/local/bin/pkgproxy
sudo ln -sf pkgproxy /usr/local/bin/pkgproxyctl

Or build it with Go: go install gitlab.com/zygoon/pkgproxy/cmd/pkgproxy@latest, then ln -s pkgproxy "$(go env GOPATH)/bin/pkgproxyctl".

From the setup page

A machine without pkgproxyctl gets the same scripts from the daemon's setup page (The setup page):

URLWhat it serves
http://<host>:3142/setup/<tool>#!/bin/sh and the lines pkgproxyctl setup <tool> prints when piped, as text/plain
http://<host>:3142/setup/<tool>?undo=1The -undo script

The tools are apt, snap, go, docker (unless the Docker mirror is off), lxd, lxd-remote (only with image remotes and the HTTPS listener on), snapcraft, rockcraft and charmcraft; status is not offered. The scripts name the address the visitor reached pkgproxy at, or the public-url setting. Pipe those for apt, snap and docker to sudo sh, and the others to sh.

Roads from here

LXD

SETUP AND THE COMMAND LINE

OPERATIONS

  • Statistics — checking that a client uses the cache