Setting up clients
command line for users for administrators for CI operators
One pkgproxyctl setup command per client prints a guide on a terminal and a shell script in a pipe; running it twice is running it once and -undo puts the machine back.
On the map: setup and the command line
01 What this topic covers
One pkgproxyctl setup command per client, printed as a guide or piped as a script, safe to run again and to undo.
pkgproxyctl setup is the command that points a client on this machine at pkgproxy. If you do not know what pkgproxy does yet, watch A request first. You see the guide it prints and the script you pipe to the shell, how that script is guarded, the command for each client, and the address it names. Then you see how to run it twice or undo it, and how to check what is set up.
02 A guide on a terminal, a script in a pipe
On a terminal pkgproxyctl setup apt prints a guide; piped, it
prints a script. Either way apt ends up with an
Acquire::http::Proxy line in
/etc/apt/apt.conf.d/99pkgproxy.
Every client is set up with one command. Run pkgproxyctl setup apt on a terminal and it explains what to change and why, with the configuration line to add. Pipe the same command to sudo sh and it prints only the commands, as a shell script, and the shell runs them. So you can read what a setup does before you apply it. For apt the result is one small file, naming the proxy's address on port 3142.
03 One block, guarded and chained
Each piped script is one { … } block. It runs only where its
tool is, warns unless it runs as root, and chains its steps with
&& so that a failed step fails the script.
The script is one braced block, so the shell has read all of it before the first step runs, and no command inside can swallow the rest. It starts with a guard: on a machine without apt it says there is nothing to do and succeeds, so the same script is safe on any machine in a fleet. Run without root, it says it must be. The steps are chained, so the first one that fails stops the rest and fails the script, rather than reporting success with nothing done.
04 One command per client
apt, snap and docker change system
files and need root. go, lxd,
lxd-remote, snapcraft, rockcraft and
charmcraft change your own settings and run as you.
There is one setup command per client, and each changes one thing. Apt gets a file in its configuration directory, snapd a drop in for its service, which is then restarted, and Docker its list of registry-mirrors; those three are piped to sudo sh. The Go command gets GOPROXY in your own go environment. For LXD, setup lxd writes a profile for containers, setup lxd-remote adds image servers, and the craft tools' setups configure their build project. These run as yourself, as lxc does.
05 The address a setup names
-addr defaults to what /pkgproxy/v1/info reports,
else :3142. A wildcard host becomes the host name. A bad
address prints the error on stderr and exit 1 to the pipe.
A setup has to name the address the proxy listens on. It asks the running daemon over the stats socket, whose info answer names the listen addresses, so a daemon moved to another port is set up correctly without flags. A wildcard host, as in the default, becomes this machine's host name. The -addr flag wins when it is given, and with no daemon running the defaults are used. An address the daemon would refuse fails the setup, and the pipe gets exit 1, so the shell fails too.
06 Twice is once, and undo puts it back
pkgproxyctl setup X -undo | sh reverses setup X
(with sudo sh for apt, snap and docker). Both can be run
again safely.
Setup can be run again, after a port changes or just to be sure: running it a second time changes nothing more. The -undo flag prints the commands that reverse a setup, and after them the machine is as it was before. A second undo succeeds and changes nothing more, and so does an undo on a machine that was never set up. So a machine can run a setup or an undo again without checking its state first.
07 What is set up here
pkgproxyctl setup status | sh reports configuration only, one
line per setup command. It runs as you, without sudo.
To see what a machine is set up for, pipe pkgproxyctl setup status to sh. It prints one line per setup command with a state: set up, not set up, set up for another address, needs setup again, not installed, or cannot check, with notes under it saying where or why. Needs setup again means the setup is there but cannot work as it is, such as an LXD image remote at an old address; running that setup again fixes it. The script only reads, runs as yourself, and does not try the connection.
08 Where to go next for CI operators in place of the general version
Next: Build containers, for what setup lxd and the craft tools set up; LXD images, for the remotes setup lxd-remote adds; Statistics, to check the builds use the cache.
Each client you set up now downloads through pkgproxy, and you can read, rerun or undo every change setup made. To see what setup lxd and the craft tools put into a build container, watch Build containers. For the image servers that setup lxd-remote adds, watch LXD images. To check that the builds use the cache, watch Statistics.
09 Where to go next general version
Next: The setup page, for machines without pkgproxy; Statistics, to check that a client uses the cache; pkgproxyctl, for the other commands.
Each client you set up now downloads through pkgproxy, and you can read, rerun or undo every change setup made. To set up a machine without pkgproxy installed, watch The setup page. To check that a client really uses the cache, watch Statistics. For the other commands, watch pkgproxyctl.
Read more
What each setup changes
Every script but Go's runs only where its tool is; elsewhere it says there is nothing to do and succeeds. Go's needs no go command and sets up Go before it is installed. <host> is the address the setup names, as in pkgproxy:3142.
| Command | Runs as | What it changes |
|---|---|---|
setup apt | root (warns otherwise) | Writes /etc/apt/apt.conf.d/99pkgproxy with Acquire::http::Proxy "http://<host>";. Where apt is older than 3.1.3 it adds Acquire::Retries::Delay "false";, so apt does not hang while pkgproxy is offline. |
setup snap | root (warns otherwise) | Writes /etc/systemd/system/snapd.service.d/pkgproxy.conf setting SNAPPY_FORCE_API_URL=http://<host>, then systemctl daemon-reload and, where snapd is installed, restarts it. Needs systemd. |
setup docker | root (fails otherwise) | Puts http://<host> first in registry-mirrors in /var/snap/docker/current/config/daemon.json (Docker snap) or /etc/docker/daemon.json, creating the file if needed and keeping other settings and mirrors, then restarts Docker. A file it cannot understand is left alone, with an error. |
setup go | you | Writes GOPROXY=http://<host>/go,direct to your own Go environment file, where go env -w would, so it works before Go is installed. |
setup lxd | you, in the lxd group | Creates or replaces the LXD profile pkgproxy: environment.GOPROXY, environment.SNAPPY_FORCE_API_URL and two container hooks (below). Add the profile to containers yourself. |
setup lxd -project P | you, in the lxd group | The same settings in the default profile of LXD project P, as the craft setups do. |
setup snapcraft, setup rockcraft, setup charmcraft | you, in the lxd group | setup lxd for the LXD project of the same name: creates the project if missing (copying the default profile of the default project), deletes its base-instance containers so the next build starts from a fresh base, and sets the three settings on the project's default profile. |
setup lxd-remote | you; refuses to run under sudo | For each image remote, lxc remote add <name> https://<host-name>:3143/lxd/<name> --protocol=simplestreams, with pkgproxy's certificate from http://<host>/pkgproxy-lxd-cert.pem saved as servercerts/<name>.crt in your lxc configuration directory (~/snap/lxd/common/config or ~/.config/lxc). A remote of the same name at another address is replaced. On an LXD with image registries it also adds a cluster link named pkgproxy-… and an image registry per remote. |
setup status | you | Changes nothing: prints a report (see below). |
Inside an LXD container
The profile's hooks run on the LXD host, around each container that uses it, so nothing needs installing in the image.
| Hook | What it does in the container's root file system |
|---|---|
| before start | Writes /etc/apt/apt.conf.d/99pkgproxy-lxd (Acquire::http::Proxy and Acquire::Retries::Delay "false") and /etc/systemd/system/snapd.service.d/pkgproxy-lxd.conf (SNAPPY_FORCE_API_URL). Rewrites https:// to http:// for the archive, security, ports and old-releases hosts of ubuntu.com in /etc/apt/sources.list and /etc/apt/sources.list.d/*.sources. |
| after stop | Removes the two files it wrote. |
Containers reach pkgproxy at the address the setup names, or, when that address is this machine itself (a loopback address), at the address of the LXD bridge the project's default profile uses.
What -undo removes
| Command | -undo |
|---|---|
setup apt | Removes /etc/apt/apt.conf.d/99pkgproxy. |
setup snap | Removes the snapd drop-in, reloads systemd and restarts snapd. |
setup docker | Removes only pkgproxy's entry from registry-mirrors, and the key when no mirror is left, then restarts Docker. |
setup go | Removes the GOPROXY line from your Go environment file, as go env -u GOPROXY does, if it has one. |
setup lxd | Deletes the pkgproxy profile. |
setup lxd -project P and the craft setups | Unsets environment.GOPROXY, environment.SNAPPY_FORCE_API_URL and raw.lxc on the project's default profile. The project is kept. |
setup lxd-remote | Removes each remote that points at pkgproxy and its certificate, and puts craft-com.ubuntu.cloud-buildd and craft-com.ubuntu.cloud-buildd-daily back at https://cloud-images.ubuntu.com/buildd/releases and …/buildd/daily. Removes pkgproxy's image registries, the images LXD downloaded through them, and its cluster links. |
Options
| Option | Commands | Meaning |
|---|---|---|
-addr | all | pkgproxy's listen address. Taken from the running daemon when not given, else :3142. A wildcard host becomes this machine's host name. |
-undo | all but status | Print the commands that reverse the setup. |
-go-base-path, -base-path | go, status (-go-base-path only) | pkgproxy's Go base path, /go by default; not taken from the daemon. |
-project | lxd | LXD project whose default profile to change, instead of a pkgproxy profile. |
-lxd-base-path, -base-path | lxd-remote, status (-lxd-base-path only) | pkgproxy's LXD base path, /lxd by default; not taken from the daemon. |
-lxd-remote | lxd-remote, status | The image remotes, as given to the daemon; not taken from it. By default craft-com.ubuntu.cloud-buildd, craft-com.ubuntu.cloud-buildd-daily, pkgproxy-images, pkgproxy-ubuntu and pkgproxy-ubuntu-daily. |
-lxd-tls-addr | lxd-remote, status | pkgproxy's HTTPS address for LXD images. Taken from the daemon when not given, else :3143. |
setup status
pkgproxyctl setup status | sh prints one line per setup command with one of these states, and notes under it.
| State | Meaning |
|---|---|
| set up | Set up for the pkgproxy at -addr. |
| not set up | Nothing of this setup is there. |
| set up for another address | Set up, but for another pkgproxy address. |
| needs setup again | There, but it cannot work as it is, such as an LXD remote at an old address; run that setup again. |
| not installed | The tool is not on this machine. |
| cannot check | The script could not tell, with the reason in a note. |
for users for administrators
Getting pkgproxyctl
pkgproxyctl is the pkgproxy binary under another name. The snap installs both. Without the snap, every release from 0.12.0 on publishes the binary for Linux on amd64 and arm64, with a SHA256SUMS file to check it against, on the releases page. To install the newest release, with arch set to the machine's architecture:
arch=amd64 url=https://gitlab.com/zygoon/pkgproxy/-/releases/permalink/latest/downloads curl -fL -O "$url/pkgproxy-linux-$arch" -O "$url/SHA256SUMS" sha256sum --check --ignore-missing SHA256SUMS sudo install -m 755 "pkgproxy-linux-$arch" /usr/local/bin/pkgproxy sudo ln -sf pkgproxy /usr/local/bin/pkgproxyctl
Or build it with Go: go install gitlab.com/zygoon/pkgproxy/cmd/pkgproxy@latest, then ln -s pkgproxy "$(go env GOPATH)/bin/pkgproxyctl".
From the setup page
A machine without pkgproxyctl gets the same scripts from the daemon's setup page (The setup page):
| URL | What it serves |
|---|---|
http://<host>:3142/setup/<tool> | #!/bin/sh and the lines pkgproxyctl setup <tool> prints when piped, as text/plain |
http://<host>:3142/setup/<tool>?undo=1 | The -undo script |
The tools are apt, snap, go, docker (unless the Docker mirror is off), lxd, lxd-remote (only with image remotes and the HTTPS listener on), snapcraft, rockcraft and charmcraft; status is not offered. The scripts name the address the visitor reached pkgproxy at, or the public-url setting. Pipe those for apt, snap and docker to sudo sh, and the others to sh.
Roads from here
LXD
- LXD build containers — what setup lxd and the craft tools set up
- LXD image servers — the image remotes setup lxd-remote adds
SETUP AND THE COMMAND LINE
- pkgproxyctl — the rest of the command line
- The setup page — setting up a machine without pkgproxy installed
OPERATIONS
- Statistics — checking that a client uses the cache