apt through a forward proxy

apt for users for administrators

apt keeps its sources as they are and sends every request to pkgproxy as a forward proxy, set up with one Acquire::http::Proxy line.

On the map: apt

▶ Watch this deck — 8 scenes, ~5 min

01 What this topic covers

ABOUT THIS TOPIC A forward proxy one line in apt's configuration sends every download through pkgproxy 1 One file in apt's configuration 2 The request line names the real host 3 A forward proxy, not a mirror 4 The HTTP apt relies on for administrators 5 Checking and undoing
The topic title, A forward proxy, and its 5 scenes in order: One file in apt's configuration, The request line names the real host, A forward proxy, not a mirror, The HTTP apt relies on, Checking and undoing

How apt sends every request to pkgproxy as a forward proxy, set up with one line and undone with one command.

Apt reaches pkgproxy as a forward proxy: one line in apt's configuration sends every download through it, and the sources.list stays as it is. Here you see the one file the setup writes, and the request line that names the real host. Then why this is a forward proxy and not a mirror, the parts of HTTP that apt relies on, and how to check the setup and undo it.

02 One file in apt's configuration

SETUP APT $ pkgproxyctl setup apt | sudo sh /etc/apt/apt.conf.d/99pkgproxy Acquire::http::Proxy "http://<host>:3142"; Acquire::Retries::Delay "false"; only when apt is older than 3.1.3 /etc/apt/sources.list deb http://deb.debian.org/debian trixie main unchanged one file, one line · the sources stay as they are
pkgproxyctl setup apt piped to sudo sh writes one apt configuration file with the Acquire::http::Proxy line, adds a retry line for apt older than 3.1.3, and leaves sources.list unchanged

pkgproxyctl setup apt | sudo sh writes /etc/apt/apt.conf.d/99pkgproxy with one Acquire::http::Proxy line, plus Acquire::Retries::Delay "false"; when apt is older than 3.1.3. sources.list stays as it is.

To send apt through pkgproxy, run pkgproxyctl setup apt and pipe it to sudo sh. The script writes one file in apt's configuration directory, holding one line: Acquire::http::Proxy, set to pkgproxy's address on port 3142. On an apt older than 3.1.3 it adds a second line that makes apt retry a failed download at once, because an older apt can hang while pkgproxy is offline. The sources.list is not touched.

03 The request line names the real host

THE REQUEST LINE apt sources name the mirror GET http://deb.debian.org/debian/dists/trixie/InRelease HTTP/1.1 the request line carries the whole URL, host included pkgproxy :3142 deb.debian.org security.debian.org archive.ubuntu.com GET http://127.0.0.1/… 403 not this machine: its loopback, or its own address at pkgproxy's ports GET /debian/dists/trixie/InRelease HTTP/1.1 400 a path alone names no host
apt sends a request line with the full URL to pkgproxy, which fetches from the host it names; other hosts go through the same proxy; this machine itself is refused with 403; a request with only a path gets 400

A forward-proxy request has an absolute URI: GET http://deb.debian.org/…. pkgproxy fetches from the host it names, with Via: 1.1 pkgproxy, so any number of mirrors go through one proxy. This machine's loopback, and its own addresses at pkgproxy's ports, get 403. A request with only a path is answered 400.

Apt keeps its sources as they are, naming mirrors such as deb.debian.org. With the proxy line set, apt connects to pkgproxy instead, and its request line carries the full URL, host included, not only the path. pkgproxy reads the host from that line, fetches from it, and answers apt. One proxy serves every mirror a machine names, except the machine pkgproxy runs on: its loopback, and its own addresses at pkgproxy's ports, get a 403. A plain request with only a path, as a client sends to a web server, gets a 400.

04 A forward proxy, not a mirror

PROXY SHAPE mirror deb http://cache.local/debian … every machine's sources rewritten cache one upstream one fixed upstream per cache address forward proxy deb http://deb.debian.org/debian … sources unchanged pkgproxy deb.debian.org security.debian.org archive.ubuntu.com a forward proxy takes any host apt names · one line per machine
Left: mirror substitution rewrites sources.list to name the cache, which fronts one fixed upstream. Right: a forward proxy leaves sources.list unchanged and passes requests for many hosts through one cache

Mirror substitution rewrites sources.list to name the cache, which fronts one upstream. A forward proxy takes requests for any host through one cache, with one Acquire::http::Proxy line per machine.

There are two ways to put a cache in front of apt. A mirror replaces the address in every machine's sources with the cache's own, and the cache fronts one fixed upstream, so each archive needs its own setup and every machine's sources must be edited. A forward proxy leaves the sources alone. Any host apt names arrives at the same cache, Debian and Ubuntu archives alike, and setting it up is one line per machine.

05 The HTTP apt relies on for administrators

HTTP APT RELIES ON GET …/dists/trixie/InRelease If-Modified-Since: <date> 304 Not Modified from the cache GET …/pool/main/h/hello/hello_2.10-3_amd64.deb Range: bytes=1048576- If-Range: <Last-Modified> 206 Partial Content if it matches 200 OK if not: the whole file GET a · GET b · GET c · … answered in order pipelining: up to 10 requests in flight on one connection apt does the client half of HTTP caching · pkgproxy answers it correctly
Three exchanges: a conditional GET answered 304 from the cache; a Range request with If-Range answered 206 when it matches and 200 with the whole file when not; several pipelined requests on one connection

If-Modified-Since gets 304 from the cache. Range is honoured only when If-Range matches the entity being sent; otherwise 200 with the whole body. apt's default Acquire::http::Pipeline-Depth is 10.

Apt already does the client half of HTTP caching. When it fetches an index again it sends If-Modified-Since, and pkgproxy answers 304 from the cache when its copy is no newer. To resume a partial download apt sends Range with If-Range. pkgproxy honours the range only if If-Range matches what it is about to send; otherwise the whole file goes out with 200, so old bytes are never spliced onto new ones. Apt also pipelines up to ten requests on one connection, and pkgproxy answers them in order.

06 Checking and undoing

CHECK AND UNDO $ apt-config dump | grep -i proxy Acquire::http::Proxy "http://<host>:3142"; pkgproxy down: every apt download fails · apt does not fall back to the mirror $ pkgproxyctl setup apt -undo | sudo sh rm -f /etc/apt/apt.conf.d/99pkgproxy the proxy line and the retry line apt straight to the mirror deb.debian.org run setup without | sudo sh to read what it would change
apt-config dump shows the proxy line; while pkgproxy is down apt's downloads fail, since apt does not fall back; pkgproxyctl setup apt -undo piped to sudo sh removes the one file, and apt talks to its mirror directly again

apt-config dump | grep -i proxy shows the setting. apt does not fail over: while pkgproxy is down, its downloads fail. pkgproxyctl setup apt -undo | sudo sh removes /etc/apt/apt.conf.d/99pkgproxy. Without | sudo sh, setup explains each step instead.

To see whether a machine uses pkgproxy, ask apt-config dump for its proxy settings: the Acquire::http::Proxy line should name pkgproxy's address. Apt has no fallback from its proxy to a direct connection, so while pkgproxy is down every apt download fails. To stop, run pkgproxyctl setup apt with the -undo flag and pipe it to sudo sh. It removes the one file it wrote, retry line included, and apt goes back to its mirrors directly. Run setup without the pipe to read what it would change first.

07 Where to go next for administrators in place of the general version

WHERE NEXT apt now downloads through pkgproxy, and its sources list is the same as before you are here A forward proxy Cache keys how a request becomes a key and is checked pool and dists which apt files are kept forever and which rechecked HTTPS sources what happens to https sources
You are here, A forward proxy; next: Cache keys, pool and dists, HTTPS sources

Next: Cache keys, for keys and hash checks; pool and dists, for what is kept and rechecked; HTTPS sources, for https archives.

Apt on your machine now downloads through pkgproxy, and its sources.list is the same as before. To see how a request becomes a cache key and how downloads are checked, watch Cache keys. To see which apt files are kept forever and which are checked again, watch pool and dists. To see what happens to https sources, watch HTTPS sources.

08 Where to go next general version

WHERE NEXT apt now downloads through pkgproxy, and its sources list is the same as before you are here A forward proxy pool and dists which apt files are kept forever and which rechecked HTTPS sources what happens to https sources pkgproxyctl setup the other tools on the machine
You are here, A forward proxy; next: pool and dists, HTTPS sources, pkgproxyctl setup

Next: pool and dists, for what apt gets from the cache; HTTPS sources, for https archives; pkgproxyctl setup, for the other tools.

Apt on your machine now downloads through pkgproxy, and its sources.list is the same as before. To see which apt files are kept forever and which are checked again, watch pool and dists. To see what happens to https sources, watch HTTPS sources. To set up the other tools on the machine, watch pkgproxyctl setup.

Read more

Roads from here

CORE · the path every side shares

APT

SETUP AND THE COMMAND LINE