apt through a forward proxy
apt for users for administrators
apt keeps its sources as they are and sends every request to pkgproxy as a forward proxy, set up with one Acquire::http::Proxy line.
On the map: apt
01 What this topic covers
How apt sends every request to pkgproxy as a forward proxy, set up with one line and undone with one command.
Apt reaches pkgproxy as a forward proxy: one line in apt's configuration sends every download through it, and the sources.list stays as it is. Here you see the one file the setup writes, and the request line that names the real host. Then why this is a forward proxy and not a mirror, the parts of HTTP that apt relies on, and how to check the setup and undo it.
02 One file in apt's configuration
pkgproxyctl setup apt | sudo sh writes
/etc/apt/apt.conf.d/99pkgproxy with one
Acquire::http::Proxy line, plus
Acquire::Retries::Delay "false"; when apt is older than 3.1.3.
sources.list stays as it is.
To send apt through pkgproxy, run pkgproxyctl setup apt and pipe it to sudo sh. The script writes one file in apt's configuration directory, holding one line: Acquire::http::Proxy, set to pkgproxy's address on port 3142. On an apt older than 3.1.3 it adds a second line that makes apt retry a failed download at once, because an older apt can hang while pkgproxy is offline. The sources.list is not touched.
03 The request line names the real host
A forward-proxy request has an absolute URI: GET
http://deb.debian.org/…. pkgproxy fetches from the host it names,
with Via: 1.1 pkgproxy, so any number of mirrors go through one
proxy. This machine's loopback, and its own addresses at pkgproxy's ports,
get 403. A request with only a path is answered
400.
Apt keeps its sources as they are, naming mirrors such as deb.debian.org. With the proxy line set, apt connects to pkgproxy instead, and its request line carries the full URL, host included, not only the path. pkgproxy reads the host from that line, fetches from it, and answers apt. One proxy serves every mirror a machine names, except the machine pkgproxy runs on: its loopback, and its own addresses at pkgproxy's ports, get a 403. A plain request with only a path, as a client sends to a web server, gets a 400.
04 A forward proxy, not a mirror
Mirror substitution rewrites sources.list to name the cache,
which fronts one upstream. A forward proxy takes requests for any host
through one cache, with one Acquire::http::Proxy line per
machine.
There are two ways to put a cache in front of apt. A mirror replaces the address in every machine's sources with the cache's own, and the cache fronts one fixed upstream, so each archive needs its own setup and every machine's sources must be edited. A forward proxy leaves the sources alone. Any host apt names arrives at the same cache, Debian and Ubuntu archives alike, and setting it up is one line per machine.
05 The HTTP apt relies on for administrators
If-Modified-Since gets 304 from the cache.
Range is honoured only when If-Range matches the
entity being sent; otherwise 200 with the whole body. apt's
default Acquire::http::Pipeline-Depth is 10.
Apt already does the client half of HTTP caching. When it fetches an index again it sends If-Modified-Since, and pkgproxy answers 304 from the cache when its copy is no newer. To resume a partial download apt sends Range with If-Range. pkgproxy honours the range only if If-Range matches what it is about to send; otherwise the whole file goes out with 200, so old bytes are never spliced onto new ones. Apt also pipelines up to ten requests on one connection, and pkgproxy answers them in order.
06 Checking and undoing
apt-config dump | grep -i proxy shows the setting. apt does
not fail over: while pkgproxy is down, its downloads fail.
pkgproxyctl setup apt -undo | sudo sh removes
/etc/apt/apt.conf.d/99pkgproxy. Without | sudo sh,
setup explains each step instead.
To see whether a machine uses pkgproxy, ask apt-config dump for its proxy settings: the Acquire::http::Proxy line should name pkgproxy's address. Apt has no fallback from its proxy to a direct connection, so while pkgproxy is down every apt download fails. To stop, run pkgproxyctl setup apt with the -undo flag and pipe it to sudo sh. It removes the one file it wrote, retry line included, and apt goes back to its mirrors directly. Run setup without the pipe to read what it would change first.
07 Where to go next for administrators in place of the general version
Next: Cache keys, for keys and hash checks; pool and dists, for what is kept and rechecked; HTTPS sources, for https archives.
Apt on your machine now downloads through pkgproxy, and its sources.list is the same as before. To see how a request becomes a cache key and how downloads are checked, watch Cache keys. To see which apt files are kept forever and which are checked again, watch pool and dists. To see what happens to https sources, watch HTTPS sources.
08 Where to go next general version
Next: pool and dists, for what apt gets from the cache; HTTPS sources,
for https archives; pkgproxyctl setup, for the other tools.
Apt on your machine now downloads through pkgproxy, and its sources.list is the same as before. To see which apt files are kept forever and which are checked again, watch pool and dists. To see what happens to https sources, watch HTTPS sources. To set up the other tools on the machine, watch pkgproxyctl setup.
Read more
Roads from here
CORE · the path every side shares
- One listener — how apt shares the port with everything else
- Offline mode — apt update without the internet
APT
- apt cache keys and hash checks — how a request becomes a cache key
- pool forever and dists rechecked — what is kept forever and what is rechecked
- HTTPS apt sources — what happens to https sources
SETUP AND THE COMMAND LINE
- Setting up clients — setting up the other clients