HTTPS apt sources

apt for users for administrators

apt sends CONNECT for every https source to its http proxy; pkgproxy relays it as a blind tunnel to port 443, never to this machine itself, and caches nothing from it.

On the map: apt

▶ Watch this deck — 7 scenes, ~4 min

01 What this topic covers

ABOUT THIS TOPIC HTTPS sources apt's https sources pass through pkgproxy in a blind tunnel, uncached 1 Why https sources reach the proxy 2 A blind tunnel to port 443 3 Why not refuse it for administrators 4 Getting such a source cached
The topic title, HTTPS sources, and its 4 scenes in order: Why https sources reach the proxy, A blind tunnel to port 443, Why not refuse it, Getting such a source cached

How apt's https sources pass through pkgproxy as an uncached tunnel, and how to get them cached instead.

When apt uses pkgproxy, its https sources pass through it in a blind tunnel, and nothing in them is cached. It helps to have seen A forward proxy first. Here you see why https sources reach the proxy at all, and the tunnel to port 443. Then why pkgproxy does not simply refuse it, and how to get such a source cached.

02 Why https sources reach the proxy

HTTPS SOURCES deb http://deb.debian.org/debian trixie main deb https://repo.example.com/apt stable main apt Acquire::http::Proxy only GET http://deb.debian.org/debian/… CONNECT repo.example.com:443 pkgproxy :3142 keeping it away takes an https proxy setting on every machine so pkgproxy has to answer CONNECT
A sources list with one http and one https source; apt sends a plain GET for the first and a CONNECT to port 443 for the second, both to pkgproxy

apt sends CONNECT host:443 to whatever Acquire::http::Proxy names for any https:// source; no Acquire::https::Proxy is needed to trigger it.

A machine's sources can mix plain http archives with an https one, often a third party repository. With only the Acquire::http::Proxy line set, apt still sends its https requests to that proxy, as a CONNECT to the host on port 443. The only way to keep this traffic away from pkgproxy would be a separate https proxy setting on every machine. So pkgproxy has to answer CONNECT sensibly.

03 A blind tunnel to port 443

CONNECT CONNECT repo.example.com:443 apt pkgproxy dial :443 repo.example.com fails: 502 this machine: 403 200 Connection Established bytes copied both ways until one side closes TLS from apt to repo.example.com · pkgproxy sees only ciphertext not terminated · not read · nothing cached CONNECT example.com:8443 403 only port 443 is tunnelled
apt sends CONNECT to pkgproxy, which dials the host on port 443, answers 200 Connection Established and copies bytes both ways; a dial that reaches this machine's loopback gets 403; TLS runs from apt to the origin and nothing is cached; CONNECT to another port gets 403

pkgproxy dials first, then takes over the client's connection and copies bytes both ways. No TLS termination, no caching, no verification. A dial that fails gets 502; one that reaches this machine's loopback gets 403. Other ports get 403: pkgproxy only tunnels CONNECT to port 443.

For a CONNECT to port 443, pkgproxy first dials the host. If that fails, apt gets a 502. If the name leads to this machine's own loopback, apt gets a 403: the check is made on the address actually dialled, so no name gets around it. Otherwise pkgproxy answers 200 Connection Established and copies bytes in both directions until either side closes. The TLS session runs between apt and the origin: pkgproxy does not terminate it, cannot read it, and caches nothing. A CONNECT to any other port gets a 403.

04 Why not refuse it for administrators

WHY NOT REFUSE IT if refused CONNECT repo.example.com:443 501 Not Implemented apt update fails for every source tunnelled CONNECT repo.example.com:443 200 Connection Established apt update works still out of scope: terminating TLS, caching or checking HTTPS content port 443 only · neither the tunnel nor plain http reaches this machine its loopback and pkgproxy's own ports get 403 · a mirror elsewhere on the network still works
Before: CONNECT answered 501 and apt update failed. Now: CONNECT answered 200 and apt update works, with the https source uncached. The scope stays: no TLS termination and no HTTPS caching; neither the tunnel nor plain http reaches this machine's loopback or pkgproxy's own ports

Refusing CONNECT (with 501) would make apt update fail on any machine with one https-only source. The port-443 tunnel keeps the non-goal: no TLS termination, no HTTPS caching. Neither the tunnel nor plain http reaches this machine's loopback or pkgproxy's own ports.

Caching HTTPS is outside pkgproxy's scope, but refusing CONNECT is not an option. A refusal makes apt update fail outright on any machine with a single https-only source, which is common for third party repositories. So pkgproxy opens the tunnel and keeps its scope: it does not terminate TLS or cache HTTPS content. The tunnel goes only to port 443, and like plain http it never reaches this machine's loopback or pkgproxy's own ports, while a mirror elsewhere on the network still works.

05 Getting such a source cached

GETTING IT CACHED msg="handled request" method=CONNECT … outcome=tunnel tunnelled: nothing cached deb https://deb.debian.org/debian trixie main deb http://deb.debian.org/debian trixie main cached apt still verifies the InRelease signature and the hashes below it https only works through the tunnel, uncached
The log marks tunnelled requests with outcome=tunnel; changing a source from https to http makes it cached; apt still verifies InRelease's signature and the hashes below it; an https-only archive keeps working uncached

A tunnelled request is logged with outcome=tunnel. Switch the source to http:// where the archive offers it; apt checks signatures either way. https-only archives keep working, uncached.

Tunnelled downloads are not cached, and the log marks each one with the outcome tunnel. If the archive is also served over plain http, change the source to http, and pkgproxy caches it like any other. That does not weaken apt's checks: apt verifies the signature on InRelease and the hashes below it either way. An archive that offers only https keeps working through the tunnel, uncached.

06 Where to go next for administrators in place of the general version

WHERE NEXT https sources keep working through the tunnel, uncached, and a source switched to http is cached like the rest you are here HTTPS sources The access log finding the tunnels and the sources to switch pool and dists what an http source gets from the cache Files by checksum where pkgproxy fetches https itself
You are here, HTTPS sources; next: The access log, pool and dists, Files by checksum

Next: The access log, for finding tunnels; pool and dists, for what an http source gets; Files by checksum, for https that pkgproxy fetches itself.

Your https sources keep working through the tunnel, uncached, and a source you switch to http is cached like the rest. To find the tunnels in the log and see which sources to switch, watch The access log. To see what such a source gets from the cache, watch pool and dists. To see where pkgproxy fetches https itself, watch Files by checksum.

07 Where to go next general version

WHERE NEXT https sources keep working through the tunnel, uncached, and a source switched to http is cached like the rest you are here HTTPS sources A forward proxy how plain http sources reach pkgproxy pool and dists what an http source gets from the cache Files by checksum where pkgproxy fetches https itself
You are here, HTTPS sources; next: A forward proxy, pool and dists, Files by checksum

Next: A forward proxy, for plain http sources; pool and dists, for what they get from the cache; Files by checksum, for https that pkgproxy fetches itself.

Your https sources keep working through the tunnel, uncached, and a source you switch to http is cached like the rest. To see how plain http sources reach pkgproxy, watch A forward proxy. To see what such a source gets from the cache, watch pool and dists. To see where pkgproxy fetches https itself, watch Files by checksum.

Read more

Roads from here

START HERE

APT

FILES

OPERATIONS