Stalled upstreams

core for everyone

How the upstream idle timeout fails a request whose upstream stops sending, counting inactivity rather than total time, so a slow download finishes and a dead one is retried.

On the map: core

▶ Watch this deck — 6 scenes, ~4 min

01 What this topic covers

ABOUT THIS TOPIC Stalled upstreams an upstream that sends nothing for 60 seconds is given up on 1 Inactivity, not total time 2 Slow readers do not count 3 A stalled fetch is given up on 4 Three bounds on waiting for administrators and CI operators
The topic title, Stalled upstreams, and its 4 scenes in order: Inactivity, not total time, Slow readers do not count, A stalled fetch is given up on, Three bounds on waiting

How the upstream idle timeout fails a download that stops, without cutting off a slow one.

When an upstream stops sending, pkgproxy gives up on the request after sixty seconds of silence, however long the download has been running. Watch Fetch once first if you have not. Here you see why the -upstream-idle-timeout measures inactivity, not total time, and why a slow reader does not count against it. Then how a stalled fetch is given up on, so the next request starts afresh, and three settings that bound waiting at pkgproxy's edges.

02 Inactivity, not total time

IDLE TIMEOUT every upstream request: at most 60 s with nothing arriving inactivity, not total time: the clock restarts whenever bytes arrive … 2 GB layer at 1 MB/s 33 minutes, finishes nothing for 60 s fails: no progress 0 waits forever
Two downloads on a time line: a slow one whose bytes keep arriving finishes after 33 minutes; another whose upstream goes silent fails after 60 seconds of nothing

-upstream-idle-timeout (default 60s) fails a request only after that long without progress. A 2 GB layer at 1 MB/s takes its 33 minutes. 0 turns the check off.

Every request pkgproxy sends upstream has an idle timeout, sixty seconds by default. It bounds inactivity, not total time. Each time pkgproxy waits for the upstream, for the response headers or for the next bytes of the body, the clock starts again from zero, and only a wait of sixty seconds with nothing arriving fails the request. A two gigabyte Docker layer from a registry sending one megabyte a second takes about thirty three minutes, and that is fine. Set to zero, the timeout waits forever.

03 Slow readers do not count

WHAT THE CLOCK COUNTS timer running: waiting on the upstream awaiting headers read read read between reads between reads timer stopped between reads upstream pkgproxy client on a slow link a slow consumer never cancels a healthy upstream a silent upstream is still caught
A time line of one upstream request: the timer runs while waiting for headers and inside each read, and is stopped between reads while pkgproxy passes bytes on to a slow client

The timer runs while the headers are awaited and while each read of the body waits for bytes, and is stopped between reads. Time package proxy spends passing the bytes on to a client is not the upstream's.

Only time spent waiting on the upstream counts: while the response headers are awaited, and inside each read of the body. The time between reads does not. When pkgproxy is slow to take the next bytes, because it is passing what it read to a client on a slow link, the clock is stopped. So a client on a slow link does not make a working download fail, and an upstream that stops sending still times out.

04 A stalled fetch is given up on

A STALLED FETCH client 1 client 2 one fetch detached, shared upstream silent client 3 without a bound: everyone joins and waits 60 s of nothing fetches in progress …/hello_2.10-3_amd64.deb fails, removed snap, LXD image, file by checksum: one more try first, about 2 min in all a fresh fetch for the next request
Clients joined to one fetch whose upstream went silent; after 60 seconds the fetch fails, or after about 2 minutes for a snap, LXD image or file by checksum, and leaves the table of fetches in progress; the next request starts a new fetch

A shared download whose upstream stalls fails like on any upstream error and is removed from the downloads in progress; a snap, LXD image or file by checksum asks once more first, about 2 minutes in all. The next request for the file starts a new download rather than waiting on the old connection.

Without this bound, a connection that went silent would keep its download open forever. A download is shared and detached from any client, so every later request for that file would join it and wait as well. With the bound, a fetch whose upstream sends nothing for sixty seconds fails, like on any upstream error, and leaves the table of fetches in progress. A snap, an LXD image or a file by checksum first tries once more, so it fails after about two minutes. The next request for the file starts a fresh fetch instead of joining a dead connection.

05 Three bounds on waiting for administrators for CI operators

THREE BOUNDS clients pkgproxy upstreams -read-header-timeout 30s a client sending its request -upstream-idle-timeout 60s an upstream sending nothing -shutdown-grace 10s requests in progress, on stop each is a flag and a snap option of the same name connecting to an upstream: at most 30 s · its TLS handshake: at most 10 s
Clients on the left, pkgproxy in the middle, upstreams on the right: read-header-timeout bounds the client side, upstream-idle-timeout the upstream side, shutdown-grace the time requests get when pkgproxy stops

-read-header-timeout (30s), -upstream-idle-timeout (60s) and -shutdown-grace (10s), also as snap options of the same names. Connecting upstream is bounded at 30s, the TLS handshake at 10s.

Three settings bound waiting at the edges of pkgproxy. The read header timeout, thirty seconds, is how long a client may take to send its request headers, on every listener. The -upstream-idle-timeout, sixty seconds, covers the other side: an upstream that sends nothing. The shutdown grace, ten seconds, is how long requests in progress get to finish when pkgproxy is asked to stop. All three are flags and snap options. Connecting to an upstream gives up after thirty seconds, and its TLS handshake after ten.

06 Where to go next general version

WHERE NEXT a slow download finishes however long it takes, while one that goes silent fails after a minute, and the next request starts it again you are here Stalled upstreams Fetch once the shared download a stall would block Immutable or not a stalled recheck answered from the cache Offline mode leaving a failing upstream alone
You are here, Stalled upstreams; next: Fetch once, Immutable or not, Offline mode

Next: Fetch once, for the shared download; Immutable or not, for a stalled recheck; Offline mode, for leaving a failing upstream alone.

A slow download finishes however long it takes, while one that goes silent fails after a minute, and the next request starts it again. To see the shared download a stall would block, watch Fetch once. To see a stalled recheck answered from the cache, watch Immutable or not. To leave a failing upstream alone for a while, watch Offline mode.

Read more

Asking an upstream again

What failedAsked againUp to
any request answered 408 Request Timeoutthe same request, when its body can be sent again3 attempts in all
a snap, LXD image or file by checksum that broke off or never started: a dropped or silent connection, or 408, 429, 500, 502, 503, 504, a proxy's includedthe rest of it, with Range from the bytes kept5 tries in a row after 1, 2, 4, 8 and 16 s, within upstream-retry-for of the last byte

An upstream that never answers a snap, LXD image or file by checksum is given up on after two attempts of upstream-idle-timeout, about 2 minutes. Every byte that arrives resets the count and the clock. A download that cannot be made at all (no such host, refused, offline mode) is not asked again. Apt and Docker downloads fail at the first error.

for administrators for CI operators

Settings

SettingSnap optionDefaultBounds
-upstream-idle-timeoutupstream-idle-timeout60show long one upstream request may receive nothing; 0 waits forever
-upstream-retry-forupstream-retry-for2mhow long after the last byte a snap, LXD image or file by checksum is still asked for again; 0 never asks again

In the documentation

Roads from here

CORE · the path every side shares

DOCKER

OPERATIONS