Stalled upstreams
core for everyone
How the upstream idle timeout fails a request whose upstream stops sending, counting inactivity rather than total time, so a slow download finishes and a dead one is retried.
On the map: core
01 What this topic covers
How the upstream idle timeout fails a download that stops, without cutting off a slow one.
When an upstream stops sending, pkgproxy gives up on the request after sixty seconds of silence, however long the download has been running. Watch Fetch once first if you have not. Here you see why the -upstream-idle-timeout measures inactivity, not total time, and why a slow reader does not count against it. Then how a stalled fetch is given up on, so the next request starts afresh, and three settings that bound waiting at pkgproxy's edges.
02 Inactivity, not total time
-upstream-idle-timeout (default 60s) fails a
request only after that long without progress. A 2 GB layer at 1 MB/s
takes its 33 minutes. 0 turns the check off.
Every request pkgproxy sends upstream has an idle timeout, sixty seconds by default. It bounds inactivity, not total time. Each time pkgproxy waits for the upstream, for the response headers or for the next bytes of the body, the clock starts again from zero, and only a wait of sixty seconds with nothing arriving fails the request. A two gigabyte Docker layer from a registry sending one megabyte a second takes about thirty three minutes, and that is fine. Set to zero, the timeout waits forever.
03 Slow readers do not count
The timer runs while the headers are awaited and while each read of the body waits for bytes, and is stopped between reads. Time package proxy spends passing the bytes on to a client is not the upstream's.
Only time spent waiting on the upstream counts: while the response headers are awaited, and inside each read of the body. The time between reads does not. When pkgproxy is slow to take the next bytes, because it is passing what it read to a client on a slow link, the clock is stopped. So a client on a slow link does not make a working download fail, and an upstream that stops sending still times out.
04 A stalled fetch is given up on
A shared download whose upstream stalls fails like on any upstream error and is removed from the downloads in progress; a snap, LXD image or file by checksum asks once more first, about 2 minutes in all. The next request for the file starts a new download rather than waiting on the old connection.
Without this bound, a connection that went silent would keep its download open forever. A download is shared and detached from any client, so every later request for that file would join it and wait as well. With the bound, a fetch whose upstream sends nothing for sixty seconds fails, like on any upstream error, and leaves the table of fetches in progress. A snap, an LXD image or a file by checksum first tries once more, so it fails after about two minutes. The next request for the file starts a fresh fetch instead of joining a dead connection.
05 Three bounds on waiting for administrators for CI operators
-read-header-timeout (30s),
-upstream-idle-timeout (60s) and
-shutdown-grace (10s), also as snap options of
the same names. Connecting upstream is bounded at 30s, the
TLS handshake at 10s.
Three settings bound waiting at the edges of pkgproxy. The read header timeout, thirty seconds, is how long a client may take to send its request headers, on every listener. The -upstream-idle-timeout, sixty seconds, covers the other side: an upstream that sends nothing. The shutdown grace, ten seconds, is how long requests in progress get to finish when pkgproxy is asked to stop. All three are flags and snap options. Connecting to an upstream gives up after thirty seconds, and its TLS handshake after ten.
06 Where to go next general version
Next: Fetch once, for the shared download; Immutable or not, for a stalled recheck; Offline mode, for leaving a failing upstream alone.
A slow download finishes however long it takes, while one that goes silent fails after a minute, and the next request starts it again. To see the shared download a stall would block, watch Fetch once. To see a stalled recheck answered from the cache, watch Immutable or not. To leave a failing upstream alone for a while, watch Offline mode.
Read more
Asking an upstream again
| What failed | Asked again | Up to |
|---|---|---|
any request answered 408 Request Timeout | the same request, when its body can be sent again | 3 attempts in all |
a snap, LXD image or file by checksum that broke off or never started: a dropped or silent connection, or 408, 429, 500, 502, 503, 504, a proxy's included | the rest of it, with Range from the bytes kept | 5 tries in a row after 1, 2, 4, 8 and 16 s, within upstream-retry-for of the last byte |
An upstream that never answers a snap, LXD image or file by checksum is given up on after two attempts of upstream-idle-timeout, about 2 minutes. Every byte that arrives resets the count and the clock. A download that cannot be made at all (no such host, refused, offline mode) is not asked again. Apt and Docker downloads fail at the first error.
for administrators for CI operators
Settings
| Setting | Snap option | Default | Bounds |
|---|---|---|---|
-upstream-idle-timeout | upstream-idle-timeout | 60s | how long one upstream request may receive nothing; 0 waits forever |
-upstream-retry-for | upstream-retry-for | 2m | how long after the last byte a snap, LXD image or file by checksum is still asked for again; 0 never asks again |
In the documentation
- Settings: the upstream timeouts
- What the errors mean
Roads from here
CORE · the path every side shares
- Fetch once — the shared fetch a stall would block
- Immutable or revalidated — a stalled recheck serves the cached copy
- Offline mode — leaving a failing upstream alone
DOCKER
- Docker Hub registry mirror — large layers from a registry
OPERATIONS
- Settings — setting the timeouts