Fetch once
core for users for administrators for CI operators
How a fill shares one upstream download among every request for an object, streams it while it is written, and what happens when the upstream fails, is asked again for the rest, or sends content that does not match its hash.
On the map: core
01 What this topic covers
How one upstream download is shared by every request for a file, and what happens when it goes wrong.
When fifty machines ask for the same file, pkgproxy downloads it once and shares that download with all of them. Watch A request first if you have not. Large files stream to every waiting client, while small ones arrive whole. The headers go out early, and a resumed download joins the same fetch. Last, what happens when the upstream dies halfway through, how a snap asks again for the rest, and what becomes of content that fails its hash.
02 Large files stream, small ones arrive whole
Concurrent requests for one file share one upstream download. A fill
streams to every waiting client while it is written (apt's
pool/**, snaps, LXD image files, Docker blobs, files by
checksum); indexes, Go modules and manifests are served once complete.
pkgproxy makes one download upstream for each file, however many clients ask for it at the same time. For the large files, that download is a fill, and it streams: packages from apt's pool, snaps, LXD image files, Docker layers and files by checksum. Every client waiting for one starts to receive bytes as they arrive, so fifty CI jobs pulling the same layer start together and cost one download. Smaller files, such as apt's indexes, Go modules and Docker manifests, are fetched once too, and each client gets the file once it is complete.
03 Headers first, and resumes from the same fill
A plain request gets 200 once the upstream's headers are in;
a Range or If-Range resume gets 206
from the same fill. With the size known in advance (snaps, LXD image
files) the headers go out at once.
Every request for the key joins the same fill, whatever it asks for. A plain request gets a 200 as soon as the upstream's headers are in, with the upstream's length, and then follows the download. A client resuming a broken download asks for a range, and gets a 206 for the rest: first from the bytes already written, then from the ones still arriving. Snaps and LXD image files have a size known in advance, so their clients get headers before the upstream has even answered.
04 When the upstream dies mid-body for administrators for CI operators
A fill that gives up ends every follower's response short and caches
nothing; the next request starts a new fill. -upstream-idle-timeout
(60s) bounds upstream inactivity, not total time.
If the upstream stops in the middle of the body and the fill gives up, every client following it gets an error, and its connection is cut short, because it was promised more bytes than will come. Nothing is cached. The failed fill is dropped, so the next request starts a fresh fill instead of joining a dead connection. An attempt that receives nothing at all fails after the -upstream-idle-timeout, sixty seconds by default. Time a slow client spends between reads does not count against it. Apt and Docker downloads give up at once; the next scene shows the others asking again.
05 A broken download asks again for administrators for CI operators
A Range: bytes=<kept>- retry, 5 in a row after 1, 2,
4, 8 and 16 s, while followers wait. -upstream-retry-for
(2m after the last byte; 0 never asks again).
Also retried before the body: a dropped connection and 408, 429, 500,
502, 503, 504, a proxy's included.
Snaps, LXD images and files by checksum have their hash known before the download starts, so a fill that breaks off can ask the upstream for the rest. It sends a range request from the last byte it kept, up to five times in a row, waiting one, two, four, eight and sixteen seconds in between. The clients following the fill keep waiting for the next byte. Every byte that arrives starts the count again, and the fill stops asking two minutes after the last one, which is the -upstream-retry-for setting. The hash still covers the whole file, so a file that changed between tries is not cached.
06 Content that fails its hash for administrators for CI operators
Drain (apt, snap, LXD): followers get the origin's bytes, bounded by
-fill-drain-timeout (10m), for apt, snaps and LXD
images alike. Hold the last byte (Docker blobs, files): a
mismatch ends the response short.
A finished download that does not match its hash is not kept. What the clients already receiving it get depends on the side. For apt, snaps and LXD images, they get the upstream's bytes unchanged, and their own checks reject them. The fill stops taking new readers, waits for these ones, up to ten minutes, and then throws the bytes away. Docker blobs and files by checksum hold back the last byte until the fill commits, so on a mismatch the response ends one byte short.
07 Where to go next for administrators for CI operators in place of the general version
Next: The store, for where a download is written; Stalled upstreams, for when an upstream goes silent; pool and dists, for apt's files.
Now you know why fifty clients asking for one file cost one download, and why a download is kept only when it checks out. To see where a download is written and how it is committed, watch The store. To see when a silent upstream is given up on, watch Stalled upstreams. To see which apt files stream and which arrive whole, watch pool and dists.
08 Where to go next general version
Next: Stalled upstreams, for when an upstream goes silent; pool and dists, for apt's files; Immutable or not, for what is kept and what is rechecked.
Now you know why fifty clients asking for one file cost one download, and why a download is kept only when it checks out. To see when a silent upstream is given up on, watch Stalled upstreams. To see which apt files stream and which arrive whole, watch pool and dists. To see what is kept forever and what is checked again, watch Immutable or not.
Read more
for administrators for CI operators
When a download goes wrong
| What happens | Snaps, LXD images, files by checksum | Apt packages, Docker blobs |
|---|---|---|
the upstream breaks off mid-body, or fails before it starts with a dropped connection, 408, 429, 500, 502, 503 or 504 | asked again for the rest, up to 5 times in a row after 1, 2, 4, 8 and 16 s, within upstream-retry-for (2m) of the last byte; followers keep waiting | the fill fails, followers are cut short |
a Retry-After longer than what is left of upstream-retry-for | fails at once | — |
in store-overloaded, the host's breaker is open when a retry is due | waits for the breaker to let it through, within upstream-retry-for | — |
| an error answer instead of the file | at most 64 KiB of it is read | — |
| more bytes than the store or image server promised (snaps, LXD images) | the fill fails, followers are cut short | — |
| the bytes do not match the hash, whichever tries they came from | not cached | not cached |
A cached file that is replaced while it is sent, such as an InRelease that another machine's apt update refreshed just then, is sent with the size, ETag and Last-Modified of the copy actually sent.
In the documentation
- Settings: the upstream timeouts
- Is a tool really using pkgproxy
Roads from here
START HERE
- A request, end to end — a request end to end
CORE · the path every side shares
- The store — the writer a fill tails
- Stalled upstreams — when the upstream stalls
- Immutable or revalidated — what is kept forever and what is rechecked
APT
- pool forever and dists rechecked — which apt files are fills
SNAP
- Snap Store relay — snap blobs served through fills
FILES
- Files by checksum — holding back the last byte