Fetch once

core for users for administrators for CI operators

How a fill shares one upstream download among every request for an object, streams it while it is written, and what happens when the upstream fails, is asked again for the rest, or sends content that does not match its hash.

On the map: core

▶ Watch this deck — 8 scenes, ~5 min

01 What this topic covers

ABOUT THIS TOPIC Fetch once one download from the internet, shared by every request for a file 1 Large files stream, small ones arrive whole 2 Headers first, and resumes 3 When the upstream dies mid-body for administrators and CI operators 4 A broken download asks again for administrators and CI operators 5 Content that fails its hash for administrators and CI operators
The topic title, Fetch once, and its 5 scenes in order: Large files stream, small ones arrive whole, Headers first, and resumes, When the upstream dies mid-body, A broken download asks again, Content that fails its hash

How one upstream download is shared by every request for a file, and what happens when it goes wrong.

When fifty machines ask for the same file, pkgproxy downloads it once and shares that download with all of them. Watch A request first if you have not. Large files stream to every waiting client, while small ones arrive whole. The headers go out early, and a resumed download joins the same fetch. Last, what happens when the upstream dies halfway through, how a snap asks again for the rest, and what becomes of content that fails its hash.

02 Large files stream, small ones arrive whole

STREAMED OR WHOLE a fill: streamed while it is written upstream 1 download one fill apt pool packages snaps LXD image files Docker layers files by checksum CI job 1 CI job 2 CI job 50 fifty CI jobs pulling one layer: one download, and every job starts receiving at once fetched whole, then served upstream 1 download complete client 1 client 2 client 3 apt indexes under dists/ Go modules Docker manifests
Left: one upstream download feeds a fill that streams bytes to three CI jobs at once, for apt pool packages, snaps, LXD image files, Docker layers and files by checksum. Right: apt indexes, Go modules and Docker manifests are fetched once and served to each client when complete

Concurrent requests for one file share one upstream download. A fill streams to every waiting client while it is written (apt's pool/**, snaps, LXD image files, Docker blobs, files by checksum); indexes, Go modules and manifests are served once complete.

pkgproxy makes one download upstream for each file, however many clients ask for it at the same time. For the large files, that download is a fill, and it streams: packages from apt's pool, snaps, LXD image files, Docker layers and files by checksum. Every client waiting for one starts to receive bytes as they arrive, so fifty CI jobs pulling the same layer start together and cost one download. Smaller files, such as apt's indexes, Go modules and Docker manifests, are fetched once too, and each client gets the file once it is complete.

03 Headers first, and resumes from the same fill

ONE FILL, ANY REQUEST one fill written still arriving… GET …/hello_2.10-3_amd64.deb 200 · Content-Length once the upstream's headers are in Range: bytes=400000- 206 · the rest written bytes, then new ones snaps and LXD image files size known from the Snap Store or the image index: headers go out before the upstream answers
One fill in the middle being written; a plain GET is answered 200 and reads from the start; a Range request is answered 206 and reads from its offset, partly from written bytes and partly from bytes still arriving

A plain request gets 200 once the upstream's headers are in; a Range or If-Range resume gets 206 from the same fill. With the size known in advance (snaps, LXD image files) the headers go out at once.

Every request for the key joins the same fill, whatever it asks for. A plain request gets a 200 as soon as the upstream's headers are in, with the upstream's length, and then follows the download. A client resuming a broken download asks for a range, and gets a 206 for the rest: first from the bytes already written, then from the ones still arriving. Snaps and LXD image files have a size known in advance, so their clients get headers before the upstream has even answered.

04 When the upstream dies mid-body for administrators for CI operators

UPSTREAM FAILS upstream stops mid-body fill client 1 client 2 error: response cut short it was promised more bytes than will come nothing cached the next request a fresh fill upstream-idle-timeout 60s of no progress a slow client does not count apt, Docker: give up at once snaps, LXD images, files: ask again first →
The upstream connection breaks partway through a download; the clients following the fill have their responses cut short and nothing is cached; the fill leaves the table and the next request starts a new fill; snaps, LXD images and files by checksum ask again first

A fill that gives up ends every follower's response short and caches nothing; the next request starts a new fill. -upstream-idle-timeout (60s) bounds upstream inactivity, not total time.

If the upstream stops in the middle of the body and the fill gives up, every client following it gets an error, and its connection is cut short, because it was promised more bytes than will come. Nothing is cached. The failed fill is dropped, so the next request starts a fresh fill instead of joining a dead connection. An attempt that receives nothing at all fails after the -upstream-idle-timeout, sixty seconds by default. Time a slow client spends between reads does not count against it. Apt and Docker downloads give up at once; the next scene shows the others asking again.

05 A broken download asks again for administrators for CI operators

ASKING AGAIN snap LXD image file by checksum hash known before the download fill upstream bytes kept Range: bytes=<kept>- up to 5 tries in a row, after 1 · 2 · 4 · 8 · 16 s client 1 client 2 waiting for the next byte upstream-retry-for 2m from the last byte · every byte starts it again the hash covers the whole file, whichever try each byte came from a file that changed between tries is not cached
A fill of a snap, LXD image or file by checksum breaks off; it asks the upstream for the rest with a Range request from the bytes it kept, up to five times after waits of 1, 2, 4, 8 and 16 seconds, while its clients wait; it stops two minutes after the last byte; the hash covers the whole file

A Range: bytes=<kept>- retry, 5 in a row after 1, 2, 4, 8 and 16 s, while followers wait. -upstream-retry-for (2m after the last byte; 0 never asks again). Also retried before the body: a dropped connection and 408, 429, 500, 502, 503, 504, a proxy's included.

Snaps, LXD images and files by checksum have their hash known before the download starts, so a fill that breaks off can ask the upstream for the rest. It sends a range request from the last byte it kept, up to five times in a row, waiting one, two, four, eight and sixteen seconds in between. The clients following the fill keep waiting for the next byte. Every byte that arrives starts the count again, and the fill stops asking two minutes after the last one, which is the -upstream-retry-for setting. The hash still covers the whole file, so a file that changed between tries is not cached.

06 Content that fails its hash for administrators for CI operators

HASH MISMATCH download complete hash does not match: not kept drain apt · snaps · LXD images every byte, as the upstream sent it client its own check rejects it no new readers · wait for these up to 10m · then discard hold the last byte Docker blobs · files by checksum all but the last byte client response ends short a whole, wrong file is never handed out
Two policies for a download that fails its hash. Left, drain: apt, snap and LXD clients receive every byte and reject it themselves, while the fill waits up to ten minutes before discarding. Right, hold the last byte: Docker and files clients get all but the last byte, and the response ends short

Drain (apt, snap, LXD): followers get the origin's bytes, bounded by -fill-drain-timeout (10m), for apt, snaps and LXD images alike. Hold the last byte (Docker blobs, files): a mismatch ends the response short.

A finished download that does not match its hash is not kept. What the clients already receiving it get depends on the side. For apt, snaps and LXD images, they get the upstream's bytes unchanged, and their own checks reject them. The fill stops taking new readers, waits for these ones, up to ten minutes, and then throws the bytes away. Docker blobs and files by checksum hold back the last byte until the fill commits, so on a mismatch the response ends one byte short.

07 Where to go next for administrators for CI operators in place of the general version

WHERE NEXT fifty clients asking for one file cost one download, and a download is kept only when it checks out you are here Fetch once The store where a download is written and committed Stalled upstreams when a silent upstream is given up on pool and dists which apt files stream and which arrive whole
You are here, Fetch once; next: The store, Stalled upstreams, pool and dists

Next: The store, for where a download is written; Stalled upstreams, for when an upstream goes silent; pool and dists, for apt's files.

Now you know why fifty clients asking for one file cost one download, and why a download is kept only when it checks out. To see where a download is written and how it is committed, watch The store. To see when a silent upstream is given up on, watch Stalled upstreams. To see which apt files stream and which arrive whole, watch pool and dists.

08 Where to go next general version

WHERE NEXT fifty clients asking for one file cost one download, and a download is kept only when it checks out you are here Fetch once Stalled upstreams when a silent upstream is given up on pool and dists which apt files stream and which arrive whole Immutable or not what is kept forever and what is rechecked
You are here, Fetch once; next: Stalled upstreams, pool and dists, Immutable or not

Next: Stalled upstreams, for when an upstream goes silent; pool and dists, for apt's files; Immutable or not, for what is kept and what is rechecked.

Now you know why fifty clients asking for one file cost one download, and why a download is kept only when it checks out. To see when a silent upstream is given up on, watch Stalled upstreams. To see which apt files stream and which arrive whole, watch pool and dists. To see what is kept forever and what is checked again, watch Immutable or not.

Read more

for administrators for CI operators

When a download goes wrong

What happensSnaps, LXD images, files by checksumApt packages, Docker blobs
the upstream breaks off mid-body, or fails before it starts with a dropped connection, 408, 429, 500, 502, 503 or 504asked again for the rest, up to 5 times in a row after 1, 2, 4, 8 and 16 s, within upstream-retry-for (2m) of the last byte; followers keep waitingthe fill fails, followers are cut short
a Retry-After longer than what is left of upstream-retry-forfails at once—
in store-overloaded, the host's breaker is open when a retry is duewaits for the breaker to let it through, within upstream-retry-for—
an error answer instead of the fileat most 64 KiB of it is read—
more bytes than the store or image server promised (snaps, LXD images)the fill fails, followers are cut short—
the bytes do not match the hash, whichever tries they came fromnot cachednot cached

A cached file that is replaced while it is sent, such as an InRelease that another machine's apt update refreshed just then, is sent with the size, ETag and Last-Modified of the copy actually sent.

In the documentation

Roads from here

START HERE

CORE · the path every side shares

APT

SNAP

FILES