internal/fill
contributors for contributors
The fill package, which serves a large blob while it downloads into the store, with its table of fills in progress, tailing, range answers, the detached fetch and the handling of content that fails its hash check.
On the map: inside pkgproxy
01 What this topic covers
How internal/fill lets one download serve every client that asks for the same file while it downloads, and what happens when the file fails its hash check.
internal/fill is the package that lets one download serve every client asking for the same file, even while it downloads. If you have not seen A request in code, start there. You will see where fills sit and the table of fills in progress. Then you see how readers follow the bytes as they are written, and how Serve answers ranges from them. Last come the fetch that outlives its client, and content that fails its hash check.
02 Where fills sit in the request path
A handler calls Group.Serve with a fill.Spec:
Fetch fills the Store (often a
fill.Download), and Cached,
Streaming and Failed are the side's own answers.
Small objects go through internal/flight instead.
The fill package, in internal/fill, sits between a side's handler and its store. On a miss for a large blob, the handler calls Serve on its group of fills with a spec: the store, the key, a fetch function that downloads into the store, and three callbacks. Cached answers from a committed object, Streaming sets the side's headers before a response follows the download, and Failed answers an error. Apt's pool, snap blobs, LXD image files, Docker blobs and files by checksum come through here. Go modules, Docker manifests and apt's dists files use the simpler shared fetch of the flight package.
03 The table of fills in progress
fill.Group maps keys to *Fill under one mutex.
join joins a fill, or finds the object in the
Store, or starts run under
context.WithoutCancel. Commit happens before the fill leaves
the table.
Each side that serves large blobs keeps a table of the fills in progress, by cache key. A request takes the table's lock and looks up its key. If a fill is there, the request joins it. If not, it checks the store, and a committed object is served as a hit. Only then does it start a new fill, on its own goroutine, with a context detached from the request. A fill leaves the table only after it has committed, so a key found in neither place really is missing.
04 Following the bytes as they are written
An ObjectWriter that is also an io.ReaderAt can
be tailed. progressWriter advances written and
closes the changed channel; readers wait on it, then switch to
OpenRange on the committed object.
The store's writer can be read back while it is being written: the local store reads its temporary file, the memory store its buffer. Each write advances a counter and wakes the readers waiting for it. A reader copies up to that point, then waits for more. When the fill commits, the temporary file is closed, and a reader that is not finished switches to the committed object. A reader that reaches the end waits until the fill has committed, so its next request is a hit.
05 Answering from the tail, with ranges
Serve hands httprange.Serve an
Entity whose Open(offset, length) returns a
tailing reader; its FlushHeaders sends the
status at once. With Spec.Size there are no validators yet,
so an If-Range fails closed and the whole body is sent.
Without a tail, serveAfterwards calls Cached.
Serve answers as soon as the upstream's headers are in. Streaming sets the side's headers, then the range package plans the answer from the upstream's size, ETag and Last-Modified. A matching If-None-Match or If-Modified-Since gets a 304. A Range resume whose If-Range matches gets a 206, read from the tail at its offset, so apt can resume a package that is still downloading. A spec with a size known in advance, as snap's is, answers before the upstream does. A fill that cannot be tailed waits, and is answered from the committed object.
06 A detached fetch, and clients that leave
join starts go g.run(context.WithoutCancel(ctx), …);
each reader selects on its own r.Context().
idletimeout.Transport bounds the fetch by inactivity
(-upstream-idle-timeout, default 60s), not total time.
The fetch runs under context.WithoutCancel of the request that started it: it keeps the request's values but none of its cancellation. Each reader uses its own request's context. A client that hangs up ends only its own reader, and Serve's deferred leave drops it from the fill's count of readers. The download carries on. What bounds it is the -upstream-idle-timeout on the transport: an upstream that sends nothing for sixty seconds fails the fill, and nothing is cached. A client that retries joins the fill still running, or finds the object committed.
07 Content that fails its hash check
Fill.Abandon(timeout) forgets the fill, wakes its readers and
drains. Apt, snap and LXD pass -fill-drain-timeout
(fill.DefaultDrainTimeout, 10m, when unset). Spec.HoldLastByte reads the last byte
only from the committed object.
A fetch hashes the body while it writes it, and commits only if the hash matches. On a mismatch the side chooses what its followers get. Apt, snap and LXD call Abandon: the fill leaves the table, so a new request starts a new fill, and it waits up to a drain timeout for its readers, who get the upstream's bytes unchanged. All three take that timeout from the -fill-drain-timeout flag, ten minutes by default. Docker blobs and files by checksum set HoldLastByte instead: no reader gets the last byte before the commit, so a bad file ends short.
08 Where to go next general version
Next: Store and keys, for the store a fill writes; How it is tested, for joins and timeouts in a bubble; Fetch once, for the behaviour from outside.
You now know how one download serves every reader of a file, and why it reaches the store only when its hash matches. To see the store it writes into and the commit that makes a file visible, watch Store and keys. To see how its joins and timeouts are tested without sleeping, watch How it is tested. To see the same behaviour as a client sees it, watch Fetch once.
Read more
A key's states in a Group
fill.Group maps keys to *Fill under one mutex; join holds it while it looks up the map and, on a miss, calls Store.Stat.
| State | How it got there | What a new request gets |
|---|---|---|
| Not in the map, not in the Store | nothing fetched it, or a fetch failed | join starts go g.run(context.WithoutCancel(ctx), …); this request is the starter (started) |
| Not in the map, committed in the Store | a fill committed and left the map | Cached(w, r, meta, true) |
| In the map, headers not in yet | the fetch has not called Begin | waits on ready; with Spec.Size > 0 it answers at once instead |
| In the map, tailable | Begin got a known size and a writer that is an io.ReaderAt | Streaming(w, r, meta, !started), then httprange.Serve reading from a tailing reader |
| In the map, not tailable | unknown size, or a writer that cannot be read back | waits for done, then Cached(…, !started) or Failed (serveAfterwards) |
| Abandoned | the fetch called Abandon: forgotten from the map, closing set | starts a new fill; the readers already following keep reading until the drain timeout |
| Done | run forgets the key, then finish closes done | as the first two rows: committed, or missing again |
A fill commits to the Store before it leaves the map, so a key found in neither is really missing.
Exported API
| Name | Kind | What it does |
|---|---|---|
Group | type | The fills in progress for one Store, by key; zero value ready |
Group.Serve(w, r, Spec) | method | Answers from the Store, or joins or starts the fill and answers once the upstream's headers are in (200, 206 for a Range resume, 304) |
Spec | type | What Serve needs from a side (below) |
Fetch | func type | func(ctx, *Fill) error: download into the Store, Begin once the headers are in, write through Writer, commit only if it verifies; ctx is detached from every client |
Fill | type | One fill in progress |
Fill.Begin(meta, w) | method | Publishes the metadata (Size -1 if unknown) and, if w can be read back and the size is known, the pending bytes |
Fill.Writer(w) | method | Wraps the Store's writer so each write advances written and wakes the readers |
Fill.Err() | method | The fetch's error, once it is done |
Fill.Abandon(timeout) | method | Content not cacheable: forget the fill, let the readers already following get the bytes, wait for them up to timeout, then cut off stragglers |
Download | type | A Fetch for content-addressed objects at a plain URL |
Download.Fetch(ctx, f) | method | GET, Create, Begin, hash while copying, commit only if the hash (and size) match |
Passthrough(w, r, client, url) | func | A fresh, uncached, streamed GET for a client whose fill failed verification before anything was streamed |
BodyLength(r, meta) | func | How many bytes Serve streams for r from a fill in progress |
DefaultDrainTimeout | const | 10 minutes |
ErrHashMismatch | var | Content did not hash to what it was requested by; nothing is cached |
ErrSizeMismatch | var | Content not of Download.Size; nothing is cached |
ErrTooLarge | var | Content larger than Download.MaxSize; nothing is cached |
Spec fields
| Field | Meaning |
|---|---|
Store, Key | Where the object is, committed or pending |
Fetch | Starts the download when the object is neither stored nor being filled |
HoldLastByte | No follower gets the last byte before the commit: a bad object ends short |
Size | Known in advance: a miss is answered without waiting for the upstream's headers |
Cached(w, r, meta, hit) | Answers from the committed object; hit is false for the request that started the fill |
Streaming(w, r, meta, hit) | Sets the side's headers and records the serve before a response streams from the fill; Serve sets the status, Content-Length and Content-Range |
Failed(w, r, err) | Answers a fill that failed before anything was streamed, or a Store lookup that failed |
Download fields
| Field | Meaning |
|---|---|
Client | Nil means http.DefaultClient |
URL, Store, Key | What to fetch and where to keep it |
Algo, NewHash, Hex | The hash's name for ContentHash, its constructor, and the digest the content must have |
Now | Stamps StoredAt; nil means time.Now |
DrainTimeout | Positive: on a mismatch, Abandon(DrainTimeout); zero leaves it to Spec.HoldLastByte |
MaxSize | Positive: a larger announced size is not read, and more bytes than that fail the fetch |
Size | Positive: the content must have exactly this size; also lets followers tail an upstream that announces none |
RecordOrigin | Store the URL after redirects as OriginURL |
What each side passes to Serve
| Side | Key | Fetch | Size | On a bad hash |
|---|---|---|---|---|
apt (proxy.serveFilled, pool files only) | cachekey key | fetchStoreAndIndex, checked against the index | none | Abandon(-fill-drain-timeout) in storeResponseBody; a client that got nothing yet is answered by a fresh passthrough |
snap (snapproxy) | sha3-384/<hex> | fill.Download, sha3-384, DrainTimeout from -fill-drain-timeout | from the store's answer | Abandon |
LXD (lxdproxy/blob.go) | <remote>/<path> | fill.Download, sha256 from the catalog, DrainTimeout from -fill-drain-timeout | from the catalog | Abandon |
Docker (registryproxy.serveFilled) | blobs/sha256/<hex> | fetchBlob, checked against the digest | none | HoldLastByte |
files (filesproxy.serveFill) | sha256/<hex> or sha512/<hex> | fill.Download with MaxSize (-files-max-size), Size (size=), RecordOrigin | none | HoldLastByte |
Go's .mod and .zip, Docker manifests and apt's dists/** files are not streamed: one shared fetch through flight.Group, served once it is stored.
In the documentation
Roads from here
CORE · the path every side shares
- Fetch once — the behaviour this code implements
INSIDE PKGPROXY · for contributors
- Store and keys — the store a fill writes into and tails
- The sides' packages — the sides that call Serve
- A request in code — how a request reaches a side
- How pkgproxy is tested — synctest for timeouts and joins