internal/fill

contributors for contributors

The fill package, which serves a large blob while it downloads into the store, with its table of fills in progress, tailing, range answers, the detached fetch and the handling of content that fails its hash check.

On the map: inside pkgproxy

▶ Watch this deck — 8 scenes, ~5 min

01 What this topic covers

ABOUT THIS TOPIC internal/fill one download serves every client asking for the same file 1 Where fills sit 2 The table of fills in progress 3 Following the bytes as they are written 4 Answering from the tail, with ranges 5 A detached fetch 6 Content that fails its hash check
About this topic, internal/fill: where fills sit, the table, tailing, ranges, the detached fetch, the hash check

How internal/fill lets one download serve every client that asks for the same file while it downloads, and what happens when the file fails its hash check.

internal/fill is the package that lets one download serve every client asking for the same file, even while it downloads. If you have not seen A request in code, start there. You will see where fills sit and the table of fills in progress. Then you see how readers follow the bytes as they are written, and how Serve answers ranges from them. Last come the fetch that outlives its client, and content that fails its hash check.

02 Where fills sit in the request path

IN THE REQUEST PATH client side handler internal/fill Store h.fills.Serve(w, r, fill.Spec{ Store: h.Store, Key: key, }) Fetch:func(ctx, f) error// download into the store Cached:answer from the committed object Streaming:side's headers, then follow the download Failed:answer an error internal/fill apt pool/** snap blobs LXD image files Docker blobs files by checksum internal/flight Go modules Docker manifests apt dists/**
A client request reaches a side handler, which calls Group.Serve in internal/fill with a Spec naming the store, the key, a Fetch function and three callbacks; below, the large blobs that go through fills and the small objects that use internal/flight

A handler calls Group.Serve with a fill.Spec: Fetch fills the Store (often a fill.Download), and Cached, Streaming and Failed are the side's own answers. Small objects go through internal/flight instead.

The fill package, in internal/fill, sits between a side's handler and its store. On a miss for a large blob, the handler calls Serve on its group of fills with a spec: the store, the key, a fetch function that downloads into the store, and three callbacks. Cached answers from a committed object, Streaming sets the side's headers before a response follows the download, and Failed answers an error. Apt's pool, snap blobs, LXD image files, Docker blobs and files by checksum come through here. Go modules, Docker manifests and apt's dists files use the simpler shared fetch of the flight package.

03 The table of fills in progress

IN PROGRESS GET …/hello_2.10-3_amd64.deb the apt side's fill.Group · key → *Fill · one mutex deb.debian.org/debian/pool/…/hello_2.10-3_amd64.deb deb.debian.org/debian/pool/main/a/apt/apt_2.6.1_amd64.deb security.debian.org/debian-security/pool/… in the table join it readers + 1, under the lock not there, but in the store a hit a fill committed meanwhile in neither start a new fill go run(WithoutCancel) fetch → Commit → leave the table a key in neither place really is missing
A request takes the lock on the table of fills in progress and looks up its key; three outcomes: join the fill found, serve the committed object from the store, or start a new detached fill

fill.Group maps keys to *Fill under one mutex. join joins a fill, or finds the object in the Store, or starts run under context.WithoutCancel. Commit happens before the fill leaves the table.

Each side that serves large blobs keeps a table of the fills in progress, by cache key. A request takes the table's lock and looks up its key. If a fill is there, the request joins it. If not, it checks the store, and a committed object is served as a hit. Only then does it start a new fill, on its own goroutine, with a context detached from the request. A fill leaves the table only after it has committed, so a key found in neither place really is missing.

04 Following the bytes as they are written

TAILING upstream body Write · then written += n · wake readers temporary file, read back with ReadAt bytes written so far written reader A reads up to written reader B waits for more committed object temp file closed: switch over the end of the object is answered only after commit, so the next request is a hit
A temporary file being written, with a mark at how much is written; two readers at different offsets read up to the mark and wait; after commit, a reader switches to the committed object

An ObjectWriter that is also an io.ReaderAt can be tailed. progressWriter advances written and closes the changed channel; readers wait on it, then switch to OpenRange on the committed object.

The store's writer can be read back while it is being written: the local store reads its temporary file, the memory store its buffer. Each write advances a counter and wakes the readers waiting for it. A reader copies up to that point, then waits for more. When the fill commits, the temporary file is closed, and a reader that is not finished switches to the committed object. A reader that reaches the end waits until the fill has committed, so its next request is a hit.

05 Answering from the tail, with ranges

ANSWERING upstream headers size · ETag · Last-Modified Streaming the side's headers httprange.Serve plans against those validators 304 Not Modified If-None-Match · If-Modified-Since 206 from the tail, at offset N Range: bytes=N- with a matching If-Range pending bytes the reader, from N N written so far Spec.Size known in advance: answer before the upstream no validators yet, so If-Range fails closed: 200 serveAfterwards no length, or a writer without ReadAt wait for the fill, then Cached
The upstream's headers reach Streaming and then httprange.Serve, which answers 304 for a matching conditional request or 206 from the tail at the requested offset; a Spec.Size answers before the upstream, and a fill without a tail waits for the committed object

Serve hands httprange.Serve an Entity whose Open(offset, length) returns a tailing reader; its FlushHeaders sends the status at once. With Spec.Size there are no validators yet, so an If-Range fails closed and the whole body is sent. Without a tail, serveAfterwards calls Cached.

Serve answers as soon as the upstream's headers are in. Streaming sets the side's headers, then the range package plans the answer from the upstream's size, ETag and Last-Modified. A matching If-None-Match or If-Modified-Since gets a 304. A Range resume whose If-Range matches gets a 206, read from the tail at its offset, so apt can resume a package that is still downloading. A spec with a size known in advance, as snap's is, answers before the upstream does. A fill that cannot be tailed waits, and is answered from the committed object.

06 A detached fetch, and clients that leave

DETACHED client A fill readers counted WithoutCancel(ctx) upstream client B each reader selects on its own r.Context() hangs up: its reader ends, deferred leave() the download carries on idletimeout.Transport · -upstream-idle-timeout 60s no progress for 60s: the fetch fails, nothing is cached client A again joins the fill still running, or a hit
Client A starts a fill whose fetch runs under context.WithoutCancel; client B joins; client A hangs up and only its reader ends; the download carries on, bounded by the upstream idle timeout; A's retry joins the fill or finds the object cached

join starts go g.run(context.WithoutCancel(ctx), …); each reader selects on its own r.Context(). idletimeout.Transport bounds the fetch by inactivity (-upstream-idle-timeout, default 60s), not total time.

The fetch runs under context.WithoutCancel of the request that started it: it keeps the request's values but none of its cancellation. Each reader uses its own request's context. A client that hangs up ends only its own reader, and Serve's deferred leave drops it from the fill's count of readers. The download carries on. What bounds it is the -upstream-idle-timeout on the transport: an upstream that sends nothing for sixty seconds fails the fill, and nothing is cached. A client that retries joins the fill still running, or finds the object committed.

07 Content that fails its hash check

VERIFICATION upstream body store writer + hasher match: Commit mismatch: the side chooses Fill.Abandon(timeout) apt pool snap LXD leaves the table: a new request starts a new fill waits up to the timeout for the readers it has followers get the upstream's bytes unchanged apt: -fill-drain-timeout, 10m snap, LXD: the same flag, 10m Spec.HoldLastByte Docker blobs files by checksum the last byte only from the committed object a bad file: the response ends short never a complete wrong body
The upstream body is written to the store and hashed; a match commits; a mismatch goes one of two ways: Fill.Abandon for apt, snap and LXD, which drains the followers for a timeout, or Spec.HoldLastByte for Docker blobs and files, which ends a bad response short

Fill.Abandon(timeout) forgets the fill, wakes its readers and drains. Apt, snap and LXD pass -fill-drain-timeout (fill.DefaultDrainTimeout, 10m, when unset). Spec.HoldLastByte reads the last byte only from the committed object.

A fetch hashes the body while it writes it, and commits only if the hash matches. On a mismatch the side chooses what its followers get. Apt, snap and LXD call Abandon: the fill leaves the table, so a new request starts a new fill, and it waits up to a drain timeout for its readers, who get the upstream's bytes unchanged. All three take that timeout from the -fill-drain-timeout flag, ten minutes by default. Docker blobs and files by checksum set HoldLastByte instead: no reader gets the last byte before the commit, so a bad file ends short.

08 Where to go next general version

WHERE NEXT one download serves every reader of a file, and reaches the store only when its hash matches you are here internal/fill Store and keys the store a fill writes and commits How it is tested joins and timeouts in a synctest bubble Fetch once the same behaviour from the outside
Where next from internal/fill: Store and keys, How it is tested, Fetch once

Next: Store and keys, for the store a fill writes; How it is tested, for joins and timeouts in a bubble; Fetch once, for the behaviour from outside.

You now know how one download serves every reader of a file, and why it reaches the store only when its hash matches. To see the store it writes into and the commit that makes a file visible, watch Store and keys. To see how its joins and timeouts are tested without sleeping, watch How it is tested. To see the same behaviour as a client sees it, watch Fetch once.

Read more

A key's states in a Group

fill.Group maps keys to *Fill under one mutex; join holds it while it looks up the map and, on a miss, calls Store.Stat.

StateHow it got thereWhat a new request gets
Not in the map, not in the Storenothing fetched it, or a fetch failedjoin starts go g.run(context.WithoutCancel(ctx), …); this request is the starter (started)
Not in the map, committed in the Storea fill committed and left the mapCached(w, r, meta, true)
In the map, headers not in yetthe fetch has not called Beginwaits on ready; with Spec.Size > 0 it answers at once instead
In the map, tailableBegin got a known size and a writer that is an io.ReaderAtStreaming(w, r, meta, !started), then httprange.Serve reading from a tailing reader
In the map, not tailableunknown size, or a writer that cannot be read backwaits for done, then Cached(…, !started) or Failed (serveAfterwards)
Abandonedthe fetch called Abandon: forgotten from the map, closing setstarts a new fill; the readers already following keep reading until the drain timeout
Donerun forgets the key, then finish closes doneas the first two rows: committed, or missing again

A fill commits to the Store before it leaves the map, so a key found in neither is really missing.

Exported API

NameKindWhat it does
GrouptypeThe fills in progress for one Store, by key; zero value ready
Group.Serve(w, r, Spec)methodAnswers from the Store, or joins or starts the fill and answers once the upstream's headers are in (200, 206 for a Range resume, 304)
SpectypeWhat Serve needs from a side (below)
Fetchfunc typefunc(ctx, *Fill) error: download into the Store, Begin once the headers are in, write through Writer, commit only if it verifies; ctx is detached from every client
FilltypeOne fill in progress
Fill.Begin(meta, w)methodPublishes the metadata (Size -1 if unknown) and, if w can be read back and the size is known, the pending bytes
Fill.Writer(w)methodWraps the Store's writer so each write advances written and wakes the readers
Fill.Err()methodThe fetch's error, once it is done
Fill.Abandon(timeout)methodContent not cacheable: forget the fill, let the readers already following get the bytes, wait for them up to timeout, then cut off stragglers
DownloadtypeA Fetch for content-addressed objects at a plain URL
Download.Fetch(ctx, f)methodGET, Create, Begin, hash while copying, commit only if the hash (and size) match
Passthrough(w, r, client, url)funcA fresh, uncached, streamed GET for a client whose fill failed verification before anything was streamed
BodyLength(r, meta)funcHow many bytes Serve streams for r from a fill in progress
DefaultDrainTimeoutconst10 minutes
ErrHashMismatchvarContent did not hash to what it was requested by; nothing is cached
ErrSizeMismatchvarContent not of Download.Size; nothing is cached
ErrTooLargevarContent larger than Download.MaxSize; nothing is cached

Spec fields

FieldMeaning
Store, KeyWhere the object is, committed or pending
FetchStarts the download when the object is neither stored nor being filled
HoldLastByteNo follower gets the last byte before the commit: a bad object ends short
SizeKnown in advance: a miss is answered without waiting for the upstream's headers
Cached(w, r, meta, hit)Answers from the committed object; hit is false for the request that started the fill
Streaming(w, r, meta, hit)Sets the side's headers and records the serve before a response streams from the fill; Serve sets the status, Content-Length and Content-Range
Failed(w, r, err)Answers a fill that failed before anything was streamed, or a Store lookup that failed

Download fields

FieldMeaning
ClientNil means http.DefaultClient
URL, Store, KeyWhat to fetch and where to keep it
Algo, NewHash, HexThe hash's name for ContentHash, its constructor, and the digest the content must have
NowStamps StoredAt; nil means time.Now
DrainTimeoutPositive: on a mismatch, Abandon(DrainTimeout); zero leaves it to Spec.HoldLastByte
MaxSizePositive: a larger announced size is not read, and more bytes than that fail the fetch
SizePositive: the content must have exactly this size; also lets followers tail an upstream that announces none
RecordOriginStore the URL after redirects as OriginURL

What each side passes to Serve

SideKeyFetchSizeOn a bad hash
apt (proxy.serveFilled, pool files only)cachekey keyfetchStoreAndIndex, checked against the indexnoneAbandon(-fill-drain-timeout) in storeResponseBody; a client that got nothing yet is answered by a fresh passthrough
snap (snapproxy)sha3-384/<hex>fill.Download, sha3-384, DrainTimeout from -fill-drain-timeoutfrom the store's answerAbandon
LXD (lxdproxy/blob.go)<remote>/<path>fill.Download, sha256 from the catalog, DrainTimeout from -fill-drain-timeoutfrom the catalogAbandon
Docker (registryproxy.serveFilled)blobs/sha256/<hex>fetchBlob, checked against the digestnoneHoldLastByte
files (filesproxy.serveFill)sha256/<hex> or sha512/<hex>fill.Download with MaxSize (-files-max-size), Size (size=), RecordOriginnoneHoldLastByte

Go's .mod and .zip, Docker manifests and apt's dists/** files are not streamed: one shared fetch through flight.Group, served once it is stored.

Roads from here

CORE · the path every side shares

INSIDE PKGPROXY · for contributors