Store and keys
contributors for contributors
The store.Store interface with its fslocal and memstore implementations, the object metadata in an extended attribute of each file, how Commit makes an object visible, and how apt's cache keys and policies reach the store.
On the map: inside pkgproxy
01 What this topic covers
The store interface, its metadata and commit, and how apt's cache keys and policies reach it.
The store is where each side keeps what it downloads, behind one interface in internal/store. If you have not seen A request in code, start there. You will see the store and its keys in the code, and one interface with its two implementations. Then you see the metadata each file carries in an extended attribute, and why nothing is visible before commit. Last come apt's cache key and index packages, and how a policy reaches the store.
02 The store and its keys in the code
internal/store defines Store and
ObjectMeta; fslocal.New is called once per side
on -cache-dir/<side>; memstore serves the
tests. Keys: cachekey.PathResolver for apt, simple prefixes
such as sha3-384/ or blobs/sha256/ elsewhere.
Every side reaches its cache through one interface, in internal/store. The daemon opens one local store per side, each in its own directory under the cache directory: apt, snap, go, docker, lxd and files, each holding an objects directory and a tmp directory. A side turns a request into a key, a slash separated path inside its store. For apt that is the cachekey package's job, which also picks the policy. Tests use the memory store instead of the disk.
03 One interface, two implementations
store.Store: Stat, Open,
OpenRange, Create (an ObjectWriter
with Commit/Discard, optionally
io.ReaderAt), UpdateMeta, Delete.
fslocal on disk, memstore in tests.
Every side talks to its cache through one interface, store.Store. Stat and Open read a committed object, and OpenRange reads part of one, for resumes. Create returns a writer, which becomes visible only when it is committed, or is discarded. UpdateMeta replaces the metadata without touching the body, and Delete removes an entry. Bodies are streams, so a large file is never held in memory. The local store keeps files inside an os.Root, so no key can escape the cache directory. The memory store is for tests.
04 Object metadata in an extended attribute
store.ObjectMeta, marshalled with encoding/json
into the user.pkgproxy.meta xattr of
objects/<key>, gzipped above 3 KiB (64 KiB of JSON at
most; OriginURL is dropped if it would not fit). ContentHash is
<algo>:<digest> in the side's own terms.
<key>.meta.json holds it too, for pkgproxy 0.11
(§25.3).
What the store knows about an object without reading its body is its object meta. Size; the origin's Last-Modified, sent back when the entry is checked again; an ETag, for clients' own conditional and range requests; StoredAt, which the -revalidate-ttl counts from; the ContentHash of what was written; and Immutable, for what is kept forever. Three fields only some sides set: ContentType where the key cannot tell it, MaxAge when the origin announces its own lifetime, and OriginURL for files by checksum. The local store keeps it as JSON in the user.pkgproxy.meta extended attribute of the object's own file, compressed with gzip when it is larger than three kilobytes, and writes the same JSON to a .meta.json file beside it, for 0.11.
05 Nothing is visible before Commit
fslocal: tmp/obj-<hex>,
setAttr, Sync, Close,
Rename to objects/<key>, with
<key>.meta.json removed before and rewritten after. A
reader that has the old file open keeps reading it. defer w.Discard() is
always safe. memstore copies its buffer on
Commit and copies out on read.
Create opens a temporary file in tmp, with a random name, and the writer writes there. Commit sets the metadata as the file's extended attribute, syncs and closes it, and renames it to its place under objects. One rename switches the body and its metadata together, and a reader takes the metadata from the file it opened, so it never pairs one body with another's metadata. The .meta.json file for 0.11 is removed before the rename and written after it, so it never describes another body either. Discard removes the temporary file and does nothing after Commit, so callers defer it. The memory store buffers the writes and copies them into its map on Commit, so tests get the same contract.
06 Cache keys and the index
cachekey.PathResolver returns Resolved;
internal/apt parses Release and stanzas;
index.Store implements index.Lookup. The proxy
handler in internal/proxy wires them together.
The cachekey package turns the request URL into a resolved value: the key, the archive id, the path inside the archive, the policy, and for a by-hash path the hash from the URL. The apt package parses Release, InRelease and Packages stanzas, keeping only file names, sizes and hashes. The index package holds what was learned, in memory, and is asked for the expected hash before each write. All three are pure logic, unit tested without a network.
07 How a policy reaches the store
PolicyImmutable commits Immutable: true;
PolicyRevalidate is checked against StoredAt and
-apt-revalidate-ttl (by default -revalidate-ttl), and touch refreshes it with
UpdateMeta on a 304. PolicyPassthroughOnly goes
to passthrough. cachekey.KeyPolicy serves
stats.IsContent.
The cache key's policy decides what the store holds. Immutable, for pool and by-hash paths, is committed with Immutable set and its ContentHash, and is served from then on without asking the origin. Revalidate, for the other dists files, is committed with the origin's Last-Modified. Once its StoredAt is older than the -apt-revalidate-ttl, a conditional request goes out, and a 304 only rewrites the metadata, through UpdateMeta. Passthrough only has no key and never reaches the store. KeyPolicy reads the policy back from a key, for the statistics.
08 Where to go next general version
Next: internal/fill, for clients reading while the store is written; The sides' packages, for each side's keys; The store, for the cache on disk.
You now know how a side writes each object to a temporary file and makes it visible, with its metadata, in one commit. To see fills write into the store while clients read from it, watch internal/fill. To see how each side builds its keys, watch The sides' packages. To see the store as an administrator finds it on disk, watch The store.
Read more
store.Store
| Method | Contract |
|---|---|
Open(ctx, key) (io.ReadCloser, ObjectMeta, error) | The body and metadata of a committed object; ErrNotFound otherwise |
OpenRange(ctx, key, offset, length) (io.ReadCloser, ObjectMeta, error) | Bytes [offset, offset+length); length < 0 reads to the end |
Stat(ctx, key) (ObjectMeta, error) | Metadata only; use it instead of opening and discarding the body |
Create(ctx, key) (ObjectWriter, error) | A writer whose object is visible to Open and Stat only after Commit; a Commit over an existing object switches body and metadata together, and a reader that has the old body keeps reading it |
Delete(ctx, key) error | Removes the object and its metadata; a reader that has it open keeps reading it |
UpdateMeta(ctx, key, meta) error | Replaces a committed object's metadata, body untouched (a 304); an error wrapping ErrNotFound if there is none |
| Other API | Contract |
|---|---|
ObjectWriter | io.Writer, Commit(ctx, meta) error, Discard() error; Discard after Commit is a no-op, so defer w.Discard() is always safe |
ObjectWriter that is also io.ReaderAt | Reads back the bytes written so far, concurrently with writes, for a fill in progress; may fail once Commit or Discard returned |
store.Put(ctx, st, key, data, meta) | Create, Write, Commit of a small in-memory object, discarding on failure; sets meta.Size |
store.ErrNotFound | Returned by Open, OpenRange and Stat for a key with no committed object |
store.ObjectMeta
| Field | Type | Set by | Meaning |
|---|---|---|---|
Size | int64 | every side | Body length |
ETag | string | sides that relay it | The origin's ETag |
LastModified | time.Time | sides that relay it | The origin's Last-Modified, sent back when revalidating |
StoredAt | time.Time | every side | When it was stored or last revalidated; TTLs count from it |
ContentHash | string | apt on every entry; the other sides when stored | <algorithm>:<digest> in the side's own terms: SHA256:<hex> (apt's index names), sha3-384:<hex> (snaps), sha256:<hex> (LXD, registry digests, files, Go's mutable endpoints), h1:<base64> (Go's dirhash) |
Immutable | bool | content-addressed entries | Never revalidated |
ContentType | string, omitempty | Docker manifests | The origin's media type, where the key cannot tell it |
MaxAge | time.Duration, omitempty | when the origin announces one | Cache-Control: max-age that differs from the side's TTL |
OriginURL | string, omitempty | files by checksum | The URL fetched, after redirects; dropped by fslocal when the metadata would not fit otherwise |
fslocal on disk
fslocal.New(dir) creates dir, objects/ and tmp/ (mode 0755), opens an os.Root on dir, so no key and no symlink can reach outside it, and probes for extended attributes by setting user.pkgproxy.meta on a tmp/probe-<hex> file; it fails on a file system without them (tmpfs before Linux 6.6, NFSv3, FAT, and every directory on a system other than Linux), and the daemon then refuses to start. The daemon calls it once per side, on -cache-dir/<side>.
Store.MissingMeta, when set, is called with the key of an object file that has neither the attribute nor a .meta.json: what a copy of the cache that dropped extended attributes and the .meta.json files leaves. cmd/pkgproxy sets it on every side's store to warnMissingMeta (cachestore.go), which logs once per store, so a restored cache that looks empty says why.
<-cache-dir>/ default /var/cache/pkgproxy
apt/ snap/ go/ docker/ lxd/ files/
objects/<key> the body; ObjectMeta in its user.pkgproxy.meta xattr
objects/<key>.meta.json the same ObjectMeta as plain JSON (the sidecar), for pkgproxy 0.11
tmp/obj-<32 hex> a body being written
go/sumdb/ the checksum database client's own cache
lxd/tls/ the LXD HTTPS listener's certificate
stats.json the stats recorder's checkpoint
lxd-index.json the LXD blob hashes' checkpoint
| Operation | On disk |
|---|---|
Create | tmp/obj-<hex>, opened O_EXCL, mode 0644; ten tries at a fresh random name |
Commit | encodeMeta, setAttr on the temporary file, Sync, Close, MkdirAll of the key's directory, removes the old sidecar, Rename to objects/<key>, then writes the new sidecar; a crash in between leaves an object without one, and failing to write it fails nothing |
Discard | Closes and removes the temporary file; nothing after Commit |
Stat, Open, OpenRange | Open objects/<key> and read the metadata from that open file: the attribute, or the sidecar when it has none; a file with neither, or not a regular file, is ErrNotFound, and a file with neither is also passed to MissingMeta |
UpdateMeta | Replaces the attribute in place on the open file (fsetxattr, atomic), body untouched, then replaces the sidecar through a temporary file and a rename; lost if a Commit replaced the file meanwhile, as the new object has its own |
Delete | Removes the object, then its sidecar; a missing file is not an error |
MigrateSidecars | Walks objects/ and gives the attribute to each file that has only a sidecar, with XATTR_CREATE on the file it opened, so an attribute a Commit or UpdateMeta sets meanwhile stays; keeps the sidecars; returns how many it migrated |
Store.SidecarOnly, set by -cache-metadata=meta-json (the snap option cache-metadata), writes the sidecar alone, as 0.11 did, and reads it before the attribute: it makes a 0.11 cache for the spread test tests/install/migrate-meta, and gives up the atomic switch of body and metadata, so it is for tests only.
Upgrades: snap epochs (§25)
| epoch 0 | 0.11 and earlier: the metadata in <key>.meta.json alone |
| epoch 1 | the metadata in user.pkgproxy.meta |
epoch: 1* in snap/snapcraft.yaml | reads 0 and 1, writes 1; with the sidecars it still writes 0 too, because snap revert (snapstate.RevertToRevision) checks no epoch |
snap/hooks/post-refresh | runs pkgproxy -cache-dir "$SNAP_COMMON/cache" -migrate-cache into $SNAP_DATA/migrate-cache.log, and never fails, since snapd undoes a refresh whose hook fails |
-migrate-cache | migrateCache (migratecache.go): MigrateSidecars on each side's store, for at most -migrate-cache-timeout (2m); runs before the other settings are checked, then exits |
| retiring epoch 0 | a later release, most likely 0.13, declares plain epoch: 1, stops writing and reading sidecars and removes them; snapd offers it only to 1* revisions, so a revert from it lands on one (§25.6) |
The attribute holds JSON as it is up to maxMetaSize, 3072 bytes (ext4 keeps all of a file's attributes in one 4 KiB block); larger JSON, up to maxMetaJSON, 64 KiB, is compressed with gzip, whose magic bytes no JSON starts with. Metadata that still does not fit loses OriginURL, and is refused if it is still too large. getfattr --only-values -n user.pkgproxy.meta FILE | zcat shows a compressed value.
A key is a slash-separated relative path. cleanKey rejects one that relpath.Clean cleans to ., .., ../… or an absolute path, before os.Root enforces containment. memstore keeps committed objects in a map under a mutex, buffers a writer until Commit, and copies on the way in and out.
Keys by side
| Side | Store directory | Keys |
|---|---|---|
| apt | apt/ | <archive id>/dists/… and <archive id>/pool/…, from cachekey |
| snap | snap/ | sha3-384/<hex> (snaps), pending/sha3-384/<hex> (hash to CDN URL and size), info/<name>…, find/<hash>, assertions/<type>/<key>…, catalog/<hash>, refresh/… |
| Go | go/ | the GOPROXY URL space: <escaped module>/@v/<escaped version>.info, .mod, .zip, <module>/@v/list, <module>/@latest |
| Docker | docker/ | blobs/sha256/<hex>, manifests/sha256/<hex>, tags/<name>/_/<tag>/<id> (a digest per set of accepted media types) |
| LXD | lxd/ | <remote>/<path>, such as <remote>/images/<os>/<release>/<arch>/<variant>/<serial>/<file> |
| files | files/ | sha256/<hex> or sha512/<hex> |
cachekey: apt URL to key and policy
cachekey.PathResolver{}.Resolve(u) finds the first dists or pool path segment. The archive id is the host (with its port unless it is 80) plus every non-empty segment before that one; the key is <archive id>/<archive path>.
| Path class | Policy | Store contract |
|---|---|---|
…/pool/** | PolicyImmutable | Committed with Immutable: true, only if it matches the hash the parsed indexes list (a path they do not list is cached with a warning); streamed through a fill on a miss |
…/dists/**/by-hash/<ALGO>/<hex> | PolicyImmutable | As above; the hash comes from the URL (KnownHash), no index lookup |
other …/dists/** (Release, InRelease, Packages, Sources, …) | PolicyRevalidate | Committed with the origin's LastModified; once StoredAt is older than -apt-revalidate-ttl (default -revalidate-ttl, 5m) a conditional GET goes out, and a 304 calls UpdateMeta |
no dists or pool segment | PolicyPassthroughOnly | No key; relayed, never stored |
| Example URL | Key |
|---|---|
http://archive.ubuntu.com/ubuntu/pool/main/a/apt/apt.deb | archive.ubuntu.com/ubuntu/pool/main/a/apt/apt.deb |
http://repo.example:8080/debian/dists/stable/Release | repo.example:8080/debian/dists/stable/Release |
http://host:80//dists/x/Release | host/dists/x/Release |
cachekey.Resolved field | Meaning |
|---|---|
Key | <ArchiveID>/<ArchivePath>; empty for passthrough |
ArchivePath | From dists/ or pool/ on; what index.Lookup and Release entries key on |
ArchiveID | Host (and port) plus path prefix, such as archive.ubuntu.com/ubuntu |
Policy | PolicyImmutable, PolicyRevalidate, PolicyPassthroughOnly (String(): immutable, revalidate, passthrough-only) |
OriginURL | The URL to fetch upstream |
KnownHash, HasKnownHash | The apt.Hash a by-hash path names |
cachekey.KeyPolicy(key) reads the policy back from a stored key; stats.IsContent uses it to tell content from indexes.
In the documentation
Roads from here
CORE · the path every side shares
- The store — the behaviour this code implements
APT
- apt cache keys and hash checks — the cache keys from the outside
INSIDE PKGPROXY · for contributors
- internal/fill — fills that write into the store and tail it
- The sides' packages — how each side builds its keys and store
- The source tree — the source tree
- How pkgproxy is tested — memstore and unit tests without a network