Store and keys

contributors for contributors

The store.Store interface with its fslocal and memstore implementations, the object metadata in an extended attribute of each file, how Commit makes an object visible, and how apt's cache keys and policies reach the store.

On the map: inside pkgproxy

▶ Watch this deck — 8 scenes, ~6 min

01 What this topic covers

ABOUT THIS TOPIC Store and keys where each side keeps what it downloads, behind one interface 1 The store and its keys in the code 2 One interface, two implementations 3 Object metadata in an extended attribute 4 Nothing is visible before Commit 5 Cache keys and the index 6 How a policy reaches the store
About this topic, Store and keys: the store in the code, one interface, metadata in an extended attribute, commit, cache keys and the index, policies

The store interface, its metadata and commit, and how apt's cache keys and policies reach it.

The store is where each side keeps what it downloads, behind one interface in internal/store. If you have not seen A request in code, start there. You will see the store and its keys in the code, and one interface with its two implementations. Then you see the metadata each file carries in an extended attribute, and why nothing is visible before commit. Last come apt's cache key and index packages, and how a policy reaches the store.

02 The store and its keys in the code

STORE AND KEYS internal/store Store · ObjectMeta · ObjectWriter · Put /var/cache/pkgproxy/ -cache-dir apt/objects/ tmp/ snap/objects/ tmp/ go/objects/ tmp/ docker/objects/ tmp/ lxd/objects/ tmp/ files/objects/ tmp/ fslocal.Store one per side, every path through an os.Root keys: slash-separated paths sha3-384/<hex>snap blobs blobs/sha256/<hex>Docker blobs deb.debian.org/debian/pool/…cachekey memstore in memory, for tests
The cache directory holds one fslocal store per side, apt, snap, go, docker, lxd and files, each with objects and tmp; internal/store defines the interface; keys are paths such as an apt archive path or sha3-384 and a hash; memstore stands in for tests

internal/store defines Store and ObjectMeta; fslocal.New is called once per side on -cache-dir/<side>; memstore serves the tests. Keys: cachekey.PathResolver for apt, simple prefixes such as sha3-384/ or blobs/sha256/ elsewhere.

Every side reaches its cache through one interface, in internal/store. The daemon opens one local store per side, each in its own directory under the cache directory: apt, snap, go, docker, lxd and files, each holding an objects directory and a tmp directory. A side turns a request into a key, a slash separated path inside its store. For apt that is the cachekey package's job, which also picks the policy. Tests use the memory store instead of the disk.

03 One interface, two implementations

STORE.STORE type Store interface { Stat(ctx, key) (ObjectMeta, error) Open(ctx, key) OpenRange(ctx, key, offset, length) Create(ctx, key) (ObjectWriter, error) Write · Commit(meta) · Discard maybe io.ReaderAt, for tailing UpdateMeta(ctx, key, meta) Delete(ctx, key) } bodies are io.Reader / io.Writer streams fslocal objects/ and tmp/ on disk every path through an os.Root: no key escapes the directory memstore in memory, for tests
The store.Store interface with its methods on the left; two implementations on the right: fslocal on disk, inside an os.Root, and memstore in memory for tests

store.Store: Stat, Open, OpenRange, Create (an ObjectWriter with Commit/Discard, optionally io.ReaderAt), UpdateMeta, Delete. fslocal on disk, memstore in tests.

Every side talks to its cache through one interface, store.Store. Stat and Open read a committed object, and OpenRange reads part of one, for resumes. Create returns a writer, which becomes visible only when it is committed, or is discarded. UpdateMeta replaces the metadata without touching the body, and Delete removes an entry. Bodies are streams, so a large file is never held in memory. The local store keeps files inside an os.Root, so no key can escape the cache directory. The memory store is for tests.

04 Object metadata in an extended attribute

OBJECTMETA type ObjectMeta struct { } Sizeint64 ETagstring LastModifiedtime.Time StoredAttime.Time ContentHashstring Immutablebool ContentTypestring MaxAgetime.Duration OriginURLstring omitempty: set by some sides only objects/deb.debian.org/debian/pool/…/ hello_2.10-3_amd64.deb xattr user.pkgproxy.meta {"Size": 53212, "ETag": "\"cfdc-…\"", "LastModified": "2023-…Z", "StoredAt": "2026-11-07T…Z", "ContentHash": "SHA256:3a5f…", "Immutable": true} set on the file in tmp/, then renamed into place gzip over 3 KiB, up to 64 KiB of JSON and the same JSON in hello_2.10-3_amd64.deb.meta.json
The ObjectMeta struct with its nine fields on the left; on the right, an apt object on disk whose user.pkgproxy.meta extended attribute holds the same fields as JSON, as does a .meta.json file beside it

store.ObjectMeta, marshalled with encoding/json into the user.pkgproxy.meta xattr of objects/<key>, gzipped above 3 KiB (64 KiB of JSON at most; OriginURL is dropped if it would not fit). ContentHash is <algo>:<digest> in the side's own terms. <key>.meta.json holds it too, for pkgproxy 0.11 (§25.3).

What the store knows about an object without reading its body is its object meta. Size; the origin's Last-Modified, sent back when the entry is checked again; an ETag, for clients' own conditional and range requests; StoredAt, which the -revalidate-ttl counts from; the ContentHash of what was written; and Immutable, for what is kept forever. Three fields only some sides set: ContentType where the key cannot tell it, MaxAge when the origin announces its own lifetime, and OriginURL for files by checksum. The local store keeps it as JSON in the user.pkgproxy.meta extended attribute of the object's own file, compressed with gzip when it is larger than three kilobytes, and writes the same JSON to a .meta.json file beside it, for 0.11.

05 Nothing is visible before Commit

CREATE · COMMIT tmp/ objects/ obj-3f9c… Create · Write user.pkgproxy.meta setAttr Sync · Close Rename <key> user.pkgproxy.meta one rename switches the body and its metadata a crash leaves the old object or the new one a reader takes the metadata from the file it opened: never one body with another's metadata <key>.meta.json: removed before, written after defer w.Discard() removes the temporary file does nothing after Commit memstore buffers, copies into its map on Commit the same contract for the tests
Create writes to a temporary file under tmp; Commit sets the metadata attribute, syncs, closes and renames the file under objects, so one rename switches body and metadata; a reader takes the metadata from the file it opened; the .meta.json file for 0.11 is removed before the rename and written after; Discard removes the temporary file; memstore copies its buffer on Commit

fslocal: tmp/obj-<hex>, setAttr, Sync, Close, Rename to objects/<key>, with <key>.meta.json removed before and rewritten after. A reader that has the old file open keeps reading it. defer w.Discard() is always safe. memstore copies its buffer on Commit and copies out on read.

Create opens a temporary file in tmp, with a random name, and the writer writes there. Commit sets the metadata as the file's extended attribute, syncs and closes it, and renames it to its place under objects. One rename switches the body and its metadata together, and a reader takes the metadata from the file it opened, so it never pairs one body with another's metadata. The .meta.json file for 0.11 is removed before the rename and written after it, so it never describes another body either. Discard removes the temporary file and does nothing after Commit, so callers defer it. The memory store buffers the writes and copies them into its map on Commit, so tests get the same contract.

06 Cache keys and the index

IN THE CODE internal/cachekey PathResolver.Resolve(url) → Resolved{Key, ArchiveID, ArchivePath, Policy, KnownHash, HasKnownHash} PolicyImmutable · PolicyRevalidate · PolicyPassthroughOnly internal/apt ParseReleaseFile ParsePackagesIndex ParseSourcesIndex internal/index Lookup.ExpectedHash UpdateRelease · UpdateEntries Store: in memory pure logic, unit tested without a network: cachekey_test.go · release_test.go · index_test.go
Three packages: internal/cachekey resolves a URL to a key, archive id, path, policy and known hash; internal/apt parses Release and Packages stanzas; internal/index records and answers expected hashes; all unit tested

cachekey.PathResolver returns Resolved; internal/apt parses Release and stanzas; index.Store implements index.Lookup. The proxy handler in internal/proxy wires them together.

The cachekey package turns the request URL into a resolved value: the key, the archive id, the path inside the archive, the policy, and for a by-hash path the hash from the URL. The apt package parses Release, InRelease and Packages stanzas, keeping only file names, sizes and hashes. The index package holds what was learned, in memory, and is asked for the expected hash before each write. All three are pure logic, unit tested without a network.

07 How a policy reaches the store

POLICY TO STORE cachekey.Resolved.Policy what the store holds PolicyImmutable pool/** · dists/**/by-hash/** Commit{Immutable: true, ContentHash} served from then on without asking the origin PolicyRevalidate the other dists/** files Commit{LastModified, ETag, StoredAt} Immutable: false older than -apt-revalidate-ttl: If-Modified-Since 304: UpdateMeta(StoredAt = now) PolicyPassthroughOnly no dists/ or pool/ segment no key: proxied, never stored cachekey.KeyPolicy(key) reads the policy back from a key, for stats.IsContent
Three cachekey policies and what each leaves in the store: PolicyImmutable commits an immutable object with its hash; PolicyRevalidate commits with the origin's validators and refreshes StoredAt with UpdateMeta on a 304; PolicyPassthroughOnly has no key and is never stored; KeyPolicy reads a policy back from a key

PolicyImmutable commits Immutable: true; PolicyRevalidate is checked against StoredAt and -apt-revalidate-ttl (by default -revalidate-ttl), and touch refreshes it with UpdateMeta on a 304. PolicyPassthroughOnly goes to passthrough. cachekey.KeyPolicy serves stats.IsContent.

The cache key's policy decides what the store holds. Immutable, for pool and by-hash paths, is committed with Immutable set and its ContentHash, and is served from then on without asking the origin. Revalidate, for the other dists files, is committed with the origin's Last-Modified. Once its StoredAt is older than the -apt-revalidate-ttl, a conditional request goes out, and a 304 only rewrites the metadata, through UpdateMeta. Passthrough only has no key and never reaches the store. KeyPolicy reads the policy back from a key, for the statistics.

08 Where to go next general version

WHERE NEXT each object is written to a temporary file and appears, with its metadata, in one commit you are here Store and keys internal/fill clients reading while the store is written The sides' packages how each side builds its keys The store the cache as an administrator sees it
Where next from Store and keys: internal/fill, The sides' packages, The store

Next: internal/fill, for clients reading while the store is written; The sides' packages, for each side's keys; The store, for the cache on disk.

You now know how a side writes each object to a temporary file and makes it visible, with its metadata, in one commit. To see fills write into the store while clients read from it, watch internal/fill. To see how each side builds its keys, watch The sides' packages. To see the store as an administrator finds it on disk, watch The store.

Read more

store.Store

MethodContract
Open(ctx, key) (io.ReadCloser, ObjectMeta, error)The body and metadata of a committed object; ErrNotFound otherwise
OpenRange(ctx, key, offset, length) (io.ReadCloser, ObjectMeta, error)Bytes [offset, offset+length); length < 0 reads to the end
Stat(ctx, key) (ObjectMeta, error)Metadata only; use it instead of opening and discarding the body
Create(ctx, key) (ObjectWriter, error)A writer whose object is visible to Open and Stat only after Commit; a Commit over an existing object switches body and metadata together, and a reader that has the old body keeps reading it
Delete(ctx, key) errorRemoves the object and its metadata; a reader that has it open keeps reading it
UpdateMeta(ctx, key, meta) errorReplaces a committed object's metadata, body untouched (a 304); an error wrapping ErrNotFound if there is none
Other APIContract
ObjectWriterio.Writer, Commit(ctx, meta) error, Discard() error; Discard after Commit is a no-op, so defer w.Discard() is always safe
ObjectWriter that is also io.ReaderAtReads back the bytes written so far, concurrently with writes, for a fill in progress; may fail once Commit or Discard returned
store.Put(ctx, st, key, data, meta)Create, Write, Commit of a small in-memory object, discarding on failure; sets meta.Size
store.ErrNotFoundReturned by Open, OpenRange and Stat for a key with no committed object

store.ObjectMeta

FieldTypeSet byMeaning
Sizeint64every sideBody length
ETagstringsides that relay itThe origin's ETag
LastModifiedtime.Timesides that relay itThe origin's Last-Modified, sent back when revalidating
StoredAttime.Timeevery sideWhen it was stored or last revalidated; TTLs count from it
ContentHashstringapt on every entry; the other sides when stored<algorithm>:<digest> in the side's own terms: SHA256:<hex> (apt's index names), sha3-384:<hex> (snaps), sha256:<hex> (LXD, registry digests, files, Go's mutable endpoints), h1:<base64> (Go's dirhash)
Immutableboolcontent-addressed entriesNever revalidated
ContentTypestring, omitemptyDocker manifestsThe origin's media type, where the key cannot tell it
MaxAgetime.Duration, omitemptywhen the origin announces oneCache-Control: max-age that differs from the side's TTL
OriginURLstring, omitemptyfiles by checksumThe URL fetched, after redirects; dropped by fslocal when the metadata would not fit otherwise

fslocal on disk

fslocal.New(dir) creates dir, objects/ and tmp/ (mode 0755), opens an os.Root on dir, so no key and no symlink can reach outside it, and probes for extended attributes by setting user.pkgproxy.meta on a tmp/probe-<hex> file; it fails on a file system without them (tmpfs before Linux 6.6, NFSv3, FAT, and every directory on a system other than Linux), and the daemon then refuses to start. The daemon calls it once per side, on -cache-dir/<side>.

Store.MissingMeta, when set, is called with the key of an object file that has neither the attribute nor a .meta.json: what a copy of the cache that dropped extended attributes and the .meta.json files leaves. cmd/pkgproxy sets it on every side's store to warnMissingMeta (cachestore.go), which logs once per store, so a restored cache that looks empty says why.

<-cache-dir>/                 default /var/cache/pkgproxy
  apt/  snap/  go/  docker/  lxd/  files/
    objects/<key>             the body; ObjectMeta in its user.pkgproxy.meta xattr
    objects/<key>.meta.json   the same ObjectMeta as plain JSON (the sidecar), for pkgproxy 0.11
    tmp/obj-<32 hex>          a body being written
  go/sumdb/                   the checksum database client's own cache
  lxd/tls/                    the LXD HTTPS listener's certificate
  stats.json                  the stats recorder's checkpoint
  lxd-index.json              the LXD blob hashes' checkpoint
OperationOn disk
Createtmp/obj-<hex>, opened O_EXCL, mode 0644; ten tries at a fresh random name
CommitencodeMeta, setAttr on the temporary file, Sync, Close, MkdirAll of the key's directory, removes the old sidecar, Rename to objects/<key>, then writes the new sidecar; a crash in between leaves an object without one, and failing to write it fails nothing
DiscardCloses and removes the temporary file; nothing after Commit
Stat, Open, OpenRangeOpen objects/<key> and read the metadata from that open file: the attribute, or the sidecar when it has none; a file with neither, or not a regular file, is ErrNotFound, and a file with neither is also passed to MissingMeta
UpdateMetaReplaces the attribute in place on the open file (fsetxattr, atomic), body untouched, then replaces the sidecar through a temporary file and a rename; lost if a Commit replaced the file meanwhile, as the new object has its own
DeleteRemoves the object, then its sidecar; a missing file is not an error
MigrateSidecarsWalks objects/ and gives the attribute to each file that has only a sidecar, with XATTR_CREATE on the file it opened, so an attribute a Commit or UpdateMeta sets meanwhile stays; keeps the sidecars; returns how many it migrated

Store.SidecarOnly, set by -cache-metadata=meta-json (the snap option cache-metadata), writes the sidecar alone, as 0.11 did, and reads it before the attribute: it makes a 0.11 cache for the spread test tests/install/migrate-meta, and gives up the atomic switch of body and metadata, so it is for tests only.

Upgrades: snap epochs (§25)

epoch 00.11 and earlier: the metadata in <key>.meta.json alone
epoch 1the metadata in user.pkgproxy.meta
epoch: 1* in snap/snapcraft.yamlreads 0 and 1, writes 1; with the sidecars it still writes 0 too, because snap revert (snapstate.RevertToRevision) checks no epoch
snap/hooks/post-refreshruns pkgproxy -cache-dir "$SNAP_COMMON/cache" -migrate-cache into $SNAP_DATA/migrate-cache.log, and never fails, since snapd undoes a refresh whose hook fails
-migrate-cachemigrateCache (migratecache.go): MigrateSidecars on each side's store, for at most -migrate-cache-timeout (2m); runs before the other settings are checked, then exits
retiring epoch 0a later release, most likely 0.13, declares plain epoch: 1, stops writing and reading sidecars and removes them; snapd offers it only to 1* revisions, so a revert from it lands on one (§25.6)

The attribute holds JSON as it is up to maxMetaSize, 3072 bytes (ext4 keeps all of a file's attributes in one 4 KiB block); larger JSON, up to maxMetaJSON, 64 KiB, is compressed with gzip, whose magic bytes no JSON starts with. Metadata that still does not fit loses OriginURL, and is refused if it is still too large. getfattr --only-values -n user.pkgproxy.meta FILE | zcat shows a compressed value.

A key is a slash-separated relative path. cleanKey rejects one that relpath.Clean cleans to ., .., ../… or an absolute path, before os.Root enforces containment. memstore keeps committed objects in a map under a mutex, buffers a writer until Commit, and copies on the way in and out.

Keys by side

SideStore directoryKeys
aptapt/<archive id>/dists/… and <archive id>/pool/…, from cachekey
snapsnap/sha3-384/<hex> (snaps), pending/sha3-384/<hex> (hash to CDN URL and size), info/<name>…, find/<hash>, assertions/<type>/<key>…, catalog/<hash>, refresh/…
Gogo/the GOPROXY URL space: <escaped module>/@v/<escaped version>.info, .mod, .zip, <module>/@v/list, <module>/@latest
Dockerdocker/blobs/sha256/<hex>, manifests/sha256/<hex>, tags/<name>/_/<tag>/<id> (a digest per set of accepted media types)
LXDlxd/<remote>/<path>, such as <remote>/images/<os>/<release>/<arch>/<variant>/<serial>/<file>
filesfiles/sha256/<hex> or sha512/<hex>

cachekey: apt URL to key and policy

cachekey.PathResolver{}.Resolve(u) finds the first dists or pool path segment. The archive id is the host (with its port unless it is 80) plus every non-empty segment before that one; the key is <archive id>/<archive path>.

Path classPolicyStore contract
…/pool/**PolicyImmutableCommitted with Immutable: true, only if it matches the hash the parsed indexes list (a path they do not list is cached with a warning); streamed through a fill on a miss
…/dists/**/by-hash/<ALGO>/<hex>PolicyImmutableAs above; the hash comes from the URL (KnownHash), no index lookup
other …/dists/** (Release, InRelease, Packages, Sources, …)PolicyRevalidateCommitted with the origin's LastModified; once StoredAt is older than -apt-revalidate-ttl (default -revalidate-ttl, 5m) a conditional GET goes out, and a 304 calls UpdateMeta
no dists or pool segmentPolicyPassthroughOnlyNo key; relayed, never stored
Example URLKey
http://archive.ubuntu.com/ubuntu/pool/main/a/apt/apt.debarchive.ubuntu.com/ubuntu/pool/main/a/apt/apt.deb
http://repo.example:8080/debian/dists/stable/Releaserepo.example:8080/debian/dists/stable/Release
http://host:80//dists/x/Releasehost/dists/x/Release
cachekey.Resolved fieldMeaning
Key<ArchiveID>/<ArchivePath>; empty for passthrough
ArchivePathFrom dists/ or pool/ on; what index.Lookup and Release entries key on
ArchiveIDHost (and port) plus path prefix, such as archive.ubuntu.com/ubuntu
PolicyPolicyImmutable, PolicyRevalidate, PolicyPassthroughOnly (String(): immutable, revalidate, passthrough-only)
OriginURLThe URL to fetch upstream
KnownHash, HasKnownHashThe apt.Hash a by-hash path names

cachekey.KeyPolicy(key) reads the policy back from a stored key; stats.IsContent uses it to tell content from indexes.

Roads from here

CORE · the path every side shares

  • The store — the behaviour this code implements

APT

INSIDE PKGPROXY · for contributors