pkgproxy design: snap

Caching the Snap Store’s packages and assertions through the snap relay. Part of the pkgproxy design, whose index says which file holds each § number.

• 11. Snap package and assertion caching
  • 11.1 Why this needs a different shape than the apt proxy
  • 11.2 Snap Store API v2 surface pkgproxy acts on
  • 11.3 Response rewriting
  • 11.4 Cache keys and assertion mutability
  • 11.6 Explicit v1 scope boundaries
  • 11.7 Recording and synthesising refresh results

11. Snap package and assertion caching

The snap side lives in internal/snap (protocol facts) and internal/snapproxy (the handler); gitlab.com/zygoon/pkgproxy#4.

11.1 Why this needs a different shape than the apt proxy

§1’s non-goals rule out caching, verifying, or terminating TLS for HTTPS content, and §3 explains why: apt’s forward-proxy shape only works because plain http:// sources exist to actually cache, with CONNECT as a blind tunnel for the https:// remainder. The Snap Store (api.snapcraft.io and its download CDN) has no plain-HTTP fallback anywhere in the protocol — every request, including bulk package downloads, is HTTPS-only. A blind tunnel therefore can’t cache any of it, and TLS-terminating MITM is ruled out both on principle (it would reverse §1’s non-goal) and in practice: snapd is known to detect and reject TLS interception on its store connection.

Decision: pkgproxy exposes its own explicit HTTP endpoint (internal/snapproxy.Handler) implementing a relaying/caching subset of the Snap Store API v2, and snapd is pointed directly at it — not tunneled — via the SNAPPY_FORCE_API_URL environment variable override (a mechanism snapd’s own spread test suite uses to redirect all store traffic to an arbitrary URL). This makes the snap side a genuine reverse relay (pkgproxy is the origin snapd talks to) built on net/http/httputil.ReverseProxy, structurally different from internal/proxy’s forward-proxy Handler — not a variant of it. It was checked against a real snapd that refresh, install, find and the assertion fallback fetches all work through a plain-http:// relay.

The two still share one listener: internal/gateway tells them apart by request-target shape, which never overlaps. apt’s traffic is CONNECT or an absolute URI (r.URL.Host set); snapd’s is origin-form under /v2, /api/v1 and /v1. So there is one -addr and no snap enable flag.

Official Snap Store Proxy registration (a Canonical-issued store id and signed store assertion behind proxy.store=<id>) is declined: it needs a Canonical SSO account and online registration, more than a self-hosted cache should require.

-snap-external-url is empty by default, which makes the rewrite base the request’s Host, the address SNAPPY_FORCE_API_URL gave snapd. -snap-revalidate-ttl is separate from apt’s -revalidate-ttl: the two sides’ freshness needs are not coupled.

11.2 Snap Store API v2 surface pkgproxy acts on

Base host api.snapcraft.io, HTTPS only, anonymous/public-snap access only (a logged-in user’s answers are cached only when they show public snaps alone, and a brand store’s device session’s never — see §11.6).

• GET /v2/snaps/find: cached for the Cache-Control: max-age the store announces (five minutes, observed), by a key made of the search with its spelling normalised (the term lowercased with its whitespace collapsed, ASCII only, the fields as a set: the store answers “HELLO” and “hello”, “text editor” and “text editor”, byte for byte alike) and of the headers that select a response, as info is. What was searched before is therefore found again offline. A search of private snaps (private= anything but a false) is relayed, and so is a brand store’s device session; a logged-in user’s search is cached only when every result is public (§11.6). The answer holds no download URL, so nothing in it needs rewriting.
• GET /v2/snaps/info/<name>: relayed and rewritten, cached with RevalidateTTL: no per-client state, the same reasoning as apt’s dists/**. A logged-in user’s is cached only when its snap is public (§11.6).
• POST /v2/snaps/refresh: relayed and rewritten, not cached as a whole: it carries a per-client context[] (installed revisions, tracking channel), so a byte-identical repeat across machines is unlikely. It is the endpoint snap refresh and snap install use, and the one that returns download URLs and, for fetch-assertions actions, assertion-stream-urls. Its results are recorded per action (§11.7).
• GET /v2/assertions/<type>/<key...>: cached per snap.MutabilityFor (§11.4).
• GET of an embedded download.url (any CDN path): rewritten to /v2/download/sha3-384/<hex>, this handler’s own content-addressed blob endpoint (§11.3, §11.4).
• GET /v2/snaps/categories, /v2/snaps/category/<name>, /v2/repairs/...: cached with RevalidateTTL, keyed on the whole request target and the headers that select a response (Snap-Device-*, Snap-Classic, X-Ubuntu-*).
• GET /api/v1/snaps/{sections,names,search}, /api/v1/snaps, /api/v1/snaps/details/<name>: the legacy v1 search API, cached like the above. HAL+JSON packages with a download_sha3_384 get their download_url and anon_download_url rewritten to the blob endpoint (the blob fetch follows the store’s redirect to the CDN), and their _links hrefs re-based; icons and screenshots stay on the CDN.
• GET /api/v1/snaps/download/..., /api/v1/snaps/dm-verity/download/..., /api/v1/snaps/resources/..., /api/v1/resources/download/..., /api/v1/media/download/...: the store answers with a redirect to its CDN. The redirect is followed here and the body streamed to the client (Range and validators forwarded, credentials dropped on a hop to another host), uncached: there is no hash to key it by.
• /api/v1/snaps/auth/*, /api/v1/snaps/metadata, /v1/* (device sessions, serials), POST /v2/cohorts, any method: relayed uncached, headers and body untouched: a nonce is single-use, and the rest is per device.

Cached entries live for the Cache-Control: max-age the store announced for them (a day for the names catalogue, an hour for sections and categories, observed), else for RevalidateTTL. The first max-age directive decides (RFC 9111 §4.2.1), and one that is zero, negative or not a number counts as none: RevalidateTTL is a floor, kept on purpose, on how often the rate-limiting store is asked. The store’s load balancer rate-limits by request rate (a bare 429, no Retry-After), and all clients of a proxy share one address, so when the store cannot answer a revalidation (a transport error, 429 or 5xx) the expired entry is served instead, with a warning; with nothing to serve, the failure is relayed as it was, and never cached.

Concurrent misses for a key share one fetch (internal/flight). It is a GET whatever method the requests had (a HEAD is answered from the same entry), without any client’s If-* validators or Range, since the answer is not one client’s alone. It runs detached from the request that started it, with a timeout of its own, and each request waits only as long as its client does. An oversized body is an error, never a truncated entry. Failures are not remembered: backing off is the circuit breaker’s job (gitlab.com/zygoon/pkgproxy#57).

Everything else under /v2/ (unrecognized snap store paths) is still relayed and rewritten, uncached — the snap-side analog of PolicyPassthroughOnly (§4/§5’s bottom row), so an endpoint this package doesn’t specifically know about still works rather than 404ing.

11.3 Response rewriting

Unlike the apt side, which never touches response bytes (§1: “our only obligation is to serve those files byte-identical to the origin”), internal/snapproxy is a real origin here and must rewrite embedded URLs before they reach the client, or every blob/assertion fetch would bypass the cache entirely by going straight to the real CDN.

A relayed response is decoded as generic JSON and walked structurally, rather than modeled as typed structs for the whole API surface: any object with sibling "url"/"sha3-384" keys is a download object, and its "url" becomes /v2/download/sha3-384/<hex>. A delta (an entry of "deltas") has the same shape, so it is rewritten and cached like a full snap: the store’s deltas are relayed, never generated here. Any "assertion-stream-urls" entry is re-based to this handler’s host, its path and query already mirroring /v2/assertions/.... Everything else survives untouched, so unknown fields and API drift don’t break rewriting. A body that fails to parse is relayed unmodified (logged, not failed): one endpoint’s URLs may then bypass the cache, but no request breaks.

The hash → real CDN URL mapping is required, not optional: the real URL is opaque (sometimes tokenized) and cannot be derived from the hash. It is recorded durably, with the blob’s size, through the same store.Store every other object uses (a pending/sha3-384/<hex> record), so a client holding a rewritten URL across a pkgproxy restart still resolves. The record is left in place once the blob is cached, in keeping with caching indefinitely (§1).

Compression. The store’s API documentation says nothing about Accept-Encoding, but the store has been seen to send gzip-compressed JSON, which cannot be rewritten as it is. The client’s Accept-Encoding is therefore not passed upstream, and a body that is still encoded (gzip, deflate) is decoded before it is rewritten and cached, so the cache only holds decoded bodies and the client gets no Content-Encoding. An encoding the standard library cannot undo (br, zstd) is relayed as it is, with a warning, and not cached.

11.4 Cache keys and assertion mutability

A second, separate fslocal.Store instance, rooted at <cache-dir>/snap (the apt side is likewise rooted at <cache-dir>/apt) — not a shared keyspace with the apt side’s hostname-derived keys, simpler than proving the two can never collide. Each side gets its own top-level directory.

• Blobs: key sha3-384/<hex>, always Immutable: true. Real content-addressed storage, unlike apt’s path-normalization-plus- verification compromise (§4.2) — the hash is known before the fetch, from whichever relayed response referenced it.
• Assertions: key assertions/<type>/<path-suffix> (the request path suffix after /v2/assertions/, used as an opaque key — this package doesn’t need to understand each type’s primary-key field layout), with ?sequence=N/?max-format= folded in when present. internal/snap.MutabilityFor(assertionType, hasSequence) classifies the type. Any assertion can be re-issued under its primary key with a higher revision (a header snapd leaves out at 0), and snapd keeps the newest. snap-revision, keyed by the hash of the blob it vouches for, is cached as permanent: no snap-revision is known to have been re-issued (all 42 on one machine were at revision 0). account-key is keyed by the hash of its public key, but is re-issued: the root key and the store’s own signing key are at revision 2, as a new revision can extend a key’s until or change its constraints, so it revalidates with TTL, sequence or not. account, snap-declaration, store and validation-set are “head”/sequence-forming types that can be updated or resequenced, so an unpinned fetch revalidates with TTL, and one where ?sequence=N pins a specific point is cached as permanent. An unrecognized type defaults to revalidate, the same “never treated as forever-cacheable” conservative default §5’s bottom row uses for an unrecognized apt path shape. Whether a pinned sequence point, or a snap-revision, is ever re-issued is not confirmed against the store.
• Info: key info/<name> plus the query string and the request headers that select a different response (series and store when not the default, architecture, classic, capabilities, delta format; x--named so they cannot collide with a query parameter). That suffix is hashed: a real snap info sends a fields= list long enough to exceed a filename’s 255-byte limit.

A relayed JSON/assertion body has no pre-known hash to verify against before being cached — the same acknowledged gap apt has for its own trust root, InRelease (§4.2). .snap blobs, by contrast, are hash-verified (sha3-384) before they are committed, but not served only after it: store-then-serve sends the client nothing, not even headers, until the whole blob is fetched and hashed, and snapd’s HTTP client gives up on a large snap before that. A miss is therefore a fill (§8.1, internal/fill): the origin fetch runs detached from every client, writing to the Store and hashing, and every client, including Range resumes, follows the bytes as they are written. As the pending record carries the blob’s size (§11.3), the headers, with a correct Content-Length, go out at once. Content that fails the sha3-384 check is not cached, but the clients already following the fill still get the origin’s bytes unmodified (the same rule as apt’s §8 step 7): snapd verifies sha3-384 itself and rejects a bad blob, the same “verification is ultimately the client’s job too” principle §1 states for apt’s InRelease. A HEAD on a miss is answered from the pending record without fetching anything.

11.6 Explicit v1 scope boundaries

Requests whose answer may depend on who asks are a user’s (Authorization, a macaroon, which can show the user’s private snaps) and a device session (Snap-Device-Authorization) on a brand store, which Snap-Device-Store names and which may restrict what it shows. A device session on the public store is not one of them: snapd sends it on every request but the catalogue’s once the device has registered a serial, which every Ubuntu Core and brand device has, and it says which device asks, not what it may see. Treating it as authentication would keep the cache, and offline mode, from ever working for such a device.

A brand store’s device session is relayed, its URLs rewritten, and nothing it is answered is served from or saved to the cache, nor recorded.

A user’s request is served from the cache as any other, since only answers anyone may see are kept. What the store answers it is kept only when the store says every snap in it is public: snapd asks for private in every info, find and action, and the answer about a public snap does not depend on who asks. An info is cached when its snap says "private": false, a find when every result’s does (one with no results shows nothing), and of an install, download or refresh the result of each such snap is recorded (§11.7). An answer that shows a private snap, or does not say, goes to that user alone. Without this, a machine logged in to the store, as most desktops are, could find and install nothing offline. A user’s fetch is shared only with that user’s identical requests in flight (the macaroon is part of the fetch’s key, not of the cache key), so no other client waits for, or is handed, an answer that may show the user’s private snaps. snapd sends the user’s macaroon when it fetches assertions too, and an assertion does not say whether its snap is private. The store does give a private snap’s assertions to anyone who names them (observed: an anonymous request for the snap-declaration of a private snap answers 200, with the snap’s name), but a cache should not depend on that. So a user’s assertion that is not cached is asked for without the macaroon, as any client’s, and what the store answers that way, which anyone gets, is cached; when it does not answer so, the user’s request is relayed with the macaroon and nothing is kept. The catalogue is relayed for a user, uncached. In every case the blobs named are cached by hash alone, as the store hands them out by signed URL.

Out of scope: find’s full field/query surface (the query is cached as asked, under a key that only normalises spelling, §11.2’s table; it is not modeled or validated), GC/eviction (a project-wide non-goal, §1), and official Snap Store Proxy registration (§11.1).

11.7 Recording and synthesising refresh results

POST /v2/snaps/refresh is per client, so it cannot be cached as a whole, yet without it snap install fails while pkgproxy does not contact the store (offline mode, or the circuit breaker of store-overloaded open: §21), even for a snap whose blob is cached. What the store answers one action is not per client, though: installing a snap from a channel, or refreshing it from a revision, has the same answer for every client of the same series, architecture, store and fields. So the handler (internal/snapproxy refresh.go, synth.go):

• records, when online and the answer is a 200 to a request that is not a device session on a brand store (§11.6), the store’s own result for each install, download or refresh action (for a logged-in user’s request, only a result that says "private": false), before its URLs are rewritten, with the Snap-Store-Version, the client’s epoch and the assertion formats asked for. The key is a hash of (series, architecture, store, classic, Snap-Device-Capabilities and fields) and of (name, channel as sent, revision) for an install or download, or (snap id, channel, revision) for a refresh, which keeps the newest revision only. The channel is as sent, but for the empty one, which is stable: stable is not latest/stable, as snapd says it supports default tracks and the store reads a channel without a track as one of the snap’s default track. An action with a cohort, ignore-validation or validation sets (its own, or in the context of a refresh) is neither recorded nor answered: they change which revision the store picks;
• synthesises the response when it is not contacting the store (and, in store-overloaded mode with the breaker still closed, when the store failed and every action can be answered from a record, as otherwise the store’s own failure is relayed and snapd retries it): one result per action, with the client’s own instance-key, the recorded result with its URLs rewritten for the way the client reached this proxy and its deltas dropped (snapd tries a delta first, and retries it for a minute and a half before it falls back to the full blob), and Snap-Store-Version and X-Pkgproxy-Synthesised headers. An action that cannot be answered gets an error result of its own, with a code snapd does not know, so that it shows the message as it is (pkgproxy is not contacting the store and has no recorded answer for hello), and does not fail the other actions.

A refresh of a snap that was only ever installed through the proxy has no refresh record, as no newer revision was ever offered online. Every install is therefore also recorded under (snap id, channel, revision), and a refresh falls back to it, using whichever of the two records has the newer revision. The result is made to name the action it answers (refresh, not install). Recording a result is a check of the old record’s revision and a write, done under a per-key lock so that two clients recording at once cannot leave the older revision.

snapd reads a refresh result with the revision it already has as “no updates” (currentSnapMatchesStoreSnap), so that is what a client at the recorded revision is told. A newer recorded revision is offered only to a client with the same epoch as the one that was refreshed (the store hops epochs, a recording does not know how); an older one is never offered, as it would be a downgrade. A recorded revision is only offered when its blob is cached, so the installable set is the snaps whose answer was recorded and whose blob is cached. fetch-assertions actions are answered with no stream URLs: the assertions an install needs are fetched one by one from the cache.

An install must not fail halfway, so a revision is only offered when the cache also holds everything snapd fetches to accept the blob (snap.CheckChain, following asserts.Fetcher): the snap-revision for the blob’s digest (primary key: its sha3-384 in unpadded base64url), which must name this snap id and revision, the snap-declaration (16/<snap id>), the accounts of the developer and the publisher, and the account-key that signed each assertion with the account of that key, down to what snapd has built in (the canonical account and its root key). snapd asks for each with max-format=<N>, the N it supports for that type (asserts.MaxSupportedFormat: 7 for snap-declaration, 1 for account-key, 0 for snap-revision and account), so they are looked for under the keys serveAssertion cached them by. A refresh request says those formats in assertion-max-formats only when it also has assertions to resolve, and names only the types above 0; a type the map does not name, or every type when there is no map, is looked for under any N.

snapd asks only for the assertions it does not have, so a snap it installed before it used the proxy, or reinstalled with the assertions it kept, would be recorded with nothing to vouch for it. So once a refresh is answered online (the response flushed, so snapd does not wait), the handler walks the chain of each result it recorded and fetches from the store, anonymously, each assertion the cache lacks, with the max-format snapd would ask for it with: the one the request’s assertion-max-formats gives (0 for a type it does not name), else the one a client last asked for the type with, else snapd’s own (the values above). Each fetch is the one serveAssertion makes, shared with a client’s request for the same assertion in flight. The blob itself is not fetched ahead: that costs a snap’s size for each one snapd did not download (gitlab.com/zygoon/pkgproxy#81).