Snap Store relay

snap for users for administrators

How snapd is pointed at pkgproxy with SNAPPY_FORCE_API_URL, how store answers are relayed with their download links rewritten, which snaps, assertions and details are kept forever, rechecked or not cached, and what still needs the store.

On the map: snap

▶ Watch this deck — 8 scenes, ~5 min

01 What this topic covers

ABOUT THIS TOPIC Snap Store relay how snapd installs snaps through pkgproxy as its store 1 snapd talks to pkgproxy as its store 2 Download links point back at the cache 3 A snap, kept by its hash 4 Assertions and snap details 5 What is relayed, and what is recorded 6 What still needs the store
The topic title, Snap Store relay, and its six scenes in order: snapd pointed at pkgproxy, links rewritten, a snap kept by its hash, assertions and details, what is relayed, what still needs the store

snapd uses pkgproxy as its store over plain HTTP. The store's answers are kept, rechecked or only relayed, and a few still need the store.

The Snap Store relay is how snapd installs and refreshes snaps through pkgproxy, even though the Snap Store answers only over HTTPS. First, snapd is pointed at pkgproxy as its store. Then you see how the store's download links are rewritten to point back at the cache, and how each snap is kept by its hash. Last come the assertions and snap details that are rechecked, what is only relayed or recorded, and what still needs the store.

02 snapd talks to pkgproxy as its store

POINTED AT THE RELAY api.snapcraft.io HTTPS only $ pkgproxyctl setup snap | sudo sh # /etc/systemd/system/snapd.service.d/pkgproxy.conf [Service] Environment=SNAPPY_FORCE_API_URL=http://pkgproxy.local:3142 snapd http /v2/… pkgproxy :3142 https snapd uses pkgproxy as its store · pkgproxy intercepts no TLS
The Snap Store is HTTPS only; pkgproxyctl setup snap writes a systemd drop-in setting SNAPPY_FORCE_API_URL; snapd then sends plain HTTP requests to pkgproxy, which relays them to api.snapcraft.io over HTTPS

pkgproxyctl setup snap | sudo sh writes a systemd drop-in that sets SNAPPY_FORCE_API_URL and restarts snapd. snapd then uses pkgproxy as its store over HTTP; pkgproxy relays to the real store over HTTPS. No TLS is intercepted.

The Snap Store answers only over HTTPS, so a tunnel through pkgproxy could not cache any of it. Instead, snapd is pointed straight at pkgproxy. pkgproxyctl setup snap writes a systemd drop-in for the snapd service that sets SNAPPY_FORCE_API_URL to pkgproxy's address, port 3142, and restarts snapd. From then on snapd sends its store requests over plain HTTP to pkgproxy, which relays them to api.snapcraft.io over HTTPS and caches what it can.

04 A snap, kept by its hash

BY HASH snapd GET /v2/download/sha3-384/8f1c…e07a pkgproxy once https://…/hello_42.snap writing… clients follow the bytes headers at once: the size is known snap/sha3-384/8f1c…e07a hash matches: kept forever never fetched or checked again hash does not match: not cached snapd checks the hash itself and rejects the snap
snapd requests a snap by its hash from pkgproxy; on a miss pkgproxy fetches it once from the store's URL while clients follow the bytes; it is checked against its sha3-384 hash and kept forever, or not cached when it does not match

A miss is one shared fill from the store's URL. The snap is committed as sha3-384/<hex> only when it matches the hash, and is never revalidated. A mismatch is not cached; snapd rejects it too.

When snapd follows a rewritten link, pkgproxy looks the hash up in its cache. On a miss it fetches the snap from the real URL once, and every client asking for it, resumed downloads included, follows the bytes as they are written. Because the size is recorded, the headers go out at once. The snap is checked against its hash before it is kept, and is then kept forever. A snap that fails the check is not cached, and snapd, which checks the hash itself, rejects it.

05 Assertions and snap details

ASSERTIONS GET /v2/assertions/<type>/<key…> signed, vouch for a snap kept forever snap-revision/<snap's hash> rechecked snap-declaration/16/<snap id> account/… · account-key/… served for 5 minutes, then asked again …?sequence=N pinned to a sequence point /v2/snaps/info/hello · /v2/snaps/find for the store's max-age, else 5 minutes 5 minutes: -snap-revalidate-ttl
Two columns: snap-revision assertions and assertions pinned to a sequence are kept forever; snap-declaration, account and account-key assertions, snap info and search results are rechecked after five minutes or the store's max-age

snap-revision and sequence-pinned assertions, other than account-key, are permanent. Other assertions, info and find are revalidated after the store's max-age, else after -snap-revalidate-ttl (default: -revalidate-ttl, 5m).

Snapd also fetches assertions, the signed documents that vouch for a snap. A snap revision assertion is keyed by the hash of the snap it describes, so it is kept forever. Declarations, accounts and account keys can be reissued, so they are served from the cache for five minutes, the snap-revalidate-ttl flag, and then asked for again. One pinned to a sequence number is kept forever, except an account key, which the store can reissue under the same key. Snap details and search results are rechecked too, after the time the store announces, or that same five minutes when it announces none.

06 What is relayed, and what is recorded

RELAYED AND RECORDED POST /v2/snaps/refresh context: what this machine has actions: install hello relayed Snap Store not cached as a whole record: hello, latest/stable → revision 42 not contacting the store: answered from the record only when cached are the snap + every assertion for it install works offline relayed, not cached: logins · device sessions · store paths it does not know an answer that shows a user's private snaps is not kept
A refresh request is relayed to the store and not cached as a whole; each snap's result is recorded; offline, an install is answered from a record when the snap and its assertions are cached; logins, device sessions and unknown paths are relayed untouched

POST /v2/snaps/refresh is relayed, and each action's result recorded; offline, snapd's install is answered from a record whose snap and assertion chain are cached. Auth, device sessions and unknown paths are relayed uncached; private answers are never kept.

A refresh request lists what one machine has installed, so its answer as a whole is not cached. The store's result for each snap in it is recorded, though. When pkgproxy is not contacting the store, it answers an install or a refresh from those records, but only for a snap whose file and every assertion that vouches for it are cached. Logins and device sessions are passed through untouched, and store paths the relay does not know are relayed without being cached. Answers that show a user's private snaps are not kept.

07 What still needs the store

NOT YET: THE STORE'S HANDSHAKE snapd before most requests: a device session pkgproxy Snap Store a serial, no session yet: POST /api/v1/snaps/auth/nonces POST /api/v1/snaps/auth/sessions relayed, never cached: a nonce is used once store down, or left alone: cannot get nonce from store: store server returned status 424 a snap that is cached its download fails too planned: answer the handshake while the store is refused (#155) planned: a breaker for each store endpoint, so a failing search stays apart (#153)
snapd on a device with a serial but no session asks for a nonce and a session; pkgproxy relays both to the store uncached; with the store refused, the exchange fails with 424 and so does the download of a cached snap; answering the exchange and per-endpoint breakers are planned

A device with a serial but no session asks /api/v1/snaps/auth/nonces and …/sessions, which are relayed uncached. With the store refused, they fail, and so does the request they were for, cached or not. Planned: answering them while the store is refused (#155), and breakers per store endpoint (#153).

pkgproxy cannot yet stand in for the store completely. Before most requests, snapd makes sure the device has a store session. A device with its serial but no session yet asks the store for a nonce, signs it, and trades it for a session. pkgproxy relays these requests without keeping them, as a nonce is used only once. While the store is down, or left alone after trouble, that exchange fails, and so does the request it was for, even a download that is cached. Answering the exchange itself is planned, and so is a breaker for each store endpoint, so a failing search does not hold back installs.

08 Where to go next general version

WHERE NEXT snapd installs and refreshes snaps through pkgproxy, and each snap comes from the store only once you are here Snap Store relay Immutable or not what is kept forever and what is rechecked Offline mode installs answered from the cache with no network pkgproxyctl setup apt, Go, Docker and LXD on the same cache
You are here, Snap Store relay; next: Immutable or not, Offline mode, pkgproxyctl setup

Next: Immutable or not, for what is kept forever and what is rechecked; Offline mode, for installs with no network; pkgproxyctl setup, for the other clients.

Snapd now installs and refreshes snaps through pkgproxy, and each snap comes from the store only once. To see what decides whether something is kept forever or rechecked, watch Immutable or not. To install snaps with no network, watch Offline mode. To point apt, Go, Docker and LXD at the same cache, watch pkgproxyctl setup.

Read more

Paths answered

snapd sends every store request to the address in SNAPPY_FORCE_API_URL, so the relay answers paths under /v2, /api/v1 and /v1 on port 3142, after the Docker registry's shapes (Docker Hub registry mirror). pkgproxy relays to https://api.snapcraft.io.

PathWhat pkgproxy doesCache policy
GET /v2/download/sha3-384/<hex>Serves the snap (or delta) a rewritten link names, fetched once from the real URL recorded for the hash; Range includedkept forever, checked against the hash; a HEAD on a miss is answered from the record without fetching
GET /v2/assertions/<type>/<key>Serves an assertionsee the next table
GET /v2/snaps/info/<name>Relays, rewriting download linksrevalidated
GET /v2/snaps/findRelays a searchrevalidated; a search of private snaps is relayed uncached
POST /v2/snaps/refreshRelays installs, refreshes and downloads, rewriting download linksthe answer is not cached; each action's result is recorded
GET /v2/snaps/categories, /v2/snaps/category/<name>, /v2/repairs/…Relaysrevalidated
GET /api/v1/snaps, /api/v1/snaps/sections, /api/v1/snaps/names, /api/v1/snaps/search, /api/v1/snaps/details/<name>The older search API, relayed with download links rewrittenrevalidated
/api/v1/snaps/download/…, /api/v1/snaps/dm-verity/download/…, /api/v1/snaps/resources/…, /api/v1/resources/download/…, /api/v1/media/download/…Follows the store's redirect to its CDN and streams the bodynot cached: there is no hash to key it by
/api/v1/snaps/auth/…, /api/v1/snaps/metadata, /v1/…, POST /v2/cohortsRelays untouched: logins, device sessions and serialsnot cached
any other path under /v2Relays, rewriting download linksnot cached
Assertion typeWithout ?sequence=NWith ?sequence=N
snap-revisionkept foreverkept forever
account, snap-declaration, store, validation-setrevalidatedkept forever
account-keyrevalidatedrevalidated
any other typerevalidatedrevalidated

Revalidated means served from the cache for the max-age the store announced for the entry, else for snap-revalidate-ttl, then asked for again. When the store cannot answer a revalidation (no connection, a 429 or a 5xx), the expired entry is served, with a warning in the log.

In every relayed JSON answer, an object with both a url and a sha3-384 gets its url replaced by http://<address>/v2/download/sha3-384/<hex>, and assertion-stream-urls are re-based to the same address. The older API's download_url and anon_download_url are rewritten the same way. <address> is the one snapd reached pkgproxy at, or snap-external-url. The cache keeps the store's answer as it came, and every answer from it is rewritten for the machine asking, so a container on a bridge and the host on localhost each get their own address.

Answers that depend on who asks

RequestServed from the cacheKept
Anonymous, or a device session on the public storeyesyes
A logged-in useryesonly answers whose snaps all say "private": false; the catalogue is relayed uncached
A device session on a brand storenonothing, and no refresh results are recorded

Answers made without the store

While pkgproxy is not contacting the store (Offline mode), a POST /v2/snaps/refresh is answered from the recorded results, with an X-Pkgproxy-Synthesised header. An action is answered only when its snap file and every assertion that vouches for it are cached; otherwise it gets an error of its own, and the other actions still succeed. Actions with a cohort, ignore-validation or validation sets are never recorded or answered.

What still needs the store

Some requests pkgproxy cannot answer for the store yet. Both gaps are planned work.

  • The device-session handshake. Before most store requests, snapd makes sure the device has a session. A device that has its serial but no session yet first sends POST /api/v1/snaps/auth/nonces, then POST /api/v1/snaps/auth/sessions. pkgproxy relays both and caches neither, as a nonce is used once and a session belongs to one device. While pkgproxy is not contacting the store (offline, or the store left alone in store-overloaded), the nonce fails, and snapd fails the request it was for, even the download of a snap that is cached: cannot get nonce from store: store server returned status 424. A device that already has a session, or has no serial, is not affected. Planned: pkgproxy#155, answering the handshake while the store is refused.
  • One breaker for the whole store. In store-overloaded, the store is left alone per host, and api.snapcraft.io is one host. A snap find the store answers with 429 closes it off for snap info, install and refresh too, and installs of snaps that are not cached fail with 424 until the search recovers. Planned: pkgproxy#153, a breaker for each kind of store request.

for administrators

Settings

SettingFlagEnvironmentSnap optionDefault
Snap Store API to relay to-snap-store-urlPKGPROXY_SNAP_STORE_URLflag or environment onlyhttps://api.snapcraft.io
Address put in rewritten links, as http://host[:port]-snap-external-urlPKGPROXY_SNAP_EXTERNAL_URLflag or environment onlythe address the client used
How long details and assertions are trusted when the store announces no max-age-snap-revalidate-ttlPKGPROXY_SNAP_REVALIDATE_TTLsnap-revalidate-ttlrevalidate-ttl, 5m
How long clients already following a snap download that failed its hash check keep receiving it-fill-drain-timeoutPKGPROXY_FILL_DRAIN_TIMEOUTfill-drain-timeout10m

for administrators

On disk

Under the snap directory of the cache (/var/snap/pkgproxy/common/cache/snap in the snap, /var/cache/pkgproxy/snap otherwise), each object with its metadata in an extended attribute of its file:

snap/objects/
  sha3-384/<hex>             snaps and deltas
  pending/sha3-384/<hex>     the real URL and size a hash stands for, kept after the snap is cached
  assertions/<type>/<key>    assertions
  info/<name>…               snap details, per query and selecting headers
  find/<hash>                searches
  catalog/<hash>             categories, repairs and the older search API
  refresh/…                  recorded install and refresh results

See The store.

Roads from here

CORE · the path every side shares

DOCKER

SETUP AND THE COMMAND LINE