Snap Store relay

snap for users for administrators

How snapd is pointed at pkgproxy with SNAPPY_FORCE_API_URL, how store answers are relayed with their download links rewritten, which snaps, assertions and details are kept forever, rechecked or not cached, and how the store's device-session handshake is answered while the store is not contacted.

On the map: snap

▶ Watch this deck — 8 scenes, ~5 min

Part of journeys

01 What this topic covers

ABOUT THIS TOPIC Snap Store relay how snapd installs snaps through pkgproxy as its store 1 snapd talks to pkgproxy as its store 2 Download links point back at the cache 3 A snap, kept by its hash 4 Assertions and snap details 5 What is relayed, and what is recorded 6 The store's handshake, answered
The topic title, Snap Store relay, and its six scenes in order: snapd pointed at pkgproxy, links rewritten, a snap kept by its hash, assertions and details, what is relayed, the store's handshake answered

snapd uses pkgproxy as its store over plain HTTP. The store's answers are kept, rechecked or only relayed, and the device-session handshake is answered with placeholders while the store's auth is not contacted.

The Snap Store relay is how snapd installs and refreshes snaps through pkgproxy, even though the Snap Store answers only over HTTPS. First, snapd is pointed at pkgproxy as its store. Then you see how the store's download links are rewritten to point back at the cache, and how each snap is kept by its hash. Last come the assertions and snap details that are rechecked, what is only relayed or recorded, and how pkgproxy answers the store's handshake when the store cannot.

02 snapd talks to pkgproxy as its store

POINTED AT THE RELAY api.snapcraft.io HTTPS only $ pkgproxyctl setup snap | sudo sh # /etc/systemd/system/snapd.service.d/pkgproxy.conf [Service] Environment=SNAPPY_FORCE_API_URL=http://pkgproxy.local:3142 snapd http /v2/… pkgproxy :3142 https snapd uses pkgproxy as its store · pkgproxy intercepts no TLS
The Snap Store is HTTPS only; pkgproxyctl setup snap writes a systemd drop-in setting SNAPPY_FORCE_API_URL; snapd then sends plain HTTP requests to pkgproxy, which relays them to api.snapcraft.io over HTTPS

pkgproxyctl setup snap | sudo sh writes a systemd drop-in that sets SNAPPY_FORCE_API_URL and restarts snapd. snapd then uses pkgproxy as its store over HTTP; pkgproxy relays to the real store over HTTPS. No TLS is intercepted.

The Snap Store answers only over HTTPS, so a tunnel through pkgproxy could not cache any of it. Instead, snapd is pointed straight at pkgproxy. pkgproxyctl setup snap writes a systemd drop-in for the snapd service that sets SNAPPY_FORCE_API_URL to pkgproxy's address, port 3142, and restarts snapd. From then on snapd sends its store requests over plain HTTP to pkgproxy, which relays them to api.snapcraft.io over HTTPS and caches what it can.

04 A snap, kept by its hash

BY HASH snapd GET /v2/download/sha3-384/8f1c…e07a pkgproxy once https://…/hello_42.snap writing… clients follow the bytes headers at once: the size is known snap/sha3-384/8f1c…e07a hash matches: kept forever never fetched or checked again hash does not match: not cached snapd checks the hash itself and rejects the snap
snapd requests a snap by its hash from pkgproxy; on a miss pkgproxy fetches it once from the store's URL while clients follow the bytes; it is checked against its sha3-384 hash and kept forever, or not cached when it does not match

A miss is one shared fill from the store's URL. The snap is committed as sha3-384/<hex> only when it matches the hash, and is never revalidated. A mismatch is not cached; snapd rejects it too.

When snapd follows a rewritten link, pkgproxy looks the hash up in its cache. On a miss it fetches the snap from the real URL once, and every client asking for it, resumed downloads included, follows the bytes as they are written. Because the size is recorded, the headers go out at once. The snap is checked against its hash before it is kept, and is then kept forever. A snap that fails the check is not cached, and snapd, which checks the hash itself, rejects it.

05 Assertions and snap details

ASSERTIONS GET /v2/assertions/<type>/<key…> signed, vouch for a snap kept forever snap-revision/<snap's hash> rechecked snap-declaration/16/<snap id> account/… · account-key/… served for 5 minutes, then asked again …?sequence=N pinned to a sequence point /v2/snaps/info/hello · /v2/snaps/find for the store's max-age, else 5 minutes 5 minutes: -snap-revalidate-ttl
Two columns: snap-revision assertions and assertions pinned to a sequence are kept forever; snap-declaration, account and account-key assertions, snap info and search results are rechecked after five minutes or the store's max-age

snap-revision and sequence-pinned assertions, other than account-key, are permanent. Other assertions, info and find are revalidated after the store's max-age, else after -snap-revalidate-ttl (default: -revalidate-ttl, 5m).

Snapd also fetches assertions, the signed documents that vouch for a snap. A snap revision assertion is keyed by the hash of the snap it describes, so it is kept forever. Declarations, accounts and account keys can be reissued, so they are served from the cache for five minutes, the snap-revalidate-ttl flag, and then asked for again. One pinned to a sequence number is kept forever, except an account key, which the store can reissue under the same key. Snap details and search results are rechecked too, after the time the store announces, or that same five minutes when it announces none.

06 What is relayed, and what is recorded

RELAYED AND RECORDED POST /v2/snaps/refresh context: what this machine has actions: install hello relayed Snap Store not cached as a whole record: hello, latest/stable → revision 42 not contacting the store: answered from the record only when cached are the snap + every assertion for it install works offline relayed, not cached: logins · device sessions · store paths it does not know an answer that shows a user's private snaps is not kept
A refresh request is relayed to the store and not cached as a whole; each snap's result is recorded; offline, an install is answered from a record when the snap and its assertions are cached; logins, device sessions while the store's sign-in answers, and unknown paths are relayed untouched

POST /v2/snaps/refresh is relayed, and each action's result recorded; offline, snapd's install is answered from a record whose snap and assertion chain are cached. Auth, device sessions (while the store's auth is contacted) and unknown paths are relayed uncached; private answers are never kept.

A refresh request lists what one machine has installed, so its answer as a whole is not cached. The store's result for each snap in it is recorded, though. When pkgproxy is not contacting the store, it answers an install or a refresh from those records, but only for a snap whose file and every assertion that vouches for it are cached. Logins, and device sessions while the store's sign-in answers, are passed through untouched, and store paths the relay does not know are relayed without being cached. Answers that show a user's private snaps are not kept.

07 The store's handshake, answered

THE STORE'S HANDSHAKE snapd before most requests: a device session pkgproxy Snap Store a serial, no session yet: POST /api/v1/snaps/auth/nonces POST /api/v1/snaps/auth/sessions offline, or sign-in left alone answered by pkgproxy: {"nonce": "pkgproxy-…-n-…"} {"macaroon": "pkgproxy-…-s-…"} a snap that is cached installs store-overloaded: also when the store fails them sign-in answers again: a placeholder gets 401 refresh_device_session=1 and snapd gets a real session through pkgproxy
snapd on a device with a serial but no session asks for a nonce and a session; while the store's sign-in is not contacted, pkgproxy answers both with placeholders, so a cached snap installs; once the sign-in answers, a request with a placeholder gets 401 and snapd fetches a real session

/api/v1/snaps/auth/nonces and …/sessions are answered with placeholders, marked X-Pkgproxy-Synthesised. A placeholder later gets 401 with WWW-Authenticate: Macaroon refresh_device_session=1. In mode never the store's failure is passed on.

Before most requests, snapd makes sure the device has a store session. A device with its serial but no session yet asks the store for a nonce, signs it, and trades it for a session. While pkgproxy is offline, or the store's sign-in is left alone after trouble, pkgproxy answers both requests itself, with placeholders, so a fresh machine can still install what is cached. In store-overloaded mode it does the same when the store fails them. Once the sign-in answers again, a request carrying a placeholder is answered as if its session had expired, and snapd gets a real one through pkgproxy.

08 Where to go next general version

WHERE NEXT snapd installs and refreshes snaps through pkgproxy, and each snap comes from the store only once you are here Snap Store relay Immutable or not what is kept forever and what is rechecked Offline mode installs answered from the cache with no network pkgproxyctl setup apt, Go, Docker and LXD on the same cache
You are here, Snap Store relay; next: Immutable or not, Offline mode, pkgproxyctl setup

Next: Immutable or not, for what is kept forever and what is rechecked; Offline mode, for installs with no network; pkgproxyctl setup, for the other clients.

Snapd now installs and refreshes snaps through pkgproxy, and each snap comes from the store only once. To see what decides whether something is kept forever or rechecked, watch Immutable or not. To install snaps with no network, watch Offline mode. To point apt, Go, Docker and LXD at the same cache, watch pkgproxyctl setup.

Read more

Paths answered

snapd sends every store request to the address in SNAPPY_FORCE_API_URL, so the relay answers paths under /v2, /api/v1 and /v1 on port 3142, after the Docker registry's shapes (Docker Hub registry mirror). pkgproxy relays to https://api.snapcraft.io.

PathWhat pkgproxy doesCache policy
GET /v2/download/sha3-384/<hex>Serves the snap (or delta) a rewritten link names, fetched once from the real URL recorded for the hash; Range includedkept forever, checked against the hash; a HEAD on a miss is answered from the record without fetching
GET /v2/assertions/<type>/<key>Serves an assertionsee the next table
GET /v2/snaps/info/<name>Relays, rewriting download linksrevalidated
GET /v2/snaps/findRelays a searchrevalidated; a search of private snaps is relayed uncached
POST /v2/snaps/refreshRelays installs, refreshes and downloads, rewriting download linksthe answer is not cached; each action's result is recorded
GET /v2/snaps/categories, /v2/snaps/category/<name>, /v2/repairs/…Relaysrevalidated
GET /api/v1/snaps, /api/v1/snaps/sections, /api/v1/snaps/names, /api/v1/snaps/search, /api/v1/snaps/details/<name>The older search API, relayed with download links rewrittenrevalidated
/api/v1/snaps/download/…, /api/v1/snaps/dm-verity/download/…, /api/v1/snaps/resources/…, /api/v1/resources/download/…, /api/v1/media/download/…A snap's or its dm-verity data's download path that an answer to an anonymous request named next to its sha3-384 is served like the blob it names; any other is followed through the store's redirect to its CDN and streamedthe named ones kept forever with their blob; the rest not cached: there is no hash to key it by
/api/v1/snaps/auth/…, /api/v1/snaps/metadata, /v1/…, POST /v2/cohortsRelays untouched: logins, device sessions and serials; while the store's auth is not contacted, nonces and sessions are answered with placeholders (below)not cached
any other path under /v2Relays, rewriting download linksnot cached
Assertion typeWithout ?sequence=NWith ?sequence=N
snap-revisionkept foreverkept forever
account, snap-declaration, store, validation-setrevalidatedkept forever
account-keyrevalidatedrevalidated
any other typerevalidatedrevalidated

Revalidated means served from the cache for the max-age the store announced for the entry, else for snap-revalidate-ttl, then asked for again. When the store cannot answer a revalidation (no connection, a 429 or a 5xx), the expired entry is served, with a warning in the log, and the store is not asked for it again for a while: the failure's Retry-After, or else the entry's time to live doubled for each failure in a row, up to an hour. Meanwhile the entry is served as stale with no warning. An answer, or a change of the offline mode, ends the wait.

In every relayed JSON answer, an object with both a url and a sha3-384 gets its url replaced by http://<address>/v2/download/sha3-384/<hex>, and assertion-stream-urls are re-based to the same address. The older API's download_url and anon_download_url are rewritten the same way. <address> is the one snapd reached pkgproxy at, or snap-external-url. The cache keeps the store's answer as it came, and every answer from it is rewritten for the machine asking, so a container on a bridge and the host on localhost each get their own address.

Answers that depend on who asks

RequestServed from the cacheKept
Anonymous, or a device session on the public storeyesyes
A logged-in useryesonly answers whose snaps all say "private": false; the catalogue is relayed uncached
A device session on a brand storenonothing, and no refresh results are recorded

Answers made without the store

While pkgproxy is not contacting the store (Offline mode), a POST /v2/snaps/refresh is answered from the recorded results, with an X-Pkgproxy-Synthesised header. An action is answered only when its snap file and every assertion that vouches for it are cached; otherwise it gets an error of its own, and the other actions still succeed. Actions with a cohort, ignore-validation or validation sets are never recorded or answered. An install is recorded under the channel asked and under the channel snapd then tracks, so an offline snap refresh of a snap installed from stable (tracked as latest/stable) is answered. Installs recorded before 0.12.2 are found once they are installed again online.

A refresh and a nonce request are sent again on a new connection when the store had closed the one they went out on, as its edge does after each 429; snapd no longer fails with cannot get nonce from store for that.

The device-session handshake

Before most store requests, snapd makes sure the device has a session. A device that has its serial but no session yet first sends POST /api/v1/snaps/auth/nonces, then POST /api/v1/snaps/auth/sessions. pkgproxy relays both and caches neither, as a nonce is used once and a session belongs to one device, except:

Whenpkgproxy
Offline, or the store's auth part left alone in store-overloadedAnswers both itself with placeholders, pkgproxy-<random>-n-… and -s-…, marked X-Pkgproxy-Synthesised and logged synthesised
store-overloaded, and the store fails the request (no connection, 429 or 5xx)The same
neverPasses the store's failure on
A request carries this pkgproxy's placeholder, and the store's auth answers againAnswers 401 with WWW-Authenticate: Macaroon refresh_device_session=1 without sending it, so snapd gets a real session
A request carries this pkgproxy's placeholder, the auth is still refused but the request's own part is notSends it without device authorization

Cache hits are served whatever session they carry. A placeholder of another pkgproxy, or a real session, is passed on.

In store-overloaded, each part of the store has a breaker of its own (see Offline mode): a snap find the store answers with 429 leaves snap info, install, refresh and the device sessions working. Before pkgproxy 0.12.2 the store was one breaker, and a refused search closed off all of it. When pkgproxy does not contact the store, snapd's error names the part that is left alone: pkgproxy is giving the store's find endpoints a rest, as they looked overloaded, and the content is not in its cache. A relayed request pkgproxy does not send is logged at info level as not contacting the upstream: the offline mode refused the request, with mode, host and class, not as a failure of the store. The log line of an error the store answers names the store's backend that sent it as view=; an error without it came from in front of the store.

for administrators

Settings

SettingFlagEnvironmentSnap optionDefault
Snap Store API to relay to-snap-store-urlPKGPROXY_SNAP_STORE_URLflag or environment onlyhttps://api.snapcraft.io
Address put in rewritten links, as http://host[:port]-snap-external-urlPKGPROXY_SNAP_EXTERNAL_URLflag or environment onlythe address the client used
How long details and assertions are trusted when the store announces no max-age-snap-revalidate-ttlPKGPROXY_SNAP_REVALIDATE_TTLsnap-revalidate-ttlrevalidate-ttl, 5m
How long clients already following a snap download that failed its hash check keep receiving it-fill-drain-timeoutPKGPROXY_FILL_DRAIN_TIMEOUTfill-drain-timeout10m

for administrators

On disk

Under the snap directory of the cache (/var/snap/pkgproxy/common/cache/snap in the snap, /var/cache/pkgproxy/snap otherwise), each object with its metadata in an extended attribute of its file:

snap/objects/
  sha3-384/<hex>             snaps and deltas
  pending/sha3-384/<hex>     the real URL and size a hash stands for, kept after the snap is cached
  assertions/<type>/<key>    assertions
  info/<name>…               snap details, per query and selecting headers
  find/<hash>                searches
  catalog/<hash>             categories, repairs and the older search API
  refresh/…                  recorded install and refresh results
  store-path/<hash>          the hash a store download path stands for

The random prefix of device-session placeholders is kept in snap-session-prefix at the top of the cache directory.

See The store.

Roads from here

CORE · the path every proxy engine shares

DOCKER

SETUP AND THE COMMAND LINE

OPERATIONS