Snap Store relay
snap for users for administrators
How snapd is pointed at pkgproxy with SNAPPY_FORCE_API_URL, how store answers are relayed with their download links rewritten, which snaps, assertions and details are kept forever, rechecked or not cached, and how the store's device-session handshake is answered while the store is not contacted.
On the map: snap
▶ Watch this deck — 8 scenes, ~5 min
Part of journeys
- pkgproxy on your own computer, step 5 of 10 · previous: apt through a forward proxy · next: Docker Hub registry mirror
01 What this topic covers
snapd uses pkgproxy as its store over plain HTTP. The store's answers are kept, rechecked or only relayed, and the device-session handshake is answered with placeholders while the store's auth is not contacted.
The Snap Store relay is how snapd installs and refreshes snaps through pkgproxy, even though the Snap Store answers only over HTTPS. First, snapd is pointed at pkgproxy as its store. Then you see how the store's download links are rewritten to point back at the cache, and how each snap is kept by its hash. Last come the assertions and snap details that are rechecked, what is only relayed or recorded, and how pkgproxy answers the store's handshake when the store cannot.
02 snapd talks to pkgproxy as its store
pkgproxyctl setup snap | sudo sh writes a systemd drop-in
that sets SNAPPY_FORCE_API_URL and restarts snapd. snapd then
uses pkgproxy as its store over HTTP; pkgproxy relays to the real store over
HTTPS. No TLS is intercepted.
The Snap Store answers only over HTTPS, so a tunnel through pkgproxy could not cache any of it. Instead, snapd is pointed straight at pkgproxy. pkgproxyctl setup snap writes a systemd drop-in for the snapd service that sets SNAPPY_FORCE_API_URL to pkgproxy's address, port 3142, and restarts snapd. From then on snapd sends its store requests over plain HTTP to pkgproxy, which relays them to api.snapcraft.io over HTTPS and caches what it can.
03 Download links point back at the cache
Every object with a url and a sha3-384 gets its
URL rewritten to /v2/download/sha3-384/<hex>; the real
URL is recorded under pending/. The host is each request's own,
cached answers included, or -snap-external-url.
The store's answer to an install or a refresh says where to download each snap: a URL on the store's servers, next to the snap's SHA3-384 hash. Before the answer reaches snapd, pkgproxy rewrites that URL to its own download address, named by the hash, and records on disk which real URL and size the hash stands for. Assertion links are rewritten the same way. The new links use the address snapd reached pkgproxy by, or the snap-external-url flag when it is set. A cached answer is rewritten again for each machine that asks.
04 A snap, kept by its hash
A miss is one shared fill from the store's URL. The snap is committed as
sha3-384/<hex> only when it matches the hash, and is
never revalidated. A mismatch is not cached; snapd rejects it too.
When snapd follows a rewritten link, pkgproxy looks the hash up in its cache. On a miss it fetches the snap from the real URL once, and every client asking for it, resumed downloads included, follows the bytes as they are written. Because the size is recorded, the headers go out at once. The snap is checked against its hash before it is kept, and is then kept forever. A snap that fails the check is not cached, and snapd, which checks the hash itself, rejects it.
05 Assertions and snap details
snap-revision and sequence-pinned assertions, other than
account-key, are permanent.
Other assertions, info and find are revalidated
after the store's max-age, else after
-snap-revalidate-ttl (default: -revalidate-ttl, 5m).
Snapd also fetches assertions, the signed documents that vouch for a snap. A snap revision assertion is keyed by the hash of the snap it describes, so it is kept forever. Declarations, accounts and account keys can be reissued, so they are served from the cache for five minutes, the snap-revalidate-ttl flag, and then asked for again. One pinned to a sequence number is kept forever, except an account key, which the store can reissue under the same key. Snap details and search results are rechecked too, after the time the store announces, or that same five minutes when it announces none.
06 What is relayed, and what is recorded
POST /v2/snaps/refresh is relayed, and each action's result
recorded; offline, snapd's install is answered from a record whose snap
and assertion chain are cached. Auth, device sessions (while the store's auth is
contacted) and unknown paths are relayed uncached; private answers are never kept.
A refresh request lists what one machine has installed, so its answer as a whole is not cached. The store's result for each snap in it is recorded, though. When pkgproxy is not contacting the store, it answers an install or a refresh from those records, but only for a snap whose file and every assertion that vouches for it are cached. Logins, and device sessions while the store's sign-in answers, are passed through untouched, and store paths the relay does not know are relayed without being cached. Answers that show a user's private snaps are not kept.
07 The store's handshake, answered
/api/v1/snaps/auth/nonces and …/sessions are
answered with placeholders, marked X-Pkgproxy-Synthesised. A
placeholder later gets 401 with
WWW-Authenticate: Macaroon refresh_device_session=1. In mode
never the store's failure is passed on.
Before most requests, snapd makes sure the device has a store session. A device with its serial but no session yet asks the store for a nonce, signs it, and trades it for a session. While pkgproxy is offline, or the store's sign-in is left alone after trouble, pkgproxy answers both requests itself, with placeholders, so a fresh machine can still install what is cached. In store-overloaded mode it does the same when the store fails them. Once the sign-in answers again, a request carrying a placeholder is answered as if its session had expired, and snapd gets a real one through pkgproxy.
08 Where to go next general version
Next: Immutable or not, for what is kept forever and what is
rechecked; Offline mode, for installs with no network; pkgproxyctl
setup, for the other clients.
Snapd now installs and refreshes snaps through pkgproxy, and each snap comes from the store only once. To see what decides whether something is kept forever or rechecked, watch Immutable or not. To install snaps with no network, watch Offline mode. To point apt, Go, Docker and LXD at the same cache, watch pkgproxyctl setup.
Read more
Paths answered
snapd sends every store request to the address in SNAPPY_FORCE_API_URL, so the relay answers paths under /v2, /api/v1 and /v1 on port 3142, after the Docker registry's shapes (Docker Hub registry mirror). pkgproxy relays to https://api.snapcraft.io.
| Path | What pkgproxy does | Cache policy |
|---|---|---|
GET /v2/download/sha3-384/<hex> | Serves the snap (or delta) a rewritten link names, fetched once from the real URL recorded for the hash; Range included | kept forever, checked against the hash; a HEAD on a miss is answered from the record without fetching |
GET /v2/assertions/<type>/<key> | Serves an assertion | see the next table |
GET /v2/snaps/info/<name> | Relays, rewriting download links | revalidated |
GET /v2/snaps/find | Relays a search | revalidated; a search of private snaps is relayed uncached |
POST /v2/snaps/refresh | Relays installs, refreshes and downloads, rewriting download links | the answer is not cached; each action's result is recorded |
GET /v2/snaps/categories, /v2/snaps/category/<name>, /v2/repairs/… | Relays | revalidated |
GET /api/v1/snaps, /api/v1/snaps/sections, /api/v1/snaps/names, /api/v1/snaps/search, /api/v1/snaps/details/<name> | The older search API, relayed with download links rewritten | revalidated |
/api/v1/snaps/download/…, /api/v1/snaps/dm-verity/download/…, /api/v1/snaps/resources/…, /api/v1/resources/download/…, /api/v1/media/download/… | A snap's or its dm-verity data's download path that an answer to an anonymous request named next to its sha3-384 is served like the blob it names; any other is followed through the store's redirect to its CDN and streamed | the named ones kept forever with their blob; the rest not cached: there is no hash to key it by |
/api/v1/snaps/auth/…, /api/v1/snaps/metadata, /v1/…, POST /v2/cohorts | Relays untouched: logins, device sessions and serials; while the store's auth is not contacted, nonces and sessions are answered with placeholders (below) | not cached |
any other path under /v2 | Relays, rewriting download links | not cached |
| Assertion type | Without ?sequence=N | With ?sequence=N |
|---|---|---|
snap-revision | kept forever | kept forever |
account, snap-declaration, store, validation-set | revalidated | kept forever |
account-key | revalidated | revalidated |
| any other type | revalidated | revalidated |
Revalidated means served from the cache for the max-age the store announced for the entry, else for snap-revalidate-ttl, then asked for again. When the store cannot answer a revalidation (no connection, a 429 or a 5xx), the expired entry is served, with a warning in the log, and the store is not asked for it again for a while: the failure's Retry-After, or else the entry's time to live doubled for each failure in a row, up to an hour. Meanwhile the entry is served as stale with no warning. An answer, or a change of the offline mode, ends the wait.
Rewritten links
In every relayed JSON answer, an object with both a url and a sha3-384 gets its url replaced by http://<address>/v2/download/sha3-384/<hex>, and assertion-stream-urls are re-based to the same address. The older API's download_url and anon_download_url are rewritten the same way. <address> is the one snapd reached pkgproxy at, or snap-external-url. The cache keeps the store's answer as it came, and every answer from it is rewritten for the machine asking, so a container on a bridge and the host on localhost each get their own address.
Answers that depend on who asks
| Request | Served from the cache | Kept |
|---|---|---|
| Anonymous, or a device session on the public store | yes | yes |
| A logged-in user | yes | only answers whose snaps all say "private": false; the catalogue is relayed uncached |
| A device session on a brand store | no | nothing, and no refresh results are recorded |
Answers made without the store
While pkgproxy is not contacting the store (Offline mode), a POST /v2/snaps/refresh is answered from the recorded results, with an X-Pkgproxy-Synthesised header. An action is answered only when its snap file and every assertion that vouches for it are cached; otherwise it gets an error of its own, and the other actions still succeed. Actions with a cohort, ignore-validation or validation sets are never recorded or answered. An install is recorded under the channel asked and under the channel snapd then tracks, so an offline snap refresh of a snap installed from stable (tracked as latest/stable) is answered. Installs recorded before 0.12.2 are found once they are installed again online.
A refresh and a nonce request are sent again on a new connection when the store had closed the one they went out on, as its edge does after each 429; snapd no longer fails with cannot get nonce from store for that.
The device-session handshake
Before most store requests, snapd makes sure the device has a session. A device that has its serial but no session yet first sends POST /api/v1/snaps/auth/nonces, then POST /api/v1/snaps/auth/sessions. pkgproxy relays both and caches neither, as a nonce is used once and a session belongs to one device, except:
| When | pkgproxy |
|---|---|
Offline, or the store's auth part left alone in store-overloaded | Answers both itself with placeholders, pkgproxy-<random>-n-… and -s-…, marked X-Pkgproxy-Synthesised and logged synthesised |
store-overloaded, and the store fails the request (no connection, 429 or 5xx) | The same |
never | Passes the store's failure on |
| A request carries this pkgproxy's placeholder, and the store's auth answers again | Answers 401 with WWW-Authenticate: Macaroon refresh_device_session=1 without sending it, so snapd gets a real session |
| A request carries this pkgproxy's placeholder, the auth is still refused but the request's own part is not | Sends it without device authorization |
Cache hits are served whatever session they carry. A placeholder of another pkgproxy, or a real session, is passed on.
In store-overloaded, each part of the store has a breaker of its own (see Offline mode): a snap find the store answers with 429 leaves snap info, install, refresh and the device sessions working. Before pkgproxy 0.12.2 the store was one breaker, and a refused search closed off all of it. When pkgproxy does not contact the store, snapd's error names the part that is left alone: pkgproxy is giving the store's find endpoints a rest, as they looked overloaded, and the content is not in its cache. A relayed request pkgproxy does not send is logged at info level as not contacting the upstream: the offline mode refused the request, with mode, host and class, not as a failure of the store. The log line of an error the store answers names the store's backend that sent it as view=; an error without it came from in front of the store.
for administrators
Settings
| Setting | Flag | Environment | Snap option | Default |
|---|---|---|---|---|
| Snap Store API to relay to | -snap-store-url | PKGPROXY_SNAP_STORE_URL | flag or environment only | https://api.snapcraft.io |
Address put in rewritten links, as http://host[:port] | -snap-external-url | PKGPROXY_SNAP_EXTERNAL_URL | flag or environment only | the address the client used |
How long details and assertions are trusted when the store announces no max-age | -snap-revalidate-ttl | PKGPROXY_SNAP_REVALIDATE_TTL | snap-revalidate-ttl | revalidate-ttl, 5m |
| How long clients already following a snap download that failed its hash check keep receiving it | -fill-drain-timeout | PKGPROXY_FILL_DRAIN_TIMEOUT | fill-drain-timeout | 10m |
for administrators
On disk
Under the snap directory of the cache (/var/snap/pkgproxy/common/cache/snap in the snap, /var/cache/pkgproxy/snap otherwise), each object with its metadata in an extended attribute of its file:
snap/objects/ sha3-384/<hex> snaps and deltas pending/sha3-384/<hex> the real URL and size a hash stands for, kept after the snap is cached assertions/<type>/<key> assertions info/<name>… snap details, per query and selecting headers find/<hash> searches catalog/<hash> categories, repairs and the older search API refresh/… recorded install and refresh results store-path/<hash> the hash a store download path stands for
The random prefix of device-session placeholders is kept in snap-session-prefix at the top of the cache directory.
See The store.
In the documentation
Roads from here
CORE · the path every proxy engine shares
- Fetch once — one download shared while it is written
- Immutable or revalidated — kept forever or rechecked
- Offline mode — installs answered from what is cached
- One listener — how /v2 is shared with Docker
DOCKER
- Docker Hub registry mirror — the other proxy engine under /v2
SETUP AND THE COMMAND LINE
- Setting up clients — pointing other clients at pkgproxy
OPERATIONS
- Upstream status — how each part of the store has answered