Snap Store relay
snap for users for administrators
How snapd is pointed at pkgproxy with SNAPPY_FORCE_API_URL, how store answers are relayed with their download links rewritten, which snaps, assertions and details are kept forever, rechecked or not cached, and what still needs the store.
On the map: snap
01 What this topic covers
snapd uses pkgproxy as its store over plain HTTP. The store's answers are kept, rechecked or only relayed, and a few still need the store.
The Snap Store relay is how snapd installs and refreshes snaps through pkgproxy, even though the Snap Store answers only over HTTPS. First, snapd is pointed at pkgproxy as its store. Then you see how the store's download links are rewritten to point back at the cache, and how each snap is kept by its hash. Last come the assertions and snap details that are rechecked, what is only relayed or recorded, and what still needs the store.
02 snapd talks to pkgproxy as its store
pkgproxyctl setup snap | sudo sh writes a systemd drop-in
that sets SNAPPY_FORCE_API_URL and restarts snapd. snapd then
uses pkgproxy as its store over HTTP; pkgproxy relays to the real store over
HTTPS. No TLS is intercepted.
The Snap Store answers only over HTTPS, so a tunnel through pkgproxy could not cache any of it. Instead, snapd is pointed straight at pkgproxy. pkgproxyctl setup snap writes a systemd drop-in for the snapd service that sets SNAPPY_FORCE_API_URL to pkgproxy's address, port 3142, and restarts snapd. From then on snapd sends its store requests over plain HTTP to pkgproxy, which relays them to api.snapcraft.io over HTTPS and caches what it can.
03 Download links point back at the cache
Every object with a url and a sha3-384 gets its
URL rewritten to /v2/download/sha3-384/<hex>; the real
URL is recorded under pending/. The host is each request's own,
cached answers included, or -snap-external-url.
The store's answer to an install or a refresh says where to download each snap: a URL on the store's servers, next to the snap's SHA3-384 hash. Before the answer reaches snapd, pkgproxy rewrites that URL to its own download address, named by the hash, and records on disk which real URL and size the hash stands for. Assertion links are rewritten the same way. The new links use the address snapd reached pkgproxy by, or the snap-external-url flag when it is set. A cached answer is rewritten again for each machine that asks.
04 A snap, kept by its hash
A miss is one shared fill from the store's URL. The snap is committed as
sha3-384/<hex> only when it matches the hash, and is
never revalidated. A mismatch is not cached; snapd rejects it too.
When snapd follows a rewritten link, pkgproxy looks the hash up in its cache. On a miss it fetches the snap from the real URL once, and every client asking for it, resumed downloads included, follows the bytes as they are written. Because the size is recorded, the headers go out at once. The snap is checked against its hash before it is kept, and is then kept forever. A snap that fails the check is not cached, and snapd, which checks the hash itself, rejects it.
05 Assertions and snap details
snap-revision and sequence-pinned assertions, other than
account-key, are permanent.
Other assertions, info and find are revalidated
after the store's max-age, else after
-snap-revalidate-ttl (default: -revalidate-ttl, 5m).
Snapd also fetches assertions, the signed documents that vouch for a snap. A snap revision assertion is keyed by the hash of the snap it describes, so it is kept forever. Declarations, accounts and account keys can be reissued, so they are served from the cache for five minutes, the snap-revalidate-ttl flag, and then asked for again. One pinned to a sequence number is kept forever, except an account key, which the store can reissue under the same key. Snap details and search results are rechecked too, after the time the store announces, or that same five minutes when it announces none.
06 What is relayed, and what is recorded
POST /v2/snaps/refresh is relayed, and each action's result
recorded; offline, snapd's install is answered from a record whose snap
and assertion chain are cached. Auth, device sessions and unknown paths are
relayed uncached; private answers are never kept.
A refresh request lists what one machine has installed, so its answer as a whole is not cached. The store's result for each snap in it is recorded, though. When pkgproxy is not contacting the store, it answers an install or a refresh from those records, but only for a snap whose file and every assertion that vouches for it are cached. Logins and device sessions are passed through untouched, and store paths the relay does not know are relayed without being cached. Answers that show a user's private snaps are not kept.
07 What still needs the store
A device with a serial but no session asks
/api/v1/snaps/auth/nonces and …/sessions, which
are relayed uncached. With the store refused, they fail, and so does the
request they were for, cached or not. Planned: answering them while the
store is refused (#155), and breakers per store endpoint (#153).
pkgproxy cannot yet stand in for the store completely. Before most requests, snapd makes sure the device has a store session. A device with its serial but no session yet asks the store for a nonce, signs it, and trades it for a session. pkgproxy relays these requests without keeping them, as a nonce is used only once. While the store is down, or left alone after trouble, that exchange fails, and so does the request it was for, even a download that is cached. Answering the exchange itself is planned, and so is a breaker for each store endpoint, so a failing search does not hold back installs.
08 Where to go next general version
Next: Immutable or not, for what is kept forever and what is
rechecked; Offline mode, for installs with no network; pkgproxyctl
setup, for the other clients.
Snapd now installs and refreshes snaps through pkgproxy, and each snap comes from the store only once. To see what decides whether something is kept forever or rechecked, watch Immutable or not. To install snaps with no network, watch Offline mode. To point apt, Go, Docker and LXD at the same cache, watch pkgproxyctl setup.
Read more
Paths answered
snapd sends every store request to the address in SNAPPY_FORCE_API_URL, so the relay answers paths under /v2, /api/v1 and /v1 on port 3142, after the Docker registry's shapes (Docker Hub registry mirror). pkgproxy relays to https://api.snapcraft.io.
| Path | What pkgproxy does | Cache policy |
|---|---|---|
GET /v2/download/sha3-384/<hex> | Serves the snap (or delta) a rewritten link names, fetched once from the real URL recorded for the hash; Range included | kept forever, checked against the hash; a HEAD on a miss is answered from the record without fetching |
GET /v2/assertions/<type>/<key> | Serves an assertion | see the next table |
GET /v2/snaps/info/<name> | Relays, rewriting download links | revalidated |
GET /v2/snaps/find | Relays a search | revalidated; a search of private snaps is relayed uncached |
POST /v2/snaps/refresh | Relays installs, refreshes and downloads, rewriting download links | the answer is not cached; each action's result is recorded |
GET /v2/snaps/categories, /v2/snaps/category/<name>, /v2/repairs/… | Relays | revalidated |
GET /api/v1/snaps, /api/v1/snaps/sections, /api/v1/snaps/names, /api/v1/snaps/search, /api/v1/snaps/details/<name> | The older search API, relayed with download links rewritten | revalidated |
/api/v1/snaps/download/…, /api/v1/snaps/dm-verity/download/…, /api/v1/snaps/resources/…, /api/v1/resources/download/…, /api/v1/media/download/… | Follows the store's redirect to its CDN and streams the body | not cached: there is no hash to key it by |
/api/v1/snaps/auth/…, /api/v1/snaps/metadata, /v1/…, POST /v2/cohorts | Relays untouched: logins, device sessions and serials | not cached |
any other path under /v2 | Relays, rewriting download links | not cached |
| Assertion type | Without ?sequence=N | With ?sequence=N |
|---|---|---|
snap-revision | kept forever | kept forever |
account, snap-declaration, store, validation-set | revalidated | kept forever |
account-key | revalidated | revalidated |
| any other type | revalidated | revalidated |
Revalidated means served from the cache for the max-age the store announced for the entry, else for snap-revalidate-ttl, then asked for again. When the store cannot answer a revalidation (no connection, a 429 or a 5xx), the expired entry is served, with a warning in the log.
Rewritten links
In every relayed JSON answer, an object with both a url and a sha3-384 gets its url replaced by http://<address>/v2/download/sha3-384/<hex>, and assertion-stream-urls are re-based to the same address. The older API's download_url and anon_download_url are rewritten the same way. <address> is the one snapd reached pkgproxy at, or snap-external-url. The cache keeps the store's answer as it came, and every answer from it is rewritten for the machine asking, so a container on a bridge and the host on localhost each get their own address.
Answers that depend on who asks
| Request | Served from the cache | Kept |
|---|---|---|
| Anonymous, or a device session on the public store | yes | yes |
| A logged-in user | yes | only answers whose snaps all say "private": false; the catalogue is relayed uncached |
| A device session on a brand store | no | nothing, and no refresh results are recorded |
Answers made without the store
While pkgproxy is not contacting the store (Offline mode), a POST /v2/snaps/refresh is answered from the recorded results, with an X-Pkgproxy-Synthesised header. An action is answered only when its snap file and every assertion that vouches for it are cached; otherwise it gets an error of its own, and the other actions still succeed. Actions with a cohort, ignore-validation or validation sets are never recorded or answered.
What still needs the store
Some requests pkgproxy cannot answer for the store yet. Both gaps are planned work.
- The device-session handshake. Before most store requests, snapd makes sure the device has a session. A device that has its serial but no session yet first sends
POST /api/v1/snaps/auth/nonces, thenPOST /api/v1/snaps/auth/sessions. pkgproxy relays both and caches neither, as a nonce is used once and a session belongs to one device. While pkgproxy is not contacting the store (offline, or the store left alone instore-overloaded), the nonce fails, and snapd fails the request it was for, even the download of a snap that is cached:cannot get nonce from store: store server returned status 424. A device that already has a session, or has no serial, is not affected. Planned: pkgproxy#155, answering the handshake while the store is refused. - One breaker for the whole store. In
store-overloaded, the store is left alone per host, andapi.snapcraft.iois one host. Asnap findthe store answers with 429 closes it off forsnap info, install and refresh too, and installs of snaps that are not cached fail with 424 until the search recovers. Planned: pkgproxy#153, a breaker for each kind of store request.
for administrators
Settings
| Setting | Flag | Environment | Snap option | Default |
|---|---|---|---|---|
| Snap Store API to relay to | -snap-store-url | PKGPROXY_SNAP_STORE_URL | flag or environment only | https://api.snapcraft.io |
Address put in rewritten links, as http://host[:port] | -snap-external-url | PKGPROXY_SNAP_EXTERNAL_URL | flag or environment only | the address the client used |
How long details and assertions are trusted when the store announces no max-age | -snap-revalidate-ttl | PKGPROXY_SNAP_REVALIDATE_TTL | snap-revalidate-ttl | revalidate-ttl, 5m |
| How long clients already following a snap download that failed its hash check keep receiving it | -fill-drain-timeout | PKGPROXY_FILL_DRAIN_TIMEOUT | fill-drain-timeout | 10m |
for administrators
On disk
Under the snap directory of the cache (/var/snap/pkgproxy/common/cache/snap in the snap, /var/cache/pkgproxy/snap otherwise), each object with its metadata in an extended attribute of its file:
snap/objects/ sha3-384/<hex> snaps and deltas pending/sha3-384/<hex> the real URL and size a hash stands for, kept after the snap is cached assertions/<type>/<key> assertions info/<name>… snap details, per query and selecting headers find/<hash> searches catalog/<hash> categories, repairs and the older search API refresh/… recorded install and refresh results
See The store.
In the documentation
Roads from here
CORE · the path every side shares
- Fetch once — one download shared while it is written
- Immutable or revalidated — kept forever or rechecked
- Offline mode — installs answered from what is cached
- One listener — how /v2 is shared with Docker
DOCKER
- Docker Hub registry mirror — the other side under /v2
SETUP AND THE COMMAND LINE
- Setting up clients — pointing other clients at pkgproxy