The store

core for administrators for CI operators

Where cached files live on disk, how an object and its metadata are written and committed, and how to free space in a cache that is never evicted.

On the map: core

▶ Watch this deck — 8 scenes, ~5 min

01 What this topic covers

ABOUT THIS TOPIC The store where pkgproxy keeps what it downloads, on its machine's disk 1 Where the cache lives 2 An object and its metadata for administrators 3 Going back to 0.11 keeps the cache for administrators 4 Stored in one rename, refreshed in place for administrators 5 Never evicted, freed by hand 6 Throwing away one apt file for administrators
The topic title, The store, and its 6 scenes in order: Where the cache lives, An object and its metadata, Going back to 0.11 keeps the cache, Stored in one rename, refreshed in place, Never evicted, freed by hand, Throwing away one apt file

Where the cache lives on disk, how a download is committed, and how to free space.

The store is where pkgproxy keeps what it downloads, on the disk of the machine it runs on. Fetch once shows how a download gets there. Here you see where the cache lives, how each file carries its metadata, and why going back to 0.11 keeps the cache. Then how a download is stored in one rename and refreshed in place, why nothing is evicted and how you free space by hand, and how to throw away one apt file.

02 Where the cache lives

ON DISK /var/snap/pkgproxy/common/cache/ the snap; survives updates without the snap: /var/cache/pkgproxy, or -cache-dir apt/ objects/ deb.debian.org/debian/pool/main/h/hello/ hello_2.10-3_amd64.deb its metadata in an extended attribute of the file tmp/ downloads in progress snap/ go/ docker/ files/ lxd/ each: objects/ and tmp/ tls/ certificate of port 3143 stats.json saved statistics lxd-index.json hashes of LXD image files
A directory tree of the cache: one directory per kind, apt expanded to show objects with a cached package, its metadata in an extended attribute, and tmp; beside them stats.json, lxd-index.json and the lxd/tls certificate directory

-cache-dir: /var/snap/pkgproxy/common/cache in the snap, /var/cache/pkgproxy otherwise. Each kind has objects/ and tmp/; stats.json, lxd-index.json and lxd/tls sit beside them.

In the snap, the cache lives in /var/snap/pkgproxy/common/cache, so it survives snap updates. Without the snap it is /var/cache/pkgproxy, or wherever the cache dir flag points. Inside is one directory for each kind of download, each with an objects directory for the cached files and a temporary directory for downloads in progress. Beside them are the saved statistics, the hashes of LXD image files, and the LXD port's certificate.

03 An object and its metadata for administrators

AN ENTRY hello_2.10-3_amd64.deb the bytes, as the upstream sent them extended attribute user.pkgproxy.meta { "Size": …, "ETag": "…", "LastModified": "…", "StoredAt": "…", "ContentHash": "SHA256:…", "Immutable": true } plus, on some sides: ContentType · MaxAge · OriginURL a file system that keeps them: ext4 · XFS · btrfs · ZFS tmpfs from Linux 6.6 and copies that keep them: cp -a · rsync -aX · tar --xattrs hello_2.10-3_amd64.deb.meta.json the same JSON lost the attribute: read from .meta.json lost both: a miss, fetched again
A cached package file with its metadata as JSON in the file's extended attribute user.pkgproxy.meta, listing Size, ETag, LastModified, StoredAt, ContentHash and Immutable; a .meta.json file beside it holds the same JSON; the file system and every copy should keep extended attributes, a file that lost its attribute is read from its .meta.json file, and one that lost both is a miss

The body is objects/<key>; its metadata is JSON in the file's user.pkgproxy.meta extended attribute, and in <key>.meta.json beside it, for pkgproxy 0.11. pkgproxy refuses to start on a file system without extended attributes (tmpfs before Linux 6.6, NFSv3, FAT); copy the cache with cp -a, rsync -aX or tar --xattrs.

Each cached file is stored as itself, and what pkgproxy knows about it is kept as JSON in an extended attribute of that same file. The metadata records the size, the upstream's ETag and last modified time, when the file was stored, its verified content hash, and whether it is immutable. Some sides add the content type, a max age the upstream announced, or the URL a file was downloaded from. So the cache needs a file system that keeps extended attributes. pkgproxy also writes the same JSON to a .meta.json file beside each file. A copy that lost the attribute reads that file instead, and a file that lost both is a miss.

04 Going back to 0.11 keeps the cache for administrators

GOING BACK pkgproxy 0.11 epoch 0 writes <file>.meta.json refresh pkgproxy 0.12 epoch 1*: reads 0 and 1 writes attribute + .meta.json snap revert: 0.11 reads the cache refresh or install 0.11 over 0.12 refused by snapd only snap revert goes back later likely 0.13 epoch 1 the attribute alone revert lands on epoch 1* not on 0.11
Three snap revisions: pkgproxy 0.11, epoch 0, writes .meta.json files; 0.12, epoch 1*, reads both and writes the extended attribute and the .meta.json files; a later release, likely 0.13, epoch 1, the attribute alone. snap revert from 0.12 to 0.11 keeps the cache; snapd refuses to refresh or install 0.11 over 0.12; a revert from the later release lands on an epoch 1* revision

sudo snap revert pkgproxy goes back to 0.11 with the cache intact. After each refresh, the snap copies the .meta.json files of what 0.11 cached into the attributes, and logs how many in /var/snap/pkgproxy/current/migrate-cache.log; outside the snap, pkgproxy -migrate-cache does the same, though nothing needs it.

pkgproxy 0.11 kept each file's metadata in a .meta.json file beside it. 0.12 reads the extended attribute first, but still writes the .meta.json files, so if you go back with snap revert, 0.11 finds a cache it can read. The snap declares epoch 1*: it reads both formats, and snapd refuses to refresh or install 0.11 over it. Only snap revert goes back. A later release, most likely 0.13, will declare epoch 1 and stop writing the .meta.json files. Snapd moves every machine through an epoch 1* release first, so a revert from there lands on that release, not on 0.11.

05 Stored in one rename, refreshed in place for administrators

ONE RENAME stored tmp/obj-… + metadata fsync rename objects/<key> + metadata body and metadata switch together a reader gets the old pair or the new one interrupted: a file under tmp/, never served 304 304 Not Modified objects/<key> body untouched replace + metadata new StoredAt · one small write
Left: a finished download under tmp gets its metadata attribute, is flushed and renamed into objects, so one rename switches body and metadata together. Right: a 304 answer replaces only the attribute, leaving the body untouched

The metadata is set on the file in tmp/, which is flushed and renamed into objects/: body and metadata switch together. A 304 replaces only the user.pkgproxy.meta attribute.

A download is written to a file under tmp. When it is complete and checked, pkgproxy sets its metadata on that file, flushes it to disk and renames it into place under objects. One rename switches the body and its metadata together, so a reader gets the old pair or the new one, and an interrupted download leaves at most a file under tmp, which is never served. A recheck that the upstream answers with 304 Not Modified does not touch the body: only the attribute is replaced, with a new stored time, so a large index is refreshed by one small write.

06 Never evicted, freed by hand

DISK SPACE nothing is deleted on its own: the cache only grows $ du -sh /var/snap/pkgproxy/common/cache/* … /var/snap/pkgproxy/common/cache/apt … /var/snap/pkgproxy/common/cache/docker … $ sudo snap stop pkgproxy $ sudo rm -rf /var/snap/pkgproxy/common/cache/docker $ sudo snap start pkgproxy what is gone is downloaded again when a client next needs it removed lxd/? a new certificate: pkgproxyctl setup lxd-remote | sh stats.json keep it, or the statistics start over
A terminal: du shows the size of each kind's directory; then stop pkgproxy, remove the docker directory and start it again. Notes: removing lxd needs setup lxd-remote again; stats.json holds the statistics

Nothing is evicted. Stop pkgproxy, remove one kind's directory, start it again. After removing lxd, run pkgproxyctl setup lxd-remote | sh; keep stats.json to keep the statistics.

pkgproxy deletes nothing from its cache on its own, so the cache only grows. The du command shows how much each kind of download takes. To free the space of one kind, stop the service, remove its directory, and start it again; clients download what is gone the next time they need it. Removing the lxd directory also replaces the certificate of the LXD port, so run setup lxd-remote again. Leave stats.json in place unless you also want to reset the statistics.

07 Throwing away one apt file for administrators

ONE FILE $ pkgproxyctl cache invalidate http://deb.debian.org/debian/dists/bookworm/InRelease invalidated cache entry for http://deb.debian.org/debian/dists/bookworm/InRelease pkgproxyctl admin.sock DELETE /pkgproxy/v1/cache pkgproxy apt/objects/…/InRelease deleted the next request fetches it again apt only snaps, Go modules, Docker, LXD images and files are checked against their hashes before they are kept
pkgproxyctl cache invalidate with an InRelease URL; the request goes over the admin socket to the daemon, which deletes that one apt entry; the next apt update fetches it again

pkgproxyctl cache invalidate <url> sends DELETE /pkgproxy/v1/cache over admin.sock. It works for apt only; other kinds are verified before they are stored.

For apt there is a way to remove one file early, such as a broken package list that a mirror served. pkgproxyctl cache invalidate takes the http address apt downloads the file from, with dists or pool in its path. It asks the daemon over the admin socket, the daemon deletes that one entry, and the next request fetches it again. The other kinds of download are checked against published hashes, so a broken one is not kept in the first place.

08 Where to go next general version

WHERE NEXT every download lands in one directory per kind, and stays there until you remove it you are here The store Statistics how much the cache holds and has saved The cache page everything cached, and what is served now Settings moving the cache directory
You are here, The store; next: Statistics, The cache page, Settings

Next: Statistics, for what the cache holds and saved; The cache page, for what is cached and served now; Settings, for the cache directory.

You now know where each download lands on disk, and that it stays there until you remove it. To see how much the cache holds and has saved, watch Statistics. To see everything that is cached, and what is served from it right now, watch The cache page. To move the cache directory, watch Settings.

Read more

Where things are

WhatIn the snapWithout the snap
Cache directory (-cache-dir)/var/snap/pkgproxy/common/cache/var/cache/pkgproxy
Statistics socket/var/snap/pkgproxy/common/stats.sock/run/pkgproxy/stats.sock
Admin socket/var/snap/pkgproxy/common/admin.sock/run/pkgproxy/admin.sock
Content sockets, shared with other snaps/var/snap/pkgproxy/common/sockets/only where -content-socket-path and -lxd-tls-content-socket-path point
Settings the configure hook last applied/var/snap/pkgproxy/current/configured.env(none)
What the last refresh migrated/var/snap/pkgproxy/current/migrate-cache.log(none)

The snap's common directory survives snap updates and refreshes.

The cache directory

/var/snap/pkgproxy/common/
├── cache/
│   ├── apt/
│   │   ├── objects/
│   │   │   └── archive.ubuntu.com/ubuntu/
│   │   │       ├── dists/noble/InRelease
│   │   │       ├── dists/noble/InRelease.meta.json
│   │   │       ├── pool/main/h/hello/hello_2.10-3_amd64.deb
│   │   │       └── pool/main/h/hello/hello_2.10-3_amd64.deb.meta.json
│   │   └── tmp/                  downloads in progress
│   ├── snap/
│   │   ├── objects/
│   │   │   ├── sha3-384/HEX      snap files, by hash
│   │   │   ├── info/NAME         snap details
│   │   │   ├── find/…  refresh/…  catalog/…
│   │   │   ├── assertions/…
│   │   │   └── pending/sha3-384/HEX   where a rewritten download really points
│   │   └── tmp/
│   ├── go/
│   │   ├── objects/
│   │   │   └── MODULE/@v/        list, VERSION.info, VERSION.mod, VERSION.zip
│   │   ├── sumdb/                checksum database tiles and notes
│   │   └── tmp/
│   ├── docker/
│   │   ├── objects/
│   │   │   ├── blobs/sha256/HEX      layers and configs
│   │   │   ├── manifests/sha256/HEX
│   │   │   └── tags/REPOSITORY/_/TAG/ACCEPT
│   │   └── tmp/
│   ├── lxd/
│   │   ├── objects/
│   │   │   └── REMOTE/…          the image server's own paths
│   │   ├── tmp/
│   │   └── tls/
│   │       ├── cert.pem          the HTTPS port's self-signed certificate
│   │       └── key.pem           and its private key
│   ├── files/
│   │   ├── objects/
│   │   │   ├── sha256/HEX
│   │   │   └── sha512/HEX
│   │   └── tmp/
│   ├── stats.json                saved statistics
│   └── lxd-index.json            saved hashes of LXD image files
├── stats.sock
├── admin.sock
└── sockets/
    ├── pkgproxy.sock             the main port, for another snap
    └── pkgproxy-lxd-tls.sock     the HTTPS port, for another snap

Without the snap, what the snap keeps in cache/ is in /var/cache/pkgproxy, and stats.sock and admin.sock are in /run/pkgproxy. Every cached object is one file, with its metadata in an extended attribute of that file and in a .meta.json file beside it. A kind's directory is made when its side starts, so a side turned off, such as Docker with docker-upstream set to off, has none; lxd/tls exists only while the HTTPS port is on.

What a key looks like

KindKey under objects/Example
aptmirror host (with a port other than 80), the path before dists or pool, then the archive patharchive.ubuntu.com/ubuntu/pool/main/h/hello/hello_2.10-3_amd64.deb
snapsha3-384/ and the snap's hash; details, searches, refreshes and assertions under their own prefixessha3-384/1f2e…
gothe module path and version, case-escaped as the Go module proxy protocol names themgolang.org/x/mod/@v/v0.20.0.zip
dockerblobs/sha256/, manifests/sha256/, or tags/ with the repository, tag and a short hash of the client's accepted typestags/library/ubuntu/_/24.04/any
lxdthe remote's name, then the image server's pathpkgproxy-ubuntu/server/releases/noble/…
filesthe checksum's algorithm and its hex digest, whatever URL the file came fromsha256/9f86…

The metadata

Each object's metadata is JSON in the user.pkgproxy.meta extended attribute of its file, so it is set and replaced together with the file. getfattr -n user.pkgproxy.meta FILE shows it. Metadata of more than 3 KiB is kept compressed with gzip, up to 64 KiB of JSON: getfattr --only-values -n user.pkgproxy.meta FILE | zcat shows it.

FieldMeaning
Sizesize of the object in bytes
ETag, LastModifiedthe upstream's validators, sent back when the entry is rechecked
StoredAtwhen the object was stored or last rechecked
ContentHashthe hash of the object as stored, as ALGORITHM:DIGEST in the kind's own terms: SHA256:HEX (or the index's other algorithm) for apt, sha3-384:HEX for snaps, sha256:HEX for LXD images, Docker digests and files, h1:BASE64 for Go module zips and go.mod files, and sha256:HEX of what was stored for Go's version lists and .info files; empty for the rest. Where the kind has a published hash to check against, the object matched it before it was stored; an apt file no index lists and Go's version lists are stored with the hash unchecked
Immutablethe entry is kept forever and never rechecked
ContentTypethe media type to serve it with, for sides that keep one
MaxAgea lifetime the upstream announced, for sides that honour one
OriginURLthe URL the object was downloaded from, for sides that record it; left out if the metadata would not fit otherwise

pkgproxy also writes the same JSON, uncompressed, to <file>.meta.json beside each object, which is where pkgproxy 0.11 kept it, so that going back to 0.11 keeps the cache (below). A file without the attribute takes its metadata from that file; one without either is a miss and is fetched again, and pkgproxy logs the first one it finds in each kind's directory. Anything under tmp/ is never served and can be removed while pkgproxy is stopped.

for administrators

Upgrading and going back

the cache from 0.11every object has only its .meta.json file; 0.12 reads it as it is
after each snap refreshthe post-refresh hook runs pkgproxy -migrate-cache, which copies each .meta.json file of what 0.11 cached into the file's extended attribute, keeps the .meta.json files, and stops after two minutes; it writes how many files it migrated to /var/snap/pkgproxy/current/migrate-cache.log. A file it did not reach still works
without the snappkgproxy -migrate-cache does the same, then exits; nothing requires it
going backsudo snap revert pkgproxy: 0.11 finds every object it would have written itself
going back any other wayrefused: the snap declares epoch 1*, which reads the cache of 0.11 (epoch 0) and its own (epoch 1), and snapd refuses to refresh or install a revision of epoch 0, such as 0.11, over it
after a revert, on 0.12 againwhat 0.11 stored reads right; a file 0.11 revalidated is revalidated again sooner than it need be
latera release, most likely 0.13, will declare epoch 1: it stops writing and reading the .meta.json files, and removes them. snapd moves every machine through an epoch 1* revision first, so a revert from it lands there, not on 0.11

File systems and backups

file systems that keep extended attributesext4, XFS, btrfs, ZFS; tmpfs from Linux 6.6
file systems that do nottmpfs before Linux 6.6, NFSv3, FAT: pkgproxy refuses to start with its cache there
copying or backing up the cachecp -a, rsync -aX or tar --xattrs, which keep extended attributes
a copy that dropped themevery file reads its metadata from its .meta.json file; a file without that too is a miss and is downloaded again the next time a client needs it, and pkgproxy logs the first such file it finds in each kind's directory

for administrators

Freeing space

ToDo
see what each kind takesdu -sh /var/snap/pkgproxy/common/cache/*
free one kindsudo snap stop pkgproxy, sudo rm -rf /var/snap/pkgproxy/common/cache/KIND, sudo snap start pkgproxy
free the LXD kindas above, then pkgproxyctl setup lxd-remote again on every client, because the certificate is new
throw away one apt filepkgproxyctl cache invalidate URL, with the http:// URL apt fetches it from
keep the statisticsleave stats.json in place

See pkgproxyctl for cache invalidate and Statistics for what stats.json holds.

Roads from here

START HERE

CORE · the path every side shares

OPERATIONS