The store
core for administrators for CI operators
Where cached files live on disk, how an object and its metadata are written and committed, and how to free space in a cache that is never evicted.
On the map: core
01 What this topic covers
Where the cache lives on disk, how a download is committed, and how to free space.
The store is where pkgproxy keeps what it downloads, on the disk of the machine it runs on. Fetch once shows how a download gets there. Here you see where the cache lives, how each file carries its metadata, and why going back to 0.11 keeps the cache. Then how a download is stored in one rename and refreshed in place, why nothing is evicted and how you free space by hand, and how to throw away one apt file.
02 Where the cache lives
-cache-dir: /var/snap/pkgproxy/common/cache in
the snap, /var/cache/pkgproxy otherwise. Each kind has
objects/ and tmp/; stats.json,
lxd-index.json and lxd/tls sit beside them.
In the snap, the cache lives in /var/snap/pkgproxy/common/cache, so it survives snap updates. Without the snap it is /var/cache/pkgproxy, or wherever the cache dir flag points. Inside is one directory for each kind of download, each with an objects directory for the cached files and a temporary directory for downloads in progress. Beside them are the saved statistics, the hashes of LXD image files, and the LXD port's certificate.
03 An object and its metadata for administrators
The body is objects/<key>; its metadata is JSON in
the file's user.pkgproxy.meta extended attribute, and in
<key>.meta.json beside it, for pkgproxy 0.11. pkgproxy
refuses to start on a file system without extended attributes (tmpfs
before Linux 6.6, NFSv3, FAT); copy the cache with cp -a,
rsync -aX or tar --xattrs.
Each cached file is stored as itself, and what pkgproxy knows about it is kept as JSON in an extended attribute of that same file. The metadata records the size, the upstream's ETag and last modified time, when the file was stored, its verified content hash, and whether it is immutable. Some sides add the content type, a max age the upstream announced, or the URL a file was downloaded from. So the cache needs a file system that keeps extended attributes. pkgproxy also writes the same JSON to a .meta.json file beside each file. A copy that lost the attribute reads that file instead, and a file that lost both is a miss.
04 Going back to 0.11 keeps the cache for administrators
sudo snap revert pkgproxy goes back to 0.11 with the cache
intact. After each refresh, the snap copies the .meta.json
files of what 0.11 cached into the attributes, and logs how many in
/var/snap/pkgproxy/current/migrate-cache.log; outside the
snap, pkgproxy -migrate-cache does the same, though nothing
needs it.
pkgproxy 0.11 kept each file's metadata in a .meta.json file beside it. 0.12 reads the extended attribute first, but still writes the .meta.json files, so if you go back with snap revert, 0.11 finds a cache it can read. The snap declares epoch 1*: it reads both formats, and snapd refuses to refresh or install 0.11 over it. Only snap revert goes back. A later release, most likely 0.13, will declare epoch 1 and stop writing the .meta.json files. Snapd moves every machine through an epoch 1* release first, so a revert from there lands on that release, not on 0.11.
05 Stored in one rename, refreshed in place for administrators
The metadata is set on the file in tmp/, which is flushed
and renamed into objects/: body and metadata switch
together. A 304 replaces only the
user.pkgproxy.meta attribute.
A download is written to a file under tmp. When it is complete and checked, pkgproxy sets its metadata on that file, flushes it to disk and renames it into place under objects. One rename switches the body and its metadata together, so a reader gets the old pair or the new one, and an interrupted download leaves at most a file under tmp, which is never served. A recheck that the upstream answers with 304 Not Modified does not touch the body: only the attribute is replaced, with a new stored time, so a large index is refreshed by one small write.
06 Never evicted, freed by hand
Nothing is evicted. Stop pkgproxy, remove one kind's directory, start it
again. After removing lxd, run pkgproxyctl setup
lxd-remote | sh; keep stats.json to keep the
statistics.
pkgproxy deletes nothing from its cache on its own, so the cache only grows. The du command shows how much each kind of download takes. To free the space of one kind, stop the service, remove its directory, and start it again; clients download what is gone the next time they need it. Removing the lxd directory also replaces the certificate of the LXD port, so run setup lxd-remote again. Leave stats.json in place unless you also want to reset the statistics.
07 Throwing away one apt file for administrators
pkgproxyctl cache invalidate <url> sends
DELETE /pkgproxy/v1/cache over admin.sock. It
works for apt only; other kinds are verified before they are stored.
For apt there is a way to remove one file early, such as a broken package list that a mirror served. pkgproxyctl cache invalidate takes the http address apt downloads the file from, with dists or pool in its path. It asks the daemon over the admin socket, the daemon deletes that one entry, and the next request fetches it again. The other kinds of download are checked against published hashes, so a broken one is not kept in the first place.
08 Where to go next general version
Next: Statistics, for what the cache holds and saved; The cache page, for what is cached and served now; Settings, for the cache directory.
You now know where each download lands on disk, and that it stays there until you remove it. To see how much the cache holds and has saved, watch Statistics. To see everything that is cached, and what is served from it right now, watch The cache page. To move the cache directory, watch Settings.
Read more
Where things are
| What | In the snap | Without the snap |
|---|---|---|
Cache directory (-cache-dir) | /var/snap/pkgproxy/common/cache | /var/cache/pkgproxy |
| Statistics socket | /var/snap/pkgproxy/common/stats.sock | /run/pkgproxy/stats.sock |
| Admin socket | /var/snap/pkgproxy/common/admin.sock | /run/pkgproxy/admin.sock |
| Content sockets, shared with other snaps | /var/snap/pkgproxy/common/sockets/ | only where -content-socket-path and -lxd-tls-content-socket-path point |
| Settings the configure hook last applied | /var/snap/pkgproxy/current/configured.env | (none) |
| What the last refresh migrated | /var/snap/pkgproxy/current/migrate-cache.log | (none) |
The snap's common directory survives snap updates and refreshes.
The cache directory
/var/snap/pkgproxy/common/
├── cache/
│ ├── apt/
│ │ ├── objects/
│ │ │ └── archive.ubuntu.com/ubuntu/
│ │ │ ├── dists/noble/InRelease
│ │ │ ├── dists/noble/InRelease.meta.json
│ │ │ ├── pool/main/h/hello/hello_2.10-3_amd64.deb
│ │ │ └── pool/main/h/hello/hello_2.10-3_amd64.deb.meta.json
│ │ └── tmp/ downloads in progress
│ ├── snap/
│ │ ├── objects/
│ │ │ ├── sha3-384/HEX snap files, by hash
│ │ │ ├── info/NAME snap details
│ │ │ ├── find/… refresh/… catalog/…
│ │ │ ├── assertions/…
│ │ │ └── pending/sha3-384/HEX where a rewritten download really points
│ │ └── tmp/
│ ├── go/
│ │ ├── objects/
│ │ │ └── MODULE/@v/ list, VERSION.info, VERSION.mod, VERSION.zip
│ │ ├── sumdb/ checksum database tiles and notes
│ │ └── tmp/
│ ├── docker/
│ │ ├── objects/
│ │ │ ├── blobs/sha256/HEX layers and configs
│ │ │ ├── manifests/sha256/HEX
│ │ │ └── tags/REPOSITORY/_/TAG/ACCEPT
│ │ └── tmp/
│ ├── lxd/
│ │ ├── objects/
│ │ │ └── REMOTE/… the image server's own paths
│ │ ├── tmp/
│ │ └── tls/
│ │ ├── cert.pem the HTTPS port's self-signed certificate
│ │ └── key.pem and its private key
│ ├── files/
│ │ ├── objects/
│ │ │ ├── sha256/HEX
│ │ │ └── sha512/HEX
│ │ └── tmp/
│ ├── stats.json saved statistics
│ └── lxd-index.json saved hashes of LXD image files
├── stats.sock
├── admin.sock
└── sockets/
├── pkgproxy.sock the main port, for another snap
└── pkgproxy-lxd-tls.sock the HTTPS port, for another snap
Without the snap, what the snap keeps in cache/ is in /var/cache/pkgproxy, and stats.sock and admin.sock are in /run/pkgproxy. Every cached object is one file, with its metadata in an extended attribute of that file and in a .meta.json file beside it. A kind's directory is made when its side starts, so a side turned off, such as Docker with docker-upstream set to off, has none; lxd/tls exists only while the HTTPS port is on.
What a key looks like
| Kind | Key under objects/ | Example |
|---|---|---|
| apt | mirror host (with a port other than 80), the path before dists or pool, then the archive path | archive.ubuntu.com/ubuntu/pool/main/h/hello/hello_2.10-3_amd64.deb |
| snap | sha3-384/ and the snap's hash; details, searches, refreshes and assertions under their own prefixes | sha3-384/1f2e… |
| go | the module path and version, case-escaped as the Go module proxy protocol names them | golang.org/x/mod/@v/v0.20.0.zip |
| docker | blobs/sha256/, manifests/sha256/, or tags/ with the repository, tag and a short hash of the client's accepted types | tags/library/ubuntu/_/24.04/any |
| lxd | the remote's name, then the image server's path | pkgproxy-ubuntu/server/releases/noble/… |
| files | the checksum's algorithm and its hex digest, whatever URL the file came from | sha256/9f86… |
The metadata
Each object's metadata is JSON in the user.pkgproxy.meta extended attribute of its file, so it is set and replaced together with the file. getfattr -n user.pkgproxy.meta FILE shows it. Metadata of more than 3 KiB is kept compressed with gzip, up to 64 KiB of JSON: getfattr --only-values -n user.pkgproxy.meta FILE | zcat shows it.
| Field | Meaning |
|---|---|
Size | size of the object in bytes |
ETag, LastModified | the upstream's validators, sent back when the entry is rechecked |
StoredAt | when the object was stored or last rechecked |
ContentHash | the hash of the object as stored, as ALGORITHM:DIGEST in the kind's own terms: SHA256:HEX (or the index's other algorithm) for apt, sha3-384:HEX for snaps, sha256:HEX for LXD images, Docker digests and files, h1:BASE64 for Go module zips and go.mod files, and sha256:HEX of what was stored for Go's version lists and .info files; empty for the rest. Where the kind has a published hash to check against, the object matched it before it was stored; an apt file no index lists and Go's version lists are stored with the hash unchecked |
Immutable | the entry is kept forever and never rechecked |
ContentType | the media type to serve it with, for sides that keep one |
MaxAge | a lifetime the upstream announced, for sides that honour one |
OriginURL | the URL the object was downloaded from, for sides that record it; left out if the metadata would not fit otherwise |
pkgproxy also writes the same JSON, uncompressed, to <file>.meta.json beside each object, which is where pkgproxy 0.11 kept it, so that going back to 0.11 keeps the cache (below). A file without the attribute takes its metadata from that file; one without either is a miss and is fetched again, and pkgproxy logs the first one it finds in each kind's directory. Anything under tmp/ is never served and can be removed while pkgproxy is stopped.
for administrators
Upgrading and going back
| the cache from 0.11 | every object has only its .meta.json file; 0.12 reads it as it is |
| after each snap refresh | the post-refresh hook runs pkgproxy -migrate-cache, which copies each .meta.json file of what 0.11 cached into the file's extended attribute, keeps the .meta.json files, and stops after two minutes; it writes how many files it migrated to /var/snap/pkgproxy/current/migrate-cache.log. A file it did not reach still works |
| without the snap | pkgproxy -migrate-cache does the same, then exits; nothing requires it |
| going back | sudo snap revert pkgproxy: 0.11 finds every object it would have written itself |
| going back any other way | refused: the snap declares epoch 1*, which reads the cache of 0.11 (epoch 0) and its own (epoch 1), and snapd refuses to refresh or install a revision of epoch 0, such as 0.11, over it |
| after a revert, on 0.12 again | what 0.11 stored reads right; a file 0.11 revalidated is revalidated again sooner than it need be |
| later | a release, most likely 0.13, will declare epoch 1: it stops writing and reading the .meta.json files, and removes them. snapd moves every machine through an epoch 1* revision first, so a revert from it lands there, not on 0.11 |
File systems and backups
| file systems that keep extended attributes | ext4, XFS, btrfs, ZFS; tmpfs from Linux 6.6 |
| file systems that do not | tmpfs before Linux 6.6, NFSv3, FAT: pkgproxy refuses to start with its cache there |
| copying or backing up the cache | cp -a, rsync -aX or tar --xattrs, which keep extended attributes |
| a copy that dropped them | every file reads its metadata from its .meta.json file; a file without that too is a miss and is downloaded again the next time a client needs it, and pkgproxy logs the first such file it finds in each kind's directory |
for administrators
Freeing space
| To | Do |
|---|---|
| see what each kind takes | du -sh /var/snap/pkgproxy/common/cache/* |
| free one kind | sudo snap stop pkgproxy, sudo rm -rf /var/snap/pkgproxy/common/cache/KIND, sudo snap start pkgproxy |
| free the LXD kind | as above, then pkgproxyctl setup lxd-remote again on every client, because the certificate is new |
| throw away one apt file | pkgproxyctl cache invalidate URL, with the http:// URL apt fetches it from |
| keep the statistics | leave stats.json in place |
See pkgproxyctl for cache invalidate and Statistics for what stats.json holds.
In the documentation
Roads from here
START HERE
- A request, end to end — a request end to end
CORE · the path every side shares
- Fetch once — how a miss is written into the store
- Immutable or revalidated — what the metadata is used for
OPERATIONS
- Statistics — how much the cache holds and saved
- The cache page — what is cached and served now
- Settings — the cache directory and other settings