snap debug: tools for looking inside
command group for app developers for Ubuntu Core developers for snapd developers for administrators
The 27 subcommands of snap debug sorted into four classes by who reaches for them. Most ask snapd through one /v2/debug endpoint where GET is open to every local user and POST is root-only; ten work without the daemon and snap debug api reaches the rest.
On the map: tools district
01 Four kinds of reader
Hidden only means "not in help"; a hidden subcommand still runs. Pick the class that matches who you are.
Something's wrong with the acme-agent on the gateway, and snap list isn't telling you why. That's when you reach for snap debug: twenty seven subcommands for looking inside snapd and the sandbox. Only snap help --all lists it, and its help warns that these commands can be removed without notice. Some are hidden from help, but a hidden subcommand still runs when you type it. This deck sorts the twenty seven by who reaches for them: app developers, Ubuntu Core developers, snapd developers and support.
02 One endpoint, two verbs
GET /v2/debug?aspect= is open to every local user; POST {"action": ...} is root only. The verb, not the effect, decides who may call.
Most subcommands that talk to snapd ask one endpoint, /v2/debug, and how they ask decides who may. A GET carries an aspect, and all nine aspects are open to every local user. A POST carries an action, and all eight actions need root. The split is by verb, not by effect. So stacktraces and can-manage-refreshes need root although they only read, and a spread test checks that a normal user gets access denied. Four actions have a subcommand; the other four you reach through snap debug api. Confinement and sandbox-features ask /v2/system-info instead.
03 Three ways to look inside
When snapd is wedged, state, device-cgroup, lsm and mount-namespace still answer. snap debug state defaults to ./state.json; pass /var/lib/snapd/state.json as root.
Now suppose snapd itself is stuck, so asking it gets you nowhere. Not every subcommand needs it. Most ask snapd, but snap debug state reads a state file directly. With no argument it opens state.json in the current directory, so on the gateway you pass the real path, as root, because only root may read that file. device-cgroup, lsm and mount namespace read the kernel and the run directory. Ten subcommands in all work with snapd stopped, which is exactly when you need them: if snapd is wedged, the state file and the kernel still answer.
04 For app developers: what the sandbox gives you for app developers
confinement and sandbox-features describe the host; device-cgroup and mount-namespace describe one snap at run time.
You're building the acme-agent. What does the sandbox around it actually look like? Confinement tells you the mode the system runs snaps in, not the mode of any one snap. sandbox-features --required prints nothing and fails when a feature is missing, which makes it a guard in test scripts. device-cgroup, as root, shows the device nodes each of the agent's apps may open, one section per security tag. Mount namespace --shell drops you into the namespace snapd keeps for the snap, so you see the files as the agent does, with a minimal environment rather than the snap's. Discard throws that namespace away.
05 For Ubuntu Core developers: first boot and boot state for Ubuntu Core developers
seeding for first boot, timings --ensure=seed or --ensure=install-system for where the time went, boot-vars for try and recovery state.
You've just built the gateway's image, and first boot felt slow. Where did the time go? Snap debug seeding says whether seeding finished and how long it took. On a preseeded image, image-preseeding was measured when the image was built, and seed-completion is measured on the device from the seed-restart-time. On an unseeded device, a seed-error line shows the error of the earliest failed change. Timings --ensure=seed breaks the seed work into nested spans. And boot-vars prints a fixed set of bootloader variables, not the whole environment; --root-dir reads them from another directory, such as the mounted seed partition.
06 For snapd developers: changes, time and goroutines for snapd developers
Offline graph: state. Time: timings. Live goroutines: stacktraces, or /v2/debug/pprof/goroutine when the state lock is stuck. Run the managers now: ensure-state-soon.
The acme-agent's refresh from 41 to 42 hangs halfway, and you're fixing snapd. Snap debug state reads the change and task graph from a copy of state.json, without snapd. For one change it lists the tasks by lane and wait order, then their logs. Dot writes the graph for graphviz, and check looks for dependency cycles. Timings shows how long each task spent doing and undoing, with nested spans beneath. Stacktraces prints snapd's goroutines, as root, but takes the state lock first. If that lock is stuck, read the pprof goroutine profile with snap debug api. And ensure-state-soon asks every manager to run now.
07 For support: why it will not refresh for administrators
connectivity for reaching the store, refresh-app-awareness for refreshes waiting on a running app, can-manage-refreshes for whether refresh.timer=managed can be set.
The gateway hasn't refreshed the acme-agent in days. Where do you start? Connectivity asks snapd to reach the store hosts and prints pass, or each host it couldn't reach. It runs inside snapd, so it uses snapd's proxy settings, not your shell's. refresh-app-awareness lists snaps whose refresh waits because one of their apps is running, and the candidates waiting on them. Once the app exits, the refresh goes ahead and the list empties. And can-manage-refreshes, as root, answers whether a snap here may take over the refresh schedule. If it says true, look at refresh.timer next.
08 Endpoints without a subcommand: snap debug api for snapd developers for Ubuntu Core developers for administrators
When no subcommand exists, use snap debug api, and add --fail in scripts so HTTP errors give a non-zero exit status.
Some endpoints have no subcommand at all. Snap debug api is a raw client for any snapd path, on the main socket, the snapd-snap.socket, or a user's session agent. POST and PUT read the request body from standard input. Four debug actions and the pprof endpoints can only be reached this way, so the tests use it, for example to create a recovery system on the gateway. The response carries a change id, which you follow with snap watch. One trap: without --fail, the exit status is zero even when the request returns an HTTP error, so scripts should always pass it.
09 Where to go from here
Roads: the overlord, confinement, boot and recovery, model, gadget and kernel, the store, and how interfaces are implemented.
Where next? The overlord explains the changes, tasks, timings and ensure loop that state, timings, ensure-state-soon and stacktraces look at. Confinement covers the sandbox that sandbox-features, device-cgroup, mount namespace and lsm report on. Boot and recovery explains the variables that boot-vars reads. Model, gadget and kernel covers what model, seeding, validate-seed and gadget-disk-mapping look at. The store covers the hosts that connectivity checks, and the download cache. And how interfaces are implemented explains the base declaration that base-declaration prints.
Read more
The catalogue below this deck lists every snap debug subcommand with its usage, class, source and gotchas. This page collects what cuts across them.
for snapd developers for Ubuntu Core developers for administrators
The /v2/debug endpoint
On snapd.socket. GET is open to every local user; POST needs root, whatever the action does.
| Verb | Value | Subcommand |
|---|---|---|
GET ?aspect= | base-declaration | base-declaration, get-base-declaration |
connectivity | connectivity | |
model | model | |
change-timings (with change-id, ensure, startup, all) | timings | |
seeding | seeding | |
gadget-disk-mapping | gadget-disk-mapping | |
disks | disks | |
raa | refresh-app-awareness | |
features | features | |
POST {"action": ...} | ensure-state-soon | ensure-state-soon |
can-manage-refreshes | can-manage-refreshes | |
stacktraces | stacktraces | |
migrate-home (with snaps) | migrate-home | |
add-warning (with message) | none | |
unshow-warnings | none | |
prune | none | |
create-recovery-system (with params.recovery-system-label) | none; returns a change |
Every POST takes the state lock before it looks at the action, stacktraces included.
/v2/debug/pprof/ (root only) serves Go's cmdline, profile, symbol, trace and the profiles heap, allocs, block, threadcreate, goroutine and mutex. It does not take the state lock.
confinement and sandbox-features read /v2/system-info instead.
Working without snapd
These ten read files, the kernel or the bootloader directly: boot-vars, set-boot-vars, device-cgroup, execution, lsm, mount-namespace, paths, snap-downloads-cache, state, validate-seed.
snap debug state opens ./state.json unless given a path. The live file is /var/lib/snapd/state.json, mode 0600; copy it before reading it while snapd runs.
Hidden and internal
| Subcommands | |
|---|---|
| Always hidden from help | base-declaration, get-base-declaration, can-manage-refreshes, disks, ensure-state-soon, gadget-disk-mapping, model |
| Hidden on classic | boot-vars, set-boot-vars; both refuse to run there |
| Short help starts with "(internal)" | refresh-app-awareness, lsm; on Ubuntu Core also boot-vars, set-boot-vars |
A hidden subcommand runs when typed. snap debug itself is listed only by snap help --all.
for snapd developers for Ubuntu Core developers for administrators
snap debug api
| Option | Effect |
|---|---|
-X GET, POST, PUT | the method; POST and PUT read the body from standard input |
-H 'Name: value' | an extra header, repeatable |
--fail | exit with an error on an HTTP status of 400 or more; without it the exit status is 0 |
--snap-socket | talk to snapd-snap.socket instead of snapd.socket |
--session-agent-uid=<uid> | talk to that user's session agent; cannot be combined with --snap-socket |
--disable-auth | do not send the credentials from auth.json |
for snapd developers for Ubuntu Core developers
Known limits
- On a system that is not seeded,
seedingreports asseed-errorthe error of the earliest failed change among all finished changes and all seed changes, so the error can come from a change that is not the seed change.
Further reading
- Fix common issues: reading
state.jsonwithsnap debug statewhen snapd misbehaves. - Snap performance:
snap debug timingsfor changes, ensure activities and startup, with examples. - Debug snaps: the wider toolkit for debugging a snap, including the information script shipped with snapd.
Roads from here
GETTING SNAPS
- The store — the store hosts that connectivity checks and the download cache
INTERFACES AND CONFINEMENT
- Confinement — the confinement that sandbox-features and device-cgroup and mount-namespace report on
- How interfaces are implemented — the base declaration that base-declaration prints
RUNNING SNAPS
- The overlord — changes and tasks and timings and the ensure loop that state and timings inspect
UBUNTU CORE · boot
- Ubuntu Core: how a device boots and recovers — the boot variables and recovery systems that boot-vars reads
UBUNTU CORE · install and recovery
- Model, gadget and kernel — the model and gadget and seed behind model and seeding and validate-seed
DEVELOPING SNAPS · try, pack, run, debug
- Running and debugging an app — snap run and its debugging options
INSIDE SNAPD · the daemon itself
- The snapd REST API — the sockets and access rules behind /v2/debug