snap debug: tools for looking inside

command group for app developers for Ubuntu Core developers for snapd developers for administrators

The 27 subcommands of snap debug sorted into four classes by who reaches for them. Most ask snapd through one /v2/debug endpoint where GET is open to every local user and POST is root-only; ten work without the daemon and snap debug api reaches the rest.

On the map: tools district

▶ Watch this deck — 9 scenes, ~7 min

01 Four kinds of reader

SNAP DEBUG $ snap help --allunder Introspectiondebug Run debug commands $ snap debug --help18 listed on classic, 20 on Ubuntu Core hidden: 7 alwaysbase-declaration get-base-declarationcan-manage-refreshes disks modelensure-state-soon gadget-disk-mappingon classic also:boot-vars set-boot-varshidden, still runs when typed app developer7 subcommandssandbox-featuresmount-namespace Ubuntu Core developer7 subcommandsseedingboot-vars snapd developer8 subcommandsstatetimings support5 subcommandsconnectivityrefresh-app-awareness "Debug commands can be removed without notice and may not work on non-development systems."
snap debug is listed under Introspection in snap help --all; nine subcommands are hidden from help but still run; the 27 subcommands fall into four classes: app developer, Ubuntu Core developer, snapd developer and support

Hidden only means "not in help"; a hidden subcommand still runs. Pick the class that matches who you are.

Something's wrong with the acme-agent on the gateway, and snap list isn't telling you why. That's when you reach for snap debug: twenty seven subcommands for looking inside snapd and the sandbox. Only snap help --all lists it, and its help warns that these commands can be removed without notice. Some are hidden from help, but a hidden subcommand still runs when you type it. This deck sorts the twenty seven by who reaches for them: app developers, Ubuntu Core developers, snapd developers and support.

02 One endpoint, two verbs

/V2/DEBUG snapd/v2/debugon snapd.socket GET ?aspect=read: open to every local user base-declaration connectivity model change-timings seeding gadget-disk-mapping disks raa features POST {"action": ...}act: root only, even to read ensure-state-soon can-manage-refreshes stacktraces migrate-home add-warning unshow-warnings prune create-recovery-system orange: has a subcommandgrey: only through snap debug api $ su -c "snap debug stacktraces 2>&1" - testerror: access deniedread-only, but POST, so root timings asks change-timings · refresh-app-awareness asks raa · confinement and sandbox-features use /v2/system-info
The /v2/debug endpoint in snapd: nine GET aspects open to every local user on the left, eight root-only POST actions on the right, and a normal user getting access denied from snap debug stacktraces

GET /v2/debug?aspect= is open to every local user; POST {"action": ...} is root only. The verb, not the effect, decides who may call.

Most subcommands that talk to snapd ask one endpoint, /v2/debug, and how they ask decides who may. A GET carries an aspect, and all nine aspects are open to every local user. A POST carries an action, and all eight actions need root. The split is by verb, not by effect. So stacktraces and can-manage-refreshes need root although they only read, and a spread test checks that a normal user gets access denied. Four actions have a subcommand; the other four you reach through snap debug api. Confinement and sandbox-features ask /v2/system-info instead.

03 Three ways to look inside

THREE SOURCES ask snapdsnap debug timingssnapd/v2/debugmost subcommands read the state filesnap debug statestate.json $ snap debug state# opens ./state.json# in the current directory /var/lib/snapd/state.jsonis mode 0600: read it as root read the kernel device-cgroupdevices cgroup or BPF device map lsmlsm_list_modules syscall mount-namespace/run/snapd/ns/<snap>.mnt without snapd: 10boot-vars set-boot-vars device-cgroup execution lsmmount-namespace paths snap-downloads-cache state validate-seed snapd stopped:the other two still answer
Three sources: snap debug timings asks snapd, snap debug state reads a state file, device-cgroup, lsm and mount-namespace read the kernel; ten subcommands work with snapd stopped

When snapd is wedged, state, device-cgroup, lsm and mount-namespace still answer. snap debug state defaults to ./state.json; pass /var/lib/snapd/state.json as root.

Now suppose snapd itself is stuck, so asking it gets you nowhere. Not every subcommand needs it. Most ask snapd, but snap debug state reads a state file directly. With no argument it opens state.json in the current directory, so on the gateway you pass the real path, as root, because only root may read that file. device-cgroup, lsm and mount namespace read the kernel and the run directory. Ten subcommands in all work with snapd stopped, which is exactly when you need them: if snapd is wedged, the state file and the kernel still answer.

04 For app developers: what the sandbox gives you for app developers

APP DEVELOPERS $ snap debug confinementstrict$ snap debug sandbox-features \--required apparmor:kernel:dbusprints nothing; fails if the feature is missing:a guard in scripts $ sudo snap debug device-cgroup acme-agentSecurity tag: snap.acme-agent.agentTYPE MAJOR:MINOR ACCESS DEVICEc 1:3 rwm /dev/nullone section per security tag,only once a device cgroup or map exists $ sudo snap debug mount-namespace \--shell acme-agent -- \ls /var/snap/acme-agent/commonminimal environment, not the snap's mount namespace/run/snapd/ns/acme-agent.mnt --discardthrows the namespace away--shell and --discard cannot be combined also in this class:migrate-home · base-declaration · get-base-declaration
Terminal output of snap debug confinement, sandbox-features --required, device-cgroup for acme-agent, and mount-namespace --shell entering the snap's preserved mount namespace

confinement and sandbox-features describe the host; device-cgroup and mount-namespace describe one snap at run time.

You're building the acme-agent. What does the sandbox around it actually look like? Confinement tells you the mode the system runs snaps in, not the mode of any one snap. sandbox-features --required prints nothing and fails when a feature is missing, which makes it a guard in test scripts. device-cgroup, as root, shows the device nodes each of the agent's apps may open, one section per security tag. Mount namespace --shell drops you into the namespace snapd keeps for the snap, so you see the files as the agent does, with a minimal environment rather than the snap's. Discard throws that namespace away.

05 For Ubuntu Core developers: first boot and boot state for Ubuntu Core developers

UBUNTU CORE DEVELOPERS image-preseeding 9.318sat image build timefirst bootseed-restart-timeseed-completion 3.873son the device $ snap debug seedingseeded: truepreseeded: trueimage-preseeding: 9.318sseed-completion: 3.873s not seeded yet: seed-error shows the earliest failed change $ sudo snap debug boot-vars \--root-dir /run/mnt/ubuntu-seedsnapd_recovery_mode=run...snapd_good_recovery_systems=...a fixed key list, not the whole env; hidden on classic timings --ensure=seed: seed 8msand the nested spans beneath it also in this class:set-boot-vars · model · disksgadget-disk-mapping · validate-seed
A preseeded boot timeline with image-preseeding at build time and seed-completion from seed-restart-time, the snap debug seeding output, boot-vars read from the seed partition, and timings for the seed ensure

seeding for first boot, timings --ensure=seed or --ensure=install-system for where the time went, boot-vars for try and recovery state.

You've just built the gateway's image, and first boot felt slow. Where did the time go? Snap debug seeding says whether seeding finished and how long it took. On a preseeded image, image-preseeding was measured when the image was built, and seed-completion is measured on the device from the seed-restart-time. On an unseeded device, a seed-error line shows the error of the earliest failed change. Timings --ensure=seed breaks the seed work into nested spans. And boot-vars prints a fixed set of bootloader variables, not the whole environment; --root-dir reads them from another directory, such as the mounted seed partition.

06 For snapd developers: changes, time and goroutines for snapd developers

SNAPD DEVELOPERS $ sudo snap debug state --change=9 /var/lib/snapd/state.jsonLanes ID Status Spawn Ready Kind Summary1 11 Done ... ... download-snap Download snap "acme-agent" (42) from channel "latest/stable"1 14 Doing ... ... mount-snap Mount snap "acme-agent" (42) --dot: graphviz · --check: dependency cycles $ snap debug timings --last=installID Status Doing Undoing Summary40 Doing 910ms - task bar ^ 1ms - foo ^ 1ms - bar41 Done 210ms - task baz^ rows: nested timing spans $ sudo snap debug stacktracesstacks of snapd's goroutines; takes the state lock firstif that lock is stuck: snap debug api on/v2/debug/pprof/goroutine?debug=2 ensure-state-soon: every manager's Ensure, nowreturns at once; does not wait for it also in this class:api · features · execution · lsm
Terminal output of snap debug state for one change, snap debug timings with nested spans, a note on snap debug stacktraces and the pprof goroutine endpoint, and a note on ensure-state-soon

Offline graph: state. Time: timings. Live goroutines: stacktraces, or /v2/debug/pprof/goroutine when the state lock is stuck. Run the managers now: ensure-state-soon.

The acme-agent's refresh from 41 to 42 hangs halfway, and you're fixing snapd. Snap debug state reads the change and task graph from a copy of state.json, without snapd. For one change it lists the tasks by lane and wait order, then their logs. Dot writes the graph for graphviz, and check looks for dependency cycles. Timings shows how long each task spent doing and undoing, with nested spans beneath. Stacktraces prints snapd's goroutines, as root, but takes the state lock first. If that lock is stuck, read the pprof goroutine profile with snap debug api. And ensure-state-soon asks every manager to run now.

07 For support: why it will not refresh for administrators

SUPPORT $ snap debug connectivityConnectivity status: * PASSor one line per host: * <host>: unreachablechecked from snapd, with snapd's proxy settings test-snapd-sh.shrunning, PID 10435refresh heldlatest/edge $ snap debug refresh-app-awarenessMonitored snaps:Name Security Tag PIDtest-snapd-sh snap.test-snapd-sh.sh 10435Refresh candidates:Name Version Rev Channel Monitoredtest-snapd-sh 1.0 7 latest/edge Yes app exits: the refresh goes ahead, the output is empty $ sudo snap debug can-manage-refreshesfalsemay a snap take over the refresh schedule? also:snap-downloads-cache · paths
snap debug connectivity reporting PASS; a running test-snapd-sh app holding its refresh, shown by snap debug refresh-app-awareness; the app exits and the list empties; snap debug can-manage-refreshes prints false

connectivity for reaching the store, refresh-app-awareness for refreshes waiting on a running app, can-manage-refreshes for whether refresh.timer=managed can be set.

The gateway hasn't refreshed the acme-agent in days. Where do you start? Connectivity asks snapd to reach the store hosts and prints pass, or each host it couldn't reach. It runs inside snapd, so it uses snapd's proxy settings, not your shell's. refresh-app-awareness lists snaps whose refresh waits because one of their apps is running, and the candidates waiting on them. Once the app exits, the refresh goes ahead and the list empties. And can-manage-refreshes, as root, answers whether a snap here may take over the refresh schedule. If it says true, look at refresh.timer next.

08 Endpoints without a subcommand: snap debug api for snapd developers for Ubuntu Core developers for administrators

SNAP DEBUG API $ echo '{"action":"create-recovery-system","params":{"recovery-system-label":"1234"}}' |sudo snap debug api -X POST -H 'Content-Type: application/json' /v2/debug no subcommand, only through api:add-warning · unshow-warningsprune · create-recovery-system/v2/debug/pprof/root only "change": "..."the response carries a change idsnapdchangereboot$ sudo snap watch <id> -X GET|POST|PUT -H 'Name: value'--fail --snap-socket --disable-auth--session-agent-uid=<uid>exit 0 on HTTP errors unless --fail used in about 60 spread tests
snap debug api posting a create-recovery-system action to /v2/debug, snapd starting a change that reboots and is followed with snap watch, the actions only reachable this way, and the command's flags

When no subcommand exists, use snap debug api, and add --fail in scripts so HTTP errors give a non-zero exit status.

Some endpoints have no subcommand at all. Snap debug api is a raw client for any snapd path, on the main socket, the snapd-snap.socket, or a user's session agent. POST and PUT read the request body from standard input. Four debug actions and the pprof endpoints can only be reached this way, so the tests use it, for example to create a recovery system on the gateway. The response carries a change id, which you follow with snap watch. One trap: without --fail, the exit status is zero even when the request returns an HTTP error, so scripts should always pass it.

09 Where to go from here

ROADS snap debug The overlordstate · timingsensure-state-soon · stacktraces Confinementsandbox-features · lsmdevice-cgroup · mount-namespace Boot and recoveryboot-vars Model, gadget and kernelmodel · seeding · validate-seedgadget-disk-mapping The storeconnectivitysnap-downloads-cache How interfaces are implementedbase-declaration
Roads from snap debug: the overlord, confinement, boot and recovery, model gadget and kernel, the store, and how interfaces are implemented, each with the subcommands that lead there

Roads: the overlord, confinement, boot and recovery, model, gadget and kernel, the store, and how interfaces are implemented.

Where next? The overlord explains the changes, tasks, timings and ensure loop that state, timings, ensure-state-soon and stacktraces look at. Confinement covers the sandbox that sandbox-features, device-cgroup, mount namespace and lsm report on. Boot and recovery explains the variables that boot-vars reads. Model, gadget and kernel covers what model, seeding, validate-seed and gadget-disk-mapping look at. The store covers the hosts that connectivity checks, and the download cache. And how interfaces are implemented explains the base declaration that base-declaration prints.

Read more

The catalogue below this deck lists every snap debug subcommand with its usage, class, source and gotchas. This page collects what cuts across them.

for snapd developers for Ubuntu Core developers for administrators

The /v2/debug endpoint

On snapd.socket. GET is open to every local user; POST needs root, whatever the action does.

VerbValueSubcommand
GET ?aspect=base-declarationbase-declaration, get-base-declaration
connectivityconnectivity
modelmodel
change-timings (with change-id, ensure, startup, all)timings
seedingseeding
gadget-disk-mappinggadget-disk-mapping
disksdisks
raarefresh-app-awareness
featuresfeatures
POST {"action": ...}ensure-state-soonensure-state-soon
can-manage-refreshescan-manage-refreshes
stacktracesstacktraces
migrate-home (with snaps)migrate-home
add-warning (with message)none
unshow-warningsnone
prunenone
create-recovery-system (with params.recovery-system-label)none; returns a change

Every POST takes the state lock before it looks at the action, stacktraces included.

/v2/debug/pprof/ (root only) serves Go's cmdline, profile, symbol, trace and the profiles heap, allocs, block, threadcreate, goroutine and mutex. It does not take the state lock.

confinement and sandbox-features read /v2/system-info instead.

Working without snapd

These ten read files, the kernel or the bootloader directly: boot-vars, set-boot-vars, device-cgroup, execution, lsm, mount-namespace, paths, snap-downloads-cache, state, validate-seed.

snap debug state opens ./state.json unless given a path. The live file is /var/lib/snapd/state.json, mode 0600; copy it before reading it while snapd runs.

Hidden and internal

Subcommands
Always hidden from helpbase-declaration, get-base-declaration, can-manage-refreshes, disks, ensure-state-soon, gadget-disk-mapping, model
Hidden on classicboot-vars, set-boot-vars; both refuse to run there
Short help starts with "(internal)"refresh-app-awareness, lsm; on Ubuntu Core also boot-vars, set-boot-vars

A hidden subcommand runs when typed. snap debug itself is listed only by snap help --all.

for snapd developers for Ubuntu Core developers for administrators

snap debug api

OptionEffect
-X GET, POST, PUTthe method; POST and PUT read the body from standard input
-H 'Name: value'an extra header, repeatable
--failexit with an error on an HTTP status of 400 or more; without it the exit status is 0
--snap-sockettalk to snapd-snap.socket instead of snapd.socket
--session-agent-uid=<uid>talk to that user's session agent; cannot be combined with --snap-socket
--disable-authdo not send the credentials from auth.json

for snapd developers for Ubuntu Core developers

Known limits

  • On a system that is not seeded, seeding reports as seed-error the error of the earliest failed change among all finished changes and all seed changes, so the error can come from a change that is not the seed change.

Further reading

  • Fix common issues: reading state.json with snap debug state when snapd misbehaves.
  • Snap performance: snap debug timings for changes, ensure activities and startup, with examples.
  • Debug snaps: the wider toolkit for debugging a snap, including the information script shipped with snapd.

Roads from here

GETTING SNAPS

  • The store — the store hosts that connectivity checks and the download cache

INTERFACES AND CONFINEMENT

RUNNING SNAPS

  • The overlord — changes and tasks and timings and the ensure loop that state and timings inspect

UBUNTU CORE · boot

UBUNTU CORE · install and recovery

DEVELOPING SNAPS · try, pack, run, debug

INSIDE SNAPD · the daemon itself