Running and debugging an app
snap run step by step for app developers for snapd developers
Every snap command on your path is a link to the snap program which runs the app through snap-confine and snap-exec. Before it hands over it checks the system key and waits for a refresh and creates the user's data directories and a tracking scope. The options --shell and --strace and --gdbserver and --trace-exec each change one link of that chain so that you can look inside the sandbox.
On the map: developing snaps district
01 Three programs before yours
snap run execs snap-confine, which execs snap-exec, which execs the app. --shell, --strace, --gdbserver and --trace-exec each alter one part of that chain.
You type photo-editor and press enter. Your program isn't the first thing that runs. It's the fourth. The snap command decides what to run and gets things ready. Then it replaces itself with snap-confine, which builds the sandbox. snap-confine replaces itself with snap-exec, which, already inside the sandbox, picks the command and starts your program. It's a relay, one runner at a time, and the baton is the process itself. Each debugging option of snap run changes one leg: a shell in place of your program, strace watching from outside, a stop just before your program, or a stopwatch on every handover.
02 What snap run does before it hands over for app developers for snapd developers for administrators
snap run checks the system key, waits while the snap is inhibited for refresh, creates ~/snap/<snap>/<rev>, common and current, creates a transient systemd scope, and then execs snap-confine. The daemon does not start the app.
Before anything is confined, snap run works through a checklist. If the photo editor hangs before its window appears, look here. The command on your path is a link to snap; the name it was called by picks the app. First: do the security profiles on disk match this system? That's the system key; if not, snap run asks snapd and waits for new profiles. Second: is a refresh under way? Then the app waits, with a notification. Next come your data directories for revision 42, and a systemd scope, so a refresh can tell the app is running. Only then does snap run, not the daemon, exec snap-confine.
03 The command line snap run builds for app developers for snapd developers
snap-confine [--classic] [--base <base>] snap.<snap>.<app> <snap-exec> [--command=…] <snap>.<app> <args>. The security tag selects the sandbox; --command tells snap-exec what to run instead of the app's command.
Snap run is about to replace itself with snap-confine. It gets one chance to pass on what it knows: the command line. The base comes from your snap.yaml and says which root filesystem to build. Then the security tag, snap.photo-editor.photo-editor, which picks every profile that confines the process. Next, the program snap-confine runs when it's done: snap-exec, written as the path looks from inside the sandbox. Last come the app and whatever you typed after it. A classic snap adds the --classic flag. And the debugging options you're about to see add one word: a command option for snap-exec.
04 A shell in the same sandbox for app developers for snapd developers for administrators
snap run --shell <snap>.<app> runs /bin/bash under the app's security tag, after the app's command-chain and with its environment. The sandbox is the app's own, so a denial in the shell is a denial for the app.
The photo editor says it can't open a file that you can plainly see from your terminal. The fastest way to find out why is to stand where it stands. Snap run --shell adds one word to the command line, and snap-exec runs bash instead of your command. Nothing before that changed: the same security tag, so the same AppArmor profile, the same seccomp filter, the same namespace. Your command-chain ran first, so the environment is the one your app gets. Look around: the variables, a layout path. Then try what fails. If the shell is denied, so is the app, for the same reason.
05 strace, attached from outside for app developers for snapd developers
A shim raises SIGSTOP, sudo strace -f -p attaches, and snap run sends SIGCONT. Only strace runs as root. Output before the first exec of a program under /snap is filtered unless you pass --strace=--raw.
The shell showed you where it fails. To see the exact system call that's refused, you want strace. Strace needs root to follow snap-confine, a privileged program, but running your app as root would change the test. So snap run starts a tiny shim as you, and the shim stops itself with SIGSTOP. Then snap run starts strace through sudo, attached to the stopped process. Once strace is attached, snap run sends SIGCONT, and the chain runs as usual: as you, in the sandbox. Everything before your snap's first program is dropped from the output. Pass --raw to keep it.
06 gdbserver waits at the door for app developers for snapd developers
snap-exec runs snap-gdbserver-shim after the app's command-chain, and the shim stops; snap run prints the gdb command to connect, then runs sudo gdbserver --attach <addr> <pid>. The default address :0 means a free port is chosen. --gdb is no longer supported.
You want a debugger in place before the first line of your program runs. With a snap, before has to mean after the sandbox is in place. So this time the shim sits at the far end of the chain. snap-exec runs it in place of your command, and it stops. Snap run picks a free port, prints the gdb command to use, and starts gdbserver through sudo, attached to the stopped process. Run that in a second terminal, or point your IDE at the port. When you continue, the shim traps once, so the debugger has control, and then becomes your program. It runs as you, confined.
07 Where a slow start goes for app developers for snapd developers
--trace-exec prints Slowest N exec calls during snap run:, one line per program, and Total time:, which covers the whole run until the app exits. --debug-log sets SNAPD_DEBUG=1 and logs stage timestamps for snap, snap-confine and snap-exec.
The photo editor takes a second longer to start as a snap than from your build tree. Is that the sandbox, or something you ship? Snap run with the --trace-exec option runs the chain under strace, watching only exec calls. When the app exits, it lists the slowest ten, with a total. snap-confine and snap-exec are on that list, and so is every launcher and helper your snap runs. Often the long bar is one of yours. If the time is inside snap-confine instead, the --debug-log option turns on debug output for the whole chain, with a timestamp at each handover.
08 Where to go from here
Roads: the system key, refresh and aliases behind snap run; confinement behind snap-confine; apps and environment behind snap-exec; layouts; devmode in try and pack; and the snap debug tools.
You can now stand inside the sandbox, trace it, debug it and time it. Each link in that chain has more behind it. The system key and the refresh wait sit behind the first link, and so do aliases. Confinement is what snap-confine builds. Apps and environment is what snap-exec does with your snap.yaml, and layouts explain the paths you found from the shell. If the profile itself is what stops you, try and pack shows how to install in devmode, so denials are logged instead of enforced. And snap debug is the toolbox for looking inside snapd, not inside your app.
Read more
Options of snap run
| Option | What it does |
|---|---|
--shell | runs a shell instead of the command |
--strace[=<opts>] | runs the command under strace; --raw in the value also traces the helpers |
--gdbserver[=<addr>] | runs the command with gdbserver; default address :0 |
--trace-exec | displays exec call timing data |
--debug-log | enables debug logging during the early startup phases |
--gdb | refused: --gdb is no longer supported: use --gdbserver option instead |
for snapd developers
Hidden options
| Option | Use |
|---|---|
--command=<name> | stop, reload, post-stop or complete instead of the app's command |
--hook=<name>, -r <rev> | run a hook, optionally of a given revision |
--timer=<schedule> | run only if now is inside the schedule; otherwise print a message and exit 0 |
--experimental-gdbserver | the old name of --gdbserver |
Argument errors: need the application to run as argument 1, you can only use one of --hook, --command, and --timer, -r can only be used with --hook.
What snap run does, in order
1. Checks the system key. On a mismatch it asks snapd up to 12 times, one second apart, and may wait for a change that regenerates the profiles; the final failure is timeout waiting for snap system profiles to get updated. See the system key. 2. Waits while the snap is inhibited. For a refresh an app waits, polling every 500 ms, and a notification is shown; a service continues with the previous revision. cannot run "<snap>", snap is being removed and cannot run "<snap>", snap is disabled are the other outcomes. 3. If the current link vanished in between, retries once: race condition detected, snap-run can only retry once. 4. Creates ~/snap/<snap>/<rev>, ~/snap/<snap>/common and the current link; a failure is only a warning. 5. Copies the X authority file, exposes Kerberos tickets for non-classic snaps and starts the document portal; each failure is a warning. 6. Builds the snap-confine command line and the environment, and creates the snap's XDG_RUNTIME_DIR with mode 0700. 7. Creates a transient systemd scope for apps and hooks. Services already have a unit. 8. Releases the lock on the inhibition hint, or re-checks for a refresh that started meanwhile, and execs snap-confine.
If the scope cannot be created: for bases core to core24 the run continues and the debug log says snapd cannot track the started application and snap refreshes will not be postponed by this process; for newer bases the run fails unless the snap's device cgroup is self-managed.
The command line
snap-confine [--classic] [--base <base>] snap.<snap>.<app> <snap-exec> [--command=...] [--hook=...] <snap>.<app> <args...>
| Piece | Source |
|---|---|
--classic | the snap is installed with classic confinement |
--base | base in snap.yaml; for a kernel snap, the base of the model |
| the security tag | snap.<snap>.<app>, or the hook's tag |
| snap-exec | /usr/lib/snapd/snap-exec as seen inside the sandbox; a host path for classic snaps |
--command=shell, --command=gdbserver | from --shell and --gdbserver |
base: none cannot run anything: cannot run hooks / applications with base "none".
What snap-exec runs
--command | Runs |
|---|---|
| none | the app's command |
shell | /bin/bash |
gdbserver | snap-gdbserver-shim followed by the app's command |
stop, reload, post-stop | stop-command, reload-command, post-stop-command |
complete | /bin/bash with the completion helper, when the app has a completer |
In every row the app's command-chain entries run first, so with gdbserver the shim stops after the chain and immediately before the app's command. How the final command line and environment are built is in apps and environment.
strace details
| Fact | Value |
|---|---|
| strace used | /snap/strace-static/current/bin/strace if present, else the one in PATH |
| no strace | cannot find an installed strace, please try 'snap install strace-static' |
| no sudo | cannot use strace without sudo: ... |
| always excluded | select, pselect6, _newselect, clock_gettime, sigaltstack, gettid, gettimeofday, nanosleep (select is not listed on arm, arm64 and riscv64) |
| filter off | --strace=--raw, or passing -o or --output |
--trace-exec attaches the same way, with -ttt -e trace=execve,execveat, and prints the report when the traced process exits. Total time is the time from the first to the last line of the trace, so it includes the time the app was open.
--gdbserver needs gdbserver in PATH (please install gdbserver on your system) and sudo. The address :0 is replaced with a free TCP port found by snap run before gdbserver starts.
Stage names in the debug log
start, snap to snap-confine, snap-exec to app, and from snap-confine snap-confine enter, snap-confine mount namespace start and snap-confine mount namespace finish.
for snapd developers
In the code
| What | Where |
|---|---|
dispatch from a /snap/bin link | cmd/snapd/cli/main.go:437-445, 503-519 |
snap run options | cmd/snapd/cli/cmd_run.go:79-143 |
| system key check | cmd/snapd/cli/cmd_run.go:178-291; SNAPD_DEBUG_SYSTEM_KEY_RETRY overrides the retry count |
| waiting while inhibited | cmd/snapd/cli/inhibit.go:62-130 |
runSnapConfine | cmd/snapd/cli/cmd_run.go:1562-1899 |
| strace and gdbserver sequences | cmd/snapd/cli/cmd_run.go:1259-1380, 1062-1103 |
| snap-exec | cmd/snapctl/tool/snap-exec/main.go:95-268 |
For classic snaps, variables that the dynamic loader strips for setuid programs are renamed with the prefix SNAP_SAVED_ before snap-confine and renamed back by snap-exec.
Further reading
- Debug snaps:
snap run --shell,--strace, devmode and reading AppArmor and seccomp denials. Its file names for the compiled seccomp profile differ from what snapd writes. - Debugging startup performance:
snap run --trace-execoutput and how to read it. - Refresh awareness: why
snap runmay wait while a refresh is in progress. - Snap confinement: strict, classic and devmode from the user's side.
No published page describes snap run --gdbserver; this deck and the text the command prints are the description.
Roads from here
START HERE
- snapd — back to the overview
INTERFACES AND CONFINEMENT
- Confinement — what snap-confine sets up before your program runs
RUNNING SNAPS
- Refresh — why an app waits while its snap is being refreshed
TOOLS
- snap debug: tools for looking inside — the snap debug tools for looking inside snapd itself
ADMINISTRATION · managing a system of snaps
- Aliases — how an alias on your path resolves to an app
DEVELOPING SNAPS · try, pack, run, debug
- Apps, command lines and environment — how snap-exec builds the command line and the environment
- Trying and packing a snap — install with --devmode to log denials instead of enforcing them
- Layouts — paths you will find from a shell inside the snap
SANDBOX INTERNALS · the mount namespace
- The system key — the check snap run makes before it trusts the profiles on disk