Running and debugging an app

snap run step by step for app developers for snapd developers

Every snap command on your path is a link to the snap program which runs the app through snap-confine and snap-exec. Before it hands over it checks the system key and waits for a refresh and creates the user's data directories and a tracking scope. The options --shell and --strace and --gdbserver and --trace-exec each change one link of that chain so that you can look inside the sandbox.

On the map: developing snaps district

▶ Watch this deck — 8 scenes, ~6 min

01 Three programs before yours

SNAP RUN $ photo-editorsnap rundecides and prepares execsnap-confinebuilds the sandboxexecsnap-execpicks the commandexecphoto-editoryour program --shella shell insteadof your program --stracetrace the whole chain from outside --gdbserverstop just before your program --trace-exectime every exec
A chain of four programs joined by exec: snap run, snap-confine, snap-exec and photo-editor. Four markers show where each debugging option changes the chain: --shell replaces the program, --strace watches the whole chain, --gdbserver stops just before the program, --trace-exec times every exec

snap run execs snap-confine, which execs snap-exec, which execs the app. --shell, --strace, --gdbserver and --trace-exec each alter one part of that chain.

You type photo-editor and press enter. Your program isn't the first thing that runs. It's the fourth. The snap command decides what to run and gets things ready. Then it replaces itself with snap-confine, which builds the sandbox. snap-confine replaces itself with snap-exec, which, already inside the sandbox, picks the command and starts your program. It's a relay, one runner at a time, and the baton is the process itself. Each debugging option of snap run changes one leg: a shell in place of your program, strace watching from outside, a stop just before your program, or a stopwatch on every handover.

02 What snap run does before it hands over for app developers for snapd developers for administrators

THE FIRST LINK /snap/bin/photo-editor -> /usr/bin/snapthe name it was called by picks the appsnap run system key matches?yes: go onno: ask snapd, up to 12 times, one second apart being refreshed?wait, checking every 500 ms; a notification is showncannot run "photo-editor", snap is being removedcannot run "photo-editor", snap is disabled data directories~/snap/photo-editor/42 · common · current tracking scopea systemd scope named after the security tagand a random UUID: snapd can tell the app is running exec snap-confinesnap run does this, not the daemon
Inside snap run, called through the link /snap/bin/photo-editor to /usr/bin/snap, a timeline of checks: the system key matches or snapd is asked up to 12 times one second apart; a refresh in progress makes the app wait, while a removed or disabled snap is refused; the data directories are created; a systemd scope is created; then snap run itself execs snap-confine

snap run checks the system key, waits while the snap is inhibited for refresh, creates ~/snap/<snap>/<rev>, common and current, creates a transient systemd scope, and then execs snap-confine. The daemon does not start the app.

Before anything is confined, snap run works through a checklist. If the photo editor hangs before its window appears, look here. The command on your path is a link to snap; the name it was called by picks the app. First: do the security profiles on disk match this system? That's the system key; if not, snap run asks snapd and waits for new profiles. Second: is a refresh under way? Then the app waits, with a notification. Next come your data directories for revision 42, and a systemd scope, so a refresh can tell the app is running. Only then does snap run, not the daemon, exec snap-confine.

03 The command line snap run builds for app developers for snapd developers

ONE COMMAND LINE snap-confine \one of snapd's helper programsif it is missing:missing snap-confine: try updating your core/snapd package --base core24 \base: in snap.yamlwhich root filesystem snap.photo-editor.photo-editor \the security tagpicks profiles and namespace /usr/lib/snapd/snap-exec \a path as seen from inside the sandbox photo-editor.photo-editor file.jpgthe app, then your arguments, untouched --classic \classic snaps only, with a host snap-exec path--command=shell \where the debugging options go--command=shell or --command=gdbserver
The command line snap run passes to snap-confine, one piece per line: snap-confine, --base core24 from snap.yaml, the security tag snap.photo-editor.photo-editor, the path /usr/lib/snapd/snap-exec as seen inside the sandbox, and the app with your arguments; --classic and --command=shell or --command=gdbserver are optional insertions

snap-confine [--classic] [--base <base>] snap.<snap>.<app> <snap-exec> [--command=…] <snap>.<app> <args>. The security tag selects the sandbox; --command tells snap-exec what to run instead of the app's command.

Snap run is about to replace itself with snap-confine. It gets one chance to pass on what it knows: the command line. The base comes from your snap.yaml and says which root filesystem to build. Then the security tag, snap.photo-editor.photo-editor, which picks every profile that confines the process. Next, the program snap-confine runs when it's done: snap-exec, written as the path looks from inside the sandbox. Last come the app and whatever you typed after it. A classic snap adds the --classic flag. And the debugging options you're about to see add one word: a command option for snap-exec.

04 A shell in the same sandbox for app developers for snapd developers for administrators

--SHELL $ snap run --shell photo-editor snap runsnap-confinesnap-exec/bin/bashthe base snap's bashphoto-editor ✓same AppArmor profile✓same seccomp filter✓same mount namespace✓same environment, after the command-chain $ env | grep ^SNAP_SNAP_NAME=photo-editorSNAP_REVISION=42 $ ls /etc/photo-editorpresets.conf themesa layout path: here, and not on the host $ cat /home/you/.shots/a.jpgcat: /home/you/.shots/a.jpg: Permission deniedwhat the app cannot do, the shell cannot dohidden folders in home are outside the home interface
snap run --shell photo-editor: the chain snap run, snap-confine, snap-exec ends in /bin/bash instead of photo-editor, with the same AppArmor profile, seccomp filter, mount namespace and environment. In the shell, env shows SNAP_NAME and SNAP_REVISION, a layout path under /etc exists, and reading a hidden folder in home is denied, as it would be for the app

snap run --shell <snap>.<app> runs /bin/bash under the app's security tag, after the app's command-chain and with its environment. The sandbox is the app's own, so a denial in the shell is a denial for the app.

The photo editor says it can't open a file that you can plainly see from your terminal. The fastest way to find out why is to stand where it stands. Snap run --shell adds one word to the command line, and snap-exec runs bash instead of your command. Nothing before that changed: the same security tag, so the same AppArmor profile, the same seccomp filter, the same namespace. Your command-chain ran first, so the environment is the one your app gets. Look around: the variables, a layout path. Then try what fails. If the shell is denied, so is the app, for the same reason.

05 strace, attached from outside for app developers for snapd developers

--STRACE $ snap run --strace photo-editor as yousnap-strace-shimraises SIGSTOP, waits root, through sudosudo strace -f -p 4321attaches SIGCONTsnap-confinesnap-execphoto-editorstill as you, still confined execve("/usr/lib/snapd/snap-confine", ...) = 0...dropped: everything beforethe first exec of a program under /snapexecve("/snap/photo-editor/42/bin/desktop-launch", ...) = 0openat(AT_FDCWD, "/home/you/.shots/a.jpg", O_RDONLY) = -1 EACCES (Permission denied) --strace=--rawkeep the snap-confine and snap-exec part--strace="-ttt"pass your own options to strace
snap run --strace photo-editor: as you, snap-strace-shim raises SIGSTOP and waits; as root through sudo, strace -f -p attaches to it; snap run sends SIGCONT and the chain continues through snap-confine and snap-exec to photo-editor, still as you. The output drops everything before the first exec of a program under /snap and shows a denied openat; --strace=--raw keeps it all

A shim raises SIGSTOP, sudo strace -f -p attaches, and snap run sends SIGCONT. Only strace runs as root. Output before the first exec of a program under /snap is filtered unless you pass --strace=--raw.

The shell showed you where it fails. To see the exact system call that's refused, you want strace. Strace needs root to follow snap-confine, a privileged program, but running your app as root would change the test. So snap run starts a tiny shim as you, and the shim stops itself with SIGSTOP. Then snap run starts strace through sudo, attached to the stopped process. Once strace is attached, snap run sends SIGCONT, and the chain runs as usual: as you, in the sandbox. Everything before your snap's first program is dropped from the output. Pass --raw to keep it.

06 gdbserver waits at the door for app developers for snapd developers

--GDBSERVER $ snap run --gdbserver photo-editor snap-execsnap-gdbserver-shiminside the sandbox: raises SIGSTOPphoto-editorwaits its turn Welcome to "snap run --gdbserver".You are right before your application is run.Please open a different terminal and run:gdb -ex="target remote :43210" -ex=continue -ex="signal SIGCONT"(gdb) continueor use your favorite gdb frontend and connect to :43210 sudo gdbserver --attach :43210 4321 $ gdb -ex="target remote :43210" \-ex=continue -ex="signal SIGCONT"(gdb) continue photo-editorSIGTRAP once, then it execs your commandthe debugger has control beforeyour program's first instruction
snap run --gdbserver photo-editor: snap-exec runs snap-gdbserver-shim inside the sandbox, which raises SIGSTOP before photo-editor starts; the first terminal prints the welcome text and the gdb command; sudo gdbserver --attach :43210 attaches to the shim; in a second terminal gdb connects and continues; the shim raises SIGTRAP and then execs the app

snap-exec runs snap-gdbserver-shim after the app's command-chain, and the shim stops; snap run prints the gdb command to connect, then runs sudo gdbserver --attach <addr> <pid>. The default address :0 means a free port is chosen. --gdb is no longer supported.

You want a debugger in place before the first line of your program runs. With a snap, before has to mean after the sandbox is in place. So this time the shim sits at the far end of the chain. snap-exec runs it in place of your command, and it stops. Snap run picks a free port, prints the gdb command to use, and starts gdbserver through sudo, attached to the stopped process. Run that in a second terminal, or point your IDE at the port. When you continue, the shim traps once, so the debugger has control, and then becomes your program. It runs as you, confined.

07 Where a slow start goes for app developers for snapd developers

--TRACE-EXEC $ snap run --trace-exec photo-editor desktop-launchphoto-editorsnap-confine · snap-exec · a wrapperopen until you close it Slowest 10 exec calls during snap run:0.842s /snap/photo-editor/42/bin/desktop-launch0.064s /snap/photo-editor/42/usr/bin/fc-cache0.031s /usr/lib/snapd/snap-confine0.009s /usr/lib/snapd/snap-exec... Total time: 13.351sthe whole run, until the app exitsthe time is in the snap's own launcher, not in snapd $ snap run --debug-log photo-editorsets SNAPD_DEBUG=1 for the whole chain and logs stages:snap to snap-confine · snap-exec to app
snap run --trace-exec photo-editor: a time axis with short segments for snap-confine, snap-exec and a wrapper, a long segment for the snap's own desktop-launch script and then photo-editor while it is open; the report lists the slowest exec calls with their times and the total time of the run, and the long one is the snap's launcher; snap run --debug-log sets SNAPD_DEBUG=1 and logs stage timestamps

--trace-exec prints Slowest N exec calls during snap run:, one line per program, and Total time:, which covers the whole run until the app exits. --debug-log sets SNAPD_DEBUG=1 and logs stage timestamps for snap, snap-confine and snap-exec.

The photo editor takes a second longer to start as a snap than from your build tree. Is that the sandbox, or something you ship? Snap run with the --trace-exec option runs the chain under strace, watching only exec calls. When the app exits, it lists the slowest ten, with a total. snap-confine and snap-exec are on that list, and so is every launcher and helper your snap runs. Often the long bar is one of yours. If the time is inside snap-confine instead, the --debug-log option turns on debug output for the whole chain, with a timestamp at each handover.

08 Where to go from here

ROADS snap runsnap-confinesnap-execphoto-editor The system keyRefreshAliases Confinement Apps and environmentLayouts Try and packdevmode: log, do not enforce Snap debuginside snapdOverview
Roads hanging off the chain snap run, snap-confine, snap-exec, photo-editor: the system key, refresh and aliases under snap run; confinement under snap-confine; apps and environment under snap-exec; layouts under photo-editor; and to the side try and pack for devmode, snap debug and the overview

Roads: the system key, refresh and aliases behind snap run; confinement behind snap-confine; apps and environment behind snap-exec; layouts; devmode in try and pack; and the snap debug tools.

You can now stand inside the sandbox, trace it, debug it and time it. Each link in that chain has more behind it. The system key and the refresh wait sit behind the first link, and so do aliases. Confinement is what snap-confine builds. Apps and environment is what snap-exec does with your snap.yaml, and layouts explain the paths you found from the shell. If the profile itself is what stops you, try and pack shows how to install in devmode, so denials are logged instead of enforced. And snap debug is the toolbox for looking inside snapd, not inside your app.

Read more

Options of snap run

OptionWhat it does
--shellruns a shell instead of the command
--strace[=<opts>]runs the command under strace; --raw in the value also traces the helpers
--gdbserver[=<addr>]runs the command with gdbserver; default address :0
--trace-execdisplays exec call timing data
--debug-logenables debug logging during the early startup phases
--gdbrefused: --gdb is no longer supported: use --gdbserver option instead

for snapd developers

Hidden options

OptionUse
--command=<name>stop, reload, post-stop or complete instead of the app's command
--hook=<name>, -r <rev>run a hook, optionally of a given revision
--timer=<schedule>run only if now is inside the schedule; otherwise print a message and exit 0
--experimental-gdbserverthe old name of --gdbserver

Argument errors: need the application to run as argument 1, you can only use one of --hook, --command, and --timer, -r can only be used with --hook.

What snap run does, in order

1. Checks the system key. On a mismatch it asks snapd up to 12 times, one second apart, and may wait for a change that regenerates the profiles; the final failure is timeout waiting for snap system profiles to get updated. See the system key. 2. Waits while the snap is inhibited. For a refresh an app waits, polling every 500 ms, and a notification is shown; a service continues with the previous revision. cannot run "<snap>", snap is being removed and cannot run "<snap>", snap is disabled are the other outcomes. 3. If the current link vanished in between, retries once: race condition detected, snap-run can only retry once. 4. Creates ~/snap/<snap>/<rev>, ~/snap/<snap>/common and the current link; a failure is only a warning. 5. Copies the X authority file, exposes Kerberos tickets for non-classic snaps and starts the document portal; each failure is a warning. 6. Builds the snap-confine command line and the environment, and creates the snap's XDG_RUNTIME_DIR with mode 0700. 7. Creates a transient systemd scope for apps and hooks. Services already have a unit. 8. Releases the lock on the inhibition hint, or re-checks for a refresh that started meanwhile, and execs snap-confine.

If the scope cannot be created: for bases core to core24 the run continues and the debug log says snapd cannot track the started application and snap refreshes will not be postponed by this process; for newer bases the run fails unless the snap's device cgroup is self-managed.

The command line

snap-confine [--classic] [--base <base>] snap.<snap>.<app> <snap-exec> [--command=...] [--hook=...] <snap>.<app> <args...>

PieceSource
--classicthe snap is installed with classic confinement
--basebase in snap.yaml; for a kernel snap, the base of the model
the security tagsnap.<snap>.<app>, or the hook's tag
snap-exec/usr/lib/snapd/snap-exec as seen inside the sandbox; a host path for classic snaps
--command=shell, --command=gdbserverfrom --shell and --gdbserver

base: none cannot run anything: cannot run hooks / applications with base "none".

What snap-exec runs

--commandRuns
nonethe app's command
shell/bin/bash
gdbserversnap-gdbserver-shim followed by the app's command
stop, reload, post-stopstop-command, reload-command, post-stop-command
complete/bin/bash with the completion helper, when the app has a completer

In every row the app's command-chain entries run first, so with gdbserver the shim stops after the chain and immediately before the app's command. How the final command line and environment are built is in apps and environment.

strace details

FactValue
strace used/snap/strace-static/current/bin/strace if present, else the one in PATH
no stracecannot find an installed strace, please try 'snap install strace-static'
no sudocannot use strace without sudo: ...
always excludedselect, pselect6, _newselect, clock_gettime, sigaltstack, gettid, gettimeofday, nanosleep (select is not listed on arm, arm64 and riscv64)
filter off--strace=--raw, or passing -o or --output

--trace-exec attaches the same way, with -ttt -e trace=execve,execveat, and prints the report when the traced process exits. Total time is the time from the first to the last line of the trace, so it includes the time the app was open.

--gdbserver needs gdbserver in PATH (please install gdbserver on your system) and sudo. The address :0 is replaced with a free TCP port found by snap run before gdbserver starts.

Stage names in the debug log

start, snap to snap-confine, snap-exec to app, and from snap-confine snap-confine enter, snap-confine mount namespace start and snap-confine mount namespace finish.

for snapd developers

In the code

WhatWhere
dispatch from a /snap/bin linkcmd/snapd/cli/main.go:437-445, 503-519
snap run optionscmd/snapd/cli/cmd_run.go:79-143
system key checkcmd/snapd/cli/cmd_run.go:178-291; SNAPD_DEBUG_SYSTEM_KEY_RETRY overrides the retry count
waiting while inhibitedcmd/snapd/cli/inhibit.go:62-130
runSnapConfinecmd/snapd/cli/cmd_run.go:1562-1899
strace and gdbserver sequencescmd/snapd/cli/cmd_run.go:1259-1380, 1062-1103
snap-execcmd/snapctl/tool/snap-exec/main.go:95-268

For classic snaps, variables that the dynamic loader strips for setuid programs are renamed with the prefix SNAP_SAVED_ before snap-confine and renamed back by snap-exec.

Further reading

  • Debug snaps: snap run --shell, --strace, devmode and reading AppArmor and seccomp denials. Its file names for the compiled seccomp profile differ from what snapd writes.
  • Debugging startup performance: snap run --trace-exec output and how to read it.
  • Refresh awareness: why snap run may wait while a refresh is in progress.
  • Snap confinement: strict, classic and devmode from the user's side.

No published page describes snap run --gdbserver; this deck and the text the command prints are the description.

Roads from here

START HERE

  • snapd — back to the overview

INTERFACES AND CONFINEMENT

  • Confinement — what snap-confine sets up before your program runs

RUNNING SNAPS

  • Refresh — why an app waits while its snap is being refreshed

TOOLS

ADMINISTRATION · managing a system of snaps

  • Aliases — how an alias on your path resolves to an app

DEVELOPING SNAPS · try, pack, run, debug

SANDBOX INTERNALS · the mount namespace

  • The system key — the check snap run makes before it trusts the profiles on disk