Hotplug slots
sandbox internals for snapd developers for Ubuntu Core developers
With the experimental hotplug feature on snapd watches udev and adds a slot to the system snap when a matching device appears and removes it when the device goes. A hotplug key computed from the device's attributes lets the slot and its connections come back when the device is plugged in again. Only serial-port takes part and a gadget slot for the same device wins.
On the map: sandbox internals district
01 A slot that arrives with the device
snap set system experimental.hotplug=true turns hotplug slots on. A slot appears on the system snap when the device does and goes when it goes; a connection made once is restored when the device returns. Only serial-port implements it.
Someone in the field plugs a USB serial adapter into the gateway. acme-agent needs a slot to connect its plug to, and nobody wrote one for this adapter. Where does it come from? Normally a slot comes from a snap or from the gadget. Switch on the experimental hotplug feature, and snapd makes one itself when the device shows up, named after the device. You connect the agent to it once. Pull the adapter out and the slot goes, but snapd remembers the connection. Plug it back in and both come back. The feature is off by default, and today only the serial-port interface uses it.
02 "The monitor: enumerate, then listen" for Ubuntu Core developers for snapd developers
The monitor starts once a system snap is present and retries every 5 minutes on failure. It enumerates with udevadm info -e, then handles add and remove uevents for subsystems net, tty and usb. After enumeration, remembered slots whose device was not seen get a remove change.
Snapd has to hear about the adapter, whether it went in a second ago or before the gateway was switched on. The interface manager starts a udev monitor once a system snap is installed, retrying every five minutes on failure. It works in two phases. First it reads every existing device from udevadm info and treats each one as just added. Then it listens on a netlink socket for add and remove events from the net, tty and usb subsystems, and drops repeats. When the first phase ends, a remembered hotplug slot whose device didn't turn up is removed, as if unplugged while the gateway was off.
03 Each hotplug interface is asked for Ubuntu Core developers for snapd developers
HotplugDeviceDetected returns a proposed slot or nothing. serial-port proposes one for subsystem tty, ID_BUS=usb and a serial device node, with path, usb-vendor and usb-product. The feature flag is checked after the device has been evaluated.
Snapd itself has no idea what a serial device is. So it asks the interfaces. Each interface that supports hotplug is shown the device and either stays quiet or proposes a slot. The serial-port interface wants three things: the tty subsystem, the USB bus, and a device node that looks like a serial one, such as /dev/ttyUSB0. Our adapter passes, so it proposes a slot with the path, the usb-vendor and the usb-product. A network card gets no answer and is ignored. Only now is the feature flag checked. With it off, the log tells you to enable experimental.hotplug.
04 The hotplug key for Ubuntu Core developers for snapd developers
The default hotplug key is a version digit 0 plus the SHA-256 of the first non-empty attribute from each of four groups (name, vendor, model, identifier). With fewer than two groups the device is ignored. The key, not the device path, is stored.
Why not remember /dev/ttyUSB0? Because the numbers follow arrival order. Plug a second adapter in first, and tomorrow that name belongs to a different device. So snapd builds a key from what the device says about itself. There are four groups of udev attributes: name, vendor, model and identifier. From each group, the first attribute with a value goes into a SHA-256 hash. The key is a version digit, zero, then the digest. A device that fills fewer than two groups is ignored. The key is what snapd stores. The device path stays in memory, only to match a later remove event.
05 Plugging in is a change for Ubuntu Core developers for snapd developers for administrators
The change has tasks hotplug-seq-wait, hotplug-add-slot and hotplug-connect. The slot name is the proposed name, else the shortest of NAME, ID_MODEL_FROM_DATABASE and ID_MODEL made into a valid slot name, else the interface name. serial-port has deny-auto-connection: true, so the first connection is made by hand.
The adapter passed the checks and has a key. From here it's machinery you already know: a change, with tasks. The first, hotplug-seq-wait, only waits, so an add and a remove for one device run in order. Then hotplug-add-slot puts the slot on the system snap. Its name is the interface's proposal, or else the shortest cleaned-up form of the device's name or model, with a number added if it's taken. Last, hotplug-connect brings back connections this device had before, plus any that policy auto-connects. For serial-port, the base declaration says deny-auto-connection, so the first one is yours to make.
06 Unplug, remember, plug back in for app developers for Ubuntu Core developers for snapd developers for administrators
hotplug-disconnect marks connections hotplug-gone; hotplug-remove-slot keeps the slot in hotplug-slots, marked gone, while a connection refers to its key. On return the slot is re-added under its old name and connections that are gone and not undesired are re-created.
Pull the adapter out. The slot has to go, there's no device behind it. But should snapd forget that you connected it? A remove change runs: wait your turn, hotplug-disconnect, then hotplug-remove-slot. The agent loses access to the device, as with any disconnect. But the connection isn't deleted; it's marked hotplug-gone in state, and because a connection still refers to the key, the slot is remembered too. With nothing connected, it'd be forgotten. Plug the adapter back in, into any port, and the key matches. The slot returns under its old name, and the connection comes back. One you removed yourself with snap disconnect doesn't.
07 A gadget slot for the same device wins for Ubuntu Core developers for snapd developers
HandledByGadget lets a gadget slot claim a device: for serial-port by usb-vendor, usb-product and the optional usb-interface-number, otherwise by path. A claimed device gets no hotplug slot.
On a board you designed, you've probably declared the serial device in the gadget already. So what happens when udev announces that very adapter? Two slots for one device would be one too many. So before anything is proposed, the interface is asked whether one of the gadget's slots already handles this device. For serial-port, a gadget slot with a usb-vendor must match the device's vendor and product, and its interface number if it gives one. A slot without a vendor is compared by its path instead. A match, and no hotplug slot is made. Any other adapter still gets one.
08 Where to go from here
Roads: devices and storage, interface hooks, how interfaces are implemented, the overlord, confinement and the per-snap mount namespace.
That's a slot with a lifetime: it exists exactly as long as the device is plugged in. A device appears, an interface claims it, a key names it, a change adds the slot, and state remembers the connection. serial-port and its neighbours are in the devices and storage family. The connections run the same interface hooks as any other. The repository and the auto-connection policy are in how interfaces are implemented. Changes and their tasks are in the overlord. Confinement shows the profiles a connection ends up in, and the mount namespace leads to the rest of this district.
Read more
Turning it on
Hotplug slots are an experimental feature, off by default:
sudo snap set system experimental.hotplug=true
With the flag off, udev events are still received and evaluated. An add that would have made a slot ends in the log line hotplug device add event ignored, enable experimental.hotplug; a remove for a device that has a slot ends in hotplug device remove event ignored, enable experimental.hotplug.
Only one interface implements hotplug in this tree: serial-port. A hotplug slot belongs to the system snap and is connected with the usual snap connect. serial-port's base declaration has deny-auto-connection: true, so the first connection is made by hand.
The examples in the deck use invented udev values for a USB serial adapter (ID_VENDOR_ID=0403, ID_MODEL_ID=6001, ID_MODEL=FT232R_USB_UART, ID_SERIAL_SHORT=A1B2C3, /dev/ttyUSB0) and assume acme-agent has a plug named serial-port of interface serial-port.
for snapd developers
What an interface implements
| Interface (Go) | Method | Required | Purpose |
|---|---|---|---|
hotplug.Definer | HotplugDeviceDetected(di) | yes | return a *ProposedSlot, or nil when the device is not this interface's kind |
hotplug.HotplugKeyHandler | HotplugKey(di) | no | a custom key; an empty key falls back to the default |
hotplug.HandledByGadgetPredicate | HandledByGadget(di, slot) | no | true when a gadget slot already covers the device |
ProposedSlot has Name, Label and Attrs. Clean() validates the name when one is given and normalises the attributes. An empty label becomes the interface summary.
The default hotplug key
From each group, the first attribute with a non-empty value is written into a SHA-256 as name\0value\0.
| Group | Attributes, in order |
|---|---|
| Name | ID_V4L_PRODUCT, NAME, ID_NET_NAME, PCI_SLOT_NAME |
| Vendor | ID_VENDOR_ID, ID_VENDOR, ID_WWN, ID_WWN_WITH_EXTENSION, ID_VENDOR_FROM_DATABASE, ID_VENDOR_ENC, ID_OUI_FROM_DATABASE |
| Model | ID_MODEL_ID, ID_MODEL_ENC |
| Identifier | ID_SERIAL, ID_SERIAL_SHORT, ID_NET_NAME_MAC, ID_REVISION |
The key is one hex digit of version (0 today) followed by the 64 hex digits of the digest: 65 characters. The code notes that "any future changes to these definitions require a new key version". When fewer than two groups give a value there is no key, and the device is skipped with no valid hotplug key provided by interface %q, device %s ignored.
Summaries show a key shortened to its first 12 characters followed by ….
The key identifies the device across unplug, replug and reboot. The device path is kept only in memory, to map a later remove event back to its keys.
Changes and tasks
| Event | Change kind | Tasks |
|---|---|---|
| device added | hotplug-add-slot-serial-port | hotplug-seq-wait, hotplug-add-slot, hotplug-connect |
| device removed | hotplug-remove-serial-port | hotplug-seq-wait, hotplug-disconnect, hotplug-remove-slot |
| present slot, attributes changed | (injected into the add change) | hotplug-disconnect, hotplug-update-slot |
hotplug-seq-waitretries every 300 ms while an unfinished hotplug change for the same key with a lower sequence number exists, loggingWaiting processing of earlier hotplug event change %q affecting device with hotplug key %q. Sequence numbers come from the state keyhotplug-seq.hotplug-add-slotre-adds a remembered slot under its old name, injectshotplug-disconnectandhotplug-update-slotwhen a present slot's attributes changed, or adds a new slot.hotplug-connectinjectsconnecttasks for connections of the key that are markedhotplug-goneand notundesired, and for new auto-connections that policy allows.hotplug-disconnectinjects disconnect tasks flaggedby-hotplug; on a conflicting change it logsWaiting for conflicting change in progress: %sand retries.
Task summaries:
Create slot for device %s with hotplug key %qRecreate connections of interface %q for device %s with hotplug key %qDisable connections of interface %q, hotplug key %qRemove slot for interface %q, hotplug key %qUpdate slot of interface %q, hotplug key %q
The remove change's summary is Remove hotplug connections and slots of device %s with interface %q. The add change's summary, Add hotplug slot of interface %q for device %s with hotplug key %q, currently prints the device where the interface belongs and the interface where the device belongs (a confirmed cosmetic snapd bug, overlord/ifacestate/hotplug.go:240); read the task summaries instead. The change kind is formatted from the interface value; it is hotplug-add-slot-serial-port only because serial-port's String() returns its name, so do not read it as a pattern for other interfaces.
A device in a summary is printed as its device node, then the model or vendor (cut to 16 characters) and the short serial, for example /dev/ttyUSB0 (FT232R_USB_UART; serial: A1B2C3).
for snapd developers for administrators
State
hotplug-slots: one entry per slot name withname,interface,static-attrs,hotplug-keyandhotplug-gone.hotplug-seq: the last sequence number handed out.- Connections of a hotplug slot carry the key and are marked
hotplug-gonewhile the device is away, instead of being deleted. - A slot whose device is gone stays in
hotplug-slotswithhotplug-gone: truewhile any connection refers to its key; otherwise it is deleted. - A connection removed with
snap disconnectis deleted (or, if it was auto-connected, markedundesired), so it is not re-created when the device returns.
Look with snap debug state (see snap debug). The spread test reads .data["hotplug-slots"] from state.json.
Slot names
1. The name the interface proposed, if any. serial-port proposes none. 2. Otherwise the shortest of NAME, ID_MODEL_FROM_DATABASE and ID_MODEL made into a valid slot name: letters lower-cased, digits kept after the first letter, single dashes kept, everything else dropped, at most 20 characters, no trailing dash. 3. Otherwise the interface name.
A taken name gets -1, -2 and so on. A remembered slot keeps its name, since a name already in hotplug-slots counts as free for the same key. The spread test's QEMU adapter, ID_MODEL=QEMU_USB_SERIAL, gets qemuusbserial.
The monitor
The interface manager starts the udev monitor from its Ensure, not in preseed mode, and only once a system snap is present. A failure is retried every 5 minutes; opening the netlink socket fails with cannot start udev monitor: %s.
1. Enumerate: parse udevadm info -e and report each device as added, then signal the end of enumeration. This code does not filter the enumeration by subsystem. 2. Listen: netlink uevents with a SUBSYSTEM filter for net, tty and usb; only add and remove are handled.
A table of device paths drops a second add for the same path and a remove for an unknown one. A device without a device path is ignored with missing device path attribute. With no system snap an event is dropped with system snap not available, hotplug events ignored.
When enumeration ends, every slot in hotplug-slots whose key was not seen gets a remove change, Remove hotplug connections and slots of interface %q, as if it had been unplugged.
serial-port and the gadget
HotplugDeviceDetected proposes a slot only for subsystem tty, ID_BUS=usb and a device node matching ^/dev/tty(mxc|USB|ACM|AMA|XRUSB|S|O|SC|MSM|HS|GS|LP|RPMSG|MAX)[0-9]+$. The slot gets path (the device node), usb-vendor from ID_VENDOR_ID and usb-product from ID_MODEL_ID.
Before asking for a proposal, snapd checks the gadget's slots of the same interface with HandledByGadget:
- a gadget slot with
usb-vendormust matchID_VENDOR_ID, itsusb-productmust matchID_MODEL_ID, and when it setsusb-interface-numberthat must matchID_USB_INTERFACE_NUM; the device's values are parsed as hex; - a gadget slot without
usb-vendormatches when itspathequals the device node.
A gadget serial-port slot that has usb-vendor must have a path of the form /dev/serial-port-<name> (lower-case letters and digits): it is the symlink udev creates. A claimed device gets no hotplug slot, logged at debug level as ignoring device %s, interface %q (handled by gadget slot %s).
Not determined
- Which hooks run on the connect and disconnect tasks that
hotplug-connectandhotplug-disconnectinject (see interface hooks). - How
snap connectionsshows a hotplug slot. - Whether the enumeration by
udevadm info -eis filtered to the three subsystems elsewhere.
Further reading
- Hotplug support: enabling the feature, slot naming and remembered connections from the user's side. Its minimum snapd version disagrees with the experimental features page; the tree records neither.
- Developing hotplug interfaces: the Go methods an interface implements.
- serial-port interface: the one interface with hotplug support, and its slot attributes.
- Hardware interfaces: gadget-defined slots, the alternative on Ubuntu Core.
Roads from here
INTERFACES AND CONFINEMENT
- How interfaces are implemented — the repository that holds the slots and the policy for connecting
- Confinement — the profiles a connection ends up in
RUNNING SNAPS
- The overlord — changes and tasks and why they wait for each other
INTERFACE FAMILIES · every interface, grouped by what it touches
- "Interfaces: devices and storage" — serial-port and the other device interfaces
HOOKS · code snapd runs inside a snap
- Hooks: interface hooks — the hooks that run when a connection is made or removed
SANDBOX INTERNALS · the mount namespace
- The per-snap mount namespace — the district hub