Hotplug slots

sandbox internals for snapd developers for Ubuntu Core developers

With the experimental hotplug feature on snapd watches udev and adds a slot to the system snap when a matching device appears and removes it when the device goes. A hotplug key computed from the device's attributes lets the slot and its connections come back when the device is plugged in again. Only serial-port takes part and a gadget slot for the same device wins.

On the map: sandbox internals district

▶ Watch this deck — 8 scenes, ~6 min

01 A slot that arrives with the device

PLUG IT IN 1 · plugged inacme-agent:serial-portgateway · system snapUSB serial adapter $ sudo snap set system experimental.hotplug=true:ft232rusbuart $ sudo snap connect acme-agent:serial-port :ft232rusbuart 2 · pulled outacme-agent:serial-portgateway · system snapno slotrememberedUSB serial adapter 3 · plugged back inacme-agent:serial-portgateway · system snap:ft232rusbuartUSB serial adapterno command experimental · off by default · today only the serial-port interface uses it
Three frames of the gateway. Plugged in: with experimental.hotplug set, a USB serial adapter gives the system snap a slot :ft232rusbuart, and snap connect joins acme-agent:serial-port to it. Pulled out: the slot is gone and the connection is remembered. Plugged back in: slot and connection return without a command. Experimental, and only serial-port uses it

snap set system experimental.hotplug=true turns hotplug slots on. A slot appears on the system snap when the device does and goes when it goes; a connection made once is restored when the device returns. Only serial-port implements it.

Someone in the field plugs a USB serial adapter into the gateway. acme-agent needs a slot to connect its plug to, and nobody wrote one for this adapter. Where does it come from? Normally a slot comes from a snap or from the gadget. Switch on the experimental hotplug feature, and snapd makes one itself when the device shows up, named after the device. You connect the agent to it once. Pull the adapter out and the slot goes, but snapd remembers the connection. Plug it back in and both come back. The feature is off by default, and today only the serial-port interface uses it.

02 "The monitor: enumerate, then listen" for Ubuntu Core developers for snapd developers

HOW SNAPD HEARS ABOUT IT kernel · udev snapd · interface manager starts once a system snap is presentretries every 5 minutes on failure phase 1 · enumerateudevadm info -edeviceAddedfor every deviceenumerationDone phase 2 · listennetlink ueventsnetttyusbSUBSYSTEM filteraddremove seen: device pathsdrops duplicate addsand unknown removes a remembered slot whosedevice was not seen isremoved, as if unpluggedhow a device removed while the gateway was off is noticed
The interface manager starts a udev monitor once a system snap is present and retries every 5 minutes. Phase 1 enumerates with udevadm info -e and reports every device as added, then enumerationDone. Phase 2 reads netlink uevents for subsystems net, tty and usb, handles add and remove, and drops duplicate adds and unknown removes. After enumeration, a remembered slot whose device was not seen is removed as if unplugged

The monitor starts once a system snap is present and retries every 5 minutes on failure. It enumerates with udevadm info -e, then handles add and remove uevents for subsystems net, tty and usb. After enumeration, remembered slots whose device was not seen get a remove change.

Snapd has to hear about the adapter, whether it went in a second ago or before the gateway was switched on. The interface manager starts a udev monitor once a system snap is installed, retrying every five minutes on failure. It works in two phases. First it reads every existing device from udevadm info and treats each one as just added. Then it listens on a netlink socket for add and remove events from the net, tty and usb subsystems, and drops repeats. When the first phase ends, a remembered hotplug slot whose device didn't turn up is removed, as if unplugged while the gateway was off.

03 Each hotplug interface is asked for Ubuntu Core developers for snapd developers

IS THIS ONE MINE? the device, as udev describes itSUBSYSTEM=ttyDEVNAME=/dev/ttyUSB0ID_BUS=usbID_VENDOR_ID=0403ID_MODEL_ID=6001ID_MODEL=FT232R_USB_UARTID_SERIAL_SHORT=A1B2C3 serial-portHotplugDeviceDetected✓ subsystem tty✓ ID_BUS is usb✓ a serial device nodettyUSB, ttyACM, ttyS and others proposed slotpath = /dev/ttyUSB0usb-vendor = 0403usb-product = 6001 a network card · SUBSYSTEM=net → no interface proposes a slot → ignored experimental.hotplugon?off: the log sayshotplug device add event ignored,enable experimental.hotplugon: a hotplug key and a change follow
The adapter as udev describes it: SUBSYSTEM=tty, DEVNAME=/dev/ttyUSB0, ID_BUS=usb, ID_VENDOR_ID=0403, ID_MODEL_ID=6001, ID_MODEL=FT232R_USB_UART. serial-port's HotplugDeviceDetected checks subsystem tty, ID_BUS usb and a serial device node, and proposes a slot with path, usb-vendor and usb-product. A network card is ignored. Then the experimental.hotplug flag is checked; when it is off a log line says to enable it

HotplugDeviceDetected returns a proposed slot or nothing. serial-port proposes one for subsystem tty, ID_BUS=usb and a serial device node, with path, usb-vendor and usb-product. The feature flag is checked after the device has been evaluated.

Snapd itself has no idea what a serial device is. So it asks the interfaces. Each interface that supports hotplug is shown the device and either stays quiet or proposes a slot. The serial-port interface wants three things: the tty subsystem, the USB bus, and a device node that looks like a serial one, such as /dev/ttyUSB0. Our adapter passes, so it proposes a slot with the path, the usb-vendor and the usb-product. A network card gets no answer and is ignored. Only now is the feature flag checked. With it off, the log tells you to enable experimental.hotplug.

04 The hotplug key for Ubuntu Core developers for snapd developers

SAME DEVICE, WHATEVER THE PORT /dev/ttyUSB0depends on which devicearrived first nameID_V4L_PRODUCT, NAME, …vendorID_VENDOR_ID, ID_VENDOR, …modelID_MODEL_ID, ID_MODEL_ENCidentifierID_SERIAL, ID_SERIAL_SHORT, … none setID_VENDOR_ID=0403ID_MODEL_ID=6001ID_SERIAL_SHORT=A1B2C3ID_SERIAL not set SHA-256 overname\0value\0 for each hit01a2b3c4d5e6f…version64 hex digitsdigits illustrative fewer than two groups with a value → no key → device ignored stored in state: the key · kept in memory only: the device path, to match a remove eventan interface may supply its own key instead
/dev/ttyUSB0 is struck out because it depends on arrival order. Four groups of udev attributes, name, vendor, model and identifier; the first attribute with a value in each group, here ID_VENDOR_ID=0403, ID_MODEL_ID=6001 and ID_SERIAL_SHORT=A1B2C3, goes into a SHA-256. The key is the version digit 0 followed by 64 hex digits. Fewer than two groups means no key and the device is ignored. The key is stored; the device path is kept only in memory

The default hotplug key is a version digit 0 plus the SHA-256 of the first non-empty attribute from each of four groups (name, vendor, model, identifier). With fewer than two groups the device is ignored. The key, not the device path, is stored.

Why not remember /dev/ttyUSB0? Because the numbers follow arrival order. Plug a second adapter in first, and tomorrow that name belongs to a different device. So snapd builds a key from what the device says about itself. There are four groups of udev attributes: name, vendor, model and identifier. From each group, the first attribute with a value goes into a SHA-256 hash. The key is a version digit, zero, then the digest. A device that fills fewer than two groups is ignored. The key is what snapd stores. The device path stays in memory, only to match a later remove event.

05 Plugging in is a change for Ubuntu Core developers for snapd developers for administrators

THE ADD CHANGE hotplug-seq-waitwaits for earlier changeswith the same key hotplug-add-slotCreate slot for device /dev/ttyUSB0 (FT232R_USB_UART; serial: A1B2C3)with hotplug key "01a2b3c4d5e6…"task summary ID_MODEL=FT232R_USB_UARTlower case; letters, digits, dashes;at most 20 charactersft232rusbuarttaken? -1, -2 …:ft232rusbuart on the system snappath · usb-vendor · usb-product hotplug-connectbrings remembered connections back;new ones only if policy auto-connects $ sudo snap connect acme-agent:serial-port :ft232rusbuartserial-port's base declaration:deny-auto-connection: true
The add change has three tasks: hotplug-seq-wait waits for earlier changes with the same key; hotplug-add-slot, summary Create slot for device /dev/ttyUSB0 (FT232R_USB_UART; serial: A1B2C3) with hotplug key 01a2b3c4d5e6…, names the slot from ID_MODEL=FT232R_USB_UART as ft232rusbuart and adds it to the system snap with path, usb-vendor and usb-product; hotplug-connect restores remembered connections and makes new ones only if policy auto-connects. serial-port's base declaration has deny-auto-connection: true, so the first connection is made with snap connect

The change has tasks hotplug-seq-wait, hotplug-add-slot and hotplug-connect. The slot name is the proposed name, else the shortest of NAME, ID_MODEL_FROM_DATABASE and ID_MODEL made into a valid slot name, else the interface name. serial-port has deny-auto-connection: true, so the first connection is made by hand.

The adapter passed the checks and has a key. From here it's machinery you already know: a change, with tasks. The first, hotplug-seq-wait, only waits, so an add and a remove for one device run in order. Then hotplug-add-slot puts the slot on the system snap. Its name is the interface's proposal, or else the shortest cleaned-up form of the device's name or model, with a number added if it's taken. Last, hotplug-connect brings back connections this device had before, plus any that policy auto-connects. For serial-port, the base declaration says deny-auto-connection, so the first one is yours to make.

06 Unplug, remember, plug back in for app developers for Ubuntu Core developers for snapd developers for administrators

GONE, NOT FORGOTTEN 1 · connectedacme-agent:serial-port:ft232rusbuartsnapd's stateslot ft232rusbuartkey 01a2b3c4d5e6…acme-agent:serial-port remove changehotplug-seq-waithotplug-disconnecthotplug-remove-slot 2 · pulled outacme-agent:serial-portno slotsnapd's stateslot ft232rusbuartkey 01a2b3c4d5e6…acme-agent:serial-porthotplug-gonehotplug-gone nothing connected? then the slot is forgotten too 3 · back, any portacme-agent:serial-port:ft232rusbuarthotplug-connectsnapd's stateslot ft232rusbuartkey 01a2b3c4d5e6…acme-agent:serial-port a connection you removed with snap disconnect is not brought back
Three columns. Connected: acme-agent:serial-port is connected to :ft232rusbuart and snapd's state holds the slot with its key and the connection. A remove change runs hotplug-seq-wait, hotplug-disconnect and hotplug-remove-slot. Pulled out: no slot, and the slot and the connection stay in state marked hotplug-gone. Back in any port: the key matches, the slot returns under the same name and hotplug-connect restores the connection. With nothing connected the slot is forgotten; a connection removed with snap disconnect is not brought back

hotplug-disconnect marks connections hotplug-gone; hotplug-remove-slot keeps the slot in hotplug-slots, marked gone, while a connection refers to its key. On return the slot is re-added under its old name and connections that are gone and not undesired are re-created.

Pull the adapter out. The slot has to go, there's no device behind it. But should snapd forget that you connected it? A remove change runs: wait your turn, hotplug-disconnect, then hotplug-remove-slot. The agent loses access to the device, as with any disconnect. But the connection isn't deleted; it's marked hotplug-gone in state, and because a connection still refers to the key, the slot is remembered too. With nothing connected, it'd be forgotten. Plug the adapter back in, into any port, and the key matches. The slot returns under its old name, and the connection comes back. One you removed yourself with snap disconnect doesn't.

07 A gadget slot for the same device wins for Ubuntu Core developers for snapd developers

THE GADGET SPOKE FIRST the gadget snap's snap.yamlslots:modem-port:interface: serial-portusb-vendor: 0x0403usb-product: 0x6001path: /dev/serial-port-modem 0403:6001 /dev/ttyUSB0the adapterserial-portHandledByGadget vendor and product, compared as hexusb-interface-number too, when set a gadget slot without usb-vendor: its path must equal the device node handled bygadget slotmodem-portno hotplug slotdebug logignoring device /dev/ttyUSB0 (FT232R_USB_UART; serial: A1B2C3),interface "serial-port" (handled by gadget slot modem-port) 067b:2303 /dev/ttyUSB1no gadget slot matches → a hotplug slot as before
The gadget snap's snap.yaml declares a serial-port slot modem-port with usb-vendor 0x0403, usb-product 0x6001 and path /dev/serial-port-modem. serial-port's HandledByGadget compares it with the device 0403:6001 at /dev/ttyUSB0, as hex numbers and with usb-interface-number when set; a slot without usb-vendor is compared by path. The device is handled by the gadget slot and gets no hotplug slot. Another adapter, 067b:2303, matches no gadget slot and gets a hotplug slot as before

HandledByGadget lets a gadget slot claim a device: for serial-port by usb-vendor, usb-product and the optional usb-interface-number, otherwise by path. A claimed device gets no hotplug slot.

On a board you designed, you've probably declared the serial device in the gadget already. So what happens when udev announces that very adapter? Two slots for one device would be one too many. So before anything is proposed, the interface is asked whether one of the gadget's slots already handles this device. For serial-port, a gadget slot with a usb-vendor must match the device's vendor and product, and its interface number if it gives one. A slot without a vendor is compared by its path instead. A match, and no hotplug slot is made. Any other adapter still gets one.

08 Where to go from here

ROADS Hotplug slots Devices and storageserial-port and its neighbours Interface hooksrun on connect and disconnect How interfaces are implementedthe repository, auto-connection The overlordchanges and their tasks Confinementthe profiles a connection reachesThe mount namespacethe hub of this district
Roads from Hotplug slots: interfaces for devices and storage, interface hooks, how interfaces are implemented, the overlord, confinement and the per-snap mount namespace

Roads: devices and storage, interface hooks, how interfaces are implemented, the overlord, confinement and the per-snap mount namespace.

That's a slot with a lifetime: it exists exactly as long as the device is plugged in. A device appears, an interface claims it, a key names it, a change adds the slot, and state remembers the connection. serial-port and its neighbours are in the devices and storage family. The connections run the same interface hooks as any other. The repository and the auto-connection policy are in how interfaces are implemented. Changes and their tasks are in the overlord. Confinement shows the profiles a connection ends up in, and the mount namespace leads to the rest of this district.

Read more

Turning it on

Hotplug slots are an experimental feature, off by default:

sudo snap set system experimental.hotplug=true

With the flag off, udev events are still received and evaluated. An add that would have made a slot ends in the log line hotplug device add event ignored, enable experimental.hotplug; a remove for a device that has a slot ends in hotplug device remove event ignored, enable experimental.hotplug.

Only one interface implements hotplug in this tree: serial-port. A hotplug slot belongs to the system snap and is connected with the usual snap connect. serial-port's base declaration has deny-auto-connection: true, so the first connection is made by hand.

The examples in the deck use invented udev values for a USB serial adapter (ID_VENDOR_ID=0403, ID_MODEL_ID=6001, ID_MODEL=FT232R_USB_UART, ID_SERIAL_SHORT=A1B2C3, /dev/ttyUSB0) and assume acme-agent has a plug named serial-port of interface serial-port.

for snapd developers

What an interface implements

Interface (Go)MethodRequiredPurpose
hotplug.DefinerHotplugDeviceDetected(di)yesreturn a *ProposedSlot, or nil when the device is not this interface's kind
hotplug.HotplugKeyHandlerHotplugKey(di)noa custom key; an empty key falls back to the default
hotplug.HandledByGadgetPredicateHandledByGadget(di, slot)notrue when a gadget slot already covers the device

ProposedSlot has Name, Label and Attrs. Clean() validates the name when one is given and normalises the attributes. An empty label becomes the interface summary.

The default hotplug key

From each group, the first attribute with a non-empty value is written into a SHA-256 as name\0value\0.

GroupAttributes, in order
NameID_V4L_PRODUCT, NAME, ID_NET_NAME, PCI_SLOT_NAME
VendorID_VENDOR_ID, ID_VENDOR, ID_WWN, ID_WWN_WITH_EXTENSION, ID_VENDOR_FROM_DATABASE, ID_VENDOR_ENC, ID_OUI_FROM_DATABASE
ModelID_MODEL_ID, ID_MODEL_ENC
IdentifierID_SERIAL, ID_SERIAL_SHORT, ID_NET_NAME_MAC, ID_REVISION

The key is one hex digit of version (0 today) followed by the 64 hex digits of the digest: 65 characters. The code notes that "any future changes to these definitions require a new key version". When fewer than two groups give a value there is no key, and the device is skipped with no valid hotplug key provided by interface %q, device %s ignored.

Summaries show a key shortened to its first 12 characters followed by ….

The key identifies the device across unplug, replug and reboot. The device path is kept only in memory, to map a later remove event back to its keys.

Changes and tasks

EventChange kindTasks
device addedhotplug-add-slot-serial-porthotplug-seq-wait, hotplug-add-slot, hotplug-connect
device removedhotplug-remove-serial-porthotplug-seq-wait, hotplug-disconnect, hotplug-remove-slot
present slot, attributes changed(injected into the add change)hotplug-disconnect, hotplug-update-slot
  • hotplug-seq-wait retries every 300 ms while an unfinished hotplug change for the same key with a lower sequence number exists, logging Waiting processing of earlier hotplug event change %q affecting device with hotplug key %q. Sequence numbers come from the state key hotplug-seq.
  • hotplug-add-slot re-adds a remembered slot under its old name, injects hotplug-disconnect and hotplug-update-slot when a present slot's attributes changed, or adds a new slot.
  • hotplug-connect injects connect tasks for connections of the key that are marked hotplug-gone and not undesired, and for new auto-connections that policy allows.
  • hotplug-disconnect injects disconnect tasks flagged by-hotplug; on a conflicting change it logs Waiting for conflicting change in progress: %s and retries.

Task summaries:

  • Create slot for device %s with hotplug key %q
  • Recreate connections of interface %q for device %s with hotplug key %q
  • Disable connections of interface %q, hotplug key %q
  • Remove slot for interface %q, hotplug key %q
  • Update slot of interface %q, hotplug key %q

The remove change's summary is Remove hotplug connections and slots of device %s with interface %q. The add change's summary, Add hotplug slot of interface %q for device %s with hotplug key %q, currently prints the device where the interface belongs and the interface where the device belongs (a confirmed cosmetic snapd bug, overlord/ifacestate/hotplug.go:240); read the task summaries instead. The change kind is formatted from the interface value; it is hotplug-add-slot-serial-port only because serial-port's String() returns its name, so do not read it as a pattern for other interfaces.

A device in a summary is printed as its device node, then the model or vendor (cut to 16 characters) and the short serial, for example /dev/ttyUSB0 (FT232R_USB_UART; serial: A1B2C3).

for snapd developers for administrators

State

  • hotplug-slots: one entry per slot name with name, interface, static-attrs, hotplug-key and hotplug-gone.
  • hotplug-seq: the last sequence number handed out.
  • Connections of a hotplug slot carry the key and are marked hotplug-gone while the device is away, instead of being deleted.
  • A slot whose device is gone stays in hotplug-slots with hotplug-gone: true while any connection refers to its key; otherwise it is deleted.
  • A connection removed with snap disconnect is deleted (or, if it was auto-connected, marked undesired), so it is not re-created when the device returns.

Look with snap debug state (see snap debug). The spread test reads .data["hotplug-slots"] from state.json.

Slot names

1. The name the interface proposed, if any. serial-port proposes none. 2. Otherwise the shortest of NAME, ID_MODEL_FROM_DATABASE and ID_MODEL made into a valid slot name: letters lower-cased, digits kept after the first letter, single dashes kept, everything else dropped, at most 20 characters, no trailing dash. 3. Otherwise the interface name.

A taken name gets -1, -2 and so on. A remembered slot keeps its name, since a name already in hotplug-slots counts as free for the same key. The spread test's QEMU adapter, ID_MODEL=QEMU_USB_SERIAL, gets qemuusbserial.

The monitor

The interface manager starts the udev monitor from its Ensure, not in preseed mode, and only once a system snap is present. A failure is retried every 5 minutes; opening the netlink socket fails with cannot start udev monitor: %s.

1. Enumerate: parse udevadm info -e and report each device as added, then signal the end of enumeration. This code does not filter the enumeration by subsystem. 2. Listen: netlink uevents with a SUBSYSTEM filter for net, tty and usb; only add and remove are handled.

A table of device paths drops a second add for the same path and a remove for an unknown one. A device without a device path is ignored with missing device path attribute. With no system snap an event is dropped with system snap not available, hotplug events ignored.

When enumeration ends, every slot in hotplug-slots whose key was not seen gets a remove change, Remove hotplug connections and slots of interface %q, as if it had been unplugged.

serial-port and the gadget

HotplugDeviceDetected proposes a slot only for subsystem tty, ID_BUS=usb and a device node matching ^/dev/tty(mxc|USB|ACM|AMA|XRUSB|S|O|SC|MSM|HS|GS|LP|RPMSG|MAX)[0-9]+$. The slot gets path (the device node), usb-vendor from ID_VENDOR_ID and usb-product from ID_MODEL_ID.

Before asking for a proposal, snapd checks the gadget's slots of the same interface with HandledByGadget:

  • a gadget slot with usb-vendor must match ID_VENDOR_ID, its usb-product must match ID_MODEL_ID, and when it sets usb-interface-number that must match ID_USB_INTERFACE_NUM; the device's values are parsed as hex;
  • a gadget slot without usb-vendor matches when its path equals the device node.

A gadget serial-port slot that has usb-vendor must have a path of the form /dev/serial-port-<name> (lower-case letters and digits): it is the symlink udev creates. A claimed device gets no hotplug slot, logged at debug level as ignoring device %s, interface %q (handled by gadget slot %s).

Not determined

  • Which hooks run on the connect and disconnect tasks that hotplug-connect and hotplug-disconnect inject (see interface hooks).
  • How snap connections shows a hotplug slot.
  • Whether the enumeration by udevadm info -e is filtered to the three subsystems elsewhere.

Further reading

  • Hotplug support: enabling the feature, slot naming and remembered connections from the user's side. Its minimum snapd version disagrees with the experimental features page; the tree records neither.
  • Developing hotplug interfaces: the Go methods an interface implements.
  • serial-port interface: the one interface with hotplug support, and its slot attributes.
  • Hardware interfaces: gadget-defined slots, the alternative on Ubuntu Core.

Roads from here

INTERFACES AND CONFINEMENT

RUNNING SNAPS

  • The overlord — changes and tasks and why they wait for each other

INTERFACE FAMILIES · every interface, grouped by what it touches

HOOKS · code snapd runs inside a snap

SANDBOX INTERNALS · the mount namespace