Emulated xHCI 1.0 Controller, USB 2.0 devices, and HID 1.11 tablet

[lifning]

The version of the standard referenced throughout the comments in this module is xHCI 1.2, but we do not implement the features required of a 1.1 or 1.2 compliant host controller - that is, we are only implementing a subset of what xHCI version 1.0 requires of an xHC, as described by version 1.2 of the specification.

At present, the USB devices supported are:

• a HIDTabletDevice for use as an absolute-axis pointing device for VNC clients (the RFB protocol has no relative-mouse support that would be required for using the simpler PS/2 mouse)
• a NullUsbDevice with no actual functionality, which primarily exists for testing, such as being a means to show that USB DeviceDescriptor`s are communicated to the guest in phd-tests.

Specifications used

• xHCI: https://www.intel.com/content/dam/www/public/us/en/documents/technical-specifications/extensible-host-controler-interface-usb-xhci.pdf
• USB 2.0: https://www.usb.org/document-library/usb-20-specification
• HID: https://www.usb.org/sites/default/files/hid1_11.pdf
• HID Usage Tables: https://www.usb.org/sites/default/files/hut1_6.pdf

I've made an effort to cite relevant sections of these where applicable in comments within the implementation.

Usage

In a propolis config.toml, an xHC with an HID tablet attached can be defined thusly:

[dev.xhc0]
driver = "pci-xhci"
pci-path = "0.6.0"

[dev.usb1]
driver = "usb-hid-tablet"
xhc-device = "xhc0"
root-hub-port = 1

Conventions

Wherever possible, the xHC represents Trb data through a further level of abstraction, such as enums constructed from the raw TRB bitfields before being passed to other parts of the system that use them, such that the behavior of identifying TrbType and accessing their fields properly according to the spec lives in a conversion function rather than strewn across implementation of other xHC functionality.

The nomenclature used is generally trading the "Descriptor" suffix for "Info", e.g. the high-level enum-variant version of an EventDescriptor is EventInfo (which is passed to the EventRing to be converted into Event TRBs and written into guest memory).

For 1-based indices defined by the spec (slot ID, port ID), we use [SlotId] and [PortId] and their respective .as_index() methods to index into our internal arrays of slots and ports, such that we aspire to categorically avoid off-by-one errors of omission (of - 1). (Note that indexing into the DCBAA is not done with this method, as position 0 in it is reserved by the spec for the Scratchpad described in xHCI 1.2 section 4.20)

xHC Implementation

   +---------+
   | PciXhci |
   +---------+
        | has-a
  +-----------------------------+
  |          XhciState          |
  |-----------------------------|
  | PCI MMIO registers          |
  | XhciInterrupter             |
  | DeviceSlotTable             |
  | Usb2Ports + Usb3Ports       |
  | CommandRing                 |
  | newly attached USB devices  |
  +-----------------------------+
      | has-a               |
+-------------------+       | has-a
|  XhciInterrupter  |   +-----------------+
|-------------------|   | DeviceSlotTable |
| EventRing         |   |-----------------|
| MSI-X/INTxPin     |   | DeviceSlot(s)   |___+------------------+
+-------------------+   | DCBAAP          |   |    DeviceSlot    |
                        | Active USB devs |   |------------------|
                        +-----------------+   | TransferRing(s)  |
                                              +------------------+

DeviceSlotTable

When a USB device is attached to the xHC, it is enqueued in a list within XhciState along with its PortId. The next time the xHC runs:

• it will update the corresponding PORTSC register and inform the guest with a TRB on the EventRing, and if enabled, a hardware interrupt.
• it moves the USB device to the DeviceSlotTable in preparation for being configured and assigned a slot. When the guest xHCD rings Doorbell 0 to run an EnableSlot Command, the DeviceSlotTable assigns the first unused slot ID to it.

Hot-plugging devices live (i.e. not just attaching all devices defined by the instance spec at boot time as is done now) is not yet implemented.

Device-slot-related Command TRBs are handled by the DeviceSlotTable. The command interface methods are written as translations of the behaviors defined in xHCI 1.2 section 4.6 to Rust, with liberties taken around redundant TrbCompletionCode writes; i.e. when the outlined behavior from the spec describes the xHC placing a Success into a new TRB on the EventRing immediately at the beginning of the command's execution, and then overwriting it with a failure code in the event of a failure, our implementation postpones the creation and enqueueing of the event until after the outcome of the command's execution (and thus the Event TRB's values) are all known.

Ports

Root hub port state machines (xHCI 1.2 section 4.19.1) and port registers are managed by Usb2Port, which has separate methods for handling register writes by the guest and by the xHC itself.

A USB device may hold an XhciPortHandle, which is used to inform the xHC to update the Port Status Change register when a device returns from suspend (as well as to plumb completion events for transfer requests sent to the device in that port; speaking of:)

TRB Rings

Consumer:

The CommandRing and each slot endpoint's TransferRing are implemented as ConsumerRing<CommandInfo> and
ConsumerRing<TransferInfo>. Dequeued work items are converted from raw CommandDescriptors and TransferDescriptors, respectively).

Starting at the dequeue pointer provided by the guest, the ConsumerRing will consume non-Link TRBs (and follow Link TRBs, as in xHCI 1.2 figure 4-15) into complete work items. In the case of the CommandRing, CommandDescriptors are each only made up of one Trb, but for the TransferRing multi-TRB work items are possible, where all but the last item have the chain_bit set.

Producer:

The only type of producer ring is the EventRing. Events destined for it are fed through the XhciInterrupter, which handles enablement and rate-limiting of PCI-level machine interrupts being generated as a result of the events.

Similarly (and inversely) to the consumer rings, the EventRing converts the EventInfos enqueued in it into EventDescriptors to be written to guest memory regions defined by the EventRingSegment Table.

Doorbells

The guest writing to a DoorbellRegister makes the host controller process a consumer TRB ring (the CommandRing for doorbell 0, or the corresponding slot's TransferRing for nonzero doorbells). The ring consumption is performed by the doorbell register write handler, in process_command_ring and process_transfer_ring.

Timer registers

The value of registers defined as incrementing/decrementing per time interval, such as MFINDEX and the XhciInterrupter's IMODC, are simulated with VmGuestInstants and Durations rather than by repeated incrementation. (VmGuestInstant is a new type defined in this change with a similar interface to std::time::Instant, but using timestamps provided by the VMM.)

Migration

The types defined for serializing the device state are admittedly undercooked; advice about what a cleaner approach could look like before committing to the 'V1' of the format are quite welcome.

An inactive phd-test for migrating a guest with an active HIDTabletDevice is present in phd-tests/tests/src/xhci.rs, which was written with a temporary (reverted for this PR) hack in place to plumb the xHC payloads despite their absence in the "V0" Spec.

USB Devices

Implementors of the UsbDevice trait can be attached to the xHC. When Transfer Descriptors are executed from an endpoint's TransferRing, the appropriate trait function is called (i.e. a Normal TRB calls normal_transfer, a Data Stage TRB calls data_stage). Note that to reduce boilerplate, TRBs as defined in the xHCI are handled directly, rather than being converted into URBs.

Abstractions are provided for defining USB Descriptors and HID Report Descriptors, as well as for implementing Control Endpoints (such as the Default Control Endpoint used for requesting the Device Descriptor) and Interrupt-IN Endpoints (such as the pipe used for HID reports).

HID Tablet

HIDTabletDevice implements a USB HID 1.11 pointing device that reports pointer state events provided by VNC clients connected to propolis-server.

      +--------------------------+
      |      HIDTabletDevice     |
      +--------------------------+
         |               |     |
 +--------------------+  |     |
 |  ControlEndpoint   |  |  +-----------------+
 |--------------------|  |  | HIDTabletReport |
 | GET_DESCRIPTOR     |  |  |-----------------|    +--------------------+
 | SET_CONFIGURATION  |  |  | Pointer state   |<---| VNC pointer events |
 | HID class-specific |  |  +-----------------+    +--------------------+
 |  (Set Idle,        |  |       |
 |   Get Report)      |  |       |
 +--------------------+  |       |
                         |       v
              +---------------------+
              | InterruptInEndpoint |
              |---------------------|
              | Receive Normal TRBs |
              | Write HID Report    |
              +---------------------+

The pointer events suppiled by the RFB server are placed behind a Mutex shared with the Interrupt-IN EP, which has an internal state machine for filling request buffers from the guest with payload data from VNC:

WaitForTransferDescriptor
  |       ^            ^
(TRB) (timeout)    (written)
  v       |            |
WaitForPayload-(data)>Writing

HID descriptors

Partial definitions (enough for our Tablet) of the HID Usage Tables, and some high-level constructs for defining Report Descriptors with them, are defined in hw::usb::usbdev::hid::report.

PHD tests

In addition to a simple "does the Null device enumerate under Linux" check, I've also written a xhci_usb_tablet_vnc_pointer_events_test that connects to the guest propois-server's RFB endpoint and wiggles the mouse in the VNC client, to see if Linux the hidraw device node reflects the expected reports. Naturally, this required making PHD capable of being a VNC client.

The inactively-developed rust-vnc client crate has been vendored under our org to update its dependencies. (Note that it is only used in PHD, nowhere near production.)

DTrace support

To see a trace of all MMIO register reads/writes and TRB enqueue/dequeues:

pfexec ./scripts/xhci-trace.d -p $(pgrep propolis-server)

The name of each register as used by DTrace is &'statically defined in registers::Registers::reg_name.

---

USB and VNC support are discussed in RFDs 476 and 252, respectively.
Special thanks to Luqman getting this started in early 2023: https://github.com/luqmana/propolis/commits/xhci/