Drop OPTE port lock earlier (#10281)

Fixes a possible deadlock in the OPTE port manager. When failing to add
external IPs for a new port, we both the `PortTicket` and a lock guard
around the mananger's mapping of all ports. Unforunately, we do that in
the wrong order: `PortTicket` first, which attempts to take that same
lock in its drop implementation. This drops the port lock earlier, to
avoid a deadlock.

Fixes #10280

Author: bnaecker

illumos-utils/src/opte/port_manager.rs

@@ -414,7 +414,6 @@ impl PortManager {
             hdl
         };
         let (port, ticket) = {
-            let mut ports = self.inner.ports.lock().unwrap();
             let ticket = PortTicket::new(nic.id, nic.kind, self.inner.clone());
             let port = Port::new(PortData {
                 name: port_name.clone(),
@@ -424,7 +423,18 @@ impl PortManager {
                 vni,
                 gateway,
             });
-            let old = ports.insert((nic.id, nic.kind), port.clone());
+
+            // NOTE: We may add external IPs below, which can fail. If that
+            // does, we drop the `ticket` on the way out of this block. That
+            // attempts to acquire this lock, in order to remove itself on drop.
+            // We need to drop the lock before that, to avoid a deadlock, so
+            // let's do it right away, after inserting.
+            let old = self
+                .inner
+                .ports
+                .lock()
+                .unwrap()
+                .insert((nic.id, nic.kind), port.clone());
             assert!(
                 old.is_none(),
                 "Duplicate OPTE port detected: interface_id = {}, kind = {:?}",