feat: resolve a host's boot interface from predictions before its first lease

The admin boot actions learned to resolve a host still awaiting its first
DHCP lease in #2528 -- from its predicted boot interface -- but the
machine-controller never did: it read only `machine_interfaces` rows, so a
zero-DPU or NIC-mode host (which gets no primary row at ingestion) had
nothing to boot-target and its boot states waited. This gives the controller
the same fallback: before the first lease creates a real row, the host's
predicted boot interface answers, and the real row wins the moment it exists.

The selection is now one shared `pick_boot_prediction` -- the declared
`ExpectedHostNic.primary` (recorded on the prediction in #2657), else the
sole non-underlay prediction, else refuse to guess. Both the machine-controller
and the admin resolver route through it, so a multi-NIC host with a declared
primary resolves instead of refusing, and the two paths can't drift.

- Add `pick_boot_prediction` (api-model) and give the controller's
  `boot_interface_target`/`resolve_boot_interface` a prediction fallback, loaded
  per resolve via `load_boot_predictions`. `AwaitingNic` now fires only when
  there is no row and no usable prediction.
- Route admin `resolve_admin_boot_interface_target` through the same selector,
  so it honors the declared-primary flag for multi-NIC hosts (it used to refuse).
- The real `machine_interfaces` row always wins the moment it exists; a host
  with one expected boot NIC resolves from its sole prediction.

Tests cover `pick_boot_prediction`'s precedence, the controller's
row-wins / prediction-fallback / MAC-only / wait composition (split out so it
unit-tests without a full snapshot), and the admin declared-primary case.

Part of #2658 (epic #2660).

Signed-off-by: Chet Nichols III <chetn@nvidia.com>

Author: chet

crates/api-core/src/handlers/bmc_endpoint_explorer.rs

@@ -29,7 +29,6 @@ use model::expected_entity::ExpectedEntity;
 use model::machine::machine_search_config::MachineSearchConfig;
 use model::machine::{LoadSnapshotOptions, MachineInterfaceSnapshot};
 use model::machine_boot_interface::MachineBootInterface;
-use model::network_segment::NetworkSegmentType;
 use model::predicted_machine_interface::PredictedMachineInterface;
 use model::site_explorer::{NicMode, PreingestionState};
 use sqlx::PgConnection;
@@ -53,15 +52,14 @@ use crate::api::{Api, log_machine_id, log_request_data};
 /// machine awaiting its first DHCP lease) resolves from its
 /// `predicted_machine_interfaces` instead: the predicted NIC's MAC and
 /// recorded Redfish interface id form the same [`MachineBootInterface`] the
-/// real row will hold once the lease promotes it. Predictions answer only
-/// when unambiguous -- exactly one non-underlay prediction. Predictions now
-/// carry a `primary_interface` flag, but this resolver doesn't consult it yet,
-/// so with several (e.g. a host whose report lists SuperNICs alongside the boot
-/// NIC) the declared `ExpectedHostNic.primary` is not applied here; resolution
-/// refuses to guess and the action keeps requiring an explicit MAC, which the
-/// matching prediction's recorded id completes.
-/// The machine-controller does not consult predictions at all yet -- its
-/// boot states wait out this window -- a known follow-up.
+/// real row will hold once the lease promotes it. The candidate is chosen by
+/// the shared `pick_boot_prediction` -- the declared `ExpectedHostNic.primary`
+/// (recorded on the prediction), else the sole non-underlay prediction. With
+/// several (e.g. a host whose report lists SuperNICs alongside the boot NIC) and
+/// none declared primary the boot NIC is unknowable; resolution refuses to guess
+/// and the action keeps requiring an explicit MAC, which the matching
+/// prediction's recorded id completes. The machine-controller resolves the same
+/// way, through the same `pick_boot_prediction`.
 ///
 /// Site-explorer's stored default (`ExploredEndpoint::boot_interface()`)
 /// answers only for endpoints no machine owns. An owned machine resolves
@@ -117,15 +115,12 @@ fn resolve_admin_boot_interface_target(
                 // machine-controller's boot_interface_target.
                 return Some(target_for(picked.mac_address, picked.boot_interface()));
             }
-            // The rows offered no boot candidate: the machine's predicted
-            // NICs answer, but only when unambiguous -- exactly one
-            // non-underlay prediction. Predictions now carry a primary flag,
-            // but this resolver doesn't consult it yet, so with several the
-            // declared intent isn't applied here.
-            let mut bootable = candidates.predicted.iter().filter(|predicted| {
-                predicted.expected_network_segment_type != NetworkSegmentType::Underlay
-            });
-            if let (Some(predicted), None) = (bootable.next(), bootable.next()) {
+            // The rows offered no boot candidate: the machine's predicted NICs
+            // answer, via the shared `pick_boot_prediction` -- the declared
+            // primary, else the sole non-underlay prediction. With several and
+            // none declared primary the boot NIC is unknowable, so it returns
+            // `None` and the action keeps requiring an explicit MAC.
+            if let Some(predicted) = model::machine::pick_boot_prediction(&candidates.predicted) {
                 return Some(target_for(
                     predicted.mac_address,
                     predicted.boot_interface(),
@@ -1123,6 +1118,8 @@ pub(crate) async fn validate_and_complete_bmc_endpoint_request(
 
 #[cfg(test)]
 mod tests {
+    use model::network_segment::NetworkSegmentType;
+
     use super::*;
 
     fn row(mac: &str, primary: bool, boot_interface_id: Option<&str>) -> MachineInterfaceSnapshot {
@@ -1357,6 +1354,30 @@ mod tests {
         );
     }
 
+    // A declared-primary prediction disambiguates a multi-prediction host:
+    // `pick_boot_prediction` selects it, so resolution targets the declared NIC
+    // rather than refusing. (Multiple NON-primary predictions still refuse --
+    // see `no_mac_multiple_predictions_refuse_to_guess_a_boot_device`.)
+    #[test]
+    fn no_mac_declared_primary_prediction_wins_over_other_predictions() {
+        let declared_primary = PredictedMachineInterface {
+            primary_interface: true,
+            ..predicted("00:00:5e:00:53:01", Some("NIC.Embedded.1-1-1"))
+        };
+        let other = predicted("00:00:5e:00:53:02", Some("NIC.Slot.7-1-1"));
+        let c = BootInterfaceCandidates {
+            interfaces: vec![],
+            predicted: vec![other, declared_primary],
+        };
+        assert_eq!(
+            resolve_admin_boot_interface_target(None, Some(&c), None),
+            Some(BootInterfaceTarget::Pair(pair(
+                "00:00:5e:00:53:01",
+                "NIC.Embedded.1-1-1"
+            ))),
+        );
+    }
+
     #[test]
     fn no_mac_underlay_only_rows_let_a_sole_prediction_answer() {
         // Real rows exist but none is a boot candidate (declared bmc/oob NICs

crates/api-model/src/machine/mod.rs

@@ -65,6 +65,7 @@ use crate::machine_interface::InterfaceType;
 use crate::machine_interface_address::InterfaceAssociationType;
 use crate::network_segment::NetworkSegmentType;
 use crate::power_manager::PowerOptions;
+use crate::predicted_machine_interface::PredictedMachineInterface;
 use crate::state_history::StateHistoryRecord;
 
 pub mod slas;
@@ -306,6 +307,38 @@ fn pick_boot_interface_pair(
     pick_boot_interface(interfaces).and_then(MachineInterfaceSnapshot::boot_interface)
 }
 
+/// Pick the predicted interface a host should boot from in the window before
+/// its first DHCP lease creates a real `machine_interfaces` row. Mirrors
+/// `pick_boot_interface`'s precedence, one rung down -- predictions, not rows:
+///
+/// 1. A prediction flagged `primary_interface` wins -- the declared
+///    `ExpectedHostNic.primary`, recorded onto the prediction at minting.
+/// 2. Otherwise the sole non-underlay prediction. With several and none
+///    declared primary the boot NIC is unknowable (e.g. a host whose report
+///    lists SuperNICs alongside the boot NIC), so this returns `None` rather
+///    than guess against whichever NIC happens to sort first.
+///
+/// Public because both the machine-controller and admin boot-interface
+/// resolution apply it for the pre-first-lease window, when the real rows
+/// offer no candidate yet. Callers map the chosen prediction to a boot target
+/// via `PredictedMachineInterface::boot_interface()`.
+pub fn pick_boot_prediction(
+    predictions: &[PredictedMachineInterface],
+) -> Option<&PredictedMachineInterface> {
+    // The declared primary wins, exactly as in `pick_boot_interface`.
+    if let Some(primary) = predictions.iter().find(|p| p.primary_interface) {
+        return Some(primary);
+    }
+    // Otherwise only a sole non-underlay prediction is unambiguous.
+    let mut non_underlay = predictions
+        .iter()
+        .filter(|p| p.expected_network_segment_type != NetworkSegmentType::Underlay);
+    match (non_underlay.next(), non_underlay.next()) {
+        (Some(only), None) => Some(only),
+        _ => None,
+    }
+}
+
 impl ManagedHostStateSnapshot {
     /// Returns `true` if this managed host has at least one DPU snapshot
     /// attached -- i.e. a DPU we actively manage as a `Machine`.
@@ -3451,6 +3484,77 @@ mod tests {
         assert_eq!(pick_boot_interface_pair(&[primary]), None);
     }
 
+    /// Build a mock `PredictedMachineInterface` with the fields
+    /// `pick_boot_prediction` inspects (MAC, primary flag, segment type).
+    fn build_mock_prediction(
+        mac: &str,
+        primary: bool,
+        segment_type: NetworkSegmentType,
+    ) -> PredictedMachineInterface {
+        PredictedMachineInterface {
+            id: uuid::Uuid::nil(),
+            machine_id: MachineId::from_str(
+                "fm100ds7blqjsadm2uuh3qqbf1h7k8pmf47um6v9uckrg7l03po8mhqgvng",
+            )
+            .unwrap(),
+            mac_address: mac.parse().unwrap(),
+            expected_network_segment_type: segment_type,
+            boot_interface_id: None,
+            primary_interface: primary,
+        }
+    }
+
+    // A declared-primary prediction wins outright, mirroring pick_boot_interface
+    // -- regardless of how many other predictions there are.
+    #[test]
+    fn pick_boot_prediction_returns_the_declared_primary() {
+        let predictions = vec![
+            build_mock_prediction("05:00:00:00:00:01", false, NetworkSegmentType::HostInband),
+            build_mock_prediction("10:00:00:00:00:01", true, NetworkSegmentType::HostInband),
+            build_mock_prediction("20:00:00:00:00:01", false, NetworkSegmentType::HostInband),
+        ];
+        assert_eq!(
+            pick_boot_prediction(&predictions).map(|p| p.mac_address),
+            Some("10:00:00:00:00:01".parse().unwrap())
+        );
+    }
+
+    // With no declared primary, a sole non-underlay prediction is unambiguous.
+    #[test]
+    fn pick_boot_prediction_returns_the_sole_non_underlay_prediction() {
+        let predictions = vec![
+            build_mock_prediction("01:00:00:00:00:01", false, NetworkSegmentType::Underlay),
+            build_mock_prediction("10:00:00:00:00:01", false, NetworkSegmentType::HostInband),
+        ];
+        assert_eq!(
+            pick_boot_prediction(&predictions).map(|p| p.mac_address),
+            Some("10:00:00:00:00:01".parse().unwrap())
+        );
+    }
+
+    // Several non-underlay predictions and none declared primary: the boot NIC
+    // is unknowable, so refuse to guess rather than program boot order against
+    // whichever sorts first (the Gigawatt SuperNIC safety case).
+    #[test]
+    fn pick_boot_prediction_refuses_multiple_non_primary_predictions() {
+        let predictions = vec![
+            build_mock_prediction("10:00:00:00:00:01", false, NetworkSegmentType::HostInband),
+            build_mock_prediction("20:00:00:00:00:01", false, NetworkSegmentType::HostInband),
+        ];
+        assert!(pick_boot_prediction(&predictions).is_none());
+    }
+
+    // Underlay predictions are never a boot candidate on their own.
+    #[test]
+    fn pick_boot_prediction_ignores_underlay_only_predictions() {
+        let predictions = vec![build_mock_prediction(
+            "01:00:00:00:00:01",
+            false,
+            NetworkSegmentType::Underlay,
+        )];
+        assert!(pick_boot_prediction(&predictions).is_none());
+    }
+
     // Check the case  where only the BMC has been discovered so far (which
     // is common during early ingestion). In this case, there's no valid boot MAC
     // yet; callers fall back to the `::NoDpu` handling downstream.

crates/machine-controller/src/boot_interface.rs

@@ -17,26 +17,63 @@
 //! Resolving how to target a host's boot interface for Redfish setup calls.
 
 use carbide_redfish::boot_interface::BootInterfaceTarget;
-use model::machine::ManagedHostStateSnapshot;
+use mac_address::MacAddress;
+use model::machine::{ManagedHostStateSnapshot, pick_boot_prediction};
+use model::machine_boot_interface::MachineBootInterface;
+use model::predicted_machine_interface::PredictedMachineInterface;
 
 /// Resolve how to target this host's boot interface for Redfish setup calls.
 ///
-/// Uses the host's primary `machine_interface`: when that row has a captured
-/// Redfish interface id, the full pair is returned (enabling the MAC-first /
-/// interface-id fallback); otherwise it targets the MAC alone. Both come from the
-/// same row, so the pair can never name a different interface than the MAC.
+/// The host's own `machine_interface` row wins the moment it exists: when that
+/// row has a captured Redfish interface id, the full pair is returned (enabling
+/// the MAC-first / interface-id fallback); otherwise it targets the MAC alone.
+/// Both come from the same row, so the pair can never name a different interface
+/// than the MAC.
 ///
-/// Returns `None` only when the host has no boot interface at all (e.g. only the
-/// BMC has been discovered, or the primary NIC hasn't appeared yet).
+/// Before that first DHCP lease creates a row -- the window a zero-DPU or
+/// NIC-mode host sits in, since it gets no primary row at ingestion -- the host's
+/// `predictions` answer instead, via `pick_boot_prediction` (the declared
+/// primary, else the sole non-underlay prediction). The predicted MAC and
+/// recorded id form the same pair the real row will hold once the lease promotes
+/// it.
+///
+/// Returns `None` only when the host has no boot interface at all -- no row and
+/// no usable prediction (e.g. only the BMC has been discovered, or several
+/// predictions with no declared primary).
 pub fn boot_interface_target(
     mh_snapshot: &ManagedHostStateSnapshot,
+    predictions: &[PredictedMachineInterface],
+) -> Option<BootInterfaceTarget> {
+    boot_interface_target_from(
+        mh_snapshot.boot_interface(),
+        mh_snapshot.boot_interface_mac(),
+        predictions,
+    )
+}
+
+/// The boot-target decision, split out from the snapshot lookup so it can be
+/// unit-tested directly without constructing a full `ManagedHostStateSnapshot`
+/// -- the same split `pick_boot_interface_mac`/`_pair` use in api-model. The
+/// host's own row wins (its captured pair, else its MAC alone); only when it
+/// owns no boot row does the predicted boot interface answer.
+fn boot_interface_target_from(
+    row_pair: Option<MachineBootInterface>,
+    row_mac: Option<MacAddress>,
+    predictions: &[PredictedMachineInterface],
 ) -> Option<BootInterfaceTarget> {
-    if let Some(boot_interface) = mh_snapshot.boot_interface() {
-        return Some(BootInterfaceTarget::Pair(boot_interface));
+    if let Some(pair) = row_pair {
+        return Some(BootInterfaceTarget::Pair(pair));
+    }
+    if let Some(mac) = row_mac {
+        return Some(BootInterfaceTarget::MacOnly(mac));
     }
-    mh_snapshot
-        .boot_interface_mac()
-        .map(BootInterfaceTarget::MacOnly)
+    // No real row yet -- fall back to the host's predicted boot interface.
+    pick_boot_prediction(predictions).map(|prediction| {
+        prediction.boot_interface().map_or(
+            BootInterfaceTarget::MacOnly(prediction.mac_address),
+            BootInterfaceTarget::Pair,
+        )
+    })
 }
 
 /// What a Redfish boot step should do with a host's boot interface.
@@ -51,17 +88,21 @@ pub fn boot_interface_target(
 pub enum BootInterfaceResolution {
     /// The boot interface resolved; target it.
     Ready(BootInterfaceTarget),
-    /// A zero-DPU host whose boot NIC has not been discovered yet -- wait.
+    /// A zero-DPU host with no boot interface yet -- neither a real row nor a
+    /// usable prediction -- so wait for its boot NIC to appear.
     AwaitingNic,
     /// A host that should already have a boot interface is missing one.
     Missing,
 }
 
 /// Resolve this host's boot interface for a Redfish boot step, classifying a
 /// missing one as either "wait for the NIC" (zero-DPU) or "fault".
-pub fn resolve_boot_interface(mh_snapshot: &ManagedHostStateSnapshot) -> BootInterfaceResolution {
+pub fn resolve_boot_interface(
+    mh_snapshot: &ManagedHostStateSnapshot,
+    predictions: &[PredictedMachineInterface],
+) -> BootInterfaceResolution {
     classify_boot_interface(
-        boot_interface_target(mh_snapshot),
+        boot_interface_target(mh_snapshot, predictions),
         mh_snapshot.has_managed_dpus(),
     )
 }
@@ -82,9 +123,74 @@ fn classify_boot_interface(
 #[cfg(test)]
 mod tests {
     use mac_address::MacAddress;
+    use model::network_segment::NetworkSegmentType;
 
     use super::*;
 
+    fn prediction(mac: &str, boot_interface_id: Option<&str>) -> PredictedMachineInterface {
+        PredictedMachineInterface {
+            id: uuid::Uuid::nil(),
+            machine_id: "fm100ds7blqjsadm2uuh3qqbf1h7k8pmf47um6v9uckrg7l03po8mhqgvng"
+                .parse()
+                .unwrap(),
+            mac_address: mac.parse().unwrap(),
+            expected_network_segment_type: NetworkSegmentType::HostInband,
+            boot_interface_id: boot_interface_id.map(String::from),
+            primary_interface: false,
+        }
+    }
+
+    // The host's own row wins over any prediction: a captured id gives the pair.
+    #[test]
+    fn boot_target_prefers_the_owned_row_over_predictions() {
+        let pair = MachineBootInterface {
+            mac_address: "10:00:00:00:00:01".parse().unwrap(),
+            interface_id: "NIC.Slot.5-1".to_string(),
+        };
+        assert_eq!(
+            boot_interface_target_from(
+                Some(pair.clone()),
+                Some("10:00:00:00:00:01".parse().unwrap()),
+                &[prediction("20:00:00:00:00:01", Some("NIC.Embedded.1-1-1"))],
+            ),
+            Some(BootInterfaceTarget::Pair(pair)),
+        );
+    }
+
+    // No owned row yet: the predicted boot interface answers (pre-first-lease),
+    // as a full pair when the prediction carries the Redfish id.
+    #[test]
+    fn boot_target_falls_back_to_the_prediction_before_the_first_lease() {
+        assert_eq!(
+            boot_interface_target_from(
+                None,
+                None,
+                &[prediction("20:00:00:00:00:01", Some("NIC.Embedded.1-1-1"))],
+            ),
+            Some(BootInterfaceTarget::Pair(MachineBootInterface {
+                mac_address: "20:00:00:00:00:01".parse().unwrap(),
+                interface_id: "NIC.Embedded.1-1-1".to_string(),
+            })),
+        );
+    }
+
+    // A prediction with no captured id targets the MAC alone.
+    #[test]
+    fn boot_target_prediction_without_id_is_mac_only() {
+        assert_eq!(
+            boot_interface_target_from(None, None, &[prediction("20:00:00:00:00:01", None)]),
+            Some(BootInterfaceTarget::MacOnly(
+                "20:00:00:00:00:01".parse().unwrap()
+            )),
+        );
+    }
+
+    // No row and no usable prediction: nothing to target, so the caller waits.
+    #[test]
+    fn boot_target_is_none_without_row_or_prediction() {
+        assert_eq!(boot_interface_target_from(None, None, &[]), None);
+    }
+
     #[test]
     fn classify_waits_for_a_zero_dpu_host_without_a_boot_interface() {
         // The zero-DPU host's boot NIC has not taken its first lease yet: wait