fix(ci3): retry on-demand fleet for up to 5m (metal bench capacity) (#24440)

## Problem

The dedicated benchmark box pins `m6a.metal` (sole-tenant, to avoid
co-tenant memory-bandwidth variance). Metal capacity is thin, and
`create-fleet` can return `InsufficientInstanceCapacity` even on-demand.
The on-demand fallback in `aws_request_instance_type` tried **once** and
gave up, failing the bench run.

## Fix

Wrap the on-demand fallback in a time-budgeted retry loop, mirroring the
existing spot loop:
- `CI_ONDEMAND_TIMEOUT` (default **300s** = 5 min), `CI_ONDEMAND_POLL`
(default 20s).
- Retries the whole on-demand `create-fleet` until an instance is
acquired or the budget expires; logs the capacity error code per
attempt.
- Only reached when spot couldn't be had (unchanged), so normal
spot-first behavior is untouched.

## Verification

- Trigger a bench run (`ci.sh bench`) when metal capacity is tight;
confirm it now retries the on-demand request (logging
`InsufficientInstanceCapacity; retrying in 20s...`) rather than failing
immediately, and succeeds once capacity frees within 5 min.

Author: charlielye

ci3/aws_request_instance_type

@@ -150,18 +150,34 @@ if [ "${NO_SPOT:-0}" -ne 1 ]; then
   done
 fi
 
-# On-demand fallback (cheapest pool) only if spot couldn't be had.
+# On-demand fallback (cheapest pool) only if spot couldn't be had. Some pinned pools (e.g.
+# m6a.metal for benchmarks) can hit InsufficientInstanceCapacity even on-demand, so retry
+# the whole request until a time budget expires rather than giving up on the first miss.
 if [ -z "$iid" ]; then
-  echo "Requesting on-demand fleet $info..."
-  resp=$(aws ec2 create-fleet \
-    --type instant \
-    --target-capacity-specification "TotalTargetCapacity=1,DefaultTargetCapacityType=on-demand" \
-    --on-demand-options "AllocationStrategy=lowest-price" \
-    --launch-template-configs "$ltc" \
-    --output json 2>"$state_dir/fleet_err") || true
-  iid=$(echo "$resp" | jq -r '.Instances[0].InstanceIds[0] // empty' 2>/dev/null || true)
-  instance_type=$(echo "$resp" | jq -r '.Instances[0].InstanceType // empty')
-  lifecycle=ondemand
+  ondemand_timeout=${CI_ONDEMAND_TIMEOUT:-300}
+  ondemand_poll=${CI_ONDEMAND_POLL:-20}
+  SECONDS=0
+  while true; do
+    echo "Requesting on-demand fleet $info..."
+    resp=$(aws ec2 create-fleet \
+      --type instant \
+      --target-capacity-specification "TotalTargetCapacity=1,DefaultTargetCapacityType=on-demand" \
+      --on-demand-options "AllocationStrategy=lowest-price" \
+      --launch-template-configs "$ltc" \
+      --output json 2>"$state_dir/fleet_err") || true
+    iid=$(echo "$resp" | jq -r '.Instances[0].InstanceIds[0] // empty' 2>/dev/null || true)
+    if [ -n "$iid" ]; then
+      instance_type=$(echo "$resp" | jq -r '.Instances[0].InstanceType // empty')
+      lifecycle=ondemand
+      break
+    fi
+    if (( SECONDS + ondemand_poll >= ondemand_timeout )); then
+      echo "On-demand fleet unfulfilled after ${ondemand_timeout}s; giving up."
+      break
+    fi
+    echo "On-demand capacity unavailable$(echo "$resp" | jq -r '.Errors[0].ErrorCode // "" | if . == "" then "" else " (" + . + ")" end' 2>/dev/null); retrying in ${ondemand_poll}s (elapsed ${SECONDS}s)..."
+    sleep "$ondemand_poll"
+  done
 fi
 
 if [ -z "$iid" ]; then