|
| 1 | +# Scaling NICo with machine-a-tron: 100 → 1000 → 4500 simulated hosts |
| 2 | + |
| 3 | +> **Status: DRAFT — early feedback wanted.** Stage 1 (100 hosts × 2 DPUs = |
| 4 | +> 300 BMCs → 264 machines) is complete on dev6. Stage 2 (1000 hosts) is |
| 5 | +> running as this is written. Stage 3 targets 4500 hosts × 2 DPUs = 13,500 |
| 6 | +> BMC endpoints, in support of scaling NICo to ~4500 nodes. |
| 7 | +> |
| 8 | +> **Rebased on #2764**: this work now sits on top of Alexander's ClusterIP |
| 9 | +> migration, which removes nginx/MetalLB from the chart entirely (per-BMC |
| 10 | +> ClusterIP Services + ServiceCIDR, multi-pod sharding via `pods.<name>.cidr`). |
| 11 | +> That migration supersedes issues 10, 12 and 13 below (kept for the record — |
| 12 | +> they document why the nginx/MetalLB path was abandoned) and independently |
| 13 | +> confirms the direction of the proxy-direct pivot. The scripts' scale mode |
| 14 | +> (`bmc_proxy` + client-injected Forwarded) remains valid with `bmcServices` |
| 15 | +> disabled and is what all stage results below were measured with. |
| 16 | +> |
| 17 | +> Branch: `machine-a-tron-e2e-on-2764`. Everything below is reproducible |
| 18 | +> with two commands: |
| 19 | +> |
| 20 | +> ```bash |
| 21 | +> export KUBECONFIG=/path/to/site/kubeconfig |
| 22 | +> helm-prereqs/cleanup-machine-a-tron.sh -y |
| 23 | +> MAT_MODE=scale HOST_COUNT=1000 helm-prereqs/setup-machine-a-tron.sh -y |
| 24 | +> ``` |
| 25 | +
|
| 26 | +## What this work delivers |
| 27 | +
|
| 28 | +1. **`helm-prereqs/setup-machine-a-tron.sh`** — one idempotent script that |
| 29 | + takes a running NICo site from nothing to created machines: namespace, |
| 30 | + pull secret, CA/Vault secret refresh, the full BMC/UEFI credential chain, |
| 31 | + nico-core site-config changes, DHCP pool sizing with auto-fit, DB safety |
| 32 | + checks, helm deploy, and a verification loop that actively shepherds the |
| 33 | + ingestion pipeline (details below on why that is necessary). |
| 34 | +2. **`helm-prereqs/cleanup-machine-a-tron.sh`** — the full inverse, so |
| 35 | + from-scratch runs are reproducible (this caught several |
| 36 | + "works-second-time-only" bugs). |
| 37 | +3. **`MAT_MODE=scale`** — a scale profile |
| 38 | + (`helm-prereqs/values/machine-a-tron-scale.yaml`) using a **proxy-direct** |
| 39 | + transport architecture (see next section), simulated network segments |
| 40 | + sized for 13.5k endpoints, and raised site-explorer throughput knobs. |
| 41 | +4. A one-line RBAC fix in nico-api (`Machineatron` was missing the |
| 42 | + `AddExpectedMachine` grant) plus chart fixes to the nginx/MetalLB mode. |
| 43 | +
|
| 44 | +## The architecture decision: proxy-direct |
| 45 | +
|
| 46 | +The chart offers an nginx/MetalLB mode for large scale: one LoadBalancer |
| 47 | +Service per simulated BMC (cap 16,384), nginx terminating TLS and routing to |
| 48 | +the mock. We started there and hit four independent failure modes at just 300 |
| 49 | +endpoints (§ issues 10–13). The pivotal realization: |
| 50 | +
|
| 51 | +**`site_explorer.bmc_proxy` alone already scales.** When it is set, the |
| 52 | +Redfish client itself injects `Forwarded: host=<original BMC IP>` (RFC 7239, |
| 53 | +`crates/redfish/src/libredfish/implementation.rs`), and the mock's shared |
| 54 | +registry (`use_single_bmc_mock = true`) routes each request to the right |
| 55 | +simulated BMC. One ClusterIP Service carries the whole fleet — no nginx, no |
| 56 | +MetalLB pool, no per-BMC Services, no `externalTrafficPolicy: Local` |
| 57 | +pitfalls. |
| 58 | +
|
| 59 | +The nginx/MetalLB mode remains the right choice when simulated BMCs must |
| 60 | +coexist with **real hardware** (each mock needs a real routable IP). For a |
| 61 | +simulation-only cluster it only adds moving parts. The chart fixes we made to |
| 62 | +that mode are kept for its real users. |
| 63 | +
|
| 64 | +Result at 300 endpoints: exploration went from constant flapping |
| 65 | +(Unreachable/ConnectionRefused under load) to rock-stable 300/300. |
| 66 | +
|
| 67 | +## Complete issue log |
| 68 | +
|
| 69 | +Every issue below was found live on dev6 and is fixed on the branch, encoded |
| 70 | +in the scripts/charts with explanatory comments. |
| 71 | +
|
| 72 | +### Baseline (override-mode) end-to-end |
| 73 | +
|
| 74 | +| # | Issue | Root cause | Fix | |
| 75 | +|---|---|---|---| |
| 76 | +| 1 | Every nico-api call fails `client error (Connect)` after a site reprovision | machine-a-tron trusts the old CA (stale `nico-roots` copy) and presents a cert signed by it | Script refreshes `nico-roots` + Vault secrets from nico-system and deletes the client-cert secret so cert-manager reissues from the current CA | |
| 77 | +| 2 | Redfish redirect silently ignored | Docs said `override_target_host` — never a valid field; the real field is `bmc_proxy = "host:port"`, and it must be the **cross-namespace FQDN** (site-explorer runs in nico-system; a bare service name doesn't resolve) | Script sets `bmc_proxy` correctly; docs fixed | |
| 78 | +| 3 | site-explorer aborts every run: `MissingCredentials` | `machines/bmc/site/root` isn't in default kvSeeds; the seeded UEFI creds ship with **empty** passwords which fail validation | Script seeds the full chain | |
| 79 | +| 4 | Host BMCs 401 while DPUs explore fine | Host and DPU mock factory passwords differ (`factory_password` vs `0penBmc`); the host factory Vault path vendor segment is **lowercase** (`…/dell` — `BMCVendor`'s `Display` lowercases; the earlier capital-`Dell` seed was read by nobody) | Script seeds both factory creds on the correct paths | |
| 80 | +| 5 | machine-a-tron's expected-machine registration 403s (logged misleadingly as "likely already ingested") | `Machineatron` principal missing from the `AddExpectedMachine` RBAC grant — an oversight; it holds the sibling grants (`DiscoverDhcp`, `CreateNetworkSegment`, `GetExpectedSwitch`) | One-line fix in `internal_rbac_rules.rs`; script includes a DB fallback for nico-api builds without it | |
| 81 | +| 6 | Endpoints permanently stuck `AvoidLockout` (NICO-SITEEXPLORER-144) on a fresh deploy | Per-MAC rotated creds (`machines/bmc/<mac>/root`) survive cleanup; a fresh mock is at factory password but the per-MAC entry makes site-explorer present the old rotated one → 401 latch, self-perpetuating by design | Cleanup purges per-MAC creds; setup self-heals stale ones (only when the machine graph is truly empty — machines AND interfaces at 0) | |
| 82 | +| 7 | `DiscoverDhcp` fails for every BMC: "no rows returned…" | The `machine_dhcp_records` VIEW inner-joins a singleton control row (`machine_interfaces_deletion` id=1); manual lease cleanup had deleted it | Script restores the singleton; documented: never hand-delete lease rows | |
| 83 | +| 8 | Machines never created: admin pool exhausted | Real demand is OOB = `hosts×(1+dpus)` and admin = `hosts×(dpus+1)` (one host-PF IP per DPU **plus one per host at creation**); usable = `2^(32-mask) − reserve_first − 1` | Sizing check with auto-fit; `reserve_first` parsed from the live site config | |
| 84 | +
|
| 85 | +### Scale mode (100 hosts × 2 DPUs and up) |
| 86 | +
|
| 87 | +| # | Issue | Root cause | Fix | |
| 88 | +|---|---|---|---| |
| 89 | +| 9 | helm deploy aborts: hundreds of `connection reset by peer` | helm's default burst (100 concurrent API calls) overwhelms SOCKS/ssh tunnels when creating hundreds of Services | `--qps 15 --burst-limit 30` (env-overridable) | |
| 90 | +| 10 | nginx bmc-proxy CrashLoopBackOff: `host not found in upstream` | Chart template pointed the upstream at the bare chart name, which is not a Service | Point at the `-bmc-mock` Service (chart fix) | |
| 91 | +| 11 | Nothing listens on the mock port; probes kill the pod | `use_single_bmc_mock=false` makes each mock bind its **real BMC IP** on the pod netns (bare-metal mode). `true` is the shared-registry mode K8s needs | `useSingleBmcMock: true` | |
| 92 | +| 12 | Every registry lookup 404s: `no router configured for host: 10.233.x.x` | nginx forwarded `host=$server_addr`, but kube-proxy DNATs the LB IP to the nginx **pod IP** before the connection arrives | `Forwarded "host=$host"` — the client-requested host is the LB IP end-to-end (chart fix) | |
| 93 | +| 13 | LB IPs intermittently Unreachable in-cluster | Per-BMC Services use `externalTrafficPolicy: Local` and the chart's REQUIRED podAffinity stacked all proxies on the mat node | Required anti-affinity between proxy replicas (+ `maxUnavailable=1/maxSurge=0`; with replicas == nodes a surge pod deadlocks the rollout) — chart fix, kept for nginx-mode users | |
| 94 | +| 14 | DHCP fails: `No network segment defined for relay addresses` | Config-driven segment creation is **bootstrap-once** — skipped entirely on multi-domain sites ("Multiple domains, skipping initial network creation") | Script clone-inserts the simulated segments from same-type templates; `allocation_strategy` forced to `dynamic` (templates may be `reserved`, which rejects all dynamic DHCP) | |
| 95 | +| 15 | AvoidLockout storm on all DPU endpoints; preingestion pinned at exactly `hostCount` | The rotation dance is racy at scale: preingestion's initial BMC reset reboots the mock, which returns at the **factory** password while its per-MAC Vault entry says "rotated" | Pin mock passwords to the site root (`hostBmcPassword`/`dpuBmcPassword`) — site-explorer's documented fallback ("factory failed → sitewide root, no rotation") logs straight in; resets become harmless | |
| 96 | +| 16 | Pipeline stalls at preingestion `initial`; manager idle | `waiting_for_explorer_refresh` (set when errors are cleared) gates endpoints out of preingestion and can linger after a healthy report lands (273/300 were parked) | Verification loop unparks endpoints whose reports are clean | |
| 97 | +| 17 | Managed hosts identified but machines never created; cycles never finish | `explorations_per_run` was raised to 400 "for throughput" — but identification and creation only run **at the end of a completed explore cycle**, and 400 deep scans per cycle meant cycles stopped completing | Default lowered to 120: cycles complete in ~1–2 min and creation runs every cycle | |
| 98 | +| 18 | `Resource pool lo-ip is empty` on the 3rd machine | Machine creation allocates one loopback IP per machine; pool **definitions are seed-once** ("Declaration has drifted since seed … not re-applying") so config widening is ignored; dev6 ships **3** lo-ip addresses | Script inserts free `resource_pool` rows directly for a simulated range (16k) when the pool is smaller than the machine target | |
| 99 | +
|
| 100 | +### A note on the verification loop |
| 101 | +
|
| 102 | +The script's final phase doesn't just poll — it actively shepherds: |
| 103 | +re-clears `AvoidLockout`/`Unauthorized` latches (they are one-way by design; |
| 104 | +on real hardware an operator runs `nico-admin-cli site-explorer refresh`) and |
| 105 | +unparks healthy endpoints. On a simulation cluster with hundreds of |
| 106 | +concurrent resets/explorations, transient races are guaranteed; the loop is |
| 107 | +the "operator". Mocks have no lockout threshold, so this is safe here. |
| 108 | +
|
| 109 | +## Where we are today |
| 110 | +
|
| 111 | +| Stage | Scale | Result | |
| 112 | +|---|---|---| |
| 113 | +| Baseline | 1 host × 1 DPU (override mode) | ✅ end-to-end: machines created, full credential rotation exercised | |
| 114 | +| Stage 1 | 100 hosts × 2 DPUs = 300 BMCs (proxy-direct) | ✅ 300/300 endpoints stable, machines created and advancing through `hostinit`/`dpuinit` | |
| 115 | +| Stage 2 | 1000 hosts × 2 DPUs = 3000 BMCs | ✅ **END TO END OK — 3000/3000 machines** in a single unattended script run (~25 min total; creation ≈ 240 machines/min) | |
| 116 | +| Stage 3 | 4500 hosts × 2 DPUs = 13,500 BMCs | ✅ **exploration sweep complete: 13,500/13,500 endpoints**; 10,353 machines at time of writing with the remainder completing autonomously; fleet actively progressing through machine-controller lifecycle states (`dpuinit`, `dpudiscoveringstate`) on dev-sized postgres (4 CPU) | |
| 117 | +
|
| 118 | +Stage-3 observations worth reviewers' attention: |
| 119 | +
|
| 120 | +- **The ingestion pipeline is fully autonomous once configured.** Client |
| 121 | + connectivity to the cluster dropped twice for extended periods during |
| 122 | + stage 3; ingestion continued unattended both times (e.g. +720 machines |
| 123 | + through one outage, +4,000 through another). The shepherd loop's latch |
| 124 | + clearing — critical in earlier iterations — was a no-op for the entire |
| 125 | + stage-3 run thanks to pinned credentials. |
| 126 | +- **Measured stage-3 rates on dev6 (3 nodes, dev-sized postgres):** DHCP |
| 127 | + ~110 interfaces/min; exploration ~120–360 endpoints/cycle; creation |
| 128 | + 40–240 machines per completed explore cycle, sawtoothing with cycle |
| 129 | + phasing (identification rebuilds `explored_managed_hosts` each cycle). |
| 130 | +- **Per-MAC Vault credential lifecycle needs batching at scale** (issue 19 |
| 131 | + below): site-explorer stores one `machines/bmc/<mac>/root` entry per BMC — |
| 132 | + 13,500 entries; deleting them one API round-trip at a time takes hours, |
| 133 | + batched server-side it takes seconds. |
| 134 | +- `expected_machines` auto-registration worked at stage 3 (9,890+ registered |
| 135 | + by machine-a-tron via the API), confirming the RBAC grant path. |
| 136 | +
|
| 137 | +Additional issue found at stage 3: |
| 138 | +
|
| 139 | +| # | Issue | Root cause | Fix | |
| 140 | +|---|---|---|---| |
| 141 | +| 19 | Stage-2→3 cleanup ran for over an hour "deleting credentials" | One `kubectl exec` per per-MAC Vault deletion × thousands of entries | Batch the deletion loop server-side on the vault pod — one exec total (both cleanup and setup self-heal) | |
| 142 | +
|
| 143 | +## Open questions — feedback wanted |
| 144 | +
|
| 145 | +1. **RBAC**: is granting `Machineatron` → `AddExpectedMachine` acceptable |
| 146 | + (commit `9a9ba072a`)? Until a nico-api image with it is deployed, the |
| 147 | + script registers expected machines via direct DB insert — okay as a |
| 148 | + documented simulation-only fallback? |
| 149 | +2. **Seed-once reconcile semantics**: networks, and resource-pool |
| 150 | + definitions are all create-once; config changes on established sites are |
| 151 | + silently ignored (or warn-only). The script works around this with direct |
| 152 | + DB writes (segment clone-insert, pool row insertion). Should NICo support |
| 153 | + declarative updates for these instead? |
| 154 | +3. **AvoidLockout at scale**: one-way latches are right for real BMCs, but |
| 155 | + simulation fleets guarantee latch storms during resets. Worth a |
| 156 | + site-config escape hatch (e.g. `site_explorer.lockout_protection = false`) |
| 157 | + instead of the script's DB-level clearing? |
| 158 | +4. **Mock fidelity**: the mock returns to its configured password after a |
| 159 | + BMC reset. Real BMCs persist a rotated password across resets. Should |
| 160 | + bmc-mock persist rotated credentials so the rotation path can be exercised |
| 161 | + at scale without pinning? |
| 162 | +5. **lo-ip per machine**: is one loopback IP per machine the intended |
| 163 | + allocation at 13.5k machines, and is there guidance for sizing this pool |
| 164 | + in production site templates (dev templates ship 3)? |
| 165 | +6. **Cycle economics**: identification/creation only run at the end of a |
| 166 | + completed `explore_site` cycle, so `explorations_per_run` trades sweep |
| 167 | + throughput against creation latency in a non-obvious way. Worth |
| 168 | + documenting (or decoupling creation from the exploration cycle)? |
0 commit comments