You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
fix(status): make inference route health authoritative (#6412)
<!-- markdownlint-disable MD041 -->
## Summary
Make the in-sandbox `https://inference.local/v1/models` route
authoritative for inference health in `nemoclaw status`, `nemoclaw
doctor`, and `nemoclaw connect`. This supersedes #6203 and #6264 while
retaining credit for Harjoth's original implementation and Souvik's
typed test-seam contribution.
## Related Issue
Fixes#6192
## Changes
- Share one agent-aware `inference.local` probe and fail-closed parser
across connect, status, and doctor.
- Treat final HTTP 200-499 responses as route-reachable; treat interim
100-199 responses, HTTP 500-599, `000`, malformed output, timeout, and
unavailable probes as failing.
- Centralize route-failure labels and status exit decisions so text,
JSON, status, doctor, and connect cannot drift.
- Discard route-probe response bodies through `/dev/null` instead of a
persistent sandbox temp file.
- Verify route TLS with OpenShell's managed CA bundle; certificate
failures are authoritative unreachable results.
- Keep direct provider checks as explicitly labeled, non-authoritative
upstream diagnostics.
- Return nonzero status for unhealthy or unavailable inference routes in
text and JSON output.
- Add regression coverage for HTTP boundaries, unavailable probes,
framed OpenShell output, no-direct-probe providers, DCode, and CLI exit
behavior.
- Update command, monitoring, local-inference, and troubleshooting
documentation for the authoritative route semantics.
## Type of Change
- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)
## Quality Gates
- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: [nine-category review
passed after the only hardening finding was
fixed](#6412 (comment)).
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:
## Verification
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npx vitest run --project cli` over
the nine route/connect/status/doctor/recovery/adapter files: 163 passed;
`npx vitest run --project integration` over the eight changed
CLI/recovery files: 43 passed.
- [x] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: `npm run checks` passed;
the exact-head PR CI suite completed without failures; [typed OpenClaw
and DCode live
targets](https://github.com/NVIDIA/NemoClaw/actions/runs/28970519701)
passed; [inference-routing, diagnostics, and sandbox-operations live
jobs](https://github.com/NVIDIA/NemoClaw/actions/runs/28970503175)
passed.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) — build
passed with 0 errors and two existing Fern warnings.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
## Architecture and Runtime-Boundary Review
- **Invalid state:** Host/provider checks can succeed while the
agent-visible `https://inference.local/v1/models` route is unusable.
This PR exposes that disagreement; it does not preserve or mask it.
- **Source boundary:** OpenShell owns sandbox exec, DNS, TLS CA
injection, and proxy provisioning. NemoClaw owns interpretation and
orchestration in `status`, `doctor`, and `connect`, so it probes the
exact in-sandbox route and fails closed when OpenShell cannot return
trusted `OK` or `BROKEN` evidence.
- **Source-fix constraint:** Read-only diagnostics cannot repair
OpenShell infrastructure, and an inconclusive result is inherently
lossy. `connect` must not open SSH or mutate route state on inconclusive
evidence. This fail-closed behavior is a permanent security boundary,
not a temporary workaround.
- **Regression and removal contract:** Checked-in parser, flow, CLI,
missing-CA, 401/403, DCode argv/proxy, and redaction tests cover
deterministic fault injection. The DCode wrapper may be removed only if
OpenShell provides an agent-independent structured route-health API that
preserves its CA and login-shell contract; fail-closed handling remains.
- **Runtime justification:** A separate standalone live artifact would
duplicate the repository's existing typed E2E lanes. Exact head
`130e74f3c520aa82db86fd80e830eff362390b08` passed the [required live
jobs](https://github.com/NVIDIA/NemoClaw/actions/runs/28970503175) and
[OpenClaw and DCode live
targets](https://github.com/NVIDIA/NemoClaw/actions/runs/28970519701);
exact-head macOS E2E and arm64 image/build checks also passed. Induced
broken-route, auth, and CA faults remain deterministic checked-in tests
to avoid destructive shared-state mutation.
- **PR sequencing:** #6412 is foundational. Merge it first, then rebase
#6465 and integrate its additive route-drift warning into the final
status-snapshot shape.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added clearer, authoritative inference health checks and diagnostics
for sandbox status and doctor commands.
* `connect` now fails closed when the in-sandbox inference route cannot
be trusted, with safer redacted error details.
* **Bug Fixes**
* Improved handling of reachable, unhealthy, unreachable, and not-probed
inference states.
* Fixed status exit behavior so inference-route failures correctly
return a non-zero result.
* **Documentation**
* Updated troubleshooting and command docs to explain the new inference
health and proxy diagnostic output.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Co-authored-by: harjoth <harjoth.khara@gmail.com>
Co-authored-by: Souvik Ghosh
<138186578+souvikDevloper@users.noreply.github.com>
---------
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: cjagwani <cjagwani@nvidia.com>
Co-authored-by: harjoth <harjoth.khara@gmail.com>
Co-authored-by: Souvik Ghosh <138186578+souvikDevloper@users.noreply.github.com>
Co-authored-by: cjagwani <cjagwani@nvidia.com>
Copy file name to clipboardExpand all lines: docs/inference/use-local-inference.mdx
+5-4Lines changed: 5 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -161,8 +161,8 @@ NemoClaw configures the sandbox provider to use proxy port `11435` with the gene
161
161
OpenShell's L7 proxy injects the token at egress, so the agent inside the sandbox never sees the token directly.
162
162
163
163
All proxy endpoints require the Bearer token, including `GET /api/tags`.
164
-
Internal health and reachability checks that run through the proxy treat any HTTP response, including `401`, as proof the proxy is alive.
165
-
They fail only when nothing answers at all.
164
+
The host-side proxy liveness diagnostic is an auxiliary check; it treats an HTTP response such as `401` as proof that the proxy answered.
165
+
Separately, the authoritative `status` and `doctor` route probe runs from inside the sandbox and reports HTTP `200` through `499` (including `401` and `403`) as `healthy`, HTTP `500` through `599` as `unhealthy`, and an interim `100` through `199`, `000`, a transport failure, or an unavailable probe as `unreachable` or `not probed`.
166
166
167
167
If Ollama is already running on a non-loopback address when you start onboard, the wizard restarts it on `127.0.0.1:11434`.
168
168
The proxy becomes the only network path to the model server.
@@ -285,8 +285,9 @@ $$nemoclaw <name> status
285
285
```
286
286
287
287
The output shows the provider label (for example, "Local vLLM" or "Other OpenAI-compatible endpoint") and the active model.
288
-
For Local Ollama, status also checks the authenticated proxy when a proxy token is available.
289
-
If `Inference` is healthy but `Inference (auth proxy)` is not, rerun onboarding to repair the proxy path that sandbox requests use.
288
+
The main `Inference` line checks `inference.local` from inside the sandbox, which is the path agent requests use.
289
+
For Local Ollama, status can also print an `Inference (auth proxy)` host-side diagnostic when a proxy token is available.
290
+
If that diagnostic fails, rerun onboarding to recreate the proxy token and restart the proxy, even if the authoritative route is still responding.
Copy file name to clipboardExpand all lines: docs/monitoring/monitor-sandbox-activity.mdx
+6-5Lines changed: 6 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -28,14 +28,14 @@ Run the status command to view sandbox state, gateway health, and the active inf
28
28
$$nemoclaw <name> status
29
29
```
30
30
31
-
For local Ollama and local vLLM routes, `$$nemoclaw <name> status`also probes the host-side health endpoint.
32
-
The check catches a stopped local backend before you retry `inference.local` from inside the sandbox.
31
+
`$$nemoclaw <name> status` probes `https://inference.local/v1/models` from inside the sandbox as the authoritative inference check.
32
+
For local Ollama and local vLLM routes, it also prints labeled host-side backend diagnostics that help identify which hop failed.
33
33
34
34
Review these output fields.
35
35
36
36
- Sandbox details show the configured model, provider, GPU mode, and applied policy presets.
37
37
- Gateway and process health show whether NemoClaw can reach the OpenShell gateway and whether the in-sandbox agent process is running.
38
-
-Inference health for local Ollama and local vLLM shows `healthy` or `unreachable` with the probed local URL.
38
+
-The main inference health line reflects the in-sandbox route the agent uses; labeled upstream, local-backend, and auth-proxy lines are diagnostic only.
39
39
- NIM status shows whether a NIM container is running and healthy when that path is in use.
40
40
41
41
Run `$$nemoclaw <name> status` on the host to check sandbox state.
@@ -122,8 +122,9 @@ hermes
122
122
If the request fails, check these items.
123
123
124
124
1. Run `$$nemoclaw <name> status` to confirm the active provider and endpoint.
125
-
For local Ollama and local vLLM, check the `Inference` line first.
126
-
If it shows `unreachable`, restart the local backend before retrying from inside the sandbox.
125
+
Check the main `Inference` line first.
126
+
If it shows `unhealthy`, `unreachable`, or `not probed`, inspect the labeled upstream, local-backend, and auth-proxy diagnostics to identify the failing hop.
127
+
Restart a local backend only when its own diagnostic fails.
127
128
2. Run `$$nemoclaw <name> logs --follow` to view error messages from the blueprint runner.
128
129
3. Verify that the host can reach the inference endpoint.
129
130
4. If the agent reports a context-overflow or token-limit error, clear the conversation with `/reset` (or start a fresh one with `/new`) in the TUI before retrying.
Copy file name to clipboardExpand all lines: docs/reference/commands.mdx
+25-11Lines changed: 25 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -773,6 +773,11 @@ When the live OpenShell gateway inference route differs from the route recorded
773
773
It realigns the route only when those registry entries are compatible with the requested provider and model.
774
774
If another sandbox records a conflicting route, `connect` exits non-zero without changing the gateway and names the affected sandboxes.
775
775
Use `$$nemoclaw inference set --provider <provider> --model <model>` to make an intentional compatible route change.
776
+
Before it opens SSH, `connect` probes `https://inference.local/v1/models` from inside the sandbox with the selected agent's trusted CA and proxy context.
777
+
HTTP `200` through `499` confirms that the route is reachable.
778
+
When the probe returns a recognized broken result, `connect` attempts DNS or route repair and verifies the route again.
779
+
When the initial probe cannot return a trusted result, `connect` fails closed before health-driven repair and before opening SSH.
780
+
It prints a bounded, redacted last-probe detail and points you to `$$nemoclaw <name> doctor`.
776
781
If the sandbox is registered locally but missing from a healthy gateway, `connect` preserves the registry entry and points you to `rebuild --yes`, `onboard`, or `destroy` instead of deleting the metadata needed for recovery.
777
782
778
783
After a host reboot, the OpenShell gateway rotates its SSH host keys.
@@ -1080,7 +1085,7 @@ The JSON output includes at least `schemaVersion`, `name`, `found`, `model`, `pr
1080
1085
In that case, text output keeps OpenShell's authoritative phase but prints a `docker unpause <container>` recovery hint instead of sending you directly to rebuild.
1081
1086
For terminal runtime sandboxes, the command also checks cgroup OOM kill counters.
1082
1087
If the counter records an OOM kill, text output prints `Runtime health: degraded (... OOM kill recorded)` and points you to `$$nemoclaw <name> rebuild`; JSON output reports `terminalRuntimeHealth.kind: "degraded"` with the OOM kill count and source counter path.
1083
-
The command exits non-zero when the sandbox is missing locally, the gateway state is not `present`, the gateway reports a schema/protobuf mismatch (mirrored as `rpcIssue`), `failureLayer` is non-null, or a terminal runtime sandbox reports a recorded OOM kill.
1088
+
The command exits non-zero when the sandbox is missing locally, the gateway state is not `present`, the gateway reports a schema/protobuf mismatch (mirrored as `rpcIssue`), `failureLayer` is non-null, the authoritative in-sandbox inference route fails or cannot be probed, or a terminal runtime sandbox reports a recorded OOM kill.
1084
1089
The alias form `$$nemoclaw <name> status --json` requires the sandbox to be registered locally; the canonical form `$$nemoclaw sandbox status <name> --json` is the one to use from automation that may run against an unknown sandbox name, since it still emits a JSON document with `found: false` instead of a text error.
1085
1090
1086
1091
```bash
@@ -1089,21 +1094,25 @@ $$nemoclaw my-assistant status --json
1089
1094
$$nemoclaw sandbox status my-assistant --json
1090
1095
```
1091
1096
1092
-
The command probes every inference provider and reports one of three states on the `Inference` line:
1097
+
The command probes `https://inference.local/v1/models` from inside the sandbox as the authoritative inference health check.
1098
+
This check exercises the same route that agent traffic uses.
1099
+
The main `Inference` line reports one of these states:
1093
1100
1094
1101
| State | Meaning |
1095
1102
|-------|---------|
1096
-
|`healthy`| The provider endpoint returned a reachable response. |
1097
-
|`unreachable`| The probe failed. The output includes the endpoint URL and a remediation hint. |
1098
-
|`not probed`| The endpoint URL is not known (for example, `compatible-*` providers). |
1103
+
|`healthy`| The route returned an HTTP status from `200` through `499`. Authentication responses such as `401` and `403` confirm route reachability. |
1104
+
|`unhealthy`| The route returned an HTTP status from `500` through `599`. |
1105
+
|`unreachable`| The route had a transport failure, returned no final HTTP status (`000` or an interim `100` through `199`), or returned an invalid status outside `100` through `599`. |
1106
+
|`not probed`| NemoClaw could not run the authoritative route probe from a reachable sandbox. |
1099
1107
|`not verified`| NemoClaw could not verify the sandbox or gateway state, so it skips inference probing. |
1100
1108
1101
-
Local providers (Ollama, vLLM) probe the host-side health endpoint.
1102
-
Remote providers (NVIDIA Endpoints, OpenAI, Anthropic, Gemini) use a lightweight reachability check; any HTTP response, including `401` or `403`, counts as reachable.
1103
-
No API keys are sent.
1109
+
An authentication response confirms that the route is reachable, not that provider credentials are valid.
1110
+
The command can also print direct host-side provider checks such as `Inference (upstream)` and provider-specific subprobes.
1111
+
These checks are diagnostic only and do not override the authoritative `inference.local` result or determine the command exit status.
1104
1112
1105
-
For Local Ollama, the command also probes the authenticated proxy and prints an `Inference (auth proxy)` line when a proxy token is available.
1106
-
Use that line to distinguish a healthy backend from a broken proxy path that the sandbox uses for inference.
1113
+
Local providers add host-side backend diagnostics.
1114
+
For Local Ollama, the command can also print an `Inference (auth proxy)` diagnostic when a proxy token is available.
1115
+
Use these diagnostics to identify a failing auxiliary hop after checking the main `Inference` line.
1107
1116
1108
1117
For cloud-only providers, the output omits the NIM status line unless a NIM container is registered or an unexpected NIM container is running.
1109
1118
@@ -1231,8 +1240,13 @@ The command checks the local CLI build, Docker daemon, OpenShell CLI, NemoClaw g
1231
1240
For gateway-based agents, it also reports messaging channel conflicts.
1232
1241
</AgentOnly>
1233
1242
1243
+
For inference health, `doctor` treats the probe to `https://inference.local/v1/models` from inside the sandbox as authoritative.
1244
+
HTTP responses from `200` through `499`, including `401` and `403`, pass this check.
1245
+
HTTP `500` through `599`, interim `100` through `199`, transport failures with status `000`, invalid status values, and an unavailable authoritative probe fail the check.
1246
+
Direct provider and upstream probes are diagnostics only, so their failure does not fail `doctor` when the authoritative in-sandbox route is healthy.
1247
+
1234
1248
Warnings do not make the command fail.
1235
-
Failed checks exit non-zero so scripts can use `doctor` as a readiness gate.
1249
+
Failed checks, including a failed or unavailable authoritative inference route, exit non-zero so scripts can use `doctor` as a readiness gate.
Copy file name to clipboardExpand all lines: docs/reference/troubleshooting.mdx
+5-4Lines changed: 5 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1354,10 +1354,11 @@ Check the active provider and endpoint:
1354
1354
$$nemoclaw <name> status
1355
1355
```
1356
1356
1357
-
For local Ollama and local vLLM, `$$nemoclaw <name> status` also prints an `Inference` line that probes the host-side health endpoint directly.
1358
-
If that line shows `unreachable`, start the local backend first and then retry the request.
1359
-
For Local Ollama, current releases also print `Inference (auth proxy)` when a proxy token is available.
1360
-
If the backend is healthy but the auth proxy is `unauthorized` or `unreachable`, re-run onboarding so NemoClaw can recreate the proxy token, restart the proxy, and refresh the route.
1357
+
The main `Inference` line probes `https://inference.local/v1/models` from inside the sandbox, so it reflects the route the agent actually uses.
1358
+
If that line shows `unhealthy`, `unreachable`, or `not probed`, inspect the labeled diagnostic lines to identify the failing hop.
1359
+
For local Ollama and local vLLM, `Inference (ollama backend)` or the corresponding local-backend line reports the host-side service separately.
1360
+
For Local Ollama, current releases can also print `Inference (auth proxy)` when a proxy token is available.
1361
+
If a local backend or auth-proxy diagnostic fails, start the backend or re-run onboarding so NemoClaw can recreate the proxy token, restart the proxy, and refresh the route.
1361
1362
1362
1363
If the endpoint is correct but requests still fail, check for network policy rules that may block the connection.
1363
1364
Then verify the credential and base URL for the provider you selected during onboarding.
0 commit comments