Skip to content

Commit 67114f5

Browse files
apurvvkumariaharjothkharasouvikDevlopercjagwani
authored
fix(status): make inference route health authoritative (#6412)
<!-- markdownlint-disable MD041 --> ## Summary Make the in-sandbox `https://inference.local/v1/models` route authoritative for inference health in `nemoclaw status`, `nemoclaw doctor`, and `nemoclaw connect`. This supersedes #6203 and #6264 while retaining credit for Harjoth's original implementation and Souvik's typed test-seam contribution. ## Related Issue Fixes #6192 ## Changes - Share one agent-aware `inference.local` probe and fail-closed parser across connect, status, and doctor. - Treat final HTTP 200-499 responses as route-reachable; treat interim 100-199 responses, HTTP 500-599, `000`, malformed output, timeout, and unavailable probes as failing. - Centralize route-failure labels and status exit decisions so text, JSON, status, doctor, and connect cannot drift. - Discard route-probe response bodies through `/dev/null` instead of a persistent sandbox temp file. - Verify route TLS with OpenShell's managed CA bundle; certificate failures are authoritative unreachable results. - Keep direct provider checks as explicitly labeled, non-authoritative upstream diagnostics. - Return nonzero status for unhealthy or unavailable inference routes in text and JSON output. - Add regression coverage for HTTP boundaries, unavailable probes, framed OpenShell output, no-direct-probe providers, DCode, and CLI exit behavior. - Update command, monitoring, local-inference, and troubleshooting documentation for the authoritative route semantics. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: [nine-category review passed after the only hardening finding was fixed](#6412 (comment)). - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — `npx vitest run --project cli` over the nine route/connect/status/doctor/recovery/adapter files: 163 passed; `npx vitest run --project integration` over the eight changed CLI/recovery files: 43 passed. - [x] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — command/result: `npm run checks` passed; the exact-head PR CI suite completed without failures; [typed OpenClaw and DCode live targets](https://github.com/NVIDIA/NemoClaw/actions/runs/28970519701) passed; [inference-routing, diagnostics, and sandbox-operations live jobs](https://github.com/NVIDIA/NemoClaw/actions/runs/28970503175) passed. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — build passed with 0 errors and two existing Fern warnings. - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) ## Architecture and Runtime-Boundary Review - **Invalid state:** Host/provider checks can succeed while the agent-visible `https://inference.local/v1/models` route is unusable. This PR exposes that disagreement; it does not preserve or mask it. - **Source boundary:** OpenShell owns sandbox exec, DNS, TLS CA injection, and proxy provisioning. NemoClaw owns interpretation and orchestration in `status`, `doctor`, and `connect`, so it probes the exact in-sandbox route and fails closed when OpenShell cannot return trusted `OK` or `BROKEN` evidence. - **Source-fix constraint:** Read-only diagnostics cannot repair OpenShell infrastructure, and an inconclusive result is inherently lossy. `connect` must not open SSH or mutate route state on inconclusive evidence. This fail-closed behavior is a permanent security boundary, not a temporary workaround. - **Regression and removal contract:** Checked-in parser, flow, CLI, missing-CA, 401/403, DCode argv/proxy, and redaction tests cover deterministic fault injection. The DCode wrapper may be removed only if OpenShell provides an agent-independent structured route-health API that preserves its CA and login-shell contract; fail-closed handling remains. - **Runtime justification:** A separate standalone live artifact would duplicate the repository's existing typed E2E lanes. Exact head `130e74f3c520aa82db86fd80e830eff362390b08` passed the [required live jobs](https://github.com/NVIDIA/NemoClaw/actions/runs/28970503175) and [OpenClaw and DCode live targets](https://github.com/NVIDIA/NemoClaw/actions/runs/28970519701); exact-head macOS E2E and arm64 image/build checks also passed. Induced broken-route, auth, and CA faults remain deterministic checked-in tests to avoid destructive shared-state mutation. - **PR sequencing:** #6412 is foundational. Merge it first, then rebase #6465 and integrate its additive route-drift warning into the final status-snapshot shape. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added clearer, authoritative inference health checks and diagnostics for sandbox status and doctor commands. * `connect` now fails closed when the in-sandbox inference route cannot be trusted, with safer redacted error details. * **Bug Fixes** * Improved handling of reachable, unhealthy, unreachable, and not-probed inference states. * Fixed status exit behavior so inference-route failures correctly return a non-zero result. * **Documentation** * Updated troubleshooting and command docs to explain the new inference health and proxy diagnostic output. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> Co-authored-by: harjoth <harjoth.khara@gmail.com> Co-authored-by: Souvik Ghosh <138186578+souvikDevloper@users.noreply.github.com> --------- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> Signed-off-by: cjagwani <cjagwani@nvidia.com> Co-authored-by: harjoth <harjoth.khara@gmail.com> Co-authored-by: Souvik Ghosh <138186578+souvikDevloper@users.noreply.github.com> Co-authored-by: cjagwani <cjagwani@nvidia.com>
1 parent f2fa8ad commit 67114f5

36 files changed

Lines changed: 1878 additions & 394 deletions

docs/inference/use-local-inference.mdx

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -161,8 +161,8 @@ NemoClaw configures the sandbox provider to use proxy port `11435` with the gene
161161
OpenShell's L7 proxy injects the token at egress, so the agent inside the sandbox never sees the token directly.
162162

163163
All proxy endpoints require the Bearer token, including `GET /api/tags`.
164-
Internal health and reachability checks that run through the proxy treat any HTTP response, including `401`, as proof the proxy is alive.
165-
They fail only when nothing answers at all.
164+
The host-side proxy liveness diagnostic is an auxiliary check; it treats an HTTP response such as `401` as proof that the proxy answered.
165+
Separately, the authoritative `status` and `doctor` route probe runs from inside the sandbox and reports HTTP `200` through `499` (including `401` and `403`) as `healthy`, HTTP `500` through `599` as `unhealthy`, and an interim `100` through `199`, `000`, a transport failure, or an unavailable probe as `unreachable` or `not probed`.
166166

167167
If Ollama is already running on a non-loopback address when you start onboard, the wizard restarts it on `127.0.0.1:11434`.
168168
The proxy becomes the only network path to the model server.
@@ -285,8 +285,9 @@ $$nemoclaw <name> status
285285
```
286286

287287
The output shows the provider label (for example, "Local vLLM" or "Other OpenAI-compatible endpoint") and the active model.
288-
For Local Ollama, status also checks the authenticated proxy when a proxy token is available.
289-
If `Inference` is healthy but `Inference (auth proxy)` is not, rerun onboarding to repair the proxy path that sandbox requests use.
288+
The main `Inference` line checks `inference.local` from inside the sandbox, which is the path agent requests use.
289+
For Local Ollama, status can also print an `Inference (auth proxy)` host-side diagnostic when a proxy token is available.
290+
If that diagnostic fails, rerun onboarding to recreate the proxy token and restart the proxy, even if the authoritative route is still responding.
290291

291292
## Switch Models at Runtime
292293

docs/monitoring/monitor-sandbox-activity.mdx

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -28,14 +28,14 @@ Run the status command to view sandbox state, gateway health, and the active inf
2828
$$nemoclaw <name> status
2929
```
3030

31-
For local Ollama and local vLLM routes, `$$nemoclaw <name> status` also probes the host-side health endpoint.
32-
The check catches a stopped local backend before you retry `inference.local` from inside the sandbox.
31+
`$$nemoclaw <name> status` probes `https://inference.local/v1/models` from inside the sandbox as the authoritative inference check.
32+
For local Ollama and local vLLM routes, it also prints labeled host-side backend diagnostics that help identify which hop failed.
3333

3434
Review these output fields.
3535

3636
- Sandbox details show the configured model, provider, GPU mode, and applied policy presets.
3737
- Gateway and process health show whether NemoClaw can reach the OpenShell gateway and whether the in-sandbox agent process is running.
38-
- Inference health for local Ollama and local vLLM shows `healthy` or `unreachable` with the probed local URL.
38+
- The main inference health line reflects the in-sandbox route the agent uses; labeled upstream, local-backend, and auth-proxy lines are diagnostic only.
3939
- NIM status shows whether a NIM container is running and healthy when that path is in use.
4040

4141
Run `$$nemoclaw <name> status` on the host to check sandbox state.
@@ -122,8 +122,9 @@ hermes
122122
If the request fails, check these items.
123123

124124
1. Run `$$nemoclaw <name> status` to confirm the active provider and endpoint.
125-
For local Ollama and local vLLM, check the `Inference` line first.
126-
If it shows `unreachable`, restart the local backend before retrying from inside the sandbox.
125+
Check the main `Inference` line first.
126+
If it shows `unhealthy`, `unreachable`, or `not probed`, inspect the labeled upstream, local-backend, and auth-proxy diagnostics to identify the failing hop.
127+
Restart a local backend only when its own diagnostic fails.
127128
2. Run `$$nemoclaw <name> logs --follow` to view error messages from the blueprint runner.
128129
3. Verify that the host can reach the inference endpoint.
129130
4. If the agent reports a context-overflow or token-limit error, clear the conversation with `/reset` (or start a fresh one with `/new`) in the TUI before retrying.

docs/reference/commands.mdx

Lines changed: 25 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -773,6 +773,11 @@ When the live OpenShell gateway inference route differs from the route recorded
773773
It realigns the route only when those registry entries are compatible with the requested provider and model.
774774
If another sandbox records a conflicting route, `connect` exits non-zero without changing the gateway and names the affected sandboxes.
775775
Use `$$nemoclaw inference set --provider <provider> --model <model>` to make an intentional compatible route change.
776+
Before it opens SSH, `connect` probes `https://inference.local/v1/models` from inside the sandbox with the selected agent's trusted CA and proxy context.
777+
HTTP `200` through `499` confirms that the route is reachable.
778+
When the probe returns a recognized broken result, `connect` attempts DNS or route repair and verifies the route again.
779+
When the initial probe cannot return a trusted result, `connect` fails closed before health-driven repair and before opening SSH.
780+
It prints a bounded, redacted last-probe detail and points you to `$$nemoclaw <name> doctor`.
776781
If the sandbox is registered locally but missing from a healthy gateway, `connect` preserves the registry entry and points you to `rebuild --yes`, `onboard`, or `destroy` instead of deleting the metadata needed for recovery.
777782

778783
After a host reboot, the OpenShell gateway rotates its SSH host keys.
@@ -1080,7 +1085,7 @@ The JSON output includes at least `schemaVersion`, `name`, `found`, `model`, `pr
10801085
In that case, text output keeps OpenShell's authoritative phase but prints a `docker unpause <container>` recovery hint instead of sending you directly to rebuild.
10811086
For terminal runtime sandboxes, the command also checks cgroup OOM kill counters.
10821087
If the counter records an OOM kill, text output prints `Runtime health: degraded (... OOM kill recorded)` and points you to `$$nemoclaw <name> rebuild`; JSON output reports `terminalRuntimeHealth.kind: "degraded"` with the OOM kill count and source counter path.
1083-
The command exits non-zero when the sandbox is missing locally, the gateway state is not `present`, the gateway reports a schema/protobuf mismatch (mirrored as `rpcIssue`), `failureLayer` is non-null, or a terminal runtime sandbox reports a recorded OOM kill.
1088+
The command exits non-zero when the sandbox is missing locally, the gateway state is not `present`, the gateway reports a schema/protobuf mismatch (mirrored as `rpcIssue`), `failureLayer` is non-null, the authoritative in-sandbox inference route fails or cannot be probed, or a terminal runtime sandbox reports a recorded OOM kill.
10841089
The alias form `$$nemoclaw <name> status --json` requires the sandbox to be registered locally; the canonical form `$$nemoclaw sandbox status <name> --json` is the one to use from automation that may run against an unknown sandbox name, since it still emits a JSON document with `found: false` instead of a text error.
10851090

10861091
```bash
@@ -1089,21 +1094,25 @@ $$nemoclaw my-assistant status --json
10891094
$$nemoclaw sandbox status my-assistant --json
10901095
```
10911096

1092-
The command probes every inference provider and reports one of three states on the `Inference` line:
1097+
The command probes `https://inference.local/v1/models` from inside the sandbox as the authoritative inference health check.
1098+
This check exercises the same route that agent traffic uses.
1099+
The main `Inference` line reports one of these states:
10931100

10941101
| State | Meaning |
10951102
|-------|---------|
1096-
| `healthy` | The provider endpoint returned a reachable response. |
1097-
| `unreachable` | The probe failed. The output includes the endpoint URL and a remediation hint. |
1098-
| `not probed` | The endpoint URL is not known (for example, `compatible-*` providers). |
1103+
| `healthy` | The route returned an HTTP status from `200` through `499`. Authentication responses such as `401` and `403` confirm route reachability. |
1104+
| `unhealthy` | The route returned an HTTP status from `500` through `599`. |
1105+
| `unreachable` | The route had a transport failure, returned no final HTTP status (`000` or an interim `100` through `199`), or returned an invalid status outside `100` through `599`. |
1106+
| `not probed` | NemoClaw could not run the authoritative route probe from a reachable sandbox. |
10991107
| `not verified` | NemoClaw could not verify the sandbox or gateway state, so it skips inference probing. |
11001108

1101-
Local providers (Ollama, vLLM) probe the host-side health endpoint.
1102-
Remote providers (NVIDIA Endpoints, OpenAI, Anthropic, Gemini) use a lightweight reachability check; any HTTP response, including `401` or `403`, counts as reachable.
1103-
No API keys are sent.
1109+
An authentication response confirms that the route is reachable, not that provider credentials are valid.
1110+
The command can also print direct host-side provider checks such as `Inference (upstream)` and provider-specific subprobes.
1111+
These checks are diagnostic only and do not override the authoritative `inference.local` result or determine the command exit status.
11041112

1105-
For Local Ollama, the command also probes the authenticated proxy and prints an `Inference (auth proxy)` line when a proxy token is available.
1106-
Use that line to distinguish a healthy backend from a broken proxy path that the sandbox uses for inference.
1113+
Local providers add host-side backend diagnostics.
1114+
For Local Ollama, the command can also print an `Inference (auth proxy)` diagnostic when a proxy token is available.
1115+
Use these diagnostics to identify a failing auxiliary hop after checking the main `Inference` line.
11071116

11081117
For cloud-only providers, the output omits the NIM status line unless a NIM container is registered or an unexpected NIM container is running.
11091118

@@ -1231,8 +1240,13 @@ The command checks the local CLI build, Docker daemon, OpenShell CLI, NemoClaw g
12311240
For gateway-based agents, it also reports messaging channel conflicts.
12321241
</AgentOnly>
12331242

1243+
For inference health, `doctor` treats the probe to `https://inference.local/v1/models` from inside the sandbox as authoritative.
1244+
HTTP responses from `200` through `499`, including `401` and `403`, pass this check.
1245+
HTTP `500` through `599`, interim `100` through `199`, transport failures with status `000`, invalid status values, and an unavailable authoritative probe fail the check.
1246+
Direct provider and upstream probes are diagnostics only, so their failure does not fail `doctor` when the authoritative in-sandbox route is healthy.
1247+
12341248
Warnings do not make the command fail.
1235-
Failed checks exit non-zero so scripts can use `doctor` as a readiness gate.
1249+
Failed checks, including a failed or unavailable authoritative inference route, exit non-zero so scripts can use `doctor` as a readiness gate.
12361250
Use `--json` for machine-readable output.
12371251

12381252
<AgentOnly variant="openclaw">

docs/reference/troubleshooting.mdx

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1354,10 +1354,11 @@ Check the active provider and endpoint:
13541354
$$nemoclaw <name> status
13551355
```
13561356

1357-
For local Ollama and local vLLM, `$$nemoclaw <name> status` also prints an `Inference` line that probes the host-side health endpoint directly.
1358-
If that line shows `unreachable`, start the local backend first and then retry the request.
1359-
For Local Ollama, current releases also print `Inference (auth proxy)` when a proxy token is available.
1360-
If the backend is healthy but the auth proxy is `unauthorized` or `unreachable`, re-run onboarding so NemoClaw can recreate the proxy token, restart the proxy, and refresh the route.
1357+
The main `Inference` line probes `https://inference.local/v1/models` from inside the sandbox, so it reflects the route the agent actually uses.
1358+
If that line shows `unhealthy`, `unreachable`, or `not probed`, inspect the labeled diagnostic lines to identify the failing hop.
1359+
For local Ollama and local vLLM, `Inference (ollama backend)` or the corresponding local-backend line reports the host-side service separately.
1360+
For Local Ollama, current releases can also print `Inference (auth proxy)` when a proxy token is available.
1361+
If a local backend or auth-proxy diagnostic fails, start the backend or re-run onboarding so NemoClaw can recreate the proxy token, restart the proxy, and refresh the route.
13611362

13621363
If the endpoint is correct but requests still fail, check for network policy rules that may block the connection.
13631364
Then verify the credential and base URL for the provider you selected during onboarding.

src/commands/sandbox/oclif-command-adapters.test.ts

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -302,6 +302,33 @@ describe("sandbox oclif command adapters", () => {
302302
expect(mocks.shieldsStatus).toHaveBeenCalledWith("alpha");
303303
});
304304

305+
it("sets a nonzero JSON exit when doctor reports inference.local failure (#6192)", async () => {
306+
const previousExitCode = process.exitCode;
307+
process.exitCode = undefined;
308+
mocks.runSandboxDoctor.mockResolvedValueOnce({
309+
schemaVersion: 1,
310+
sandbox: "alpha",
311+
status: "fail",
312+
failed: 1,
313+
warnings: 0,
314+
checks: [
315+
{
316+
group: "Inference",
317+
label: "Inference route (gateway)",
318+
status: "fail",
319+
detail: "Inference gateway returned HTTP 503",
320+
},
321+
],
322+
});
323+
324+
try {
325+
await SandboxDoctorCliCommand.run(["alpha", "--json"], rootDir);
326+
expect(process.exitCode).toBe(1);
327+
} finally {
328+
process.exitCode = previousExitCode;
329+
}
330+
});
331+
305332
it("keeps doctor --json stdout clean while diagnostics recovery prints progress", async () => {
306333
const report = {
307334
schemaVersion: 1,

src/commands/sandbox/status.ts

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,11 @@
11
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
// SPDX-License-Identifier: Apache-2.0
33

4-
import { getSandboxStatusReport, showSandboxStatus } from "../../lib/actions/sandbox/status";
4+
import {
5+
getSandboxStatusReport,
6+
isInferenceHealthFailing,
7+
showSandboxStatus,
8+
} from "../../lib/actions/sandbox/status";
59
import { NemoClawCommand } from "../../lib/cli/nemoclaw-oclif-command";
610
import { sandboxNameArg } from "../../lib/sandbox/command-support";
711
import { redactForLog } from "../../lib/security/redact";
@@ -33,6 +37,7 @@ export default class SandboxStatusCommand extends NemoClawCommand {
3337
report.gatewayState !== "present" ||
3438
report.rpcIssue ||
3539
report.failureLayer ||
40+
isInferenceHealthFailing(report.inferenceHealth) ||
3641
report.terminalRuntimeHealth?.kind === "degraded"
3742
) {
3843
process.exitCode = 1;

0 commit comments

Comments
 (0)