fix(sandbox/recover): enforce Hermes env-file secret boundary on probe path (#5530)

## Summary

`nemohermes <name> recover` did not re-evaluate the documented Hermes
secret boundary when the gateway was already running. `recover` →
`connect --probe-only` → `checkAndRecoverSandboxProcesses` took the `if
(running)` branch and returned after refreshing the port-forward, never
invoking `buildHermesEnvFileBoundaryGuard()` (only the relaunch path
embeds it). A raw `TELEGRAM_BOT_TOKEN` injected into
`/sandbox/.hermes/.env` survived recovery and the gateway kept serving.
The probe path now executes a standalone secret-boundary check via the
root `openshell sandbox exec` channel (so the kill snippet has authority
over the gateway-user process), brings the gateway down on refusal, and
surfaces a non-zero exit on `recover`.

## Related Issue

Fixes #5525

## Changes

- `hermes-recovery-boundary.ts` exports
`buildHermesEnvFileBoundaryStandaloneCheck()` plus
`SECRET_BOUNDARY_{OK,REFUSED,VALIDATOR_MISSING}_MARKER` constants. The
standalone snippet runs the existing validator on
`/sandbox/.hermes/.env`, kills any running Hermes gateway/dashboard on
refusal, and emits a single stdout marker.
- `process-recovery.ts` adds
`enforceHermesSecretBoundaryOnRunningGateway`, invoked from
`checkAndRecoverSandboxProcesses` before the `if (running)` early-return
when the active agent is Hermes. Refusal short-circuits the function
with `secretBoundaryRefused: true` and skips the forward-refresh path.
- `connect.ts` `runSandboxConnectProbe` surfaces a refusal as exit 1
with a clear remediation line pointing at the
`openshell:resolve:env:<name>` placeholder.
- `docs/reference/commands.mdx` documents the Hermes-only
secret-boundary re-evaluation on `recover`; the nemohermes variant is
regenerated.
- New tests: 4 in `process-recovery.test.ts` (refused / passed /
validator-missing / OpenClaw no-op); 3 in
`runtime-hermes-secret-boundary-behavioural.test.ts` exercising the
standalone snippet against real bash with stubbed `python3`/`pkill`.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Verification

- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Git hooks passed during commit and push, or `npx prek run
--from-ref main --to-ref HEAD` passes
- [x] Targeted tests pass for changed behavior
- [ ] Full `npm test` passes (broad runtime changes only)
- [x] Tests added or updated for new or changed behavior
- [x] No secrets, API keys, or credentials committed
- [x] Docs updated for user-facing behavior changes
- [ ] `npm run docs` builds without warnings (doc changes only)
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Tinson Lai <tinsonl@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Updated `recover` command documentation to describe secret boundary
re-evaluation behavior.

* **New Features**
* Added Hermes secret boundary validation during the recovery process
that checks for improperly configured raw secrets in environment files.

* **Bug Fixes**
* Improved error detection with clearer guidance to replace raw secret
values with proper placeholders before recovery proceeds.

* **Tests**
  * Added test coverage for secret boundary validation scenarios.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Tinson Lai <tinsonl@nvidia.com>

Author: laitingsheng

docs/reference/commands-nemohermes.mdx

@@ -501,6 +501,11 @@ If the gateway is already running, the command exits zero with a probe message a
 nemohermes my-assistant recover
 ```
 
+`recover` re-evaluates the documented Hermes secret boundary against `/sandbox/.hermes/.env` on every run, including when the gateway is already healthy.
+If the file contains raw secret-shaped values (for example a pasted Telegram, Discord, or Slack bot token in place of the expected `openshell:resolve:env:<name>` placeholder), the command stops the running gateway, exits non-zero, and prints the offending key.
+Replace each flagged value with the `openshell:resolve:env:<name>` placeholder and re-run.
+Older Hermes sandbox images that predate the standalone validator are detected and left untouched: `recover` prints a `[boundary]` warning naming the sandbox and noting that `/sandbox/.hermes/.env` was not re-evaluated, then proceeds with the rest of the recovery path rather than blocking the gateway. Re-image the sandbox to a current Hermes build to enable the per-run boundary re-evaluation described above.
+
 ### `nemohermes <name> status`
 
 Show sandbox status, health, and inference configuration.

docs/reference/commands.mdx

@@ -630,6 +630,15 @@ If the gateway is already running, the command exits zero with a probe message a
 $$nemoclaw my-assistant recover
 ```
 
+<AgentOnly variant="hermes">
+
+`recover` re-evaluates the documented Hermes secret boundary against `/sandbox/.hermes/.env` on every run, including when the gateway is already healthy.
+If the file contains raw secret-shaped values (for example a pasted Telegram, Discord, or Slack bot token in place of the expected `openshell:resolve:env:<name>` placeholder), the command stops the running gateway, exits non-zero, and prints the offending key.
+Replace each flagged value with the `openshell:resolve:env:<name>` placeholder and re-run.
+Older Hermes sandbox images that predate the standalone validator are detected and left untouched: `recover` prints a `[boundary]` warning naming the sandbox and noting that `/sandbox/.hermes/.env` was not re-evaluated, then proceeds with the rest of the recovery path rather than blocking the gateway. Re-image the sandbox to a current Hermes build to enable the per-run boundary re-evaluation described above.
+
+</AgentOnly>
+
 ### `$$nemoclaw <name> status`
 
 Show sandbox status, health, and inference configuration.

src/lib/actions/sandbox/connect-flow.test.ts

@@ -29,6 +29,8 @@ type ConnectHarnessOptions = {
     wasRunning?: boolean;
     recovered?: boolean;
     forwardRecovered?: boolean;
+    secretBoundaryRefused?: boolean;
+    secretBoundaryReason?: "raw-secret" | "inconclusive";
   };
   spawnStatus?: number | null;
 };
@@ -222,4 +224,110 @@ describe("connectSandbox flow", () => {
     );
     expect(exitSpy).toHaveBeenCalledWith(1);
   });
+
+  it("probe-only mode exits with raw-secret remediation when the Hermes boundary refuses recovery", async () => {
+    const harness = createConnectHarness({
+      processCheck: {
+        checked: true,
+        wasRunning: true,
+        recovered: false,
+        forwardRecovered: false,
+        secretBoundaryRefused: true,
+        secretBoundaryReason: "raw-secret",
+      },
+    });
+    const agentRuntime = requireDist("../../../../dist/lib/agent/runtime.js");
+    vi.spyOn(agentRuntime, "getSessionAgent").mockReturnValue({ name: "hermes" });
+    vi.spyOn(agentRuntime, "getAgentDisplayName").mockReturnValue("Hermes");
+    const errorSpy = vi.spyOn(console, "error");
+
+    await expect(harness.connectSandbox("alpha", { probeOnly: true })).rejects.toThrow(
+      "process.exit(1)",
+    );
+
+    expect(harness.runAutoPairSpy).not.toHaveBeenCalled();
+    expect(harness.spawnSyncSpy).not.toHaveBeenCalledWith(
+      "openshell",
+      ["sandbox", "connect", "alpha"],
+      expect.any(Object),
+    );
+    const errorOutput = errorSpy.mock.calls.map((call) => String(call[0] ?? "")).join("\n");
+    expect(errorOutput).toContain(
+      "Probe failed: refused to confirm Hermes gateway in 'alpha' — /sandbox/.hermes/.env contains raw secret-shaped values.",
+    );
+    expect(errorOutput).toContain(
+      "Replace raw secret values with openshell:resolve:env:<name> placeholders and re-run.",
+    );
+    const logOutput = harness.logSpy.mock.calls.map((call) => String(call[0] ?? "")).join("\n");
+    expect(logOutput).not.toContain("Probe complete");
+    expect(exitSpy).toHaveBeenCalledWith(1);
+  });
+
+  it("non-probe connect exits before Ollama/inference-route/auto-pair when the Hermes boundary refuses", async () => {
+    const harness = createConnectHarness({
+      processCheck: {
+        checked: true,
+        wasRunning: true,
+        recovered: false,
+        forwardRecovered: false,
+        secretBoundaryRefused: true,
+        secretBoundaryReason: "raw-secret",
+      },
+    });
+    const agentRuntime = requireDist("../../../../dist/lib/agent/runtime.js");
+    vi.spyOn(agentRuntime, "getSessionAgent").mockReturnValue({ name: "hermes" });
+    vi.spyOn(agentRuntime, "getAgentDisplayName").mockReturnValue("Hermes");
+    const errorSpy = vi.spyOn(console, "error");
+
+    await expect(harness.connectSandbox("alpha")).rejects.toThrow("process.exit(1)");
+
+    expect(harness.ensureOllamaAuthProxySpy).not.toHaveBeenCalled();
+    expect(harness.runAutoPairSpy).not.toHaveBeenCalled();
+    expect(harness.spawnSyncSpy).not.toHaveBeenCalledWith(
+      "openshell",
+      ["sandbox", "connect", "alpha"],
+      expect.any(Object),
+    );
+    const errorOutput = errorSpy.mock.calls.map((call) => String(call[0] ?? "")).join("\n");
+    expect(errorOutput).toContain(
+      "Connect failed: refused to confirm Hermes gateway in 'alpha' — /sandbox/.hermes/.env contains raw secret-shaped values.",
+    );
+    expect(errorOutput).toContain(
+      "Replace raw secret values with openshell:resolve:env:<name> placeholders and re-run.",
+    );
+    expect(exitSpy).toHaveBeenCalledWith(1);
+  });
+
+  it("probe-only mode exits with inconclusive guidance when the Hermes boundary check could not run", async () => {
+    const harness = createConnectHarness({
+      processCheck: {
+        checked: true,
+        wasRunning: true,
+        recovered: false,
+        forwardRecovered: false,
+        secretBoundaryRefused: true,
+        secretBoundaryReason: "inconclusive",
+      },
+    });
+    const agentRuntime = requireDist("../../../../dist/lib/agent/runtime.js");
+    vi.spyOn(agentRuntime, "getSessionAgent").mockReturnValue({ name: "hermes" });
+    vi.spyOn(agentRuntime, "getAgentDisplayName").mockReturnValue("Hermes");
+    const errorSpy = vi.spyOn(console, "error");
+
+    await expect(harness.connectSandbox("alpha", { probeOnly: true })).rejects.toThrow(
+      "process.exit(1)",
+    );
+
+    expect(harness.runAutoPairSpy).not.toHaveBeenCalled();
+    const errorOutput = errorSpy.mock.calls.map((call) => String(call[0] ?? "")).join("\n");
+    expect(errorOutput).toContain(
+      "Probe failed: secret-boundary check did not complete for Hermes gateway in 'alpha'.",
+    );
+    expect(errorOutput).toContain(
+      "Inspect the validator output above and re-run `nemoclaw <sandbox> recover`.",
+    );
+    const logOutput = harness.logSpy.mock.calls.map((call) => String(call[0] ?? "")).join("\n");
+    expect(logOutput).not.toContain("Probe complete");
+    expect(exitSpy).toHaveBeenCalledWith(1);
+  });
 });

src/lib/actions/sandbox/connect.ts

@@ -27,7 +27,6 @@ import { findReachableOllamaHost, probeLocalProviderHealth } from "../../inferen
 import { ensureOllamaAuthProxy, probeOllamaAuthProxyHealth } from "../../inference/ollama/proxy";
 import { LOCAL_INFERENCE_TIMEOUT_SECS } from "../../onboard/env";
 import { resolveSandboxGatewayName } from "../../onboard/gateway-binding";
-import { getSandboxTargetGatewayName } from "./gateway-target";
 import { isWsl } from "../../platform";
 import { ROOT } from "../../runner";
 import * as sandboxVersion from "../../sandbox/version";
@@ -53,6 +52,7 @@ import {
 import { preflightVllmModelEnvOrExit } from "./connect-vllm-preflight";
 import { isDockerRuntimeDown, printDockerRuntimeDownGuidance } from "./gateway-failure-classifier";
 import { ensureLiveSandboxOrExit, printGatewayLifecycleHint } from "./gateway-state";
+import { getSandboxTargetGatewayName } from "./gateway-target";
 import { printGatewayWedgeDiagnostics } from "./gateway-wedge-diagnostics";
 import { checkAndRecoverSandboxProcesses, executeSandboxExecCommand } from "./process-recovery";
 import { applyOpenShellVmDnsMonkeypatch, shouldApplyVmDnsMonkeypatch } from "./vm-dns-monkeypatch";
@@ -183,6 +183,33 @@ export function parseSandboxConnectArgs(
   return options;
 }
 
+function exitOnSecretBoundaryRefusal(
+  sandboxName: string,
+  agentName: string,
+  processCheck: Record<string, unknown>,
+  contextLabel: "Probe" | "Connect",
+): never {
+  console.error("");
+  const reason =
+    "secretBoundaryReason" in processCheck
+      ? (processCheck.secretBoundaryReason as "raw-secret" | "inconclusive" | undefined)
+      : undefined;
+  if (reason === "raw-secret") {
+    console.error(
+      `  ${contextLabel} failed: refused to confirm ${agentName} gateway in '${sandboxName}' — /sandbox/.hermes/.env contains raw secret-shaped values.`,
+    );
+    console.error(
+      "  Replace raw secret values with openshell:resolve:env:<name> placeholders and re-run.",
+    );
+  } else {
+    console.error(
+      `  ${contextLabel} failed: secret-boundary check did not complete for ${agentName} gateway in '${sandboxName}'.`,
+    );
+    console.error("  Inspect the validator output above and re-run `nemoclaw <sandbox> recover`.");
+  }
+  process.exit(1);
+}
+
 function runSandboxConnectProbe(sandboxName: string): void {
   const processCheck = checkAndRecoverSandboxProcesses(sandboxName, { quiet: true });
   const agent = agentRuntime.getSessionAgent(sandboxName);
@@ -193,6 +220,9 @@ function runSandboxConnectProbe(sandboxName: string): void {
     );
     process.exit(1);
   }
+  if ("secretBoundaryRefused" in processCheck && processCheck.secretBoundaryRefused) {
+    exitOnSecretBoundaryRefusal(sandboxName, agentName, processCheck, "Probe");
+  }
   if (processCheck.wasRunning) {
     ensureSandboxInferenceRoute(sandboxName, { quiet: true });
     // Defense-in-depth scope-upgrade approval on the probe-only / `recover`
@@ -861,7 +891,11 @@ export async function connectSandbox(
     /* non-fatal — don't block connect on session detection failure */
   }
 
-  checkAndRecoverSandboxProcesses(sandboxName);
+  const processCheck = checkAndRecoverSandboxProcesses(sandboxName);
+  if ("secretBoundaryRefused" in processCheck && processCheck.secretBoundaryRefused) {
+    const agentName = agentRuntime.getAgentDisplayName(agentRuntime.getSessionAgent(sandboxName));
+    exitOnSecretBoundaryRefusal(sandboxName, agentName, processCheck, "Connect");
+  }
   // Ensure Ollama auth proxy is running (recovers from host reboots)
   ensureOllamaAuthProxy();
 

src/lib/actions/sandbox/process-recovery.ts

@@ -11,6 +11,12 @@ import {
   runOpenshell,
 } from "../../adapters/openshell/runtime";
 import { OPENSHELL_PROBE_TIMEOUT_MS } from "../../adapters/openshell/timeouts";
+import {
+  buildHermesEnvFileBoundaryStandaloneCheck,
+  SECRET_BOUNDARY_OK_MARKER,
+  SECRET_BOUNDARY_REFUSED_MARKER,
+  SECRET_BOUNDARY_VALIDATOR_MISSING_MARKER,
+} from "../../agent/hermes-recovery-boundary";
 import * as agentRuntime from "../../agent/runtime";
 import { G, R } from "../../cli/terminal-style";
 import { DASHBOARD_PORT } from "../../core/ports";
@@ -482,12 +488,115 @@ function recoverHermesDashboardProcessIfEnabled(sandboxName: string): boolean |
   return recoverHermesDashboardProcess(sandboxName, { executeCommand: executeSandboxCommand });
 }
 
+function isHermesAgent(agent: ReturnType<typeof agentRuntime.getSessionAgent>): boolean {
+  return !!agent && agent.name === "hermes";
+}
+
+type SecretBoundaryRefusalReason = "raw-secret" | "inconclusive";
+
+type HermesSecretBoundaryEnforcement =
+  | { refused: false }
+  | { refused: true; reason: SecretBoundaryRefusalReason; stderr: string };
+
+function printValidatorStderr(stderr: string): void {
+  if (!stderr.trim()) return;
+  for (const line of stderr.split(/\r?\n/)) {
+    if (line.trim()) console.error(`  ${line}`);
+  }
+}
+
+/**
+ * Re-run the Hermes env-file secret-boundary validator against a running
+ * gateway, before the probe path returns control to the caller. The
+ * relaunch path already runs the same validator inline as part of
+ * `buildRecoveryScript`, but the probe path returns early as soon as the
+ * gateway is reported healthy, so a poisoned `.env` injected after cold
+ * start would otherwise never be re-evaluated. The check is invoked via
+ * `openshell sandbox exec` (root) so the validator's kill snippet can
+ * actually signal the gateway-user process when refusing — a sandbox-user
+ * SSH shell cannot (test/e2e-gateway-isolation.sh test 13). Every
+ * refusal diagnostic — validator `[SECURITY]` stderr, the helper's own
+ * context line, and the remediation hint — is written to `console.error`
+ * unconditionally, so the offending key (e.g. `TELEGRAM_BOT_TOKEN (line
+ * N)`) and the reason for refusal always reach the operator, including
+ * on the quiet probe/recover path. Returns `null` only when the persisted
+ * sandbox registry entry is not Hermes (no boundary to enforce). When
+ * the registry says Hermes but the in-memory agent definition failed to
+ * load (`getSessionAgent()` returned `null` from its catch path), the
+ * helper fails safe with an inconclusive refusal rather than silently
+ * skipping the boundary. A running Hermes gateway whose root exec
+ * channel is unreachable is also treated as a fail-safe inconclusive
+ * refusal rather than a healthy path. Non-zero validator status without
+ * a `SECRET_BOUNDARY_REFUSED` marker is reported as inconclusive, not as
+ * a raw-secret refusal, so a shell or validator crash does not
+ * masquerade as a poisoned env file.
+ */
+function enforceHermesSecretBoundaryOnRunningGateway(
+  sandboxName: string,
+  agent: ReturnType<typeof agentRuntime.getSessionAgent>,
+): HermesSecretBoundaryEnforcement | null {
+  const persistedAgent = registry.getSandbox(sandboxName)?.agent;
+  if (persistedAgent !== "hermes") return null;
+  if (!isHermesAgent(agent)) {
+    console.error("");
+    console.error(
+      `  ${R}Hermes agent definition could not be loaded for sandbox '${sandboxName}'.${R}`,
+    );
+    console.error("  Refusing recovery to keep the validator-enforced boundary intact.");
+    return { refused: true, reason: "inconclusive", stderr: "" };
+  }
+  const script = buildHermesEnvFileBoundaryStandaloneCheck();
+  const result = executeSandboxExecCommand(sandboxName, script, 30000);
+  if (!result) {
+    console.error("");
+    console.error(
+      `  ${R}Secret-boundary check could not run against the Hermes gateway in '${sandboxName}'.${R}`,
+    );
+    console.error("  Refusing recovery to keep the validator-enforced boundary intact.");
+    return { refused: true, reason: "inconclusive", stderr: "" };
+  }
+  const stdoutMarker = result.stdout
+    .split(/\r?\n/)
+    .reverse()
+    .find((line) => line.trim().startsWith("SECRET_BOUNDARY_"));
+  if (stdoutMarker === SECRET_BOUNDARY_REFUSED_MARKER) {
+    printValidatorStderr(result.stderr);
+    console.error("");
+    console.error(
+      `  ${R}Secret-boundary check refused recovery of Hermes gateway in '${sandboxName}'.${R}`,
+    );
+    console.error("  /sandbox/.hermes/.env contains raw secret-shaped values. Replace them with");
+    console.error(
+      "  openshell:resolve:env:<name> placeholders and re-run `nemoclaw <sandbox> recover`.",
+    );
+    return { refused: true, reason: "raw-secret", stderr: result.stderr };
+  }
+  if (stdoutMarker === SECRET_BOUNDARY_OK_MARKER) {
+    return { refused: false };
+  }
+  if (stdoutMarker === SECRET_BOUNDARY_VALIDATOR_MISSING_MARKER) {
+    console.error(
+      `  [boundary] Hermes secret-boundary validator missing in sandbox '${sandboxName}'; recover proceeded without re-evaluating /sandbox/.hermes/.env. Re-image the sandbox to enable per-run enforcement.`,
+    );
+    return { refused: false };
+  }
+  printValidatorStderr(result.stderr);
+  console.error("");
+  console.error(
+    `  ${R}Secret-boundary check did not complete cleanly for Hermes gateway in '${sandboxName}'.${R}`,
+  );
+  console.error(
+    "  Refusing recovery; inspect the validator output above before re-running `nemoclaw <sandbox> recover`.",
+  );
+  return { refused: true, reason: "inconclusive", stderr: result.stderr };
+}
+
 /**
  * Detect and recover from a sandbox that survived a gateway restart but
  * whose OpenClaw processes are not running. Also re-establishes the
  * host-side dashboard port-forward when it has gone dead independently
  * of the gateway. Returns an object describing the outcome:
- * `{ checked, wasRunning, recovered, forwardRecovered }`.
+ * `{ checked, wasRunning, recovered, forwardRecovered, secretBoundaryRefused?, secretBoundaryReason? }`.
  */
 export function checkAndRecoverSandboxProcesses(
   sandboxName: string,
@@ -499,6 +608,19 @@ export function checkAndRecoverSandboxProcesses(
   }
   const recoveryAgent = agentRuntime.getSessionAgent(sandboxName);
   const recoveryPort = resolveSandboxDashboardPort(sandboxName);
+  if (running) {
+    const enforcement = enforceHermesSecretBoundaryOnRunningGateway(sandboxName, recoveryAgent);
+    if (enforcement?.refused) {
+      return {
+        checked: true,
+        wasRunning: true,
+        recovered: false,
+        forwardRecovered: false,
+        secretBoundaryRefused: true,
+        secretBoundaryReason: enforcement.reason,
+      };
+    }
+  }
   if (running) {
     // Gateway is alive but the host-side forward can still be dead or
     // owned by another sandbox. Probe and re-establish only when