Skip to content

fix(sandbox): harden PR 6065 advisor follow-ups#6068

Merged
ericksoa merged 10 commits into
mainfrom
fix/6065-advisor-followups
Jul 8, 2026
Merged

fix(sandbox): harden PR 6065 advisor follow-ups#6068
ericksoa merged 10 commits into
mainfrom
fix/6065-advisor-followups

Conversation

@jyaunches

@jyaunches jyaunches commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Follow-up to merged PR #6065 to address the unaddressed PR Review Advisor warnings:

  • Adds safe gateway-log append helper for PID 1/root recovery warnings; refuses symlink/non-regular/replaced log targets instead of appending through them.
  • Adds guard-chain recovery tests proving regular log append still works and unsafe targets are refused.
  • Adds focused reconciliation coverage proving NEMOCLAW_MODEL_OVERRIDE wins over conflicting gateway state, plus no-override reconciliation still runs.

Test plan

  • npx vitest run --project integration test/nemoclaw-start-reconcile.test.ts test/nemoclaw-start-guard-recovery.test.ts

Refs #6065

Summary by CodeRabbit

  • Bug Fixes
    • Hardened gateway log writing to refuse unsafe targets (non-regular, swapped, invalid), while sanitizing and newline-terminating log lines in UTF-8.
    • Recovery now records restoration warnings in the gateway log when safe, without blocking startup if logging fails.
    • Reconcile behavior improved: explicit model overrides are preserved; otherwise configuration is reconciled from the live gateway model.
  • Tests
    • Expanded gateway recovery coverage with regular/symlink/directory/missing log scenarios, validating behavior via gateway-log contents.
    • Added reconcile tests for overridden vs non-overridden behavior.
    • Updated gateway health watchdog tests to direct critical logging to controlled per-test log files.

Advisor disposition / E2E evidence

PR Review Advisor PRA-1 / PRA-2 runtime-validation warnings are resolved by the combination of in-PR helper hardening coverage plus live E2E evidence:

  • Source boundary: append_openclaw_gateway_log_line is the permanent PID 1 append boundary and hardcodes canonical /tmp/gateway.log; it intentionally refuses symlink, non-regular, replaced, or missing log targets rather than honoring inherited path seams.
  • Source-fix constraint: startup remains responsible for pre-creating canonical /tmp/gateway.log; recovery/watchdog append paths must not create a missing log because creation at recovery time would re-open the unsafe target/ownership ambiguity this PR removes.
  • Regression/unit coverage: test/nemoclaw-start-guard-recovery.test.ts covers regular append, symlink refusal, non-regular refusal, missing-log no-create, replaced-target refusal, inherited env ignored, and newline sanitization. test/nemoclaw-start-gateway-health.test.ts covers watchdog CRITICAL breadcrumb append through the same safe helper.
  • Runtime validation: current-head E2E revalidation passed on 9cda35bdaa6d426d160d262ac35c99c7449c1560: run https://github.com/NVIDIA/NemoClaw/actions/runs/28874860688 passed required jobs gateway-guard-recovery and issue-2478-crash-loop-recovery, and run https://github.com/NVIDIA/NemoClaw/actions/runs/28874896411 passed required target ubuntu-repo-cloud-openclaw. The earlier same-PR artifact for issue-2478-crash-loop-recovery showed the real sandbox respawn path emitted [gateway-recovery] WARNING ... (#2478/#2701) and mirrored the same line as [gateway-log:] [gateway-recovery] WARNING ..., proving the real PID 1 startup/recovery lifecycle pre-created and preserved the canonical gateway log before append.
  • Removal condition: keep the helper-level contract tests until gateway recovery/watchdog logging is no longer routed through /tmp/gateway.log or the startup ownership contract is replaced by a different diagnostics channel with equivalent live E2E coverage.

@github-code-quality

github-code-quality Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in the branch is 96%. Coverage data for the branch is not yet available.

Show a code coverage summary of the most covered files.
File 15634ef +/-
nemoclaw/src/se...cret-scanner.ts 100%
nemoclaw/src/commands/slash.ts 100%
nemoclaw/src/li...bprocess-env.ts 100%
nemoclaw/src/bl...eprint/state.ts 98%
nemoclaw/src/onboard/config.ts 98%
nemoclaw/src/bl...int/snapshot.ts 97%
nemoclaw/src/bl...print/runner.ts 95%
nemoclaw/src/co...ration-state.ts 94%
nemoclaw/src/bl...ate-networks.ts 94%
nemoclaw/src/index.ts 94%

TypeScript / code-coverage/cli

The overall coverage in the branch is 76%. Coverage data for the branch is not yet available.

Show a code coverage summary of the most covered files.
File 15634ef +/-
src/lib/onboard/preflight.ts 82%
src/lib/state/o...oard-session.ts 82%
src/lib/actions...all/run-plan.ts 81%
src/lib/actions...licy-channel.ts 79%
src/lib/actions...box/snapshot.ts 79%
src/lib/state/sandbox.ts 75%
src/lib/onboard...er-gpu-patch.ts 69%
src/lib/policy/index.ts 65%
src/lib/shields/index.ts 60%
src/lib/onboard.ts 28%

Updated July 08, 2026 17:47 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@github-actions

github-actions Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor (Nemotron Ultra) — No blocking findings

Merge posture: No blocking advisor findings
Primary next action: Add or justify PRA-T1 and any related test follow-ups.
Open items: 0 required · 0 warnings · 2 suggestions · 1 test follow-up
Since last review: 1 prior item resolved · 0 still apply · 0 new items found

Action checklist

  • PRA-T1 Add or justify test follow-up: Add integration test for watchdog CRITICAL kill breadcrumb reaching gateway log
  • PRA-1 In-scope improvement: Add integration test for watchdog CRITICAL kill breadcrumb reaching gateway log in test/nemoclaw-start-guard-recovery.test.ts:255
  • PRA-2 In-scope improvement: Document the hardcoded /tmp/gateway.log contract in the helper's doc comment in scripts/nemoclaw-start.sh:4470

Findings index

ID Severity Category Location Required action
PRA-1 Improvement tests test/nemoclaw-start-guard-recovery.test.ts:255 Add a test in nemoclaw-start-gateway-health.test.ts that runs the watchdog kill scenario and asserts the CRITICAL message appears in the test gateway.log file.
PRA-2 Improvement docs scripts/nemoclaw-start.sh:4470 Add a shell comment above append_openclaw_gateway_log_line() stating: 'Hardcodes /tmp/gateway.log. Ignores _NEMOCLAW_GATEWAY_LOG. Never creates the file. Production startup owns creation via _nemoclaw_safe_create_tmp_file.'
Review findings by urgency: 0 required fixes, 0 items to resolve/justify, 2 in-scope improvements

⚠️ Resolve or justify before merge

Investigate these in the current review; either fix them, explain why they are not applicable, or document the accepted risk.

  • None.

💡 In-scope improvements

These are lower-risk, not throwaway. Prefer fixing them in this PR when they are local to changed code; defer only with rationale or a linked follow-up.

PRA-1 Improvement — Add integration test for watchdog CRITICAL kill breadcrumb reaching gateway log

  • Location: test/nemoclaw-start-guard-recovery.test.ts:255
  • Category: tests
  • Problem: Watchdog tests verify stderr logging but don't assert the CRITICAL message reaches the gateway log file via the new helper. The guard-recovery test 'mirrors the guard-chain restore warning into the gateway log file' covers the recovery path; watchdog kill path should have equivalent coverage.
  • Impact: Without this, a regression in watchdog log append (e.g., helper signature change) wouldn't be caught until E2E.
  • Suggested action: Add a test in nemoclaw-start-gateway-health.test.ts that runs the watchdog kill scenario and asserts the CRITICAL message appears in the test gateway.log file.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Run the watchdog kill test and grep the wedgeLog for '[gateway-watchdog] CRITICAL'
  • Missing regression test: Watchdog kill scenario asserts CRITICAL breadcrumb in gateway.log via append_openclaw_gateway_log_line
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: test/nemoclaw-start-gateway-health.test.ts runWatchdog() creates wedgeLogFile but only checks wedgeLog for CRITICAL in stderr, not the file

PRA-2 Improvement — Document the hardcoded /tmp/gateway.log contract in the helper's doc comment

  • Location: scripts/nemoclaw-start.sh:4470
  • Category: docs
  • Problem: The Python helper has a good inline comment explaining the source boundary, but the shell function wrapper lacks a doc comment stating that it ignores _NEMOCLAW_GATEWAY_LOG and always uses /tmp/gateway.log. This contract is critical for security.
  • Impact: Future maintainers might re-introduce the env var or change the path without understanding the security boundary.
  • Suggested action: Add a shell comment above append_openclaw_gateway_log_line() stating: 'Hardcodes /tmp/gateway.log. Ignores _NEMOCLAW_GATEWAY_LOG. Never creates the file. Production startup owns creation via _nemoclaw_safe_create_tmp_file.'
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Read the function header in scripts/nemoclaw-start.sh
  • Missing regression test: N/A - documentation improvement
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Function at line 4470 has no shell-level doc comment explaining the hardcoded path contract
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Add integration test for watchdog CRITICAL kill breadcrumb reaching gateway log — Add a test in nemoclaw-start-gateway-health.test.ts that runs the watchdog kill scenario and asserts the CRITICAL message appears in the test gateway.log file.
Since last review details

Current findings, using the urgency labels above:

PRA-1 Improvement — Add integration test for watchdog CRITICAL kill breadcrumb reaching gateway log

  • Location: test/nemoclaw-start-guard-recovery.test.ts:255
  • Category: tests
  • Problem: Watchdog tests verify stderr logging but don't assert the CRITICAL message reaches the gateway log file via the new helper. The guard-recovery test 'mirrors the guard-chain restore warning into the gateway log file' covers the recovery path; watchdog kill path should have equivalent coverage.
  • Impact: Without this, a regression in watchdog log append (e.g., helper signature change) wouldn't be caught until E2E.
  • Suggested action: Add a test in nemoclaw-start-gateway-health.test.ts that runs the watchdog kill scenario and asserts the CRITICAL message appears in the test gateway.log file.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Run the watchdog kill test and grep the wedgeLog for '[gateway-watchdog] CRITICAL'
  • Missing regression test: Watchdog kill scenario asserts CRITICAL breadcrumb in gateway.log via append_openclaw_gateway_log_line
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: test/nemoclaw-start-gateway-health.test.ts runWatchdog() creates wedgeLogFile but only checks wedgeLog for CRITICAL in stderr, not the file

PRA-2 Improvement — Document the hardcoded /tmp/gateway.log contract in the helper's doc comment

  • Location: scripts/nemoclaw-start.sh:4470
  • Category: docs
  • Problem: The Python helper has a good inline comment explaining the source boundary, but the shell function wrapper lacks a doc comment stating that it ignores _NEMOCLAW_GATEWAY_LOG and always uses /tmp/gateway.log. This contract is critical for security.
  • Impact: Future maintainers might re-introduce the env var or change the path without understanding the security boundary.
  • Suggested action: Add a shell comment above append_openclaw_gateway_log_line() stating: 'Hardcodes /tmp/gateway.log. Ignores _NEMOCLAW_GATEWAY_LOG. Never creates the file. Production startup owns creation via _nemoclaw_safe_create_tmp_file.'
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Read the function header in scripts/nemoclaw-start.sh
  • Missing regression test: N/A - documentation improvement
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Function at line 4470 has no shell-level doc comment explaining the hardcoded path contract

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@github-actions

github-actions Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Changes requested

Merge posture: Do not merge yet
Primary next action: Resolve or justify PRA-1: Source-of-truth review needed: Gateway recovery/watchdog breadcrumb append to /tmp/gateway.log.
Open items: 0 required · 2 warnings · 0 suggestions · 8 test follow-ups
Since last review: 1 prior item resolved · 1 still applies · 0 new items found

Action checklist

  • PRA-1 Resolve or justify: Source-of-truth review needed: Gateway recovery/watchdog breadcrumb append to /tmp/gateway.log
  • PRA-2 Resolve or justify: Add runtime validation for canonical /tmp/gateway.log startup contract in scripts/nemoclaw-start.sh:4468
  • PRA-T1 Add or justify test follow-up: Runtime validation
  • PRA-T2 Add or justify test follow-up: Runtime validation
  • PRA-T3 Add or justify test follow-up: Runtime validation
  • PRA-T4 Add or justify test follow-up: Add runtime validation for canonical /tmp/gateway.log startup contract
  • PRA-T5 Add or justify test follow-up: Acceptance clause
  • PRA-T6 Add or justify test follow-up: Acceptance clause
  • PRA-T7 Add or justify test follow-up: Acceptance clause
  • PRA-T8 Add or justify test follow-up: Acceptance clause

Findings index

ID Severity Category Location Required action
PRA-1 Resolve/justify architecture Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
PRA-2 Resolve/justify tests scripts/nemoclaw-start.sh:4468 Add or identify in-repo runtime/integration validation for the real entrypoint lifecycle without replacing the helper's canonical /tmp/gateway.log path, or explicitly justify why the extracted-function harnesses are sufficient for this startup ownership and recovery-ordering boundary. Keep the new helper-level negative tests; the runtime check should complement them.
Review findings by urgency: 0 required fixes, 2 items to resolve/justify, 0 in-scope improvements

⚠️ Resolve or justify before merge

Investigate these in the current review; either fix them, explain why they are not applicable, or document the accepted risk.

PRA-1 Resolve/justify — Source-of-truth review needed: Gateway recovery/watchdog breadcrumb append to /tmp/gateway.log

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: Helper-level regression tests cover regular append, symlink refusal, non-regular refusal, missing-log no-create, replacement refusal, FIFO nonblocking refusal, inherited-env ignored, newline sanitization, and watchdog CRITICAL append through the extracted helper. Missing follow-up: in-repo runtime validation that the real root and non-root startup paths pre-create canonical /tmp/gateway.log with expected ownership/mode before append attempts.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: scripts/nemoclaw-start.sh:4476-4480 states the source boundary; scripts/nemoclaw-start.sh:4496-4499 refuses to create missing logs; test/nemoclaw-start-guard-recovery.test.ts covers unsafe helper states through extracted-function harnesses; no changed test exercises the real canonical startup lifecycle without replacing the helper path.

PRA-2 Resolve/justify — Add runtime validation for canonical /tmp/gateway.log startup contract

  • Location: scripts/nemoclaw-start.sh:4468
  • Category: tests
  • Problem: The new append helper is hardened, but it deliberately exits successfully when /tmp/gateway.log is missing and relies on real root/non-root PID 1 startup to pre-create the canonical log before recovery or watchdog breadcrumbs are emitted. The changed tests extract the helper and rewrite its hardcoded /tmp/gateway.log to temporary paths, so they do not prove the production startup ordering, owner, or mode contract.
  • Impact: A future startup regression could silently drop guard-chain or watchdog recovery breadcrumbs while the helper-level tests still pass. That weakens security-relevant diagnostics for sandbox recovery and can hide owner/mode drift in the canonical gateway log path.
  • Recommended action: Add or identify in-repo runtime/integration validation for the real entrypoint lifecycle without replacing the helper's canonical /tmp/gateway.log path, or explicitly justify why the extracted-function harnesses are sufficient for this startup ownership and recovery-ordering boundary. Keep the new helper-level negative tests; the runtime check should complement them.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Read append_openclaw_gateway_log_line in scripts/nemoclaw-start.sh, the non-root _nemoclaw_safe_create_tmp_file /tmp/gateway.log startup path at scripts/nemoclaw-start.sh:4794, the root _nemoclaw_safe_create_tmp_file /tmp/gateway.log 644 gateway:gateway startup path at scripts/nemoclaw-start.sh:4934, and the changed extractGatewayLogAppendFunction helpers in test/nemoclaw-start-guard-recovery.test.ts and test/nemoclaw-start-gateway-health.test.ts; confirm whether any automated test boots the real startup lifecycle without path rewriting.
  • Missing regression test: Add behavior-specific coverage such as: root-mode startup pre-creates gateway-owned canonical /tmp/gateway.log before guard recovery mirrors the missing or unsafe guard-chain warning; non-root startup pre-creates canonical /tmp/gateway.log before the gateway watchdog appends a CRITICAL alive-but-deaf breadcrumb; production PID 1 ignores inherited _NEMOCLAW_GATEWAY_LOG and _NEMOCLAW_GATEWAY_LOG_TEST_MODE while preserving canonical /tmp/gateway.log owner and mode during append.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Read append_openclaw_gateway_log_line in scripts/nemoclaw-start.sh, the non-root _nemoclaw_safe_create_tmp_file /tmp/gateway.log startup path at scripts/nemoclaw-start.sh:4794, the root _nemoclaw_safe_create_tmp_file /tmp/gateway.log 644 gateway:gateway startup path at scripts/nemoclaw-start.sh:4934, and the changed extractGatewayLogAppendFunction helpers in test/nemoclaw-start-guard-recovery.test.ts and test/nemoclaw-start-gateway-health.test.ts; confirm whether any automated test boots the real startup lifecycle without path rewriting.
  • Evidence: scripts/nemoclaw-start.sh:4496-4499 treats FileNotFoundError as success and does not create the log; scripts/nemoclaw-start.sh:4122 and :4519 route watchdog/recovery breadcrumbs through the helper; startup pre-create calls still exist at :4794 and :4934; changed tests pre-create temporary log files and inject extracted helper bodies with the log path replaced.

💡 In-scope improvements

These are lower-risk, not throwaway. Prefer fixing them in this PR when they are local to changed code; defer only with rationale or a linked follow-up.

  • None.
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Runtime validation — Root-mode startup pre-creates gateway-owned canonical /tmp/gateway.log with mode 0644 before guard-chain recovery mirrors the missing or unsafe guard-chain warning.. The changed source is a sandbox PID 1 startup/recovery/watchdog shell path. The added helper-level and extracted-function tests are strong, but they do not prove the production root/non-root entrypoint pre-creates canonical /tmp/gateway.log before the new append helper is invoked.
  • PRA-T2 Runtime validation — Non-root startup pre-creates canonical /tmp/gateway.log with mode 0644 before the gateway watchdog appends a CRITICAL alive-but-deaf breadcrumb.. The changed source is a sandbox PID 1 startup/recovery/watchdog shell path. The added helper-level and extracted-function tests are strong, but they do not prove the production root/non-root entrypoint pre-creates canonical /tmp/gateway.log before the new append helper is invoked.
  • PRA-T3 Runtime validation — Production PID 1 ignores inherited _NEMOCLAW_GATEWAY_LOG and _NEMOCLAW_GATEWAY_LOG_TEST_MODE while preserving canonical /tmp/gateway.log owner and mode during recovery/watchdog append.. The changed source is a sandbox PID 1 startup/recovery/watchdog shell path. The added helper-level and extracted-function tests are strong, but they do not prove the production root/non-root entrypoint pre-creates canonical /tmp/gateway.log before the new append helper is invoked.
  • PRA-T4 Add runtime validation for canonical /tmp/gateway.log startup contract — Add or identify in-repo runtime/integration validation for the real entrypoint lifecycle without replacing the helper's canonical /tmp/gateway.log path, or explicitly justify why the extracted-function harnesses are sufficient for this startup ownership and recovery-ordering boundary. Keep the new helper-level negative tests; the runtime check should complement them.
  • PRA-T5 Acceptance clause — Follow-up to merged PR fix(sandbox): honor model override in reconcile and log guard-chain warning to file #6065 to address the unaddressed PR Review Advisor warnings: — add test evidence or identify existing coverage. The source-boundary/helper-hardening warning is largely addressed by code comments plus helper-level tests for symlink, non-regular, missing, replaced, FIFO, inherited-env, and sanitization cases. The prior runtime-validation warning still applies because the in-repo tests do not exercise the real canonical /tmp/gateway.log startup pre-create lifecycle.
  • PRA-T6 Acceptance clause — Runtime validation: current-head E2E revalidation passed on `9cda35bdaa6d426d160d262ac35c99c7449c1560`: run https://github.com/NVIDIA/NemoClaw/actions/runs/28874860688 passed required jobs `gateway-guard-recovery` and `issue-2478-crash-loop-recovery`, and run https://github.com/NVIDIA/NemoClaw/actions/runs/28874896411 passed required target `ubuntu-repo-cloud-openclaw`. — add test evidence or identify existing coverage. External E2E job pass/fail and artifacts are not used as code-review evidence here. The in-repo diff contains helper-level and extracted-function tests but no changed runtime/integration test that exercises the real root/non-root PID 1 startup lifecycle with canonical /tmp/gateway.log.
  • PRA-T7 Acceptance clause — 1. Wrap the ciao NetworkManager calls in try/catch inside OpenClaw, fall back to no mDNS if `os.networkInterfaces()` throws. Probably the smallest diff. — add test evidence or identify existing coverage. This linked issue [DGX Spark] Gateway crash loop on startup: @homebridge/ciao networkInterfaces() returns EPERM in OpenShell sandbox #2478 possible-fix clause is not implemented in the changed files; this PR does not modify OpenClaw ciao code and appears to be addressing PR fix(sandbox): honor model override in reconcile and log guard-chain warning to file #6065 advisor follow-ups rather than the original ciao implementation.
  • PRA-T8 Acceptance clause — 2. Add an `OPENCLAW_DISABLE_MDNS=1` env var (or a config flag) that skips loading ciao entirely. Most explicit user-facing fix. — add test evidence or identify existing coverage. No OPENCLAW_DISABLE_MDNS env var or config flag is added in the changed files. This appears to be linked-issue context rather than this PR's direct acceptance target.
Since last review details

Current findings, using the urgency labels above:

PRA-1 Resolve/justify — Source-of-truth review needed: Gateway recovery/watchdog breadcrumb append to /tmp/gateway.log

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: Helper-level regression tests cover regular append, symlink refusal, non-regular refusal, missing-log no-create, replacement refusal, FIFO nonblocking refusal, inherited-env ignored, newline sanitization, and watchdog CRITICAL append through the extracted helper. Missing follow-up: in-repo runtime validation that the real root and non-root startup paths pre-create canonical /tmp/gateway.log with expected ownership/mode before append attempts.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: scripts/nemoclaw-start.sh:4476-4480 states the source boundary; scripts/nemoclaw-start.sh:4496-4499 refuses to create missing logs; test/nemoclaw-start-guard-recovery.test.ts covers unsafe helper states through extracted-function harnesses; no changed test exercises the real canonical startup lifecycle without replacing the helper path.

PRA-2 Resolve/justify — Add runtime validation for canonical /tmp/gateway.log startup contract

  • Location: scripts/nemoclaw-start.sh:4468
  • Category: tests
  • Problem: The new append helper is hardened, but it deliberately exits successfully when /tmp/gateway.log is missing and relies on real root/non-root PID 1 startup to pre-create the canonical log before recovery or watchdog breadcrumbs are emitted. The changed tests extract the helper and rewrite its hardcoded /tmp/gateway.log to temporary paths, so they do not prove the production startup ordering, owner, or mode contract.
  • Impact: A future startup regression could silently drop guard-chain or watchdog recovery breadcrumbs while the helper-level tests still pass. That weakens security-relevant diagnostics for sandbox recovery and can hide owner/mode drift in the canonical gateway log path.
  • Recommended action: Add or identify in-repo runtime/integration validation for the real entrypoint lifecycle without replacing the helper's canonical /tmp/gateway.log path, or explicitly justify why the extracted-function harnesses are sufficient for this startup ownership and recovery-ordering boundary. Keep the new helper-level negative tests; the runtime check should complement them.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Read append_openclaw_gateway_log_line in scripts/nemoclaw-start.sh, the non-root _nemoclaw_safe_create_tmp_file /tmp/gateway.log startup path at scripts/nemoclaw-start.sh:4794, the root _nemoclaw_safe_create_tmp_file /tmp/gateway.log 644 gateway:gateway startup path at scripts/nemoclaw-start.sh:4934, and the changed extractGatewayLogAppendFunction helpers in test/nemoclaw-start-guard-recovery.test.ts and test/nemoclaw-start-gateway-health.test.ts; confirm whether any automated test boots the real startup lifecycle without path rewriting.
  • Missing regression test: Add behavior-specific coverage such as: root-mode startup pre-creates gateway-owned canonical /tmp/gateway.log before guard recovery mirrors the missing or unsafe guard-chain warning; non-root startup pre-creates canonical /tmp/gateway.log before the gateway watchdog appends a CRITICAL alive-but-deaf breadcrumb; production PID 1 ignores inherited _NEMOCLAW_GATEWAY_LOG and _NEMOCLAW_GATEWAY_LOG_TEST_MODE while preserving canonical /tmp/gateway.log owner and mode during append.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Read append_openclaw_gateway_log_line in scripts/nemoclaw-start.sh, the non-root _nemoclaw_safe_create_tmp_file /tmp/gateway.log startup path at scripts/nemoclaw-start.sh:4794, the root _nemoclaw_safe_create_tmp_file /tmp/gateway.log 644 gateway:gateway startup path at scripts/nemoclaw-start.sh:4934, and the changed extractGatewayLogAppendFunction helpers in test/nemoclaw-start-guard-recovery.test.ts and test/nemoclaw-start-gateway-health.test.ts; confirm whether any automated test boots the real startup lifecycle without path rewriting.
  • Evidence: scripts/nemoclaw-start.sh:4496-4499 treats FileNotFoundError as success and does not create the log; scripts/nemoclaw-start.sh:4122 and :4519 route watchdog/recovery breadcrumbs through the helper; startup pre-create calls still exist at :4794 and :4934; changed tests pre-create temporary log files and inject extracted helper bodies with the log path replaced.

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@github-actions

github-actions Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: gateway-guard-recovery, issue-2478-crash-loop-recovery
Optional E2E: sandbox-operations

Dispatch hint: gateway-guard-recovery,issue-2478-crash-loop-recovery

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

Optional E2E

  • sandbox-operations (live sandbox E2E): Useful adjacent confidence for broad sandbox lifecycle behavior because scripts/nemoclaw-start.sh participates in status/logs/recovery and in-sandbox OpenClaw gateway process recovery, but the more focused recovery targets above cover the changed guard-chain path.

New E2E recommendations

  • security-boundary (high): The PR adds unit coverage for unsafe /tmp/gateway.log symlink/directory/replacement/FIFO cases, but there is no existing live E2E target that stages an unsafe gateway log inside a real sandbox and verifies PID 1 recovery refuses to write through it while still recovering safely.
    • Suggested test: Add a live gateway-log-path-boundary recovery target that creates unsafe /tmp/gateway.log variants in a sandbox, triggers connect --probe-only recovery, and asserts no sensitive target is modified.
  • gateway-watchdog (medium): The existing tests mock the alive-but-deaf listener-drop watchdog path. No existing live E2E appears to force a real gateway process to drop its HTTP listener while staying alive and prove the watchdog kills it for respawn.
    • Suggested test: Add a live gateway-serving-watchdog target that wedges only the gateway HTTP listener, waits for the watchdog CRITICAL breadcrumb, and asserts the respawn loop restores a healthy gateway.

Dispatch hint

  • Workflow: .github/workflows/e2e.yaml
  • jobs input: gateway-guard-recovery,issue-2478-crash-loop-recovery

@github-actions

github-actions Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

E2E Target Recommendation

Required E2E targets: gateway-guard-recovery, ubuntu-repo-cloud-openclaw
Optional E2E targets: gateway-health-honest

Dispatch required E2E targets:

  • gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=gateway-guard-recovery
  • gh workflow run e2e.yaml --ref <pr-head-ref> --field targets=ubuntu-repo-cloud-openclaw

Workflow run

Full E2E target advisor summary

E2E Target Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E targets

  • gateway-guard-recovery: scripts/nemoclaw-start.sh changes PID 1 gateway guard-chain recovery and gateway log append behavior; the gateway-guard-recovery live job is the dedicated recovery target for this surface.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=gateway-guard-recovery
  • ubuntu-repo-cloud-openclaw: scripts/nemoclaw-start.sh is shared startup machinery for the OpenClaw container/gateway; the baseline Ubuntu OpenClaw target exercises normal onboarding, gateway launch, health, and inference after the startup changes.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field targets=ubuntu-repo-cloud-openclaw

Optional E2E targets

  • gateway-health-honest: Adjacent gateway health live coverage for startup/health reporting paths touched by the gateway watchdog and recovery changes.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=gateway-health-honest

Relevant changed files

  • scripts/nemoclaw-start.sh

@wscurran wscurran added area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery bug-fix PR fixes a bug or regression labels Jul 1, 2026
@wscurran

wscurran commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

@coderabbitai

coderabbitai Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds a fail-closed gateway log append helper, routes startup logging through it, expands gateway-log recovery and watchdog test coverage, and adds reconcile tests for model override precedence.

Changes

Gateway log appending safety

Layer / File(s) Summary
Fail-closed log append helper and guard-chain wiring
scripts/nemoclaw-start.sh
Adds append_openclaw_gateway_log_line using a python3 -I block that rejects non-regular files and checks device/inode before writing via O_APPEND; routes the guard-chain warning through this helper on a best-effort basis.
Watchdog log target wiring
test/nemoclaw-start-gateway-health.test.ts
Parameterizes the extracted gateway-log append helper in watchdog tests, redirects generated watchdog logging to the per-test gateway log, and initializes the per-test log file for the watchdog cases.
Gateway-log recovery harness and append checks
test/nemoclaw-start-guard-recovery.test.ts
Extends the recovery harness with gateway-log target variants, wires _NEMOCLAW_GATEWAY_LOG and the extracted append helper into the generated script, switches the main assertion to read the gateway log file, and adds tests for unsafe, replaced, missing, and sanitized gateway-log writes.

Reconcile model override tests

Layer / File(s) Summary
Override precedence and gateway-derived model reconciliation tests
test/nemoclaw-start-reconcile.test.ts
Adds tests confirming an explicit NEMOCLAW_MODEL_OVERRIDE preserves config against a conflicting gateway model, and that without an override the gateway model updates config fields and hash.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Suggested labels: security

Suggested reviewers: kjw3, cv

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is related to the PR as a follow-up hardening change, though it is broader than the specific gateway-log and reconcile updates.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/6065-advisor-followups

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
test/nemoclaw-start-reconcile.test.ts (1)

226-246: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Solid override-precedence test; minor clarity nitpick.

Using NEMOCLAW_MODEL_OVERRIDE: "anthropic/claude-sonnet-4-6" — identical to the initial config's existing primary — still proves the override wins (via the unchanged-hash assertion confirming no write occurred), but a distinct override value would make the intent more immediately obvious to future readers without relying on the hash check alone.

♻️ Optional: differentiate override from initial config's model
     const initial = {
-      agents: { defaults: { model: { primary: "anthropic/claude-sonnet-4-6" } } },
+      agents: { defaults: { model: { primary: "anthropic/claude-sonnet-4-5" } } },
       models: {
         providers: {
           inference: {
             api: "openai-completions",
-            models: [{ id: "anthropic/claude-sonnet-4-6", name: "anthropic/claude-sonnet-4-6" }],
+            models: [{ id: "anthropic/claude-sonnet-4-5", name: "anthropic/claude-sonnet-4-5" }],
           },
         },
       },
     };
     const { result, config, hash } = runReconcile(initial, {
       env: { NEMOCLAW_MODEL_OVERRIDE: "anthropic/claude-sonnet-4-6" },
       gatewayModel: "nvidia/nemotron-3-super-120b-a12b",
     });
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/nemoclaw-start-reconcile.test.ts` around lines 226 - 246, The
override-precedence test in runReconcile is valid, but the chosen
NEMOCLAW_MODEL_OVERRIDE value matches the initial agents.defaults.model.primary
and makes the intent less obvious. Update the test to use a distinct override
model string while keeping the same assertions on result, config, and hash, so
it is clear that the explicit override in runReconcile wins over the live
gateway’s conflicting model.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/nemoclaw-start-reconcile.test.ts`:
- Around line 226-246: The override-precedence test in runReconcile is valid,
but the chosen NEMOCLAW_MODEL_OVERRIDE value matches the initial
agents.defaults.model.primary and makes the intent less obvious. Update the test
to use a distinct override model string while keeping the same assertions on
result, config, and hash, so it is clear that the explicit override in
runReconcile wins over the live gateway’s conflicting model.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 947e118e-1735-492f-83d1-bf21a9ecccac

📥 Commits

Reviewing files that changed from the base of the PR and between bdcfb1f and 5bbb972.

📒 Files selected for processing (3)
  • scripts/nemoclaw-start.sh
  • test/nemoclaw-start-guard-recovery.test.ts
  • test/nemoclaw-start-reconcile.test.ts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
test/nemoclaw-start-guard-recovery.test.ts (2)

22-43: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Assert the log_file rewrite actually matched.

String.prototype.replace with a string pattern silently no-ops if the production line drifts (e.g. indentation/quoting change). If that happens, functionSource keeps local log_file="/tmp/gateway.log" and every harness invocation would lstat/open the real /tmp/gateway.log — producing confusing failures and, in the "regular" case, a stray write to the host log. A single positive assertion pins the seam.

♻️ Guard against a silent no-op rewrite
   const functionSource = extractShellFunction(source, "append_openclaw_gateway_log_line").replace(
     '  local log_file="/tmp/gateway.log"',
     `  local log_file=${JSON.stringify(gatewayLog)}`,
   );
+  expect(functionSource, "gateway log path rewrite did not match").toContain(
+    `local log_file=${JSON.stringify(gatewayLog)}`,
+  );
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/nemoclaw-start-guard-recovery.test.ts` around lines 22 - 43, The rewrite
in extractGatewayLogAppendFunction is currently a silent no-op if the expected
log_file line changes, so add an explicit assertion that the replacement of
append_openclaw_gateway_log_line succeeded before returning functionSource. Use
the existing extractShellFunction and the local log_file rewrite logic as the
seam, and verify the source no longer contains the default /tmp/gateway.log
binding after substitution so tests fail fast instead of accidentally hitting
the real host log.

317-322: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Source-text assertion; keep as a regression guard but consider anchoring on behavior.

Per path instructions, prefer observable outcomes through the public boundary over source-text assertions. This test only greps nemoclaw-start.sh, so it can't catch a routing regression that still matches the string but behaves incorrectly. It does satisfy the migration requirement of proving the superseded >> "$LOG" path is removed, so it's acceptable as a complementary guard alongside the behavioral symlink/directory/missing tests — just be aware it locks source text rather than behavior.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/nemoclaw-start-guard-recovery.test.ts` around lines 317 - 322, The
watchdog gateway-log test is still a source-text assertion, so keep it only as a
complementary regression guard and add/lean on a behavioral check instead. In
nemoclaw-start-guard-recovery.test.ts, anchor the assertion around the
observable output of the start path exercised by the public boundary rather than
only grepping nemoclaw-start.sh, using the existing symlink/directory/missing
recovery cases and the append_openclaw_gateway_log_line flow to verify routing.
Keep the removal check for the superseded redirection path, but make the primary
validation behavior-based so a future routing regression won’t slip through
while the source string still matches.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/nemoclaw-start-guard-recovery.test.ts`:
- Around line 22-43: The rewrite in extractGatewayLogAppendFunction is currently
a silent no-op if the expected log_file line changes, so add an explicit
assertion that the replacement of append_openclaw_gateway_log_line succeeded
before returning functionSource. Use the existing extractShellFunction and the
local log_file rewrite logic as the seam, and verify the source no longer
contains the default /tmp/gateway.log binding after substitution so tests fail
fast instead of accidentally hitting the real host log.
- Around line 317-322: The watchdog gateway-log test is still a source-text
assertion, so keep it only as a complementary regression guard and add/lean on a
behavioral check instead. In nemoclaw-start-guard-recovery.test.ts, anchor the
assertion around the observable output of the start path exercised by the public
boundary rather than only grepping nemoclaw-start.sh, using the existing
symlink/directory/missing recovery cases and the
append_openclaw_gateway_log_line flow to verify routing. Keep the removal check
for the superseded redirection path, but make the primary validation
behavior-based so a future routing regression won’t slip through while the
source string still matches.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b0f7979f-2635-4ce3-a9e9-44634df76313

📥 Commits

Reviewing files that changed from the base of the PR and between df5bd21 and 67d2210.

📒 Files selected for processing (2)
  • scripts/nemoclaw-start.sh
  • test/nemoclaw-start-guard-recovery.test.ts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
test/nemoclaw-start-gateway-health.test.ts (1)

50-56: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Add an assertion that the log_file substitution actually matched.

extractGatewayLogAppendFunction uses String.replace to rewrite the hardcoded /tmp/gateway.log path, but unlike the sibling extractShellFunction (which calls expect(...).not.toBe(-1)), there's no check that the replacement text was found. If the production literal ever changes (e.g., a formatting/quoting tweak in scripts/nemoclaw-start.sh), this silently no-ops: the generated helper would still target the real /tmp/gateway.log, and since the helper never creates a missing file, the test would silently fail to observe the log write (or, worse, write into a shared path) instead of failing loudly with a clear diagnostic.

🛠️ Proposed fix
 function extractGatewayLogAppendFunction(src: string, gatewayLog: string): string {
-  return extractShellFunction(src, "append_openclaw_gateway_log_line").replace(
-    '  local log_file="/tmp/gateway.log"',
-    `  local log_file=${JSON.stringify(gatewayLog)}`,
-  );
+  const fn = extractShellFunction(src, "append_openclaw_gateway_log_line");
+  const marker = '  local log_file="/tmp/gateway.log"';
+  expect(fn.includes(marker), "Expected log_file assignment in append_openclaw_gateway_log_line").toBe(true);
+  return fn.replace(marker, `  local log_file=${JSON.stringify(gatewayLog)}`);
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/nemoclaw-start-gateway-health.test.ts` around lines 50 - 56,
`extractGatewayLogAppendFunction` currently rewrites the
`append_openclaw_gateway_log_line` shell helper with a blind String.replace, so
add a check that the hardcoded `log_file` literal is actually present before
replacing it, similar to `extractShellFunction`. If the expected
`/tmp/gateway.log` text is missing, fail the test immediately with a clear
assertion so changes in `scripts/nemoclaw-start.sh` don’t silently leave the
helper pointing at the real path.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/nemoclaw-start-gateway-health.test.ts`:
- Around line 50-56: `extractGatewayLogAppendFunction` currently rewrites the
`append_openclaw_gateway_log_line` shell helper with a blind String.replace, so
add a check that the hardcoded `log_file` literal is actually present before
replacing it, similar to `extractShellFunction`. If the expected
`/tmp/gateway.log` text is missing, fail the test immediately with a clear
assertion so changes in `scripts/nemoclaw-start.sh` don’t silently leave the
helper pointing at the real path.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 36643c17-7d4b-4831-ac36-b511227c04e0

📥 Commits

Reviewing files that changed from the base of the PR and between 67d2210 and 4835e4a.

📒 Files selected for processing (2)
  • test/nemoclaw-start-gateway-health.test.ts
  • test/nemoclaw-start-guard-recovery.test.ts
💤 Files with no reviewable changes (1)
  • test/nemoclaw-start-guard-recovery.test.ts

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

E2E Target Results — ✅ All requested jobs passed

Run: 28871463038
Workflow ref: fix/6065-advisor-followups
Requested targets: (default — all supported)
Requested jobs: gateway-guard-recovery,issue-2478-crash-loop-recovery
Summary: 2 passed, 0 failed, 0 cancelled, 0 skipped

Job Result
gateway-guard-recovery ✅ success
issue-2478-crash-loop-recovery ✅ success

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

E2E Target Results — ✅ All selected jobs passed

Run: 28874896411
Workflow ref: fix/6065-advisor-followups
Requested targets: ubuntu-repo-cloud-openclaw
Requested jobs: (default — all default-enabled free-standing jobs; explicit-only jobs openshell-gateway-auth-contract, mcp-bridge-dev, hermes-gpu-startup, sandbox-rlimits-connect, and jetson-nvmap-gpu are skipped unless selected)
Summary: 1 passed, 0 failed, 0 cancelled, 0 skipped

Job Result
live ✅ success

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

E2E Target Results — ✅ All requested jobs passed

Run: 28874860688
Workflow ref: fix/6065-advisor-followups
Requested targets: (default — all supported)
Requested jobs: gateway-guard-recovery,issue-2478-crash-loop-recovery
Summary: 2 passed, 0 failed, 0 cancelled, 0 skipped

Job Result
gateway-guard-recovery ✅ success
issue-2478-crash-loop-recovery ✅ success

@cv cv added the v0.0.77 Release target label Jul 8, 2026
@ericksoa ericksoa added v0.0.78 Release target and removed v0.0.77 Release target labels Jul 8, 2026
Signed-off-by: cjagwani <cjagwani@nvidia.com>
@cjagwani

cjagwani commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Security review — exact head b9ea4851

Verdict: Code review is clean after the maintainer follow-up; approval remains pending fresh exact-head CI. The log append boundary fails closed on unsafe path types without turning diagnostic failure into a sandbox restart bypass.

# Category Severity File Result
1 TOCTOU / availability Medium, resolved scripts/nemoclaw-start.sh Added O_NONBLOCK so a regular file swapped to a FIFO after lstat cannot block root before fstat; added a behavioral FIFO-swap regression.
2 Test isolation Low, resolved gateway-health + guard-recovery tests Assert the canonical log-path rewrite matched before running extracted shell, preventing accidental writes to the host /tmp/gateway.log.

Nine-category review:

  1. Secrets and credentials — PASS. No credential material added or logged; messages are fixed operational diagnostics.
  2. Input validation and sanitization — PASS. CR/LF are flattened, the canonical path is not inherited, and argv carries the message without shell evaluation.
  3. Authentication and authorization — PASS. No auth surface changed; the PID 1 boundary does not grant callers path control.
  4. Dependencies — PASS. No dependency changes; Python runs isolated with -I.
  5. Error handling and logging — PASS. Missing/unsafe/replaced targets do not get created or followed; logging failure remains non-fatal to recovery.
  6. Cryptography and data protection — PASS. Not applicable; no cryptographic behavior changes.
  7. Configuration and secure defaults — PASS. The helper hardcodes /tmp/gateway.log, uses append-only/no-follow/close-on-exec/nonblocking flags, and preserves startup-owned creation.
  8. Security testing — PASS. Covers regular, symlink, directory, missing, regular replacement, FIFO replacement, inherited-env, and newline cases; exact-head live E2E evidence remains linked in the PR.
  9. Holistic posture — PASS. The lstat/open/fstat identity check plus nonblocking open closes the discovered swap-to-FIFO denial-of-service window; no sandbox escape, policy bypass, or secret exposure found.

Local verification on Node 22: 61/61 affected integration tests passed, bash -n passed, Biome format/lint passed, and the test-size budget passed.

Signed-off-by: cjagwani <cjagwani@nvidia.com>

@cjagwani cjagwani left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved exact head 15634efa after the nine-category security review. The canonical log path is fixed and not environment-controlled; no-follow + append + close-on-exec + nonblocking open is followed by regular-file and inode identity verification; unsafe/missing/replaced targets do not get created or written; logging failures remain non-fatal to recovery. The added FIFO-swap regression closes the discovered blocking TOCTOU case, and all exact-head required checks are green.

@ericksoa ericksoa merged commit 7bf427e into main Jul 8, 2026
41 checks passed
@ericksoa ericksoa deleted the fix/6065-advisor-followups branch July 8, 2026 17:58
@cjagwani cjagwani mentioned this pull request Jul 9, 2026
21 tasks
cv pushed a commit that referenced this pull request Jul 9, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Prepares the user documentation for NemoClaw v0.0.78 by replacing the
unreleased section with release highlights and synchronizing the
affected inference, lifecycle, messaging, and CLI reference pages with
merged behavior.

## Changes

- Publish the v0.0.78 release-notes section with links to the most
specific user guides for each shipped behavior.
- Document authoritative Deep Agents route health, Nemotron Ultra
profile behavior, and Hermes compatible-endpoint context metadata.
- Document forced rebuild recovery after total backup failure and the
ownership-safe tunnel/full-stop behavior.
- Keep command examples and shared agent variants aligned with the
current OpenClaw, Hermes, and Deep Agents interfaces.

Source mapping:

- [#3787](#3787) ->
`docs/about/release-notes.mdx`: Record reliable workspace template
seeding during sandbox startup.
- [#4960](#4960) ->
`docs/about/release-notes.mdx`: Record safer detection of rewritten
OpenClaw gateway processes.
- [#5676](#5676) ->
`docs/about/release-notes.mdx`: Record warning-tolerant agent-list JSON
handling.
- [#5857](#5857) ->
`docs/about/release-notes.mdx`: Record synchronization of explicit
OpenClaw main-agent model state.
- [#5929](#5929) ->
`docs/about/release-notes.mdx`: Record copyable SSH port-forward
guidance for remote dashboards.
- [#6068](#6068) ->
`docs/about/release-notes.mdx`: Record custom-image plugin provenance
reconciliation.
- [#6116](#6116) ->
`docs/about/release-notes.mdx`: Record live-loopback dashboard-forward
recovery.
- [#6122](#6122) ->
`docs/about/release-notes.mdx`: Announce validated, round-trippable
policy YAML output.
- [#6211](#6211) ->
`docs/manage-sandboxes/lifecycle.mdx`, `docs/reference/commands.mdx`,
`docs/about/release-notes.mdx`: Explain the explicit no-backup `rebuild
--force` recovery boundary.
- [#6283](#6283) ->
`docs/about/release-notes.mdx`: Record Hermes WebUI port alignment.
- [#6293](#6293) ->
`docs/inference/switch-inference-providers.mdx`,
`docs/about/release-notes.mdx`: Document compatible-endpoint
context-length probing for Hermes.
- [#6320](#6320) ->
`docs/about/release-notes.mdx`: Record bounded gateway-recovery waits.
- [#6377](#6377) ->
`docs/reference/commands.mdx`, `docs/about/release-notes.mdx`: Explain
rebuild diagnostics and prepared MCP-destroy recovery.
- [#6412](#6412) ->
`docs/get-started/quickstart-langchain-deepagents-code.mdx`,
`docs/about/release-notes.mdx`: Document authoritative agent-visible
inference route health.
- [#6421](#6421) ->
`docs/about/release-notes.mdx`: Record the longer quiet-pull window for
managed vLLM images.
- [#6431](#6431) ->
`docs/inference/model-capability-audit.mdx`,
`docs/about/release-notes.mdx`: Document the version-pinned Nemotron
Ultra profile plugin.
- [#6439](#6439) ->
`docs/about/release-notes.mdx`: Summarize the authenticated, pinned
credential-capture helper boundary.
- [#6450](#6450) ->
`docs/manage-sandboxes/messaging-channels.mdx`,
`docs/reference/commands.mdx`, `docs/about/release-notes.mdx`: Document
host-forward cleanup and ownership-safe gateway-port release.
- [#6474](#6474) ->
`docs/manage-sandboxes/messaging-channels.mdx`,
`docs/about/release-notes.mdx`: Record composable OpenClaw messaging
runtime loaders.
- [#6475](#6475) ->
`docs/about/release-notes.mdx`: Record removal of the unavailable Kimi
K2.6 production endpoint option.
- [#6480](#6480) ->
`docs/about/release-notes.mdx`: Record stderr routing for the plugin
registration banner.
- [#6481](#6481) ->
`docs/about/release-notes.mdx`: Record post-pull Ollama model discovery
checks.
- [#6482](#6482) ->
`docs/about/release-notes.mdx`: Record Ollama model warm-up after daemon
restart.
- [#6486](#6486) ->
`docs/about/release-notes.mdx`: Publish the opt-in, thread-scoped Deep
Agents auto-approval boundary.
- [#6490](#6490) ->
`docs/about/release-notes.mdx`: Record diagnostics for custom images
missing the managed runtime.
- [#6494](#6494) ->
`docs/inference/model-capability-audit.mdx`,
`docs/about/release-notes.mdx`: Document nonempty tool-call content
preservation and placeholder rejection.
- [#6497](#6497) ->
`docs/get-started/quickstart-langchain-deepagents-code.mdx`,
`docs/about/release-notes.mdx`: Document isolated Deep Agents
route-probe output.
- [#6506](#6506) ->
`docs/get-started/quickstart-langchain-deepagents-code.mdx`,
`docs/about/release-notes.mdx`: Document observability-preserving
managed route probes.
- [#6508](#6508) ->
`docs/about/release-notes.mdx`: Link the new extension taxonomy and
SDK-readiness reference from the release summary.

Release-source verification: GitHub reports all 29 cited source PRs as
merged with base `main`, and every merge commit is an ancestor of
`origin/main` at `17bf9a6a9688b3b1d69cf4b37d3f23110acb055e`. No
source-mapping mismatches were found.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [x] Doc only (includes code sample changes)

## Quality Gates

<!-- Check exactly one tests line and one docs line. Check other lines
when applicable. Add every requested justification or approval
reference. -->
- [ ] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [x] Tests not applicable — justification: Documentation-only
release-prep changes; `npm run docs` validates variants, routes, and
Fern content.
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

<!-- Check each applicable item only when supported by the requested
evidence. Run targeted tests once per relevant change set and rerun
after later edits or hook autofixes that can affect the tested behavior.
Do not rerun hook-covered checks. -->
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification: Tests
are not applicable to this documentation-only change set.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result:
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) — exited
0 with zero errors; Fern reported the existing unauthenticated
redirect-check and light-mode contrast warnings.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
<!-- DCO sign-off is required in this PR description, and every commit
must appear as Verified in GitHub. Run: git config user.name && git
config user.email -->
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

---------

Signed-off-by: cjagwani <cjagwani@nvidia.com>
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
## Summary

Follow-up to merged PR NVIDIA#6065 to address the unaddressed PR Review
Advisor warnings:

- Adds safe gateway-log append helper for PID 1/root recovery warnings;
refuses symlink/non-regular/replaced log targets instead of appending
through them.
- Adds guard-chain recovery tests proving regular log append still works
and unsafe targets are refused.
- Adds focused reconciliation coverage proving `NEMOCLAW_MODEL_OVERRIDE`
wins over conflicting gateway state, plus no-override reconciliation
still runs.

## Test plan

- [x] `npx vitest run --project integration
test/nemoclaw-start-reconcile.test.ts
test/nemoclaw-start-guard-recovery.test.ts`

Refs NVIDIA#6065

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Hardened gateway log writing to refuse unsafe targets (non-regular,
swapped, invalid), while sanitizing and newline-terminating log lines in
UTF-8.
* Recovery now records restoration warnings in the gateway log when
safe, without blocking startup if logging fails.
* Reconcile behavior improved: explicit model overrides are preserved;
otherwise configuration is reconciled from the live gateway model.
* **Tests**
* Expanded gateway recovery coverage with
regular/symlink/directory/missing log scenarios, validating behavior via
gateway-log contents.
  * Added reconcile tests for overridden vs non-overridden behavior.
* Updated gateway health watchdog tests to direct critical logging to
controlled per-test log files.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

## Advisor disposition / E2E evidence

PR Review Advisor `PRA-1` / `PRA-2` runtime-validation warnings are
resolved by the combination of in-PR helper hardening coverage plus live
E2E evidence:

- Source boundary: `append_openclaw_gateway_log_line` is the permanent
PID 1 append boundary and hardcodes canonical `/tmp/gateway.log`; it
intentionally refuses symlink, non-regular, replaced, or missing log
targets rather than honoring inherited path seams.
- Source-fix constraint: startup remains responsible for pre-creating
canonical `/tmp/gateway.log`; recovery/watchdog append paths must not
create a missing log because creation at recovery time would re-open the
unsafe target/ownership ambiguity this PR removes.
- Regression/unit coverage: `test/nemoclaw-start-guard-recovery.test.ts`
covers regular append, symlink refusal, non-regular refusal, missing-log
no-create, replaced-target refusal, inherited env ignored, and newline
sanitization. `test/nemoclaw-start-gateway-health.test.ts` covers
watchdog CRITICAL breadcrumb append through the same safe helper.
- Runtime validation: current-head E2E revalidation passed on
`9cda35bdaa6d426d160d262ac35c99c7449c1560`: run
https://github.com/NVIDIA/NemoClaw/actions/runs/28874860688 passed
required jobs `gateway-guard-recovery` and
`issue-2478-crash-loop-recovery`, and run
https://github.com/NVIDIA/NemoClaw/actions/runs/28874896411 passed
required target `ubuntu-repo-cloud-openclaw`. The earlier same-PR
artifact for `issue-2478-crash-loop-recovery` showed the real sandbox
respawn path emitted `[gateway-recovery] WARNING ... (NVIDIA#2478/NVIDIA#2701)` and
mirrored the same line as `[gateway-log:] [gateway-recovery] WARNING
...`, proving the real PID 1 startup/recovery lifecycle pre-created and
preserved the canonical gateway log before append.
- Removal condition: keep the helper-level contract tests until gateway
recovery/watchdog logging is no longer routed through `/tmp/gateway.log`
or the startup ownership contract is replaced by a different diagnostics
channel with equivalent live E2E coverage.

---------

Signed-off-by: cjagwani <cjagwani@nvidia.com>
Co-authored-by: cjagwani <cjagwani@nvidia.com>
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Prepares the user documentation for NemoClaw v0.0.78 by replacing the
unreleased section with release highlights and synchronizing the
affected inference, lifecycle, messaging, and CLI reference pages with
merged behavior.

## Changes

- Publish the v0.0.78 release-notes section with links to the most
specific user guides for each shipped behavior.
- Document authoritative Deep Agents route health, Nemotron Ultra
profile behavior, and Hermes compatible-endpoint context metadata.
- Document forced rebuild recovery after total backup failure and the
ownership-safe tunnel/full-stop behavior.
- Keep command examples and shared agent variants aligned with the
current OpenClaw, Hermes, and Deep Agents interfaces.

Source mapping:

- [NVIDIA#3787](NVIDIA#3787) ->
`docs/about/release-notes.mdx`: Record reliable workspace template
seeding during sandbox startup.
- [NVIDIA#4960](NVIDIA#4960) ->
`docs/about/release-notes.mdx`: Record safer detection of rewritten
OpenClaw gateway processes.
- [NVIDIA#5676](NVIDIA#5676) ->
`docs/about/release-notes.mdx`: Record warning-tolerant agent-list JSON
handling.
- [NVIDIA#5857](NVIDIA#5857) ->
`docs/about/release-notes.mdx`: Record synchronization of explicit
OpenClaw main-agent model state.
- [NVIDIA#5929](NVIDIA#5929) ->
`docs/about/release-notes.mdx`: Record copyable SSH port-forward
guidance for remote dashboards.
- [NVIDIA#6068](NVIDIA#6068) ->
`docs/about/release-notes.mdx`: Record custom-image plugin provenance
reconciliation.
- [NVIDIA#6116](NVIDIA#6116) ->
`docs/about/release-notes.mdx`: Record live-loopback dashboard-forward
recovery.
- [NVIDIA#6122](NVIDIA#6122) ->
`docs/about/release-notes.mdx`: Announce validated, round-trippable
policy YAML output.
- [NVIDIA#6211](NVIDIA#6211) ->
`docs/manage-sandboxes/lifecycle.mdx`, `docs/reference/commands.mdx`,
`docs/about/release-notes.mdx`: Explain the explicit no-backup `rebuild
--force` recovery boundary.
- [NVIDIA#6283](NVIDIA#6283) ->
`docs/about/release-notes.mdx`: Record Hermes WebUI port alignment.
- [NVIDIA#6293](NVIDIA#6293) ->
`docs/inference/switch-inference-providers.mdx`,
`docs/about/release-notes.mdx`: Document compatible-endpoint
context-length probing for Hermes.
- [NVIDIA#6320](NVIDIA#6320) ->
`docs/about/release-notes.mdx`: Record bounded gateway-recovery waits.
- [NVIDIA#6377](NVIDIA#6377) ->
`docs/reference/commands.mdx`, `docs/about/release-notes.mdx`: Explain
rebuild diagnostics and prepared MCP-destroy recovery.
- [NVIDIA#6412](NVIDIA#6412) ->
`docs/get-started/quickstart-langchain-deepagents-code.mdx`,
`docs/about/release-notes.mdx`: Document authoritative agent-visible
inference route health.
- [NVIDIA#6421](NVIDIA#6421) ->
`docs/about/release-notes.mdx`: Record the longer quiet-pull window for
managed vLLM images.
- [NVIDIA#6431](NVIDIA#6431) ->
`docs/inference/model-capability-audit.mdx`,
`docs/about/release-notes.mdx`: Document the version-pinned Nemotron
Ultra profile plugin.
- [NVIDIA#6439](NVIDIA#6439) ->
`docs/about/release-notes.mdx`: Summarize the authenticated, pinned
credential-capture helper boundary.
- [NVIDIA#6450](NVIDIA#6450) ->
`docs/manage-sandboxes/messaging-channels.mdx`,
`docs/reference/commands.mdx`, `docs/about/release-notes.mdx`: Document
host-forward cleanup and ownership-safe gateway-port release.
- [NVIDIA#6474](NVIDIA#6474) ->
`docs/manage-sandboxes/messaging-channels.mdx`,
`docs/about/release-notes.mdx`: Record composable OpenClaw messaging
runtime loaders.
- [NVIDIA#6475](NVIDIA#6475) ->
`docs/about/release-notes.mdx`: Record removal of the unavailable Kimi
K2.6 production endpoint option.
- [NVIDIA#6480](NVIDIA#6480) ->
`docs/about/release-notes.mdx`: Record stderr routing for the plugin
registration banner.
- [NVIDIA#6481](NVIDIA#6481) ->
`docs/about/release-notes.mdx`: Record post-pull Ollama model discovery
checks.
- [NVIDIA#6482](NVIDIA#6482) ->
`docs/about/release-notes.mdx`: Record Ollama model warm-up after daemon
restart.
- [NVIDIA#6486](NVIDIA#6486) ->
`docs/about/release-notes.mdx`: Publish the opt-in, thread-scoped Deep
Agents auto-approval boundary.
- [NVIDIA#6490](NVIDIA#6490) ->
`docs/about/release-notes.mdx`: Record diagnostics for custom images
missing the managed runtime.
- [NVIDIA#6494](NVIDIA#6494) ->
`docs/inference/model-capability-audit.mdx`,
`docs/about/release-notes.mdx`: Document nonempty tool-call content
preservation and placeholder rejection.
- [NVIDIA#6497](NVIDIA#6497) ->
`docs/get-started/quickstart-langchain-deepagents-code.mdx`,
`docs/about/release-notes.mdx`: Document isolated Deep Agents
route-probe output.
- [NVIDIA#6506](NVIDIA#6506) ->
`docs/get-started/quickstart-langchain-deepagents-code.mdx`,
`docs/about/release-notes.mdx`: Document observability-preserving
managed route probes.
- [NVIDIA#6508](NVIDIA#6508) ->
`docs/about/release-notes.mdx`: Link the new extension taxonomy and
SDK-readiness reference from the release summary.

Release-source verification: GitHub reports all 29 cited source PRs as
merged with base `main`, and every merge commit is an ancestor of
`origin/main` at `17bf9a6a9688b3b1d69cf4b37d3f23110acb055e`. No
source-mapping mismatches were found.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [x] Doc only (includes code sample changes)

## Quality Gates

<!-- Check exactly one tests line and one docs line. Check other lines
when applicable. Add every requested justification or approval
reference. -->
- [ ] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [x] Tests not applicable — justification: Documentation-only
release-prep changes; `npm run docs` validates variants, routes, and
Fern content.
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

<!-- Check each applicable item only when supported by the requested
evidence. Run targeted tests once per relevant change set and rerun
after later edits or hook autofixes that can affect the tested behavior.
Do not rerun hook-covered checks. -->
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification: Tests
are not applicable to this documentation-only change set.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result:
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) — exited
0 with zero errors; Fern reported the existing unauthenticated
redirect-check and light-mode contrast warnings.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
<!-- DCO sign-off is required in this PR description, and every commit
must appear as Verified in GitHub. Run: git config user.name && git
config user.email -->
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

---------

Signed-off-by: cjagwani <cjagwani@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery bug-fix PR fixes a bug or regression v0.0.78 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants