feat(state): add key-level restore ownership to agent manifests (#6445)

<!-- markdownlint-disable MD041 -->
## Summary

Agent manifests can now declare key-level restore ownership for
mixed-ownership state files. Managed restore preserves only
manifest-owned user preferences while the current target manifest
remains authoritative for generated configuration. Deep Agents Code is
migrated to this schema, and the restore path now fails closed on unsafe
manifests, paths, TOML, filesystem races, and target/backup mismatches.

## Related Issue

Resolves #6334

## Changes

- Add a strict, discriminated state_files restore schema for
key-allowlist and openclaw-config strategies, including canonical
relative-path validation and rejection of duplicate or
ancestor-overlapping ownership declarations.
- Require the current target manifest to authorize the backup agent,
config directory, state-file path, strategy, and restore ownership
before SSH or filesystem mutation.
- Make managed key-level merge the default. Whole-file restore is
available only through the explicit
allowCustomImageWholeStateFileRestore capability for custom-image
restore.
- Read bounded backup TOML directly from Python stdin, parse with
tomllib, create a private unpredictable O_EXCL stage, traverse parent
directories without following symlinks, and revalidate current/stage
metadata before atomic replacement.
- Preserve all safe leading comments, preserve non-dictionary fresh
ancestors instead of clobbering them, and keep fresh managed tables and
headers authoritative.
- Migrate the Deep Agents Code manifest and remove the agent-specific
restore-policy callback.
- Document managed key-level restore, target-manifest authority, and the
custom-Dockerfile whole-file exception.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: maintainer-assisted
review of the behavior at dd65850
passed all nine security categories; current head
f9e4537 adds only a formatting-only
blank-line deletion to satisfy the onboard.ts net-growth guardrail.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes the DCO sign-off declaration and every
commit appears as Verified in GitHub
- [x] Normal pre-commit, commit-msg, and pre-push hooks passed, or npm
run check:diff passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — 175 focused CLI assertions plus 71
affected integration assertions passed; the final phase/contract rerun
passed 23/23. npm run typecheck:cli, npm run build:cli, npm run
test-size:check, and git diff --check also passed.
- [ ] Applicable broad gate passed — not applicable locally; targeted
CLI/integration coverage exercised every changed restore path, and
repository CI is running on the exact head.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] npm run docs builds without warnings (doc changes only) — build
passed with 0 errors and 2 pre-existing hidden-page warnings
- [x] Doc pages follow the style guide (doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
— no new pages

---
Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added manifest-driven, fine-grained state-file restore rules with
typed user key allowlists and “freshness” requirements.
* Custom Dockerfile sandboxes can restore whole state files during
rebuilds when enabled.
* **Bug Fixes**
* Snapshot restore now fails closed for unknown, unsafe, stale, or
conflicting state data, with stronger atomic replacement safeguards.
* **Documentation**
* Updated sandbox backup/restore guidance to reflect key-level ownership
semantics, allowed preference preservation, and restore failure
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Charan Jagwani <cjagwani@nvidia.com>
Co-authored-by: Carlos Villela <cvillela@nvidia.com>

Author: 4 people

agents/langchain-deepagents-code/manifest.yaml

@@ -48,12 +48,11 @@ state_dirs:
 
 # ── Top-level durable state files ───────────────────────────────
 # config.toml mixes DCode preferences with NemoClaw-managed model routing.
-# Managed re-onboard restore carries forward only boolean ui.show_scrollbar,
-# ui.show_url_open_toast, and threads.relative_time preferences, plus
-# threads.sort_order when it is updated_at or created_at. Fresh models/update
-# tables and provider metadata remain authoritative. All other backup keys,
-# including ui.theme and behavior-bearing, unknown, or security-sensitive keys,
-# are dropped.
+# Ownership is split into three explicit buckets. Users own only the allowlisted
+# ui.show_scrollbar, ui.show_url_open_toast, threads.relative_time, and
+# threads.sort_order preferences below. NemoClaw owns the fresh models/update
+# tables and generated provider headers. Agent-runtime, unknown, executable, and
+# security-sensitive backup keys are not restorable and are dropped.
 # .env and user-authored .deepagents/.mcp.json content are intentionally omitted
 # because they may contain service credentials. NemoClaw writes only direct-HTTP
 # bridge endpoint config and OpenShell placeholders to its separate
@@ -62,6 +61,29 @@ state_dirs:
 # not user-authored durable state.
 state_files:
   - path: config.toml
+    restore:
+      merge: key-allowlist
+      require_fresh_tables:
+        - models
+        - update
+      # Checked positionally: the first N leading '#' lines (N = entries below)
+      # must match. Further leading comments are safety-checked and preserved.
+      require_fresh_headers:
+        - "# Generated by NemoClaw. This file contains no provider secrets."
+        - match: prefix
+          value: "# NemoClaw provider route: "
+      user_keys:
+        - key: ui.show_scrollbar
+          type: boolean
+        - key: ui.show_url_open_toast
+          type: boolean
+        - key: threads.relative_time
+          type: boolean
+        - key: threads.sort_order
+          type: enum
+          values:
+            - updated_at
+            - created_at
 user_managed_files:
   - .deepagents/.env
   - .deepagents/.mcp.json

agents/openclaw/manifest.yaml

@@ -68,6 +68,8 @@ state_dirs:
 # credentials.
 state_files:
   - path: openclaw.json
+    restore:
+      merge: openclaw-config
 user_managed_files:
   - .env
   - .mcp.json

docs/manage-sandboxes/backup-restore.mdx

@@ -61,6 +61,10 @@ The Hermes database can contain session metadata and message history needed for
 </AgentOnly>
 Snapshots also preserve sandbox registry metadata that affects rebuild behavior, including custom policy presets applied with `policy-add --from-file` or `policy-add --from-dir`.
 When you restore a snapshot, NemoClaw replays those recorded custom presets with their stored YAML content, so you do not need the original preset files on disk for the restored sandbox to keep the same policy state.
+The target sandbox's current agent manifest remains authoritative for state-file restore behavior.
+NemoClaw rejects the restore when the snapshot's agent, config directory, state-file path, or state-file strategy conflicts with that manifest.
+For managed images, NemoClaw applies the current manifest's managed config merge rules by default and does not fall back to whole-file replacement.
+For Deep Agents targets, whole-file config replacement is limited to sandboxes created from a custom Dockerfile.
 
 ```bash
 $$nemoclaw my-assistant snapshot create
@@ -126,7 +130,11 @@ NemoClaw rejects unsafe symlinks and hard links inside sandbox state during back
 Credential-bearing Deep Agents files such as `.deepagents/.env` and user-authored `.deepagents/.mcp.json` are intentionally excluded from snapshots.
 Deep Agents auth state files such as `.deepagents/.state/auth.json` and `.deepagents/.state/chatgpt-auth.json` are also excluded because the managed launcher refuses to start when upstream credential state is present.
 The managed `.deepagents/.nemoclaw-mcp.json` projection and `hooks.json` are also excluded because NemoClaw reconstructs managed MCP state and disables executable Deep Agents Code hooks in the managed harness.
-NemoClaw recreates generated `config.toml`, managed MCP projection state, and provider credentials from host-side onboarding and OpenShell provider state during rebuild.
+NemoClaw recreates the current inference route headers, `models` and `update` tables, managed MCP projection state, and provider credentials from host-side onboarding and OpenShell provider state during rebuild.
+For a NemoClaw-managed Deep Agents image, it restores only the allowlisted `ui.show_scrollbar`, `ui.show_url_open_toast`, `threads.relative_time`, and `threads.sort_order` preferences from the previous `config.toml` when their values pass validation.
+Unknown, runtime-controlled, executable, and security-sensitive backup keys are dropped instead of replacing freshly generated settings on that managed path.
+A Deep Agents target created from a custom Dockerfile restores `config.toml` as a whole file because the custom image owns its config schema, so review that backup before restoring it.
+On the managed key-level restore path, if either config is malformed, required managed data is missing, or the restore detects an unsafe link or file replacement, NemoClaw marks the restore as failed and does not fall back to copying the backed-up file wholesale.
 Before a Deep Agents rebuild changes the sandbox, NemoClaw verifies the recorded inference route, provider, model, reasoning settings, web search selection, base image, and policy inputs.
 If a late check fails, NemoClaw restores the previous MCP state and keeps the existing sandbox intact.
 

docs/manage-sandboxes/workspace-files.mdx

@@ -192,7 +192,13 @@ The Deep Agents manifest declares these durable directories:
 /sandbox/.deepagents/agent/skills/
 ```
 
-It also declares `/sandbox/.deepagents/config.toml` as a durable top-level state file because the file contains non-secret NemoClaw-generated provider and model configuration.
+It also declares `/sandbox/.deepagents/config.toml` as a durable top-level state file with key-level ownership.
+The target sandbox's current Deep Agents manifest defines this ownership policy, so a snapshot cannot weaken it.
+NemoClaw keeps the newly generated inference route headers and the `models` and `update` tables authoritative during rebuild.
+On a NemoClaw-managed image, only the allowlisted `ui.show_scrollbar`, `ui.show_url_open_toast`, `threads.relative_time`, and `threads.sort_order` preferences can be restored from the previous file.
+Runtime-controlled, unknown, executable, and security-sensitive backup keys are dropped on that managed path.
+A Deep Agents target created from a custom Dockerfile restores `config.toml` as a whole file because the custom image owns its config schema.
+On the managed key-level restore path, if config validation or safe atomic replacement fails, NemoClaw marks the restore as failed instead of falling back to a whole-file copy.
 Credential-bearing files such as `.deepagents/.env` and user-authored `.deepagents/.mcp.json` are intentionally omitted from snapshots.
 Managed MCP state is rebuilt from the host-side NemoClaw registry and OpenShell provider state instead of treated as user-authored durable state.
 Memory files such as `/sandbox/.deepagents/agent/AGENTS.md` and `/sandbox/.deepagents/agent/memories/` are upstream Deep Agents Code files.
@@ -204,7 +210,7 @@ Back up important Deep Agents state before destroying the sandbox.
 ## Editing State
 
 Prefer NemoClaw host commands for generated configuration such as model, provider, managed MCP, and policy settings.
-Direct edits to `/sandbox/.deepagents/config.toml` can be overwritten by rebuilds because NemoClaw owns the managed inference route.
+Direct edits to NemoClaw-owned or non-allowlisted keys in `/sandbox/.deepagents/config.toml` can be overwritten by rebuilds.
 Use `$$nemoclaw <name> connect` when you need to inspect runtime files interactively, or use `openshell sandbox download` and `openshell sandbox upload` for manual file transfer.
 
 Use Deep Agents Code commands for upstream-managed memories and skills.

src/lib/actions/sandbox/rebuild-pipeline.ts

@@ -299,6 +299,8 @@ async function rebuildSandboxUnlocked(
 
       const restored = runRebuildRestorePhase({
         sandboxName,
+        targetAgentType: rebuildAgent || "openclaw",
+        targetImageIsCustom: Boolean(fromDockerfile),
         backupManifest: backup.backupManifest,
         policyPresets: targetPolicyPresets,
         customPolicies:

src/lib/actions/sandbox/rebuild-restore-forwarding.test.ts

@@ -0,0 +1,42 @@
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+import { afterEach, describe, expect, it, vi } from "vitest";
+
+import * as sandboxState from "../../state/sandbox";
+import { runRebuildRestorePhase } from "./rebuild-restore-phase";
+
+afterEach(() => {
+  vi.restoreAllMocks();
+});
+
+describe("rebuild restore target forwarding", () => {
+  it("forwards the recreated target identity and explicit custom-image capability", () => {
+    vi.spyOn(console, "log").mockImplementation(() => undefined);
+    const restoreRecreatedSandboxState = vi
+      .spyOn(sandboxState, "restoreRecreatedSandboxState")
+      .mockReturnValue({
+        success: true,
+        restoredDirs: [],
+        restoredFiles: [],
+        failedDirs: [],
+        failedFiles: [],
+      });
+
+    runRebuildRestorePhase({
+      sandboxName: "alpha",
+      targetAgentType: "langchain-deepagents-code",
+      targetImageIsCustom: true,
+      backupManifest: { agentType: "openclaw", backupPath: "/tmp/rebuild-backup" } as never,
+      policyPresets: [],
+      customPolicies: [],
+      reconcileManagedDcodeObservability: false,
+      log: vi.fn(),
+    });
+
+    expect(restoreRecreatedSandboxState).toHaveBeenCalledWith("alpha", "/tmp/rebuild-backup", {
+      targetAgentType: "langchain-deepagents-code",
+      allowCustomImageWholeStateFileRestore: true,
+    });
+  });
+});

src/lib/actions/sandbox/rebuild-restore-phase.test.ts

@@ -15,6 +15,19 @@ import { runRebuildRestorePhase } from "./rebuild-restore-phase";
 const BUILTIN_OBSERVABILITY_CONTENT =
   "network_policies:\n  observability-otlp-local:\n    name: observability-otlp-local\n";
 
+type StandardRestoreOptions = Omit<
+  Parameters<typeof runRebuildRestorePhase>[0],
+  "targetAgentType" | "targetImageIsCustom"
+>;
+
+function runStandardRebuildRestorePhase(options: StandardRestoreOptions) {
+  return runRebuildRestorePhase({
+    ...options,
+    targetAgentType: "openclaw",
+    targetImageIsCustom: false,
+  });
+}
+
 describe("rebuild policy restore fidelity", () => {
   beforeEach(() => {
     vi.spyOn(policies, "loadPresetForSandbox").mockImplementation((_sandboxName, presetName) =>
@@ -40,7 +53,7 @@ describe("rebuild policy restore fidelity", () => {
       error: "could not read fresh OpenClaw plugin install registry",
     });
 
-    const result = runRebuildRestorePhase({
+    const result = runStandardRebuildRestorePhase({
       sandboxName: "alpha",
       backupManifest: { agentType: "openclaw", backupPath: "/tmp/rebuild-backup" } as never,
       policyPresets: [],
@@ -78,7 +91,7 @@ describe("rebuild policy restore fidelity", () => {
       content: `network_policies:\n  ${name}-custom:\n    name: ${name}-custom\n`,
       sourcePath: `/tmp/${name}.yaml`,
     }));
-    const result = runRebuildRestorePhase({
+    const result = runStandardRebuildRestorePhase({
       sandboxName: "alpha",
       backupManifest: {
         agentType: "openclaw",
@@ -126,7 +139,7 @@ describe("rebuild policy restore fidelity", () => {
         sourcePath: "/tmp/custom-egress.yaml",
       },
     ];
-    const result = runRebuildRestorePhase({
+    const result = runStandardRebuildRestorePhase({
       sandboxName: "alpha",
       backupManifest: null,
       policyPresets: [],
@@ -159,7 +172,7 @@ describe("rebuild policy restore fidelity", () => {
         "network_policies:\n  mcp-bridge-search:\n    endpoints:\n      - host: mcp.example.com\n        allowed_ips: [203.0.113.10]\n",
       sourcePath: MCP_BRIDGE_POLICY_SOURCE,
     };
-    const result = runRebuildRestorePhase({
+    const result = runStandardRebuildRestorePhase({
       sandboxName: "alpha",
       backupManifest: null,
       policyPresets: [],
@@ -188,7 +201,7 @@ describe("rebuild policy restore fidelity", () => {
       .mockReturnValueOnce("absent");
     const removePreset = vi.spyOn(policies, "removePreset").mockReturnValue(true);
 
-    const result = runRebuildRestorePhase({
+    const result = runStandardRebuildRestorePhase({
       sandboxName: "alpha",
       backupManifest: null,
       policyPresets: ["npm"],
@@ -213,7 +226,7 @@ describe("rebuild policy restore fidelity", () => {
       .mockReturnValueOnce(null);
     vi.spyOn(policies, "removePreset").mockReturnValue(true);
 
-    const result = runRebuildRestorePhase({
+    const result = runStandardRebuildRestorePhase({
       sandboxName: "alpha",
       backupManifest: null,
       policyPresets: ["npm"],
@@ -243,7 +256,7 @@ describe("rebuild policy restore fidelity", () => {
         throw new Error("apply failed");
       });
 
-    const result = runRebuildRestorePhase({
+    const result = runStandardRebuildRestorePhase({
       sandboxName: "alpha",
       backupManifest: null,
       policyPresets: ["npm", "bad", "throw"],
@@ -264,7 +277,7 @@ describe("rebuild policy restore fidelity", () => {
     vi.spyOn(console, "error").mockImplementation(() => undefined);
     vi.spyOn(policies, "applyPreset").mockReturnValue(true);
 
-    const result = runRebuildRestorePhase({
+    const result = runStandardRebuildRestorePhase({
       sandboxName: "alpha",
       backupManifest: null,
       policyPresets: ["observability-otlp-local"],
@@ -285,7 +298,7 @@ describe("rebuild policy restore fidelity", () => {
     vi.spyOn(policies, "applyPreset").mockReturnValue(true);
     vi.spyOn(policies, "getPresetContentGatewayState").mockReturnValue(null);
 
-    const result = runRebuildRestorePhase({
+    const result = runStandardRebuildRestorePhase({
       sandboxName: "alpha",
       backupManifest: null,
       policyPresets: ["observability-otlp-local"],
@@ -301,7 +314,7 @@ describe("rebuild policy restore fidelity", () => {
   it("does not remove or persist DCode base-policy keys detected as broad presets", () => {
     const removePreset = vi.spyOn(policies, "removePreset");
 
-    const result = runRebuildRestorePhase({
+    const result = runStandardRebuildRestorePhase({
       sandboxName: "alpha",
       backupManifest: null,
       policyPresets: [],
@@ -326,7 +339,7 @@ describe("rebuild policy restore fidelity", () => {
       sourcePath: "/tmp/operator-collector.yaml",
     };
 
-    const result = runRebuildRestorePhase({
+    const result = runStandardRebuildRestorePhase({
       sandboxName: "alpha",
       backupManifest: null,
       policyPresets: [],
@@ -358,7 +371,7 @@ describe("rebuild policy restore fidelity", () => {
       sourcePath: "/tmp/corp-otel.yaml",
     };
 
-    const result = runRebuildRestorePhase({
+    const result = runStandardRebuildRestorePhase({
       sandboxName: "alpha",
       backupManifest: null,
       policyPresets: [],
@@ -392,7 +405,7 @@ describe("rebuild policy restore fidelity", () => {
       .mockReturnValueOnce("absent");
     const removePreset = vi.spyOn(policies, "removePreset").mockReturnValue(true);
 
-    const result = runRebuildRestorePhase({
+    const result = runStandardRebuildRestorePhase({
       sandboxName: "alpha",
       backupManifest: null,
       policyPresets: ["observability-otlp-local"],
@@ -421,7 +434,7 @@ describe("rebuild policy restore fidelity", () => {
       .mockReturnValueOnce("absent");
     const removePreset = vi.spyOn(policies, "removePreset").mockReturnValue(true);
 
-    const result = runRebuildRestorePhase({
+    const result = runStandardRebuildRestorePhase({
       sandboxName: "alpha",
       backupManifest: null,
       policyPresets: [],
@@ -449,7 +462,7 @@ describe("rebuild policy restore fidelity", () => {
     vi.spyOn(policies, "getPresetContentGatewayState").mockReturnValue("drift");
     const removePreset = vi.spyOn(policies, "removePreset");
 
-    const result = runRebuildRestorePhase({
+    const result = runStandardRebuildRestorePhase({
       sandboxName: "alpha",
       backupManifest: null,
       policyPresets: [],
@@ -476,7 +489,7 @@ describe("rebuild policy restore fidelity", () => {
       .mockReturnValueOnce(null);
     vi.spyOn(policies, "removePreset").mockReturnValue(true);
 
-    const result = runRebuildRestorePhase({
+    const result = runStandardRebuildRestorePhase({
       sandboxName: "alpha",
       backupManifest: null,
       policyPresets: ["observability-otlp-local"],

src/lib/actions/sandbox/rebuild-restore-phase.ts

@@ -16,6 +16,8 @@ import type { RebuildSandboxEntry } from "./rebuild-flow-helpers";
 
 export interface RebuildRestorePhaseInput {
   sandboxName: string;
+  targetAgentType: string;
+  targetImageIsCustom: boolean;
   backupManifest: RebuildBackupManifest;
   policyPresets: string[];
   customPolicies: NonNullable<RebuildSandboxEntry["customPolicies"]>;
@@ -176,6 +178,8 @@ function reconcileFinalManagedObservability(
 export function runRebuildRestorePhase(input: RebuildRestorePhaseInput): RebuildRestorePhaseResult {
   const {
     sandboxName,
+    targetAgentType,
+    targetImageIsCustom,
     backupManifest,
     policyPresets,
     customPolicies,
@@ -190,7 +194,10 @@ export function runRebuildRestorePhase(input: RebuildRestorePhaseInput): Rebuild
     const restore = sandboxState.restoreRecreatedSandboxState(
       sandboxName,
       backupManifest.backupPath,
-      { targetAgentType: backupManifest.agentType },
+      {
+        targetAgentType,
+        ...(targetImageIsCustom ? { allowCustomImageWholeStateFileRestore: true } : {}),
+      },
     );
     log(
       `Restore result: success=${restore.success}, restored=${restore.restoredDirs.join(",")}; files=${restore.restoredFiles.join(",")}, failed=${restore.failedDirs.join(",")}; failedFiles=${restore.failedFiles.join(",")}${restore.error ? `; error=${restore.error}` : ""}`,

src/lib/actions/sandbox/snapshot-auto-create-failure.test.ts

@@ -48,7 +48,10 @@ vi.mock("../../inference/nim", () => ({
   stopNimContainer: vi.fn(),
   stopNimContainerByName: vi.fn(),
 }));
-vi.mock("../../messaging/channels", () => ({ listMessagingProviderSuffixes: vi.fn(() => []) }));
+vi.mock("../../messaging/channels", () => ({
+  listMessagingProviderSuffixes: vi.fn(() => []),
+  listMessagingCredentialMetadata: vi.fn(() => []),
+}));
 vi.mock("../../policy", () => ({
   applyPreset: vi.fn(() => true),
   applyPresetContent: vi.fn(() => true),