Skip to content

Commit 6ec0dbe

Browse files
cvgithub-advanced-security[bot]cjagwani
authored
refactor(onboard): extract web search verification probe (#3305)
## Summary Extract the post-create web-search verification probe out of the large onboarding module. This keeps web-search setup and verification in focused helper modules while leaving onboarding flow orchestration unchanged. ## Changes - Add `src/lib/onboard/web-search-verify.ts` for best-effort Hermes/OpenClaw web-search runtime verification. - Update `src/lib/onboard.ts` to delegate web-search verification through the extracted helper. - Add unit tests covering active Hermes/OpenClaw detection, disabled/malformed OpenClaw config warnings, unsupported-agent warnings, and probe-error handling. - Update the source-shape regression to include the extracted verification module. ## Type of Change - [x] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Verification - [x] `npx prek run --all-files` passes - [x] `npm test` passes - [x] Tests added or updated for new or changed behavior - [x] No secrets, API keys, or credentials committed - [ ] Docs updated for user-facing behavior changes - [ ] `make docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- Signed-off-by: Carlos Villela <cvillela@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Runtime verification that web search is actually enabled inside sandboxes; logs success or emits non-fatal warnings when verification cannot be completed. * **Tests** * Expanded automated tests covering multiple agent/config scenarios, parsing edge cases, and probe failures to ensure robust verification behavior. [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/NVIDIA/NemoClaw/pull/3305) <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: cjagwani <cjagwani@nvidia.com>
1 parent 5a1e8fc commit 6ec0dbe

4 files changed

Lines changed: 176 additions & 71 deletions

File tree

‎src/lib/onboard.ts‎

Lines changed: 7 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,9 @@ const {
4444
const {
4545
agentSupportsWebSearch,
4646
}: typeof import("./onboard/web-search-support") = require("./onboard/web-search-support");
47+
const {
48+
verifyWebSearchInsideSandbox: verifyWebSearchInsideSandboxWithDeps,
49+
}: typeof import("./onboard/web-search-verify") = require("./onboard/web-search-verify");
4750
const {
4851
buildDirectGpuPolicyYaml,
4952
buildDirectSandboxGpuProofCommands,
@@ -2406,80 +2409,14 @@ async function configureWebSearch(
24062409
return { fetchEnabled: true };
24072410
}
24082411

2409-
/**
2410-
* Post-creation probe: verify web search is actually functional inside the
2411-
* sandbox. Hermes silently ignores unknown web.backend values, so checking
2412-
* the config file alone is insufficient — we need to ask the runtime.
2413-
*
2414-
* For Hermes: runs `hermes dump` and checks for an active web backend.
2415-
* For OpenClaw: checks that the tools.web.search block is present in the config.
2416-
*
2417-
* This is a best-effort warning — it does not abort onboarding.
2418-
*/
24192412
function verifyWebSearchInsideSandbox(
24202413
sandboxName: string,
24212414
agent: AgentDefinition | null | undefined,
24222415
): void {
2423-
const agentName = agent?.name || "openclaw";
2424-
try {
2425-
if (agentName === "hermes") {
2426-
// `hermes dump` outputs config_overrides and active toolsets.
2427-
// Look for the web backend in its output.
2428-
const dump = runCaptureOpenshell(
2429-
["sandbox", "exec", "-n", sandboxName, "--", "hermes", "dump"],
2430-
{
2431-
ignoreError: true,
2432-
timeout: 10_000,
2433-
},
2434-
);
2435-
if (!dump) {
2436-
console.warn(" ⚠ Could not verify web search config inside sandbox (hermes dump failed).");
2437-
return;
2438-
}
2439-
// A working web backend shows as an explicit config override or active-toolset entry.
2440-
// Avoid broad /web.*search/ matching so warning text never looks like success.
2441-
const hasWebBackend =
2442-
/^\s*web\.backend:\s*\S+/m.test(dump) ||
2443-
/^\s*active toolsets:\s*.*\bweb\b/im.test(dump) ||
2444-
/^\s*toolsets:\s*.*\bweb\b/im.test(dump);
2445-
if (!hasWebBackend) {
2446-
console.warn(
2447-
" ⚠ Web search was configured but Hermes does not report an active web backend.",
2448-
);
2449-
console.warn(" The agent may not have accepted the web search configuration.");
2450-
console.warn(` Check: ${cliName()} ${sandboxName} exec hermes dump`);
2451-
} else {
2452-
console.log(" ✓ Web search is active inside sandbox");
2453-
}
2454-
} else if (agentName === "openclaw") {
2455-
// OpenClaw: verify tools.web.search block exists in the baked config.
2456-
const configCheck = runCaptureOpenshell(
2457-
["sandbox", "exec", "-n", sandboxName, "--", "cat", "/sandbox/.openclaw/openclaw.json"],
2458-
{ ignoreError: true, timeout: 10_000 },
2459-
);
2460-
if (!configCheck) {
2461-
console.warn(" ⚠ Could not verify web search config inside sandbox.");
2462-
return;
2463-
}
2464-
try {
2465-
const parsed = JSON.parse(configCheck);
2466-
if (parsed?.tools?.web?.search?.enabled) {
2467-
console.log(" ✓ Web search is active inside sandbox");
2468-
} else {
2469-
console.warn(
2470-
" ⚠ Web search was configured but tools.web.search is not enabled in openclaw.json.",
2471-
);
2472-
}
2473-
} catch {
2474-
console.warn(" ⚠ Could not parse openclaw.json to verify web search config.");
2475-
}
2476-
} else {
2477-
console.warn(` ⚠ Web search verification is not implemented for agent '${agentName}'.`);
2478-
}
2479-
} catch {
2480-
// Best-effort — don't let probe failures derail onboarding.
2481-
console.warn(" ⚠ Web search verification probe failed (non-fatal).");
2482-
}
2416+
verifyWebSearchInsideSandboxWithDeps(sandboxName, agent, {
2417+
runCaptureOpenshell,
2418+
cliName,
2419+
});
24832420
}
24842421

24852422
// getSandboxInferenceConfig — moved to onboard-providers.ts

‎src/lib/onboard/web-search-verify.test.ts‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
import { describe, expect, it, vi } from "vitest";
5+
6+
import { verifyWebSearchInsideSandbox, type WebSearchVerifyDeps } from "./web-search-verify";
7+
8+
function deps(output: string | null) {
9+
return {
10+
runCaptureOpenshell: vi.fn(() => output),
11+
cliName: vi.fn(() => "nemoclaw"),
12+
log: vi.fn(),
13+
warn: vi.fn(),
14+
} satisfies WebSearchVerifyDeps;
15+
}
16+
17+
describe("verifyWebSearchInsideSandbox", () => {
18+
it("reports active Hermes web backend", () => {
19+
const d = deps("web.backend: brave\n");
20+
21+
verifyWebSearchInsideSandbox("alpha", { name: "hermes" }, d);
22+
23+
expect(d.log).toHaveBeenCalledWith(" ✓ Web search is active inside sandbox");
24+
expect(d.warn).not.toHaveBeenCalled();
25+
});
26+
27+
it("warns when Hermes does not report active web backend", () => {
28+
const d = deps("active toolsets: shell\n");
29+
30+
verifyWebSearchInsideSandbox("alpha", { name: "hermes" }, d);
31+
32+
expect(d.warn).toHaveBeenCalledWith(
33+
" ⚠ Web search was configured but Hermes does not report an active web backend.",
34+
);
35+
expect(d.warn).toHaveBeenCalledWith(" Check: nemoclaw alpha exec hermes dump");
36+
});
37+
38+
it("reports active OpenClaw web search config", () => {
39+
const d = deps(JSON.stringify({ tools: { web: { search: { enabled: true } } } }));
40+
41+
verifyWebSearchInsideSandbox("alpha", { name: "openclaw" }, d);
42+
43+
expect(d.log).toHaveBeenCalledWith(" ✓ Web search is active inside sandbox");
44+
});
45+
46+
it("warns when OpenClaw config is malformed or disabled", () => {
47+
const malformed = deps("not-json");
48+
verifyWebSearchInsideSandbox("alpha", { name: "openclaw" }, malformed);
49+
expect(malformed.warn).toHaveBeenCalledWith(
50+
" ⚠ Could not parse openclaw.json to verify web search config.",
51+
);
52+
53+
const disabled = deps(JSON.stringify({ tools: { web: { search: { enabled: false } } } }));
54+
verifyWebSearchInsideSandbox("alpha", { name: "openclaw" }, disabled);
55+
expect(disabled.warn).toHaveBeenCalledWith(
56+
" ⚠ Web search was configured but tools.web.search is not enabled in openclaw.json.",
57+
);
58+
});
59+
60+
it("warns for unknown agents and catches probe errors", () => {
61+
const unknown = deps(null);
62+
verifyWebSearchInsideSandbox("alpha", { name: "other" }, unknown);
63+
expect(unknown.warn).toHaveBeenCalledWith(
64+
" ⚠ Web search verification is not implemented for agent 'other'.",
65+
);
66+
67+
const throwing = deps(null);
68+
throwing.runCaptureOpenshell = vi.fn(() => {
69+
throw new Error("boom");
70+
});
71+
verifyWebSearchInsideSandbox("alpha", { name: "openclaw" }, throwing);
72+
expect(throwing.warn).toHaveBeenCalledWith(" ⚠ Web search verification probe failed (non-fatal).");
73+
});
74+
});

‎src/lib/onboard/web-search-verify.ts‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
export type WebSearchVerifyAgent = {
5+
name?: string | null;
6+
} | null | undefined;
7+
8+
export type WebSearchVerifyDeps = {
9+
runCaptureOpenshell: (args: string[], options: { ignoreError: true; timeout: number }) => string | null;
10+
cliName: () => string;
11+
log?: (message?: string) => void;
12+
warn?: (message?: string) => void;
13+
};
14+
15+
/**
16+
* Post-creation probe: verify web search is actually functional inside the
17+
* sandbox. Hermes silently ignores unknown web.backend values, so checking
18+
* the config file alone is insufficient — we need to ask the runtime.
19+
*
20+
* For Hermes: runs `hermes dump` and checks for an active web backend.
21+
* For OpenClaw: checks that the tools.web.search block is present in the config.
22+
*
23+
* This is a best-effort warning — it does not abort onboarding.
24+
*/
25+
export function verifyWebSearchInsideSandbox(
26+
sandboxName: string,
27+
agent: WebSearchVerifyAgent,
28+
deps: WebSearchVerifyDeps,
29+
): void {
30+
const log = deps.log ?? console.log;
31+
const warn = deps.warn ?? console.warn;
32+
const agentName = agent?.name || "openclaw";
33+
try {
34+
if (agentName === "hermes") {
35+
// `hermes dump` outputs config_overrides and active toolsets.
36+
// Look for the web backend in its output.
37+
const dump = deps.runCaptureOpenshell(
38+
["sandbox", "exec", "-n", sandboxName, "--", "hermes", "dump"],
39+
{
40+
ignoreError: true,
41+
timeout: 10_000,
42+
},
43+
);
44+
if (!dump) {
45+
warn(" ⚠ Could not verify web search config inside sandbox (hermes dump failed).");
46+
return;
47+
}
48+
// A working web backend shows as an explicit config override or active-toolset entry.
49+
// Avoid broad /web.*search/ matching so warning text never looks like success.
50+
const hasWebBackend =
51+
/^\s*web\.backend:\s*\S+/m.test(dump) ||
52+
/^\s*active toolsets:\s*.*\bweb\b/im.test(dump) ||
53+
/^\s*toolsets:\s*.*\bweb\b/im.test(dump);
54+
if (!hasWebBackend) {
55+
warn(" ⚠ Web search was configured but Hermes does not report an active web backend.");
56+
warn(" The agent may not have accepted the web search configuration.");
57+
warn(` Check: ${deps.cliName()} ${sandboxName} exec hermes dump`);
58+
} else {
59+
log(" ✓ Web search is active inside sandbox");
60+
}
61+
} else if (agentName === "openclaw") {
62+
// OpenClaw: verify tools.web.search block exists in the baked config.
63+
const configCheck = deps.runCaptureOpenshell(
64+
["sandbox", "exec", "-n", sandboxName, "--", "cat", "/sandbox/.openclaw/openclaw.json"],
65+
{ ignoreError: true, timeout: 10_000 },
66+
);
67+
if (!configCheck) {
68+
warn(" ⚠ Could not verify web search config inside sandbox.");
69+
return;
70+
}
71+
try {
72+
const parsed = JSON.parse(configCheck);
73+
if (parsed?.tools?.web?.search?.enabled) {
74+
log(" ✓ Web search is active inside sandbox");
75+
} else {
76+
warn(" ⚠ Web search was configured but tools.web.search is not enabled in openclaw.json.");
77+
}
78+
} catch {
79+
warn(" ⚠ Could not parse openclaw.json to verify web search config.");
80+
}
81+
} else {
82+
warn(` ⚠ Web search verification is not implemented for agent '${agentName}'.`);
83+
}
84+
} catch {
85+
// Best-effort — don't let probe failures derail onboarding.
86+
warn(" ⚠ Web search verification probe failed (non-fatal).");
87+
}
88+
}

‎test/onboard.test.ts‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4952,10 +4952,16 @@ const { setupInference } = require(${onboardPath});
49524952
});
49534953

49544954
it("uses named sandbox exec for dashboard and web-search probes", () => {
4955-
const source = fs.readFileSync(
4955+
const onboardSource = fs.readFileSync(
49564956
path.join(import.meta.dirname, "..", "src", "lib", "onboard.ts"),
49574957
"utf-8",
49584958
);
4959+
const webSearchVerifySource = fs.readFileSync(
4960+
path.join(import.meta.dirname, "..", "src", "lib", "onboard", "web-search-verify.ts"),
4961+
"utf-8",
4962+
);
4963+
const source = `${onboardSource}
4964+
${webSearchVerifySource}`;
49594965

49604966
assert.match(source, /"sandbox",\s*"exec",\s*"-n",\s*sandboxName,\s*"--",\s*"curl"/);
49614967
assert.match(source, /"sandbox",\s*"exec",\s*"-n",\s*sandboxName,\s*"--",\s*"hermes"/);

0 commit comments

Comments
 (0)