Skip to content

Commit 997e2b2

Browse files
George-iamclaude
andcommitted
fix: setup_complete missing scanners_run/scanners_failed + extra test coverage
Verification audit found that the setup_complete payload was missing two required spec fields and that the mcp_tool error category was wired nowhere. Also expanded test coverage to validate all Phase 1 + Phase 2 payload shapes against the spec. ## Bugs fixed 1. setup_complete missing scanners_run/scanners_failed (src/cli.ts, src/tools/init.ts) The spec requires both fields. InitResult interface gained the two counters, initProjectWithLLM now increments scannersRun on every non-skipped scanner Promise.allSettled result and scannersFailed on every rejection. cli.ts setup handler reads them from the result and threads into the sendSetupTelemetry payload. 2. mcp_tool error category not wired anywhere (src/server.ts) reportError("mcp_tool", ...) was reserved in the bounded category enum but no call site existed. Wrapped server.tool() with a single monkey-patch right after McpServer construction so every registered tool handler is auto-wrapped in a try/catch that fires reportError with category="mcp_tool", classifyError(err), fatal=true on throw. Verified end-to-end via smoke test: forcing axme_save_memory to throw EACCES on a non-existent path produces a single mcp_tool error event with error_class=permission_denied. ## Tests added (test/telemetry.test.ts) - lifecycle strict counts: first run sends exactly 1 install + 1 startup - lifecycle strict counts: second run sends only 1 startup, no install - update event: previous_version field present when version changed - processStartupSent guard: 3 sendStartupEvents calls = 1 startup - ci=true detection in real sent payload - ci=false default in real sent payload - audit_complete payload shape: all 10 spec fields present - setup_complete payload shape: all 9 spec fields present - offline queue cap at 100 events with oldest-dropped semantics - classifyError extra slugs: api_error, disk_full, permission_denied - reportError payload contains only bounded fields, no message/stack ## Verification - npm test: 475 tests, 104 suites, 0 failures - npm run lint: clean - npm run build: clean - npx tsc --noEmit: clean - E2E staging: 7/7 scenarios passed against axme-gateway-staging Cloud Run URL (install + startup, rerun no install, update with previous_version, opt-out AXME_TELEMETRY_DISABLED + DO_NOT_TRACK leaves no state, offline queue + flush, setup_complete on auth-fail fast path) - E2E Scenario D real audit: ran live LLM audit-session subprocess against staging endpoint on session d5e6391c (15.5MB transcript, verify-only mode). Result: outcome=success, durationMs=574371, promptTokens=130731, costUsd=1.0232553, chunks=1, decisions_saved=1, memories_saved=0, safety_saved=0, dropped_count=0. audit_complete event landed on staging, telemetry-queue.jsonl absent post-run. - Anti-spam check: full MCP boot lifecycle (initialize + tools/list + 3 tool calls + shutdown) produces exactly 1 install + 1 startup. No duplicates from subprocess paths. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 914685e commit 997e2b2

4 files changed

Lines changed: 404 additions & 0 deletions

File tree

src/cli.ts

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -298,6 +298,8 @@ async function main() {
298298
let setupMethod: "llm" | "deterministic" = "deterministic";
299299
let setupPhaseFailed: string | null = null;
300300
let setupPresetsApplied = 0;
301+
let setupScannersRun = 0;
302+
let setupScannersFailed = 0;
301303
// Use the blocking variant so the event lands BEFORE process.exit() runs.
302304
// The fire-and-forget sendTelemetry uses setImmediate, which is killed
303305
// by process.exit() before the network request is even started.
@@ -308,6 +310,8 @@ async function main() {
308310
outcome: setupOutcome,
309311
duration_ms: Date.now() - setupStartMs,
310312
method: setupMethod,
313+
scanners_run: setupScannersRun,
314+
scanners_failed: setupScannersFailed,
311315
phase_failed: setupPhaseFailed,
312316
presets_applied: setupPresetsApplied,
313317
is_workspace: isWorkspace,
@@ -354,6 +358,9 @@ async function main() {
354358
const anyLlm = projectResults.some(r => r.oracle.llm) || workspaceResult.decisions.fromScan > 0;
355359
setupMethod = anyLlm ? "llm" : "deterministic";
356360
setupPresetsApplied = projectResults.reduce((s, r) => s + (r.decisions.fromPresets || 0), 0);
361+
// Sum scanner counts across workspace + all projects
362+
setupScannersRun = workspaceResult.scannersRun + projectResults.reduce((s, r) => s + r.scannersRun, 0);
363+
setupScannersFailed = workspaceResult.scannersFailed + projectResults.reduce((s, r) => s + r.scannersFailed, 0);
357364
} else {
358365
const result = await initProjectWithLLM(projectPath, { onProgress: console.log, force: forceSetup });
359366
if (!result.created && result.durationMs === 0) {
@@ -370,6 +377,8 @@ async function main() {
370377
// Track telemetry: oracle.llm tells us whether LLM path was used
371378
setupMethod = result.oracle.llm ? "llm" : "deterministic";
372379
setupPresetsApplied = (result.decisions.fromPresets || 0);
380+
setupScannersRun = result.scannersRun;
381+
setupScannersFailed = result.scannersFailed;
373382
}
374383
} catch (err) {
375384
setupOutcome = "failed";

src/server.ts

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -232,6 +232,35 @@ const server = new McpServer(
232232
{ instructions: buildInstructions() },
233233
);
234234

235+
// Wrap every tool handler with a try/catch that reports caught exceptions to
236+
// telemetry under the `mcp_tool` category. We monkey-patch server.tool() once
237+
// here instead of touching all 19 individual tool registrations below. The
238+
// MCP SDK still receives any thrown error and returns it to the client, so
239+
// no behavior changes — we only add an extra observability hook.
240+
//
241+
// Why fatal=true: an exception that bubbles out of a tool handler means the
242+
// tool call did not complete its intended work — the user-visible operation
243+
// has aborted. That matches our "fatal vs degraded" definition.
244+
const _origRegisterTool: any = server.tool.bind(server);
245+
(server as any).tool = function (...args: any[]): any {
246+
// Last argument is always the handler function
247+
const handler = args[args.length - 1];
248+
if (typeof handler === "function") {
249+
args[args.length - 1] = async (...handlerArgs: any[]): Promise<any> => {
250+
try {
251+
return await handler(...handlerArgs);
252+
} catch (err) {
253+
try {
254+
const { reportError, classifyError } = await import("./telemetry.js");
255+
reportError("mcp_tool", classifyError(err), true);
256+
} catch { /* never throw from telemetry */ }
257+
throw err;
258+
}
259+
};
260+
}
261+
return _origRegisterTool.apply(server, args);
262+
};
263+
235264
// --- Helper: resolve paths with defaults from server state ---
236265

237266
function pp(project_path?: string): string {

src/tools/init.ts

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,10 @@ export interface InitResult {
3434
cost: CostInfo;
3535
durationMs: number;
3636
errors: string[];
37+
/** Number of LLM scanners that actually executed (max 4: oracle/decision/safety/deploy). 0 in deterministic-only fallback. Used for telemetry. */
38+
scannersRun: number;
39+
/** Number of LLM scanners that failed (rejected or returned an error). Used for telemetry. */
40+
scannersFailed: number;
3741
}
3842

3943
/**
@@ -61,6 +65,7 @@ export async function initProjectWithLLM(projectPath: string, opts?: {
6165
memories: { count: listMemories(projectPath).length, fromPresets: 0 },
6266
safety: { created: false, llm: true, summary: "already initialized" },
6367
config: false, cost: zeroCost(), durationMs: 0, errors: [],
68+
scannersRun: 0, scannersFailed: 0,
6469
};
6570
}
6671
}
@@ -77,6 +82,7 @@ export async function initProjectWithLLM(projectPath: string, opts?: {
7782
memories: { count: 0, fromPresets: 0 },
7883
safety: { created: false, llm: false, summary: "setup already running" },
7984
config: false, cost: zeroCost(), durationMs: 0, errors: ["Setup already in progress"],
85+
scannersRun: 0, scannersFailed: 0,
8086
};
8187
}
8288
atomicWrite(lockPath, new Date().toISOString());
@@ -172,8 +178,14 @@ export async function initProjectWithLLM(projectPath: string, opts?: {
172178

173179
// Process results
174180
log(` [${projectName}] Scanners complete, processing results...`);
181+
// Telemetry counters: how many of the 4 scanners actually ran (non-skipped)
182+
// and how many failed (rejected). Used by setup_complete telemetry payload.
183+
let scannersRun = 0;
184+
let scannersFailed = 0;
175185
for (const settled of scanners) {
176186
if (settled.status === "rejected") {
187+
scannersFailed++;
188+
scannersRun++;
177189
const err = settled.reason;
178190
const msg = err?.message ?? String(err);
179191
const stack = err?.stack ? `\n${err.stack.split("\n").slice(0, 3).join("\n")}` : "";
@@ -182,6 +194,7 @@ export async function initProjectWithLLM(projectPath: string, opts?: {
182194
}
183195
const val = settled.value;
184196
if ("skipped" in val) continue;
197+
scannersRun++;
185198

186199
if (val.type === "oracle" && val.result) {
187200
writeOracleFiles(projectPath, val.result.files);
@@ -263,6 +276,8 @@ export async function initProjectWithLLM(projectPath: string, opts?: {
263276
cost: totalCost,
264277
durationMs: Date.now() - startTime,
265278
errors,
279+
scannersRun,
280+
scannersFailed,
266281
};
267282
}
268283

@@ -352,6 +367,8 @@ export async function initWorkspaceWithLLM(workspacePath: string, opts?: {
352367
cost: zeroCost(),
353368
durationMs: 0,
354369
errors: [`Init failed: ${settled.reason?.message ?? settled.reason}`],
370+
scannersRun: 0,
371+
scannersFailed: 4,
355372
});
356373
}
357374
}
@@ -420,5 +437,6 @@ export function initProjectDeterministic(projectPath: string, opts?: { presets?:
420437
memories: { count: listMemories(projectPath).length, fromPresets: presetsMemoryCount },
421438
safety: { created: true, llm: false, summary: "" },
422439
config: configCreated, cost: zeroCost(), durationMs: Date.now() - startTime, errors: [],
440+
scannersRun: 0, scannersFailed: 0,
423441
};
424442
}

0 commit comments

Comments
 (0)