Skip to content

Commit f7c159e

Browse files
authored
feat: npm run audit — read-only drift detector for state/dashboard divergence (#27)
* feat: npm run audit command for orphan/duplicate/drift detection Adds `npm run audit -- <org>` — single-command audit for the state-vs-dashboard drift conditions that have been accumulating cruft in customer-fork repos. Detects (read-only): - orphan local YAML files (no state entry — Scenario B leftovers) - state ghosts (state UUID missing on dashboard) - state UUID collisions (cascade-duplicate fingerprint) - content-identical resources (same lastPulledHash) - sibling base-slug clusters (cascade-risk warning) - dashboard orphans (UUID not in state; suppressed by .vapi-ignore) - assistants with inline model.tools (suspected duplicate-spawn surface) Exit code: 0 if clean, 1 if any findings. Designed for DI: state loader, local file lister, remote fetcher are all injectable, making tests filesystem-free and network-free. Promotes `listExistingResourceIds` in src/pull.ts from `function` to `export function` (one-word edit) to avoid duplicating the directory walker. Tests in tests/audit.test.ts will be added in a follow-up commit on this branch. * test: add coverage for npm run audit (7 checks + formatter + integration) Covers all 7 audit checks via DI fixtures (no filesystem, no network): - orphan-yaml (3 cases) - state-ghost (3 cases inc. fetchRemote=false short-circuit) - state-uuid-collision (2 cases) - content-identical (3 cases inc. missing-hash safety) - sibling-base-slug (3 cases inc. cross-ref overlap) - dashboard-orphan (4 cases inc. .vapi-ignore suppression) - inline-tools (4 cases inc. async-Promise branch) Plus: 1 integration test combining multiple checks, 1 exit-code mapping test, 3 formatter tests. * refactor: extract exitCodeForFindings helper, pin test to it Closes the gap surfaced by the test-writer phase: audit-cmd.ts had inlined `findings.length === 0 ? 0 : 1` at every exit-code call site, so the exit-code test in tests/audit.test.ts could only assert on a parallel re-derivation rather than the real CLI behavior. Extracts a tiny exported `exitCodeForFindings(findings)` helper and routes both exit sites through it. Test imports and pins to the helper, so future changes to the severity bar (e.g. a `--strict` flag in v2) will surface in the existing assertion instead of silently drifting. No behavior change. 155/155 tests pass. * fix(audit): use Promise.allSettled for per-type fetches + README entry Addresses two non-blocking code-review findings before opening the PR: 1. **Fail-fast → fail-graceful for dashboard fetches.** Switched the parallel per-type API calls from `Promise.all` to `Promise.allSettled`. A transient 500 / 429 / network blip on one resource type used to abort the entire audit, leaving the operator with zero findings instead of findings-for-the-types-that-succeeded. Now: each failed fetch emits a `fetch-failed` finding (severity: warn, message includes the underlying error). The per-type loop checks `remoteByType.has(type)` before running state-ghost and dashboard-orphan checks — preventing the would-be false-positive where an empty-array fallback marks every state entry as a ghost. New rule: `AuditRule = ... | "fetch-failed"`. 2. **README command table missing `audit`.** Added a row under the `validate` entry so operators discover the command from the same surface that lists `pull`/`push`/`cleanup`/`rollback`/etc. New test pinning the fail-graceful path: one type's `remoteFetcher` throws → exactly 1 `fetch-failed` finding for that type, 0 false-positive state-ghost findings for any state entry of that type, and other types' checks proceed normally. Suite: 156/156 pass (+1 test).
1 parent 0c7a7aa commit f7c159e

6 files changed

Lines changed: 1216 additions & 1 deletion

File tree

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@ Every command works in two modes:
7272
| --- | --- | --- | --- |
7373
| `npm run setup` ||| First-time org wizard — creates `.env.<org>` and `resources/<org>/`. |
7474
| `npm run validate` || `npm run validate -- <org>` | Schema-check local YAML/MD with no network call. **Run before every `apply`.** |
75+
| `npm run audit` || `npm run audit -- <org> [--type <t>]` | Read-only drift detector — orphan local YAML, state ghosts, UUID collisions, content-identical clusters, sibling base-slug clusters, dashboard orphans, assistants with inline `model.tools`. Exit 1 on any finding; safe to wire into CI. |
7576
| `npm run apply` || `npm run apply -- <org> [--force]` | **Default deploy verb.** Pull → merge → push in one safe pass; resilient against dashboard drift. |
7677
| `npm run pull` || `npm run pull -- <org> [flags]` | Fetch remote state into local files / state file. Local-first by default — won't clobber local edits. |
7778
| `npm run push` || `npm run push -- <org> [flags]` | Raw push without a pre-pull. **Skip unless you just ran `pull` and are certain state is fresh** — otherwise prefer `apply`. |

package.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
"call": "bash -c 'exec tsx src/call-cmd.ts \"$@\" 2> >(grep --line-buffered -v \"buffer underflow\" >&2)' --",
1414
"cleanup": "tsx src/cleanup-cmd.ts",
1515
"validate": "tsx src/validate-cmd.ts",
16+
"audit": "tsx src/audit-cmd.ts",
1617
"sim": "tsx src/sim-cmd.ts",
1718
"rollback": "tsx src/rollback-cmd.ts",
1819
"build": "tsc --noEmit",

src/audit-cmd.ts

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
// CLI entry: `npm run audit -- <org>`
2+
//
3+
// Read-only audit for the state-vs-dashboard drift conditions that have been
4+
// accumulating cruft in customer-fork repos. Mirrors `src/validate-cmd.ts` for
5+
// argument parsing and env banner so the operator experience is consistent
6+
// across the engine.
7+
//
8+
// Exit code: 0 if no findings, 1 if any (warn or error). No `--strict` flag in
9+
// v1 — a single severity bar keeps the surface small while we observe what
10+
// shows up in real customer state.
11+
12+
import { resolve } from "path";
13+
import { fileURLToPath } from "url";
14+
import {
15+
type AuditFinding,
16+
formatFinding,
17+
runAudit,
18+
summarizeFindings,
19+
} from "./audit.ts";
20+
import { APPLY_FILTER, VAPI_BASE_URL, VAPI_ENV } from "./config.ts";
21+
import type { ResourceType } from "./types.ts";
22+
import { VALID_RESOURCE_TYPES } from "./types.ts";
23+
24+
// Single source of truth for the exit-code contract. Exported so tests can pin
25+
// behavior without duplicating the predicate.
26+
export function exitCodeForFindings(findings: AuditFinding[]): 0 | 1 {
27+
return findings.length === 0 ? 0 : 1;
28+
}
29+
30+
function groupFindings(
31+
findings: AuditFinding[],
32+
): Map<ResourceType, AuditFinding[]> {
33+
const grouped = new Map<ResourceType, AuditFinding[]>();
34+
for (const f of findings) {
35+
const arr = grouped.get(f.type) ?? [];
36+
arr.push(f);
37+
grouped.set(f.type, arr);
38+
}
39+
// Stable inner ordering: by rule, then by first resourceId.
40+
for (const arr of grouped.values()) {
41+
arr.sort((a, b) => {
42+
if (a.rule !== b.rule) return a.rule.localeCompare(b.rule);
43+
const aFirst = a.resourceIds[0] ?? "";
44+
const bFirst = b.resourceIds[0] ?? "";
45+
return aFirst.localeCompare(bFirst);
46+
});
47+
}
48+
return grouped;
49+
}
50+
51+
async function main(): Promise<void> {
52+
console.log(
53+
"═══════════════════════════════════════════════════════════════",
54+
);
55+
console.log(`🔎 Vapi GitOps Audit - Environment: ${VAPI_ENV}`);
56+
console.log(` API: ${VAPI_BASE_URL}`);
57+
console.log(
58+
"═══════════════════════════════════════════════════════════════\n",
59+
);
60+
61+
// Respect --type filter (parsed by config.ts into APPLY_FILTER). When the
62+
// operator passes one or more --type flags we audit only those types; the
63+
// default sweep covers every entry in VALID_RESOURCE_TYPES.
64+
const types: ResourceType[] = APPLY_FILTER.resourceTypes?.length
65+
? APPLY_FILTER.resourceTypes
66+
: [...VALID_RESOURCE_TYPES];
67+
68+
if (APPLY_FILTER.resourceTypes?.length) {
69+
console.log(`🔧 Type filter: ${types.join(", ")}\n`);
70+
}
71+
72+
const findings = await runAudit({ types });
73+
74+
console.log(summarizeFindings(findings));
75+
76+
if (exitCodeForFindings(findings) === 0) {
77+
process.exit(0);
78+
}
79+
80+
// Group findings by resource type → rule for human-readable output.
81+
const grouped = groupFindings(findings);
82+
83+
// Iterate types in the configured filter order so the operator can scan top-down.
84+
for (const type of types) {
85+
const arr = grouped.get(type);
86+
if (!arr?.length) continue;
87+
console.log(`\n${type} (${arr.length} finding(s)):`);
88+
for (const f of arr) {
89+
console.log(formatFinding(f));
90+
}
91+
}
92+
93+
// Any finding → exit 1. v1 has no --strict gate; a warning still indicates
94+
// operator-actionable drift.
95+
process.exit(exitCodeForFindings(findings));
96+
}
97+
98+
const isMainModule =
99+
process.argv[1] !== undefined &&
100+
resolve(process.argv[1]) === fileURLToPath(import.meta.url);
101+
102+
if (isMainModule) {
103+
main().catch((error) => {
104+
console.error(
105+
"\n❌ Audit failed:",
106+
error instanceof Error ? error.message : error,
107+
);
108+
process.exit(1);
109+
});
110+
}

0 commit comments

Comments
 (0)