test: add deterministic MCP trace corruption factory#160
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a deterministic MCP trace corruption manifest along with a generation script and a validation test suite. The feedback suggests enhancing the generation script's robustness by adding error handling for JSON operations, verifying the presence of all required fixture files, and properly handling potential null values. Improvements were also proposed for the test suite's privacy-token validation logic and the manifest's output encoding.
|
Graph handling note: The
This keeps graph validation strict at the fixture/file/field level while remaining manifest-only: no graph mutation files are generated in this PR. |
Motivation
Description
scripts/generate_mcp_trace_corruptions.pythat scans checked-in MCP fixtures, evaluates operator applicability structurally, and emits a stable manifest atartifacts/mcp_trace_corruption_manifest.json.DROP_APPROVAL_GATE,SWAP_TOOL_ORDER,TRUNCATE_RECOVERY_PATH,REMOVE_DEPENDENCY_EDGE,INSERT_UNVALIDATED_ACTION, andCOLLAPSE_CAPABILITY_BOUNDARY, each mapped to existing registered failure labels and expected contract violations.artifacts/mcp_trace_corruption_manifest.jsoncontaining ordered corruption entries and a deterministic summary.tests/test_mcp_trace_corruption_manifest.pyto assert manifest presence, stable top-level schema, deterministic ordering, uniquecorruption_ids, allowed operators, registered failure labels, relative/existing fixture paths, absence of time/env fields, and generator reproducibility; keep this PR manifest-only (no mutated fixture files) and do not change the evidence index in this PR.Testing
python scripts/generate_mcp_trace_corruptions.pyto produceartifacts/mcp_trace_corruption_manifest.jsonsuccessfully.pytest -q tests/test_mcp_trace_corruption_manifest.pyand the new tests passed.npm run check, which completed the build/typecheck and the full Python test suite (290 tests) successfully.Codex Task