Skip to content

test: add deterministic MCP trace corruption factory#160

Merged
ProfRandom92 merged 3 commits into
mainfrom
codex/add-deterministic-mcp-trace-corruption-factory
May 21, 2026
Merged

test: add deterministic MCP trace corruption factory#160
ProfRandom92 merged 3 commits into
mainfrom
codex/add-deterministic-mcp-trace-corruption-factory

Conversation

@ProfRandom92

Copy link
Copy Markdown
Owner

Motivation

  • Turn existing MCP trace replay fixtures into deterministic, labeled corruption cases without adding runtime orchestration, LLM judging, embeddings, fuzzy matching, or schema redesign.
  • Provide a narrow, reviewable first-step corruption factory that maps structural mutation operators to the repository's existing failure taxonomy.

Description

  • Add a pure-Python, standard-library-only generator at scripts/generate_mcp_trace_corruptions.py that scans checked-in MCP fixtures, evaluates operator applicability structurally, and emits a stable manifest at artifacts/mcp_trace_corruption_manifest.json.
  • Introduce an initial operator set: DROP_APPROVAL_GATE, SWAP_TOOL_ORDER, TRUNCATE_RECOVERY_PATH, REMOVE_DEPENDENCY_EDGE, INSERT_UNVALIDATED_ACTION, and COLLAPSE_CAPABILITY_BOUNDARY, each mapped to existing registered failure labels and expected contract violations.
  • Commit the deterministic, pretty-printed manifest artifacts/mcp_trace_corruption_manifest.json containing ordered corruption entries and a deterministic summary.
  • Add focused tests in tests/test_mcp_trace_corruption_manifest.py to assert manifest presence, stable top-level schema, deterministic ordering, unique corruption_ids, allowed operators, registered failure labels, relative/existing fixture paths, absence of time/env fields, and generator reproducibility; keep this PR manifest-only (no mutated fixture files) and do not change the evidence index in this PR.

Testing

  • Ran python scripts/generate_mcp_trace_corruptions.py to produce artifacts/mcp_trace_corruption_manifest.json successfully.
  • Ran pytest -q tests/test_mcp_trace_corruption_manifest.py and the new tests passed.
  • Ran full npm run check, which completed the build/typecheck and the full Python test suite (290 tests) successfully.

Codex Task

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a deterministic MCP trace corruption manifest along with a generation script and a validation test suite. The feedback suggests enhancing the generation script's robustness by adding error handling for JSON operations, verifying the presence of all required fixture files, and properly handling potential null values. Improvements were also proposed for the test suite's privacy-token validation logic and the manifest's output encoding.

Comment thread scripts/generate_mcp_trace_corruptions.py Outdated
Comment thread scripts/generate_mcp_trace_corruptions.py Outdated
Comment thread scripts/generate_mcp_trace_corruptions.py Outdated
Comment thread scripts/generate_mcp_trace_corruptions.py Outdated
Comment thread tests/test_mcp_trace_corruption_manifest.py Outdated

Copy link
Copy Markdown
Owner Author

Graph handling note:

The dependency_graph.json path is now covered by the same deterministic hardening as the trace/state inputs:

  • dependency_graph.json is part of REQUIRED_FIXTURE_FILES, so incomplete MCP fixtures fail explicitly instead of being silently skipped.
  • _load_json validates that the graph file exists, is valid JSON, and decodes to a JSON object.
  • _supports_operator reads dependency_graph.edges through _as_list; null is accepted as an empty list, but any non-list value fails with a deterministic RuntimeError.
  • Individual edge entries are guarded with isinstance(edge, dict) before source/target access, so malformed list items do not crash the generator.

This keeps graph validation strict at the fixture/file/field level while remaining manifest-only: no graph mutation files are generated in this PR.

@ProfRandom92 ProfRandom92 merged commit 8333b4c into main May 21, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant