Skip to content

chore: align MCP corruption manifest with evidence index#161

Merged
ProfRandom92 merged 1 commit into
mainfrom
codex/align-mcp-corruption-manifest-with-evidence-index
May 21, 2026
Merged

chore: align MCP corruption manifest with evidence index#161
ProfRandom92 merged 1 commit into
mainfrom
codex/align-mcp-corruption-manifest-with-evidence-index

Conversation

@ProfRandom92

Copy link
Copy Markdown
Owner

Motivation

  • A new committed artifact artifacts/mcp_trace_corruption_manifest.json must be either registered in the deterministic evidence index or explicitly justified as out of scope per repository evidence-index discipline.
  • The change aims to register the manifest when it fits existing schema semantics while preserving deterministic generation and schema shape.

Description

  • Added artifacts/mcp_trace_corruption_manifest.json to ARTIFACT_SPECS in scripts/generate_evidence_index.py with generator set to scripts/generate_mcp_trace_corruptions.py and evidence_category set to corruption_manifest.
  • Extended fixture-family extraction by adding _family_from_fixture_slug and enhancing _extract_fixture_families to infer families from corruptions[*].source_fixture slugs so the manifest can report fixture_families deterministically.
  • Regenerated artifacts/evidence_index.json via the generator (deterministic output) so the new artifact appears in the committed index without hand-editing the generated file.
  • Changes are limited to scripts/generate_evidence_index.py and the regenerated artifacts/evidence_index.json and do not alter schema shape or runtime behavior.

Testing

  • Ran python scripts/generate_evidence_index.py and it produced artifacts/evidence_index.json successfully.
  • Ran pytest -q tests/test_evidence_index.py and the tests passed (8 passed).
  • Ran full project verification via npm run check and full test suite, both completed successfully (pytest reported 291 passed).

Codex Task

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for MCP trace corruption manifests by updating the evidence index and the generation script. It includes logic to extract fixture families from corruption sources. Feedback was provided to improve the robustness of the regex used for family extraction, which currently fails on baseline fixtures, and to ensure stricter JSON list processing by raising a RuntimeError for invalid types in the 'corruptions' field.

return {str(fixture["family"]) for fixture in manifest["fixtures"]}

def _family_from_fixture_slug(slug: str) -> str | None:
match = re.match(r"^(?P<family>.+)_[^_]+_v\d+$", slug)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The regex r"^(?P<family>.+)_[^_]+_v\d+$" is incorrect for fixture slugs that do not include a degradation level (e.g., baseline fixtures like mcp_trace_replay_v1). In such cases, it would incorrectly extract mcp_trace as the family instead of mcp_trace_replay. Additionally, it is fragile if a family name contains multiple underscores. Using a non-greedy match for the family and explicitly handling the optional degradation levels is more robust.

Suggested change
match = re.match(r"^(?P<family>.+)_[^_]+_v\d+$", slug)
match = re.match(r"^(?P<family>.+?)(?:_(?:mild|moderate|degraded))?_v\d+$", slug)

Comment on lines +113 to +123
if isinstance(payload.get("corruptions"), list):
for corruption in payload["corruptions"]:
if not isinstance(corruption, dict):
continue
source_fixture = corruption.get("source_fixture")
if not isinstance(source_fixture, str):
continue
fixture_slug = source_fixture.rsplit("/", 1)[-1]
family = _family_from_fixture_slug(fixture_slug)
if family:
families.add(family)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

This block violates the general rule regarding JSON list processing. It should treat null as an empty list and raise a RuntimeError for other non-list types to maintain strictness. Additionally, use type guards like isinstance(item, dict) before accessing fields within list items to prevent crashes on malformed data.

Suggested change
if isinstance(payload.get("corruptions"), list):
for corruption in payload["corruptions"]:
if not isinstance(corruption, dict):
continue
source_fixture = corruption.get("source_fixture")
if not isinstance(source_fixture, str):
continue
fixture_slug = source_fixture.rsplit("/", 1)[-1]
family = _family_from_fixture_slug(fixture_slug)
if family:
families.add(family)
corruptions = payload.get("corruptions")
if corruptions is not None:
if not isinstance(corruptions, list):
raise RuntimeError("Field 'corruptions' must be a list if present")
for corruption in corruptions:
if not isinstance(corruption, dict):
continue
source_fixture = corruption.get("source_fixture")
if not isinstance(source_fixture, str):
continue
fixture_slug = source_fixture.rsplit("/", 1)[-1]
family = _family_from_fixture_slug(fixture_slug)
if family:
families.add(family)
References
  1. When processing JSON data, treat null values for expected list fields as empty lists, but raise a RuntimeError for other non-list types to maintain strictness. Use type guards like isinstance(item, dict) before accessing fields within list items to prevent crashes on malformed data.

@ProfRandom92 ProfRandom92 merged commit 7794f69 into main May 21, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant