Add artifact evidence index#156
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a deterministic evidence index system, including a generation script, the resulting JSON index, and a comprehensive test suite. The scripts/generate_evidence_index.py script aggregates metadata from various artifacts, while tests/test_evidence_index.py ensures schema stability, determinism, and data integrity. Feedback was provided to improve the readability of the artifact filtering logic in the generation script by utilizing the walrus operator.
| artifacts = [ | ||
| entry | ||
| for spec in sorted(ARTIFACT_SPECS, key=lambda item: item["path"]) | ||
| for entry in [_build_artifact_entry(spec, manifest_families)] | ||
| if entry is not None | ||
| ] |
There was a problem hiding this comment.
The nested list comprehension used here to filter None values is a bit convoluted. Since the project targets Python 3.9+ (as evidenced by the use of dict[str, Any]), you can use the assignment expression (walrus operator) to make this more readable and idiomatic.
artifacts = [
entry
for spec in sorted(ARTIFACT_SPECS, key=lambda item: item["path"])
if (entry := _build_artifact_entry(spec, manifest_families)) is not None
]
Motivation
Description
scripts/generate_evidence_index.py, a pure-Python deterministic generator that uses a static artifact mapping, inspects only checked-in artifacts, extracts top-level JSON keys and explicitfamilies/familyvalues, computes manifest-alignment, and writes a stable pretty JSON index toartifacts/evidence_index.json.artifacts/evidence_index.jsoncontaining sorted artifact entries, JSON/SVG classification, evidence-bearing vs visualization-only flags, manifest alignment indicators, and deterministic/no-LLM/no-external-API signals.tests/test_evidence_index.pythat verifies artifact existence, generator reproducibility, top-level schema stability, determinism and sanitization (no timestamps/generated_at/env/user/host/hash/digest/absolute paths), path existence, sorted entries, JSON top-level key extraction, SVG visualization-only marking, and summary-count correctness.scripts/generate_evidence_index.py,artifacts/evidence_index.json, andtests/test_evidence_index.py; scope note: deterministic evidence index only with no fixture, taxonomy, README, validator, workflow, or hash/digest changes.Testing
python scripts/generate_evidence_index.pyand confirmed the output pathartifacts/evidence_index.jsonwas produced as committed.pytest tests/test_evidence_index.py -qand related artifact tests (tests/test_failure_taxonomy.py,tests/test_fixture_manifest.py,tests/test_capability_boundary_replay_artifact.py,tests/test_tool_ordering_replay_artifact.py,tests/test_graph_diff_artifact.py) and observed they passed; full project checks includingnpm run checkand the full pytest suite also passed.Codex Task