Add harness certification matrix#32
Conversation
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThis PR adds a harness certification workflow: a new CLI command generates per-harness evidence JSON and an evidence index, the TUI shows harness status rows and a matrix, the certification script runs and verifies the new output, and the README and architecture docs describe the workflow. ChangesHarness Certification Matrix
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant User
participant tree-ring CLI
participant certify_harnesses
participant Filesystem
participant TUI
User->>tree-ring CLI: tree-ring integrations certify --source-root .
tree-ring CLI->>certify_harnesses: HarnessCertificationRequest
certify_harnesses->>Filesystem: scan markers and read .tree-ring guidance
certify_harnesses->>Filesystem: write harness/<id>.json
certify_harnesses->>Filesystem: merge evidence-index.json
certify_harnesses-->>tree-ring CLI: HarnessCertificationReport
User->>TUI: open /evidence view
TUI->>Filesystem: read evidence-index.json
TUI-->>User: render harness matrix rows
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements Phase 2 of the Tree Ring Harness Certification Matrix, introducing the tree-ring integrations certify CLI command to generate non-mutating harness compatibility evidence for several agent frameworks. It adds the harness_evidence module to handle probe records, updates the TUI to render the harness matrix under the /evidence view, and integrates these checks into the repository's certification script. The feedback suggests two key improvements in harness_evidence.rs: first, updating merge_harness_index to insert or update entries in the index.harness map rather than completely overwriting it, which prevents losing existing records; second, converting the markdown guidance text to lowercase before checking for commands to make the detection case-insensitive and more robust.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| index.harness = records | ||
| .iter() | ||
| .map(|record| { | ||
| ( | ||
| record.harness_id.clone(), | ||
| EvidenceRecordRef { | ||
| category: "harness".to_string(), | ||
| status: record.status, | ||
| label: record.name.clone(), | ||
| path: PathBuf::from(format!("harness/{}.json", record.harness_id)), | ||
| summary_path: None, | ||
| generated_at: record.generated_at.clone(), | ||
| }, | ||
| ) | ||
| }) | ||
| .collect::<BTreeMap<_, _>>(); |
There was a problem hiding this comment.
The current implementation of merge_harness_index completely overwrites the index.harness map with the newly certified harnesses. If there are other harness records already present in evidence-index.json (either from manual additions or other tools), they will be lost. To align with the PR description of merging records, we should insert/update the map entries instead of replacing the entire map.
for record in records {
index.harness.insert(
record.harness_id.clone(),
EvidenceRecordRef {
category: "harness".to_string(),
status: record.status,
label: record.name.clone(),
path: PathBuf::from(format!("harness/{}.json", record.harness_id)),
summary_path: None,
generated_at: record.generated_at.clone(),
},
);
}| HarnessGuidanceEvidence { | ||
| agents_md, | ||
| skill_md, | ||
| cli_md, | ||
| recall_guidance: combined.contains("tree-ring recall"), | ||
| remember_guidance: combined.contains("tree-ring remember"), | ||
| } |
There was a problem hiding this comment.
The check for guidance commands is currently case-sensitive. If a user capitalizes the commands in their markdown documentation (e.g., Tree-ring recall or TREE-RING REMEMBER), the certification will fail or skip incorrectly. Converting the combined string to lowercase before performing the contains checks makes the detection significantly more robust.
let combined_lower = combined.to_lowercase();
HarnessGuidanceEvidence {
agents_md,
skill_md,
cli_md,
recall_guidance: combined_lower.contains("tree-ring recall"),
remember_guidance: combined_lower.contains("tree-ring remember"),
}
1 participant
Summary
tree-ring integrations certifyto generate non-mutating harness evidence for Codex, Claude Code, OpenCode, Goose, Pi, and Agent Zero/A0.target/tree-ring-certification/harness/*.jsonrecords and merge them intoevidence-index.jsonwithout replacingmetrics.jsoncertification semantics./evidence, and document the command/output paths and skip-state truthfulness.Certification Evidence
6,292,960bytes6,216KB6,172KB3.409ms / 6.210ms7.711ms / 13.922mspass_count=5,fail_count=0,skip_count=1Verification
cargo fmt --checkcargo test --lockedcargo clippy --locked --all-targetsgit diff --checksh scripts/certify-tree-ring.shSummary by CodeRabbit
tree-ring integrations certifyto generate non-mutating harness certification evidence with JSON/human-readable reporting.