Add memory quality proof loop and ambient ring fullness#35
Conversation
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
|
Warning Review limit reached
Next review available in: 37 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughIntroduces a quality-scenario evaluation pipeline with fixture-based certification reports, managed memory-quality guidance, filtered FTS recall fallback, and fullness-aware TUI ring brightness. ChangesQuality proof loop
Filtered recall fallback
TUI ring fullness
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant Certification
participant QualityRunner
participant SQLiteStore
participant MemoryRetriever
participant QualityEvaluator
Certification->>QualityRunner: Execute fixture pack
QualityRunner->>SQLiteStore: Seed scenario memories
QualityRunner->>MemoryRetriever: Recall scenario query
MemoryRetriever-->>QualityRunner: Return recall candidates
QualityRunner->>QualityEvaluator: Evaluate scenario
QualityEvaluator-->>QualityRunner: Produce quality report
QualityRunner-->>Certification: Write JSON and Markdown artifacts
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a deterministic memory quality proof loop for Tree Ring Memory, adding a quality scenario model and evaluator to tree-ring-memory-core, default quality fixtures, and a certification-only runner. It also integrates quality metrics into the certification suite, updates generated agent guidance with explicit quality gates, and adds relative fullness styling to the TUI. The reviewer feedback highlights several valuable improvement opportunities, including ensuring Windows compatibility for skill file recognition, optimizing memory allocations in scenario validation, using idiomatic slice methods, disabling unused ranking explanations in the runner, and making the certification script's JSON parsing more robust.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| fn is_generated_skill_file(content: &str) -> bool { | ||
| content.starts_with("---\n") && content.contains(SKILL_FRONT_MATTER_MARKER) | ||
| } |
There was a problem hiding this comment.
On Windows systems, files are often checked out with CRLF (\r\n) line endings. Checking content.starts_with("---\n") will fail to recognize these files as generated skill files. It is safer and more portable to support both LF and CRLF line endings.
fn is_generated_skill_file(content: &str) -> bool {
(content.starts_with("---
") || content.starts_with("---
")) && content.contains(SKILL_FRONT_MATTER_MARKER)
}| let candidate_index_by_id = scenario | ||
| .write_candidates | ||
| .iter() | ||
| .enumerate() | ||
| .map(|(index, candidate)| (candidate.id.as_str(), index)) | ||
| .collect::<HashMap<_, _>>(); | ||
| let mut decision_index_by_id = HashMap::new(); | ||
|
|
||
| for (index, decision) in scenario.expected_write_decisions.iter().enumerate() { | ||
| if let Some(previous_index) = | ||
| decision_index_by_id.insert(decision.memory_id.as_str(), index) | ||
| { | ||
| return Err(TreeRingError::Validation(format!( | ||
| "quality scenario {} expected_write_decisions[{index}] duplicates memory_id {} from expected_write_decisions[{previous_index}]", | ||
| scenario.name, decision.memory_id | ||
| ))); | ||
| } | ||
| if !candidate_index_by_id.contains_key(decision.memory_id.as_str()) { | ||
| return Err(TreeRingError::Validation(format!( | ||
| "quality scenario {} expected_write_decisions[{index}] memory_id {} does not match any write_candidate", | ||
| scenario.name, decision.memory_id | ||
| ))); | ||
| } | ||
| } |
There was a problem hiding this comment.
The candidate_index_by_id map is constructed solely to check for the presence of decision.memory_id using contains_key. However, the unique candidate IDs are already collected in the candidate_ids set on lines 586-594. We can reuse candidate_ids directly, which avoids the redundant allocation and construction of candidate_index_by_id entirely.
let mut decision_index_by_id = HashMap::new();
for (index, decision) in scenario.expected_write_decisions.iter().enumerate() {
if let Some(previous_index) =
decision_index_by_id.insert(decision.memory_id.as_str(), index)
{
return Err(TreeRingError::Validation(format!(
"quality scenario {} expected_write_decisions[{index}] duplicates memory_id {} from expected_write_decisions[{previous_index}]",
scenario.name, decision.memory_id
)));
}
if !candidate_ids.contains(decision.memory_id.as_str()) {
return Err(TreeRingError::Validation(format!(
"quality scenario {} expected_write_decisions[{index}] memory_id {} does not match any write_candidate",
scenario.name, decision.memory_id
)));
}
}| if !required_evidence_refs.is_empty() | ||
| && candidate.event_type.starts_with("evaluation_") | ||
| && !required_evidence_refs | ||
| .iter() | ||
| .any(|required| required == &candidate.source.ref_) | ||
| { | ||
| return "require_evidence".to_string(); | ||
| } |
There was a problem hiding this comment.
Instead of manually iterating over required_evidence_refs with .iter().any(...), you can use the more idiomatic and concise .contains(...) method on slices.
if !required_evidence_refs.is_empty()
&& candidate.event_type.starts_with("evaluation_")
&& !required_evidence_refs.contains(&candidate.source.ref_)
{
return "require_evidence".to_string();
}| let recalls = MemoryRetriever::new(&store) | ||
| .recall( | ||
| prompt, | ||
| None, | ||
| None, | ||
| None, | ||
| None, | ||
| None, | ||
| false, | ||
| false, | ||
| RECALL_LIMIT, | ||
| true, | ||
| ) |
There was a problem hiding this comment.
The explain_ranking parameter (the last argument to recall) is set to true. However, the returned RecallResult is immediately mapped to QualityRecall which only retains the memory and score fields, discarding the ranking explanation. Setting this to false avoids the unnecessary overhead of calculating and populating the ranking explanation map.
| let recalls = MemoryRetriever::new(&store) | |
| .recall( | |
| prompt, | |
| None, | |
| None, | |
| None, | |
| None, | |
| None, | |
| false, | |
| false, | |
| RECALL_LIMIT, | |
| true, | |
| ) | |
| let recalls = MemoryRetriever::new(&store) | |
| .recall( | |
| prompt, | |
| None, | |
| None, | |
| None, | |
| None, | |
| None, | |
| false, | |
| false, | |
| RECALL_LIMIT, | |
| false, | |
| ) |
| grep -Fx ' "quality_pass": true,' "$QUALITY_OUT/quality-report.json" > /dev/null \ | ||
| || fail "memory quality scenarios did not pass" |
There was a problem hiding this comment.
Using grep -Fx with an exact string like ' "quality_pass": true,' is fragile because any change in the struct field order, serialization formatting (e.g., indentation, spacing), or line endings (e.g., CRLF on Windows) will break the certification script. It is much more robust to use grep -F '"quality_pass": true' to check for the presence of the field anywhere in the JSON.
| grep -Fx ' "quality_pass": true,' "$QUALITY_OUT/quality-report.json" > /dev/null \ | |
| || fail "memory quality scenarios did not pass" | |
| grep -F '"quality_pass": true' "$QUALITY_OUT/quality-report.json" > /dev/null \ | |
| || fail "memory quality scenarios did not pass" |
1 participant
Summary
Proof integrity
Verification
cargo fmt --all -- --checkgit diff --checkcargo test --locked --workspace --all-targets(223 tests passed)cargo clippy --locked --workspace --all-targets -- -D warningsquality_pass: truesh scripts/certify-tree-ring.shReview
Independent whole-branch review found no Critical or Important issues. Remaining non-blocking risk is direct fault-injection coverage for internal failure branches that already emit fixed sanitized error classes.
Summary by CodeRabbit
New Features
Bug Fixes
Documentation