week_6: Module C (The Librarian) — C.4 decision engine + golden-set decision gate#990
Open
PRAteek-singHWY wants to merge 6 commits into
Open
week_6: Module C (The Librarian) — C.4 decision engine + golden-set decision gate#990PRAteek-singHWY wants to merge 6 commits into
PRAteek-singHWY wants to merge 6 commits into
Conversation
… once report_retrieval_recall and report_calibration each built the live pipeline (DB + embedding model + cross-encoder) independently, loading it twice and reranking every positive row twice per --use_live_embeddings run. Build it once in main and pass (retriever, reranker) into both reports, matching _build_live_pipeline's stated intent. Behavior-preserving: recall@20 285/292, rerank top-1 220/292, ECE 0.046 PASS unchanged.
…ecision gate C.3 (Week 5) produces one honest, calibrated confidence per chunk; C.4 turns it into the action — auto-link into the OpenCRE graph, or route to a human — which is the accuracy gate of the whole pipeline. - decision_engine.py: pure `decide(confidence, candidates, *, threshold, adversarial, update_ambiguous) -> DecisionResult`. Links the top-1 iff confidence >= threshold AND candidates exist AND no blocking flag; otherwise reviews with a reason_code. Reason precedence NO_CANDIDATES > ADVERSARIAL_FLAG > UPDATE_AMBIGUOUS > BELOW_THRESHOLD. Frozen result, versioned ENGINE_NAME, custom DecisionError — mirrors the C.1/C.2/C.3 model-free seams. Does not import the C.3 scaler (confidence-in -> decision-out), so it is hermetically testable. - decision_engine_test.py: 14 hermetic tests — table-driven over every confidence/flag combination, the inclusive >= boundary, all four reason codes, precedence order, and the input guards. - evaluate_librarian.py: additive report_decision_accuracy — fits T on positive+hard_negative, runs the live C.1->C.4 decision over the golden set, and reports overall agreement plus auto-link recall vs review recall (a single accuracy hides that at tau=0.80 the softmax top-1 mass of a correct-but-close winner is often ~0.5, so correct positives route to review — the safe direction; W7 tunes tau). Informational, not a gate: the SafetyGuard flags are not wired yet, so flag-based reason codes lag until that lands. Emitters (LinkProposal/ReviewItem writers) and the C.0->C.4 pipeline glue follow in a stacked week_6b PR.
PRAteek-singHWY
requested review from
Pa04rth,
northdpole,
paoga87 and
robvanderveer
as code owners
July 23, 2026 12:46
Contributor
Summary by CodeRabbit
WalkthroughChangesAdds temperature-scaling calibration and ECE measurement, introduces confidence-based auto-link/review routing, and extends the live librarian evaluation script with shared pipeline construction, calibration reporting, and decision-accuracy reporting. Librarian calibration and routing
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
application/utils/librarian/calibration/temperature.py (1)
63-68: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueMinor duplication between
_softmax_topandprobabilities.Both implement the same "asarray + empty check + softmax" logic independently.
confidence()could derive fromprobabilities()instead of a separate module-level helper, keeping the empty-shortlist guard in one place.♻️ Suggested consolidation
def confidence(self, logits: Sequence[float]) -> float: """P(the top candidate is correct) — the top-1 mass of the softmax. This is the number the W6 decision engine thresholds on. """ - return _softmax_top(logits, self.temperature) + return float(self.probabilities(logits).max())Also applies to: 105-117
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@application/utils/librarian/calibration/temperature.py` around lines 63 - 68, Consolidate the duplicated shortlist conversion, empty-check, and softmax logic by removing or bypassing `_softmax_top` and deriving `confidence()` from the existing `probabilities()` implementation. Ensure `probabilities()` remains the single guard for empty candidate shortlists while preserving the top-1 probability result and temperature behavior.scripts/evaluate_librarian.py (1)
234-244: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy liftLive retrieve+rerank (and the temperature fit) is redundantly recomputed 2-3x per row.
report_decision_accuracyrebuilds the exact samecal_rowsset and rerunsretriever.retrieve/reranker.rerankper row to refit a secondTemperatureScaler, duplicating workreport_calibration(called right before it inmain, L458-459) already did. Then its owngradedloop rerunsretrieve/rerankagain for rows that overlap withcal_rows(e.g. positive-slice rows with an expected decision). Since retrieval/reranking against a live embedding model + cross-encoder is the expensive part this harness gates behind--use_live_embeddings, this triples model calls for no functional benefit — the fit and audits are deterministic given the same inputs.Consider having
report_calibrationreturn the fittedTemperatureScaler(and/or the per-row audits) soreport_decision_accuracyreuses them instead of recomputing, and caching each row'sretrieve+rerankresult by row id so thecal_rows/gradedloops don't redo live calls for the same row.Also applies to: 292-333
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/evaluate_librarian.py` around lines 234 - 244, Refactor report_calibration and report_decision_accuracy to reuse the fitted TemperatureScaler and per-row rerank audits instead of rerunning retrieve and rerank. Have report_calibration return the scaler and/or cached audits, pass them from main into report_decision_accuracy, and ensure overlapping cal_rows and graded rows retrieve each row only once, keyed by a stable row identifier.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@application/utils/librarian/calibration/__init__.py`:
- Around line 1-13: Update the package docstring in the module-level
documentation to describe the implemented shortlist-wide softmax calibration
used by temperature.py, replacing the single-logit sigmoid formula and related
claims. Explain that logits are scaled by a fitted scalar temperature and
normalized across each candidate shortlist, while preserving the existing
purpose, NLL fitting, and ECE context.
---
Nitpick comments:
In `@application/utils/librarian/calibration/temperature.py`:
- Around line 63-68: Consolidate the duplicated shortlist conversion,
empty-check, and softmax logic by removing or bypassing `_softmax_top` and
deriving `confidence()` from the existing `probabilities()` implementation.
Ensure `probabilities()` remains the single guard for empty candidate shortlists
while preserving the top-1 probability result and temperature behavior.
In `@scripts/evaluate_librarian.py`:
- Around line 234-244: Refactor report_calibration and report_decision_accuracy
to reuse the fitted TemperatureScaler and per-row rerank audits instead of
rerunning retrieve and rerank. Have report_calibration return the scaler and/or
cached audits, pass them from main into report_decision_accuracy, and ensure
overlapping cal_rows and graded rows retrieve each row only once, keyed by a
stable row identifier.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yml
Review profile: CHILL
Plan: Pro Plus
Run ID: 3c77096d-5a49-4d8f-9eee-e6a67592cb12
📒 Files selected for processing (7)
application/tests/librarian/decision_engine_test.pyapplication/tests/librarian/temperature_test.pyapplication/utils/librarian/__init__.pyapplication/utils/librarian/calibration/__init__.pyapplication/utils/librarian/calibration/temperature.pyapplication/utils/librarian/decision_engine.pyscripts/evaluate_librarian.py
Comment on lines
+1
to
+13
| """Module C.3 — confidence calibration (Week 5). | ||
|
|
||
| C.2 (the cross-encoder, W4) emits a raw ranking logit per candidate — great for | ||
| ordering, meaningless as confidence (a +1.5 is not "82% sure"). C.3 turns that | ||
| logit into an honest probability via **temperature scaling**: ``p = sigmoid(z/T)`` | ||
| with a single scalar ``T`` fit by negative-log-likelihood on the golden set, and | ||
| proves the result honest with **ECE < 0.10**. | ||
|
|
||
| The W6 decision engine thresholds that probability (auto-link vs. human review), | ||
| so calibration is what makes the threshold trustworthy. Kept dependency-light | ||
| (numpy + scipy) and model-free so it stays hermetically testable — mirrors the | ||
| C.1/C.2 seams. | ||
| """ |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Docstring describes the rejected sigmoid approach, not the actual softmax-over-shortlist implementation.
This package docstring says calibration is p = sigmoid(z/T) on a single logit. But temperature.py's own module docstring explicitly explains why that single-logit sigmoid approach doesn't work and why it instead calibrates softmax(logits / T) over the whole shortlist. This file should describe the actual algorithm to avoid misleading readers.
📝 Suggested fix
-logit into an honest probability via **temperature scaling**: ``p = sigmoid(z/T)``
-with a single scalar ``T`` fit by negative-log-likelihood on the golden set, and
+logit into an honest probability via **temperature scaling**: calibrating the
+*softmax over the whole shortlist*, ``p = softmax(logits / T)``, with confidence
+being the top-1 mass — a single scalar ``T`` fit by negative-log-likelihood on
+the golden set, and
proves the result honest with **ECE < 0.10**.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| """Module C.3 — confidence calibration (Week 5). | |
| C.2 (the cross-encoder, W4) emits a raw ranking logit per candidate — great for | |
| ordering, meaningless as confidence (a +1.5 is not "82% sure"). C.3 turns that | |
| logit into an honest probability via **temperature scaling**: ``p = sigmoid(z/T)`` | |
| with a single scalar ``T`` fit by negative-log-likelihood on the golden set, and | |
| proves the result honest with **ECE < 0.10**. | |
| The W6 decision engine thresholds that probability (auto-link vs. human review), | |
| so calibration is what makes the threshold trustworthy. Kept dependency-light | |
| (numpy + scipy) and model-free so it stays hermetically testable — mirrors the | |
| C.1/C.2 seams. | |
| """ | |
| """Module C.3 — confidence calibration (Week 5). | |
| C.2 (the cross-encoder, W4) emits a raw ranking logit per candidate — great for | |
| ordering, meaningless as confidence (a +1.5 is not "82% sure"). C.3 turns that | |
| logit into an honest probability via **temperature scaling**: calibrating the | |
| *softmax over the whole shortlist*, ``p = softmax(logits / T)``, with confidence | |
| being the top-1 mass — a single scalar ``T`` fit by negative-log-likelihood on | |
| the golden set, and proves the result honest with **ECE < 0.10**. | |
| The W6 decision engine thresholds that probability (auto-link vs. human review), | |
| so calibration is what makes the threshold trustworthy. Kept dependency-light | |
| (numpy + scipy) and model-free so it stays hermetically testable — mirrors the | |
| C.1/C.2 seams. | |
| """ |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@application/utils/librarian/calibration/__init__.py` around lines 1 - 13,
Update the package docstring in the module-level documentation to describe the
implemented shortlist-wide softmax calibration used by temperature.py, replacing
the single-logit sigmoid formula and related claims. Explain that logits are
scaled by a fitted scalar temperature and normalized across each candidate
shortlist, while preserving the existing purpose, NLL fitting, and ECE context.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi @northdpole — Week 6 of Module C. Week 5 produced one honest confidence per chunk; this PR turns that number into the actual decision — auto-link into the graph, or route to a human — which is the accuracy gate of the whole pipeline.
Overview
Week 3 built the search step (C.1), Week 4 the rerank step (C.2), and Week 5 the calibration step (C.3) — a single scalar
Tthat maps the reranked shortlist to a trustworthyconfidence = softmax(logits / T).The problem: a calibrated confidence is only useful if something acts on it. Auto-linking a wrong CRE pollutes the graph; sending everything to a human defeats the point. We need a rule that auto-links when it's safe and escalates when it isn't.
This PR's role: build the decision step (C.4) —
decision_engine.decide(). It links the top-1 candidate iffconfidence >= thresholdand there is a candidate and no blocking safety flag; otherwise it routes to review with areason_code. It's a pure function of(confidence, candidates, flags, threshold) -> DecisionResult— it does not import the C.3 scaler (confidence-in → decision-out), so it stays model-free and hermetically testable, mirroring the C.1/C.2/C.3 seams. Reason-code precedence is total:NO_CANDIDATES > ADVERSARIAL_FLAG > UPDATE_AMBIGUOUS > BELOW_THRESHOLD.The harness gains a decision-accuracy gate: run the live C.1→C.4 decision over the golden set and measure how often
decide()lands on the expected auto-link-vs-review call.Scope: 1 new module + 1 new test + additive harness wiring. The SafetyGuard flags (
adversarial/update_ambiguous) are accepted bydecide()but not yet produced — nothing sets them, so they default to False (declared-degraded until that lands). No frontend, no migration, no behaviour change to OpenCRE proper.What changed
decision_engine.py(new)decide(confidence, candidate_cre_ids, *, threshold, adversarial, update_ambiguous) -> DecisionResult. Links the top-1 iffconfidence >= thresholdAND candidates exist AND no blocking flag; else reviews with areason_code. FrozenDecisionResult, versionedENGINE_NAME, customDecisionError, input guards on confidence/threshold. Model-free and confidence-agnostic so it is hermetically testable — mirrors the C.1embed_fn/ C.2score_fn/ C.3 scaler seams.evaluate_librarian.pyreport_decision_accuracy: fitsTon positive+hard_negative, runs the live C.1→C.4 decision over the golden set, and reports overall agreement plus auto-link recall vs review recall (a single accuracy hides that attau=0.80the softmax top-1 mass of a correct-but-close winner is often ~0.5, so correct positives route to review — the safe direction). Informational, not a hard gate: tuningtauis the Week-7 experiment, and flag-based reason codes lag until the SafetyGuard lands.report_calibration(W5) untouched.decision_engine_test.py(new)>=boundary, all four reason codes, the precedence order, and the input guards.How the pieces connect
flowchart TB conf["C.3 calibrated confidence<br/>+ reranked candidates + flags"] subgraph C4["C.4 — decision engine (this PR)"] rule["decide(): confidence at or above tau ?<br/>AND candidates exist AND no blocking flag"] res["DecisionResult<br/>(decision, confidence, cre_ids, reason_code)"] rule --> res end conf --> rule res --> link["linked -> LinkProposal (W6b emits)"] res --> review["review -> ReviewItem + reason_code<br/>NO_CANDIDATES / ADVERSARIAL_FLAG /<br/>UPDATE_AMBIGUOUS / BELOW_THRESHOLD"]Results
Reading the C.4 line by direction, because a single accuracy hides the story:
tau=0.80many correct-but-close positives fall below the bar and route to review (the safe direction). This is a conservative starting point; Week 7's threshold sweep is exactly the lever that lifts it.What is intentionally not here
DecisionResultinto the RFCLinkProposal/ReviewItemand wiring C.0→C.4. Ships stacked asweek_6b.ood/conformal/ update-detection) that would populateadversarial/update_ambiguous.Tfor the live decision path, and live B→C integration + graph writes (W8) — the pipeline stays dry-run.How to verify locally