SPIKE: Investigate Auto-Acceptance Threshold

[m-goggins]

Background

TTC relies on a retriever to identify candidate matches and a reranker to select the best standardization result. Initial analysis suggests that the retriever performs well, while the reranker struggles in some scenarios.

This behavior may not be solely attributable to reranker quality. Rerankers are generally most effective when making nuanced distinctions between multiple plausible candidates. They are less effective when:

• The correct answer is already obvious from retriever outputs.
• All retrieved candidates are poor matches.
• The retriever returns candidates with uniformly low similarity scores, making it difficult for the reranker to identify meaningful signal.

One potential optimization is introducing an auto-acceptance threshold based on the retriever's top cosine similarity score. The retriever frequently identifies highly relevant candidates before the reranker is applied. In cases where the retriever's top candidate has both a high cosine similarity score and a large margin over the second-best candidate, the reranker may provide little additional value.If the highest-scoring retrieved candidate is above a certain threshold, the system could skip the reranker and just return the top result from the retriever.

Acceptance Criteria

• Distribution analysis

  • Quantify the % and count of inputs that would be accepted under different retriever similarity thresholds (e.g., 0.8, 0.7, etc.) to measure how much of the training pipeline traffic would be affected if we used that threshold as an auto-acceptance rule.
  • Produce summary stats for the distribution of margins between top1 and top 2 scores
  • Create visualizations showing the distribution of the margins between the top 2 scores

• Threshold impact analysis

  • Quantify the percentage and count of inputs that would be automatically accepted under candidate top-score thresholds (e.g., ≥0.8, ≥0.85, ≥0.9, ≥0.95).
  • Quantify the percentage and count of inputs that would be automatically accepted under candidate margin thresholds (e.g., ≥0.05, ≥0.1, ≥0.15).
  • Quantify the percentage and count of inputs that satisfy combinations of score and margin thresholds.

• Agreement analysis

  • Measure how often the retriever's top candidate matches the reranker's final selected candidate.
  • Analyze retriever-reranker agreement rates across:
    • Top-score ranges
    • Margin ranges
    • Combined score-and-margin ranges
    • Identify areas where retriever-reranker agreement is consistently high.

• Manually inspect representative examples from low-scoring ranges

  • Perform a manual review of representative examples from high-score and high-margin ranges.
  • Review examples where:
    • The retriever and reranker agree.
    • The retriever and reranker disagree.
  • See if there are any patterns in the agreements and/or disagreements

• Produce a recommendation deck for APHL summarizing the analysis, proposed theshold(s) if applicable, rationale, and expect impact on coverage and prediction quality (with concrete examples if possible)

  • Determine whether there is evidence supporting an auto-acceptance threshold based on the top retriever cosine similarity score.