Skip to content

fix(scorer): require hard signals for likely-test - #3

Merged
Kzoeps merged 1 commit into
mainfrom
fix/scorer-hard-signals-likely-test
Jun 1, 2026
Merged

fix(scorer): require hard signals for likely-test#3
Kzoeps merged 1 commit into
mainfrom
fix/scorer-hard-signals-likely-test

Conversation

@Kzoeps

@Kzoeps Kzoeps commented Jun 1, 2026 •
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown
Collaborator

Summary

  • Make likely-test signal-based instead of low-score-based
  • Keep configured TEST_PDS_HOSTS as hard likely-test evidence
  • Allow test, testing, and tested in profile descriptions
  • Keep short metadata test words as validation notes
  • Update docs/UI copy and add scorer/authenticity tests

Validation

  • npm test
  • npm run build
  • Ran isolated local Tap + labeler + dashboard stack and verified TEST_PDS_HOSTS/trusted PDS behavior locally

Notes

  • HF classifier behavior is intentionally left unchanged for now.

Summary by CodeRabbit

  • Documentation

    • Updated test detection and quality tier documentation to reflect revised labeling rules.

  • Behavior Updates

    • "⚠ Likely Test" badge now triggered by explicit hard test signals (placeholder domains, test patterns) rather than score ranges.
    • Validation notes are now informational only and no longer affect organization tier assignments.

@coderabbitai

coderabbitai Bot commented Jun 1, 2026 •
edited
Loading

Copy link
Copy Markdown

Review Change Stack

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Free

Run ID: c26f5432-b8c4-4d7a-b76c-7aacbf108827

📥 Commits

Reviewing files that changed from the base of the PR and between 72f1bf0 and 415c641.

📒 Files selected for processing (10)
  • README.md
  • src/app/docs/page.tsx
  • src/components/ScoreBreakdown.tsx
  • src/lib/constants.ts
  • src/lib/hf-classifier.ts
  • src/lib/scorer.ts
  • src/lib/scoring-authenticity.ts
  • src/lib/types.ts
  • tests/scorer-tiering.test.ts
  • tests/scoring-authenticity.test.ts
📝 Walkthrough

Walkthrough

This PR refactors the organization labeler's scoring system to distinguish "hard test signals" (evidence that forces the likely-test tier) from "softer validation notes" (data-quality issues shown to operators without changing tier). Constants, authenticity evaluation, scorer integration, HF classifier, tests, and documentation are updated to implement the new two-signal model.

Changes

Scoring and Tiering Logic Refactor

Layer / File(s) Summary
Constants and pattern reorganization
src/lib/constants.ts		 Tier alias ScoreBasedQualityTier excludes likely-test; score thresholds now map standard: 0–69 and high-quality: 70+. Introduces TEST_WORD_PATTERNS and separates field-specific authenticity patterns: SHORT_FIELD_AUTHENTICITY_TEXT_PATTERNS (includes test words) and DESCRIPTION_AUTHENTICITY_TEXT_PATTERNS (placeholder-focused).
Type definitions and documentation
src/lib/types.ts		 Updates ScoreResult.testSignals documentation to describe hard evidence forcing the likely-test tier.
Authenticity evaluation refactoring
src/lib/scoring-authenticity.ts		 AuthenticityGateResult now returns separate testSignals and validationNotes instead of single signals array. Validators distinguish placeholder text (pushing to testSignals) from invalid optional data (pushing to validationNotes). Pass condition changed to testSignals.length === 0.
Scorer integration with new authenticity model
src/lib/scorer.ts		 Merges validation notes from both record and authenticity sources; computes tier via tierForScore(totalScore, testSignals); removes AUTHENTICITY_FAILURE_TIER constant and returns testSignals in result.
HF classifier tier recomputation
src/lib/hf-classifier.ts		 Imports tierForScore and recomputes tier during HF reclassification; triggers _onReclassify callback only when tier changes.
Authenticity evaluation test updates
tests/scoring-authenticity.test.ts		 Updates assertions to check testSignals and validationNotes separately; verifies placeholder patterns and test words populate testSignals, while invalid optional fields populate validationNotes without failing the gate.
Scoring and tiering test coverage
tests/scorer-tiering.test.ts		 Adds fixture helpers and tests verifying: sparse non-test records tier as standard, complete high-scoring records tier as high-quality, hard test signals override to likely-test, and tierForScore boundary behavior.
Documentation and UI updates
README.md, src/app/docs/page.tsx, src/components/ScoreBreakdown.tsx		 Updates docs, API docs page, and UI text to describe the new model: hard test signals override tiering, validation notes are informational only and do not change tier.

🎯 3 (Moderate) | ⏱️ ~25 minutes

🐰 A rabbit's refactor so fine,
Separates signals from notes so divine,
Test evidence hard, validation so sweet,
No more ambiguous, the tier's complete! ✨

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

Comment @coderabbitai help to get the list of available commands and usage tips.

@sonarqubecloud

sonarqubecloud Bot commented Jun 1, 2026

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@Kzoeps
Kzoeps merged commit a84b9ed into main Jun 1, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Kzoeps