EPIC: Semantic Anchor Evaluations across LLMs

[raifdmueller]

Goal

Systematically evaluate whether our semantic anchors work across different LLMs. Build a multiple-choice evaluation framework that is deterministic, cheap, and reproducible.

Status: Phase 2 complete, Phases 3-4 partially done

Phase 1: Pilot ✅

• Pilot: Manual evaluation with 5 anchors and 2 models #330 — Manual pilot with 5 anchors, 7 models — CLOSED

Phase 2: Question authoring ✅

• Write Level 1 (Recognition) questions for Tier 3 anchors #331 — Level 1 (Recognition) questions for 66 anchors — CLOSED
• Write Level 2 (Application) questions for top anchors #332 — Level 2 (Application) questions for 59 anchors — CLOSED
• Write Level 3 (Differentiation) questions for conflict groups #333 — Level 3 (Differentiation) questions for conflict groups — OPEN
• Write Level 4 (Consistency) variants for anchor aliases and languages #334 — Level 4 (Consistency) variants (aliases + language) — OPEN

Phase 3: Automation ✅

• Build evaluation runner script #335 — Build evaluation runner script — CLOSED
• Build results heatmap for the website #336 — Build results heatmap for the website — OPEN (static HTML report exists)

Phase 4: Execution ✅

• Run full evaluation and publish results #337 — Run full evaluation across 10 models — CLOSED

Results (193 questions × 10 models)

Model	Score
claude-opus-4-6	99%
gpt-5.4-2026-03-05	99%
claude-sonnet-4-20250514	99%
claude-haiku-4-5-20251001	98%
gpt-4o	97%
gpt-5.4-mini-2026-03-17	97%
mistral-large-2512	96%
devstral-2512	96%
mistral-medium-2508	85%
mistral-small-2603	74%

Related issues

• Evaluation insights: integrate findings into anchor catalog #362 — Integrate evaluation insights into anchor catalog
• feat: Evaluation framework for Semantic Contracts (compliance testing) #370 — Evaluation framework for Semantic Contracts (compliance testing)

[raifdmueller]

Analysis of Sub-Issues

Dependencies & Critical Path

#330 Pilot
  ├─→ #331 L1 Questions ─┐
  ├─→ #332 L2 Questions ──┼─→ #334 L4 Variants
  └─→ #333 L3 Questions ─┘        │
                                    ├─→ #335 Runner ─→ #337 Run & Publish
                                    └─→ #336 Heatmap ─┘

Critical path: #330 → #331 → #335 → #337. Everything else can run in parallel once the Pilot is done.

Risk Assessment

Risk	Impact	Mitigation
Pilot shows MC format doesn't work	High — invalidates the whole approach	That's exactly why we pilot first. If MC fails, pivot to structured open-ended with manual scoring
Distractor quality varies across anchors	Medium — inflated scores for easy distractors	#331 quality criteria address this. Pilot (#330) validates on 5 examples first
Models don't answer with a single letter	Low — parsing issue, not conceptual	Parser extracts first A–D. Fallback: add "Answer: " prefix to prompt
API costs exceed budget	Low — estimated $5–25	Local models (Llama, DeepSeek) are free. Can drop Mistral to cut costs
Results become stale after model updates	Medium — ongoing cost	Addressed in concept doc. Re-run when budget allows, prioritize commercial models

Observations & Suggestions

#330 (Pilot)

• Good scope, right level of effort. One concern: the pilot tests MC mechanics but doesn't validate whether the scoring pipeline works at scale. Consider adding a task: "Write a minimal 10-line scoring script to verify the parse-and-compare logic works before building the full runner."
• The weak Tier 1 anchor should be chosen deliberately — pick one where we expect failure, so we can validate that the evaluation catches it.

#331 (L1 Questions)

• The auto-generation step is the highest-leverage work in the project. If the script produces good drafts, the human effort drops from 2-3 hours to 30 minutes of review. Consider prioritizing this script.
• "~30 Tier 3 anchors" — the exact count should be verified. The tier evaluation memory lists 95 anchors total with specific tier assignments.

#332 (L2 Questions)

• This is the hardest issue. Hand-crafting 10 realistic scenarios with calibrated paraphrases is the bottleneck. Consider: could we crowdsource these from the community (GitHub Discussion)?
• The candidate list is good but could include one non-technical anchor (e.g., Pyramid Principle or BLUF) to validate the approach beyond software engineering.

#333 (L3 Questions)

• Conflict group 7 ("Chain of Thought standalone vs. task-local") isn't actually a differentiation question — it's about when to use CoT, not about distinguishing it from another anchor. Consider replacing with a real pair, e.g., MoSCoW vs. INVEST (prioritization vs. story quality).
• For groups with 3+ members (DRY/SPOT/SSOT, Socratic/Feynman/Rubber Duck), we'll need multiple questions to cover all pairwise distinctions.

#334 (L4 Variants)

• Depends on Write Level 1 (Recognition) questions for Tier 3 anchors #331, Write Level 2 (Application) questions for top anchors #332, Write Level 3 (Differentiation) questions for conflict groups #333 — this is correct but means it can only start after all questions are written. In practice, variants for L1 questions can be written as soon as Write Level 1 (Recognition) questions for Tier 3 anchors #331 is done, without waiting for Write Level 2 (Application) questions for top anchors #332/Write Level 3 (Differentiation) questions for conflict groups #333. Consider splitting the dependency.

#335 (Runner)

• The "Python or Node.js" decision should be made early. Recommendation: Python. Better API client libraries (anthropic, openai, google-generativeai, mistralai all have first-class Python SDKs). Ollama also exposes an OpenAI-compatible API that the openai package can call.
• Missing: a --model flag to run against a single model (useful for incremental runs and debugging).
• Missing: a --dry-run flag that shows which prompts would be sent without sending them.
• Should support resuming interrupted runs (e.g., after an API error on call 1500 of 2350).

#336 (Heatmap)

• ECharts is the right choice — already in the project, supports heatmaps natively.
• "Link each cell to detail view" is a significant UX effort. Consider starting with a simple tooltip on hover (question, answer, model version) and deferring the full detail view.
• Filterable by level is important but adds complexity. Start with a single combined view, add filters later.

#337 (Run & Publish)

• "Write summary blog post / changelog entry" — this should be its own task or at least clearly optional. The main deliverable is the heatmap, not a blog post.
• Missing task: "Review anomalies and create follow-up issues for anchors that need rework."
• The model-specific notes on anchor pages need a design decision: where exactly on the page? A banner? A collapsible section? This should be specified.

Effort Summary

Phase	Estimated Effort	Dependencies
Phase 1: Pilot	1 afternoon	—
Phase 2: Questions	~8 hours total	Pilot passed
Phase 3: Automation	~2 days	Questions exist
Phase 4: Run & Publish	1 day + API costs	Everything

Total: ~4-5 days of work + $5–25 API costs. This is a very reasonable investment for the value it provides.

Recommended Execution Order

1. Pilot: Manual evaluation with 5 anchors and 2 models #330 Pilot (gate: go/no-go)
2. Write Level 1 (Recognition) questions for Tier 3 anchors #331 L1 Questions + auto-generation script (highest leverage)
3. Build evaluation runner script #335 Runner script (can start once a few L1 questions exist)
4. Write Level 2 (Application) questions for top anchors #332, Write Level 3 (Differentiation) questions for conflict groups #333 L2 + L3 Questions (in parallel)
5. Write Level 4 (Consistency) variants for anchor aliases and languages #334 L4 Variants
6. Build results heatmap for the website #336 Heatmap
7. Run full evaluation and publish results #337 Full run