docs: add "Anchors and Training Data" article (from #582 discussion)#586
Conversation
…scussion) A website article on how a semantic anchor's strength depends on how densely the concept sits in an LLM's training data, with a reproducible A-E experiment across Claude Haiku 4.5, Sonnet 4.6 and Opus 4.8. Wired into render-docs, the router and the main nav (EN), and linked from the Cockburn Use Cases anchor popup (EN + DE). Prompted by the discussion in LLM-Coding#582. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
WalkthroughEin neuer Artikel „Training Data vs Practice" wird zur Dokumentation hinzugefügt und über eine neue Website-Route mit vollständiger Navigation und Übersetzung verfügbar gemacht. Die Änderung umfasst den Kern-Artikel, Cross-References in bestehenden Ankern, Routing, Header-Navigation, Internationalisierung und Build-Integration. ChangesTraining Data vs Practice Article & Website Integration
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@website/src/main.js`:
- Around line 330-337: Der Sprachwechsel aktualisiert die neue Doc-Route nicht;
öffne die Switch in handleLanguageChange() und füge einen Fall für die Route
'/training-data-vs-practice' hinzu so dass beim Sprachwechsel die Seite neu
geladen wird (z.B. durch Aufruf von renderTrainingDataPage() oder durch erneutes
Laden des Dokuments via
loadDocContent('docs/training-data-vs-practice.adoc')/mit dem passenden
lokalisierten Pfad); referenziere renderTrainingDataPage und loadDocContent im
neuen case, damit der Artikel-Inhalt beim Wechsel auf EN/DE korrekt ersetzt
wird.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yml
Review profile: CHILL
Plan: Pro
Run ID: 03a2ef15-0d65-4034-ab90-72bf4e8a000e
📒 Files selected for processing (10)
docs/anchors/cockburn-use-cases.adocdocs/anchors/cockburn-use-cases.de.adocdocs/changelog.adocdocs/training-data-vs-practice.adocscripts/render-docs.jswebsite/src/components/header.jswebsite/src/main.jswebsite/src/translations/de.jsonwebsite/src/translations/en.jsonwebsite/src/utils/router.js
| function renderTrainingDataPage() { | ||
| const pageContent = document.getElementById('page-content') | ||
| if (!pageContent) return | ||
|
|
||
| pageContent.innerHTML = renderDocPage() | ||
| updateActiveNavLink() | ||
| loadDocContent('docs/training-data-vs-practice.adoc') | ||
| } |
There was a problem hiding this comment.
Sprachwechsel aktualisiert diese neue Doc-Route aktuell nicht.
Für /training-data-vs-practice fehlt im handleLanguageChange()-Switch ein Reload-Pfad. Ergebnis: Beim Umschalten EN/DE bleibt der Artikel-Inhalt in der alten Sprache, obwohl der Rest der UI übersetzt wird.
🔧 Vorschlag
function handleLanguageChange() {
const currentRoute = getCurrentRouteSync()
if (currentRoute === '/about') {
loadDocContent('docs/about.adoc')
@@
} else if (currentRoute === '/harness-inventory') {
loadDocContent('docs/harness-inventory.adoc')
+ } else if (currentRoute === '/training-data-vs-practice') {
+ loadDocContent('docs/training-data-vs-practice.adoc')
} else if (currentRoute === '/contracts') {
renderContractsPageHandler()
} else if (currentRoute === '/') {
initCardGridVisualization()
}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@website/src/main.js` around lines 330 - 337, Der Sprachwechsel aktualisiert
die neue Doc-Route nicht; öffne die Switch in handleLanguageChange() und füge
einen Fall für die Route '/training-data-vs-practice' hinzu so dass beim
Sprachwechsel die Seite neu geladen wird (z.B. durch Aufruf von
renderTrainingDataPage() oder durch erneutes Laden des Dokuments via
loadDocContent('docs/training-data-vs-practice.adoc')/mit dem passenden
lokalisierten Pfad); referenziere renderTrainingDataPage und loadDocContent im
neuen case, damit der Artikel-Inhalt beim Wechsel auf EN/DE korrekt ersetzt
wird.
There was a problem hiding this comment.
Pull request overview
This PR adds a new long-form documentation article (“An Anchor Delivers Only as Far as the Prior Reaches”) and wires it into the website so it’s routable, visible in navigation, linked from the Cockburn Use Cases anchor, and recorded in the changelog.
Changes:
- Add new AsciiDoc article
docs/training-data-vs-practice.adocand render it into the website docs build. - Register a new SPA route (
/training-data-vs-practice) and expose it in desktop/mobile navigation (EN/DE). - Link to the new article from the Cockburn Use Cases anchor (EN/DE) and add a changelog entry.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| website/src/utils/router.js | Adds route title mapping for the new article route. |
| website/src/translations/en.json | Adds EN nav label for the new page. |
| website/src/translations/de.json | Adds DE nav label for the new page. |
| website/src/main.js | Registers the new route and adds a page renderer that loads the pre-rendered doc. |
| website/src/components/header.js | Adds the new page link to desktop “More” menu and mobile menu. |
| scripts/render-docs.js | Ensures the new AsciiDoc file is pre-rendered to HTML at build time. |
| docs/training-data-vs-practice.adoc | New article content (with experiment + prompts). |
| docs/changelog.adoc | Adds a changelog entry for the new article. |
| docs/anchors/cockburn-use-cases.adoc | Adds “Further Reading” link to the new article. |
| docs/anchors/cockburn-use-cases.de.adoc | Adds “Weiterführend” link to the new article. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| "nav.agentskill": "AgentSkill", | ||
| "nav.harnessInventory": "Harness Inventory", | ||
| "nav.trainingData": "Anchors & Training Data", | ||
| "nav.workflow": "Spec-Driven Dev", |
| "nav.agentskill": "AgentSkill", | ||
| "nav.harnessInventory": "Harness-Inventar", | ||
| "nav.trainingData": "Anker & Trainingsdaten", | ||
| "nav.workflow": "Spec-Driven Dev", |
| ==== | ||
| *The short version.* A semantic anchor works by triggering a concept the model already learned. Its power is therefore proportional to how _densely_ that concept appears in the training data. We tested this directly: naming "Cockburn use cases" reshapes a generic answer into a full fully-dressed use case (the anchor delivers), while naming "Use-Case 3.0" delivers nothing distinct — the model silently falls back to the nearest concept it does know. That is why an anchor's popup describes the _triggered_ definition, not the state of the art, and why weak-prior terms belong in a *contract* (which supplies its own meaning), not an anchor. | ||
| ==== |
| '/harness-inventory': 'The Harness Inventory — Semantic Anchors', | ||
| '/training-data-vs-practice': 'Anchors and Training Data — Semantic Anchors', | ||
| '/evaluations': 'Evaluations — Semantic Anchors', |
Review Text (GitHub-ready)Cross-model validation: GPT-5, GPT-5-mini, and Gemini 2.5 Flash confirm and extend the findings(German version below) I ran the same A–E experiment against OpenAI GPT-5, GPT-5-mini (via OpenAI API), and Google Gemini 2.5 Flash (via AI Studio), all with empty context — no system prompt, no custom instructions, fresh session per prompt. The results confirm the article's thesis across three model families and surface one notable divergence worth discussing. Finding 1: The Cockburn anchor delivers universally (A → C transition)All five models tested (Claude Opus/Haiku, GPT-5/5-mini, Gemini) show the same structural shift:
Without the anchor: requirements lists, checklists, compliance guides — no use-case structure whatsoever. This is the portability insurance the article describes. Even GPT-5-mini, a weaker model, delivers the full Cockburn structure when the anchor is named. The anchor pins the behavior across model families and model sizes. Finding 2: "Use-Case 3.0" — three distinct failure modes across model familiesThis is where it gets interesting. The article documents Claude's behavior (Opus hedges transparently, Haiku substitutes silently). The other models show different failure patterns for the same thin prior:
Three failure modes for a thin prior:
Finding 3: The GPT-5 divergence — dense prior or confabulation?GPT-5 is the outlier. When asked "What is Use-Case 3.0?", it confidently returns 10 specific principles including "Single shared model," "Example- and test-driven," "Scalable and fractal." Gemini does the same (10 principles, attributed to "Jacobson and Cockburn"). This raises the question your article explicitly frames: Is the prior genuinely denser in GPT-5/Gemini (trained on the 2024 IJI eBook?), or are these models confabulating plausible principles from the 2.0 prior? A verification against the actual Jacobson/Spence/de Mendonca 2024 source would settle this. If the 10 principles match the published text, GPT-5 has a real 3.0 prior and the anchor could work for that model. If they don't match, it's sophisticated confabulation — which is arguably the most dangerous failure mode because it's the hardest to detect. Either way, this validates the article's core claim: you cannot know whether an anchor delivers until you test it per model. The same term activates real knowledge in one model and fabrication in another. Finding 4: Gemini shows an additional behavior — visual outputFor framing D (Use-Case 2.0 slices), Gemini spontaneously generated an ASCII sequence diagram and a Jacobson-attributed image. It also offered interactive follow-up ("Would you like to see a detailed step-by-step sequence flow for Slice 1?"). This suggests Gemini's prior for Use-Case 2.0 is extremely dense — richer than Claude's in terms of modalities activated. Finding 5: Attribution is universally correctAll models (P3) correctly identify Jacobson as inventor (OOPSLA 1986/87, OOSE 1992) and Cockburn as the writing-craft codifier (2001). Not a single model misattributes. This confirms what the article states: the misattribution lives in casual human shorthand, not in the models. Implications for the article
Methodology note
Deutsche VersionCross-Modell-Validierung: GPT-5, GPT-5-mini und Gemini 2.5 Flash bestätigen und erweitern die ErgebnisseIch habe dasselbe A–E Experiment gegen OpenAI GPT-5, GPT-5-mini (über OpenAI API) und Google Gemini 2.5 Flash (über AI Studio) durchgeführt — alle mit leerem Kontext, kein System-Prompt, neue Session pro Prompt. Kernergebnisse:
Implikation für den Artikel: Die Drei-Schichten-Unterscheidung (Anchor / Contract / Article) hält über alle getesteten Modellfamilien. Die stille Substitution ist kein Claude-spezifisches Verhalten sondern eine allgemeine Eigenschaft bei dünnem Prior. Der Hinweis auf GPT-5's möglicherweise dichterer 3.0-Prior wäre ein wertvoller Ausblick: Anker-Validität kann sich über die Zeit ändern. |
|
@JensGrote this is excellent — thank you. Reproducing the whole A–E battery against GPT-5, GPT-5-mini and Gemini 2.5 Flash is exactly the cross-family validation the article was missing (it only tested Claude tiers), and your three-failure-mode breakdown is sharper than what's written today. Two things stand out:
On your open question (genuine 3.0 prior vs confabulation): I checked the source. The published Use-Case Foundation (2024) lists nine principles, with quite different wording — "use cases apply to systems of all types", "stakeholder involvement is essential", "tells the whole story", "trigger conversations", "prioritize readability" (IJI guide). GPT-5's "10 principles" ("Single shared model", "Example- and test-driven", "Scalable and fractal") do not match that set — different count, different names. So it leans confabulation, not a denser prior — which makes your GPT-5 result the strongest example of the dangerous mode, not a counterexample. A line-by-line check against the IJI PDF would settle it definitively. This deserves to be in the article — in your words and with your data. Would you open a PR adding a "Cross-model validation" section? You have the raw outputs ( |
- main.js: reload the article on language switch (handleLanguageChange was missing a case for /training-data-vs-practice, unlike every other doc route) - router.js: align the browser/tab title with the article's actual title - article: drop the "full fully-dressed" redundancy Addresses CodeRabbit + Copilot review feedback on #586. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reproduces the A–E anchor activation battery against GPT-5, GPT-5-mini and Gemini 2.5 Flash. Confirms the mechanism is model-family-independent and documents a third failure mode (confabulation) not visible in the Claude-only test. Addresses: LLM-Coding#586 (comment)
What
A new website article — An Anchor Delivers Only as Far as the Prior Reaches — on how a semantic anchor's strength depends on how densely the concept sits in an LLM's training data. It grew directly out of the discussion in #582 about the Cockburn Use Cases anchor.
It includes a reproducible A–E experiment (prompts included, run it yourself) across Claude Haiku 4.5, Sonnet 4.6 and Opus 4.8, showing:
Why it matters for the catalog
It draws a clean line between anchors (dense priors), contracts (vocabulary supplied in text, safe even when the prior is weak), and articles (meta-knowledge about a term). This is the reasoning behind keeping the Cockburn Use Cases anchor as-is rather than renaming or modernising it with Use-Case 2.0/3.0.
Changes
docs/training-data-vs-practice.adoc— the articlerender-docs, the router, and the main nav (EN)Method note
All model outputs were produced in a clean room (
claude -p --setting-sources "" --strict-mcp-configfrom a neutral directory) so the project's ownCLAUDE.mdcould not bias the results — the article documents this, including a contamination we caught and removed.@simasch — this is the article I promised on #582. Your pull request is what prompted it and you're credited throughout. I'd value your review of how your point and your role are represented before this goes live.
@JensGrote — would value your eyes too.
🤖 Generated with Claude Code
Summary by CodeRabbit
Release Notes
New Features
Documentation