Write Level 3 (Differentiation) questions for conflict groups

[raifdmueller]

Part of EPIC #329 — depends on #330 (Pilot)

Goal

Write multiple-choice questions that test whether a model can distinguish similar anchors within conflict groups.

Conflict groups to cover

1. TDD London School vs. TDD Chicago School
2. Clean Architecture vs. Hexagonal Architecture
3. ADR according to Nygard vs. MADR
4. DRY vs. SPOT vs. SSOT
5. BLUF vs. Pyramid Principle
6. Socratic Method vs. Feynman Technique vs. Rubber Duck Debugging
7. Chain of Thought (standalone vs. task-local)
8. Problem Space NVC vs. JTBD vs. User Story Mapping

Approach

• Each question describes characteristics of one anchor and asks which anchor it matches
• Distractors are the other members of the same conflict group
• Auto-generate candidates from Related Anchors metadata, refine by hand

Output

• Differentiation section filled in the YAML specs for ~8 conflict groups

Estimated effort

~8 questions. ~1-2 hours.

[JensGrote]

Contribution: Level 3 (Differentiation) questions for all 8 conflict groups

(German version below)

Here are the Level 3 questions for all 8 conflict groups defined in the issue. Each question describes characteristics of one anchor and asks the model to identify which anchor matches — with the other conflict group members as distractors.

---

Format

Each question follows the pattern from existing specs — to be added as a differentiation section in the respective anchor's YAML file.

---

1. TDD London School vs. TDD Chicago School

Note: If #556 (IOSP) lands as an anchor, this conflict group should expand to a triple: London vs. Chicago vs. IOSP-based testing (Operations are trivially unit-testable without mocks; Integrations need no unit tests). The IOSP approach changes the "what to mock?" question fundamentally — worth a Level 3 question once it's in the catalog.

# Add to: evaluations/specs/tdd-london-school.yaml
  differentiation:
    conflict_group: [tdd-london-school, tdd-chicago-school]
    question: |
      A developer writes tests that verify how objects collaborate — mocking
      dependencies and asserting that specific methods are called with expected
      arguments. The tests start from the outermost layer and work inward,
      discovering interfaces as they go. Which TDD approach is this?
    options:
      A: TDD, London School
      B: TDD, Chicago School
    correct: A

# Add to: evaluations/specs/tdd-chicago-school.yaml (new file if needed)
  differentiation:
    conflict_group: [tdd-london-school, tdd-chicago-school]
    question: |
      A developer writes tests that assert on the final state of objects after
      an operation — using real collaborating objects wherever possible and
      avoiding mocks. Tests are structured bottom-up, starting with the smallest
      units and composing upward. Which TDD approach is this?
    options:
      A: TDD, London School
      B: TDD, Chicago School
    correct: B

---

2. Clean Architecture vs. Hexagonal Architecture

# Add to a shared or clean-architecture spec
  differentiation:
    conflict_group: [clean-architecture, hexagonal-architecture]
    question: |
      An architect describes a system with concentric rings where the dependency
      rule states that source code dependencies must point inward only. The
      innermost ring contains Entities, then Use Cases, then Interface Adapters,
      then Frameworks & Drivers. Which architecture pattern is this?
    options:
      A: Clean Architecture
      B: Hexagonal Architecture (Ports & Adapters)
    correct: A

# Add to hexagonal-architecture spec
  differentiation:
    conflict_group: [clean-architecture, hexagonal-architecture]
    question: |
      An architect describes a system where the application core defines Ports
      (interfaces) and external concerns connect via Adapters. The metaphor
      emphasizes that there is no inherent "top" or "bottom" — all external
      interactions (UI, DB, messaging) are structurally equivalent and
      interchangeable. Which architecture pattern is this?
    options:
      A: Clean Architecture
      B: Hexagonal Architecture (Ports & Adapters)
    correct: B

---

3. ADR according to Nygard vs. MADR

  differentiation:
    conflict_group: [adr-according-to-nygard, madr]
    question: |
      A team documents architecture decisions using a minimal format with
      sections: Title, Status, Context, Decision, Consequences. The format
      is intentionally brief — a short text file that captures the "why"
      behind a decision. Which ADR format is this?
    options:
      A: ADR according to Nygard
      B: MADR (Markdown Any Decision Records)
    correct: A

  differentiation:
    conflict_group: [adr-according-to-nygard, madr]
    question: |
      A team documents decisions using a structured template with explicit
      sections for "Considered Options" (listing alternatives), "Decision
      Outcome" (chosen option with justification), and optional "Pros and
      Cons of the Options" tables. The format supports any decision, not
      just architectural ones. Which ADR format is this?
    options:
      A: ADR according to Nygard
      B: MADR (Markdown Any Decision Records)
    correct: B

---

4. DRY vs. SPOT vs. SSOT

  differentiation:
    conflict_group: [dry-principle, spot-principle, ssot-principle]
    question: |
      A principle states: "Every piece of knowledge must have a single,
      unambiguous, authoritative representation within a system." It applies
      broadly to code, documentation, and data schemas. Duplication of
      knowledge (not just code) is the root problem. Which principle is this?
    options:
      A: DRY (Don't Repeat Yourself)
      B: SPOT (Single Point of Truth)
      C: SSOT (Single Source of Truth)
    correct: A

  differentiation:
    conflict_group: [dry-principle, spot-principle, ssot-principle]
    question: |
      A principle from the Unix/C community states that there should be
      exactly one place where a particular fact or policy is expressed in
      code. If you need to change something, you change it in exactly one
      spot. Originally articulated in the context of systems programming
      and the Art of Unix Programming. Which principle is this?
    options:
      A: DRY (Don't Repeat Yourself)
      B: SPOT (Single Point of Truth)
      C: SSOT (Single Source of Truth)
    correct: B

  differentiation:
    conflict_group: [dry-principle, spot-principle, ssot-principle]
    question: |
      A data governance principle states that every data element should be
      mastered in exactly one system, which serves as the authoritative
      reference for all other systems. Other systems may cache copies, but
      the canonical version lives in one designated place. Commonly applied
      to databases, master data management, and enterprise integration.
      Which principle is this?
    options:
      A: DRY (Don't Repeat Yourself)
      B: SPOT (Single Point of Truth)
      C: SSOT (Single Source of Truth)
    correct: C

---

5. BLUF vs. Pyramid Principle

  differentiation:
    conflict_group: [bluf, pyramid-principle]
    question: |
      A communication technique from military writing standards: state the
      most important information first (the "bottom line"), then provide
      supporting details. The reader knows the conclusion before reading
      the reasoning. Primarily applied to emails, memos, and status
      reports. Which communication anchor is this?
    options:
      A: BLUF (Bottom Line Up Front)
      B: Pyramid Principle (Barbara Minto)
    correct: A

  differentiation:
    conflict_group: [bluf, pyramid-principle]
    question: |
      A structured communication framework from McKinsey consulting: start
      with the governing thought (answer), then group supporting arguments
      into mutually exclusive, collectively exhaustive (MECE) clusters,
      each with its own supporting evidence. The result is a hierarchical
      tree of arguments. Which communication anchor is this?
    options:
      A: BLUF (Bottom Line Up Front)
      B: Pyramid Principle (Barbara Minto)
    correct: B

---

6. Socratic Method vs. Feynman Technique vs. Rubber Duck Debugging

  differentiation:
    conflict_group: [socratic-method, feynman-technique, five-whys]
    question: |
      A technique for deepening understanding by asking a series of probing
      questions that expose assumptions, reveal contradictions, and guide
      the other person toward their own conclusions. The practitioner never
      provides answers directly — only questions. Which anchor is this?
    options:
      A: Socratic Method
      B: Feynman Technique
      C: Five Whys (Ohno)
    correct: A

  differentiation:
    conflict_group: [socratic-method, feynman-technique, five-whys]
    question: |
      A learning technique with four steps: (1) choose a concept, (2)
      explain it as if teaching a child using simple language, (3) identify
      gaps where the explanation breaks down, (4) go back to the source
      and simplify further. The goal is to expose shallow understanding
      through the act of teaching. Which anchor is this?
    options:
      A: Socratic Method
      B: Feynman Technique
      C: Five Whys (Ohno)
    correct: B

  differentiation:
    conflict_group: [socratic-method, feynman-technique, five-whys]
    question: |
      A root-cause analysis technique from the Toyota Production System:
      when a problem occurs, ask "why?" repeatedly (typically five times)
      until the fundamental cause is reached. Each answer becomes the basis
      for the next "why?" question. The goal is to move past symptoms to
      the systemic root cause. Which anchor is this?
    options:
      A: Socratic Method
      B: Feynman Technique
      C: Five Whys (Ohno)
    correct: C

---

7. Chain of Thought (standalone vs. task-local)

  differentiation:
    conflict_group: [chain-of-thought]
    question: |
      A prompting technique where the user explicitly instructs the LLM to
      show intermediate reasoning steps before arriving at a conclusion.
      Common triggers: "Let's think step by step", "Show your reasoning",
      or providing worked examples with visible reasoning chains. The
      technique is:
    options:
      A: Chain of Thought as a standalone prompting strategy (applicable to any task)
      B: Chain of Thought as a task-local embedded technique (used implicitly within a larger methodology)
    correct: A

Note: This conflict group is somewhat unusual — it's the same anchor used in two modes. The differentiation tests whether the LLM understands CoT as a general strategy vs. its embedded use within other methodologies (e.g., within TDD reasoning or architecture evaluation).

---

8. Problem Space NVC vs. JTBD vs. User Story Mapping

  differentiation:
    conflict_group: [problem-space-nvc, jobs-to-be-done, user-story-mapping]
    question: |
      A requirements technique that focuses on identifying the underlying
      human needs behind a request — separating observations from feelings,
      feelings from needs, and needs from strategies. The practitioner uses
      empathic listening to discover what the user truly needs, rather than
      what they initially ask for. Which anchor is this?
    options:
      A: Nonviolent Communication (Rosenberg)
      B: Jobs To Be Done (JTBD)
      C: User Story Mapping
    correct: A

  differentiation:
    conflict_group: [problem-space-nvc, jobs-to-be-done, user-story-mapping]
    question: |
      A product design framework that frames customer motivation as
      "hiring" a product to accomplish a specific job. The focus is on the
      progress the customer seeks in a particular circumstance — not the
      product's features or the customer's demographics. The canonical
      example is a milkshake "hired" for a boring commute. Which anchor is
      this?
    options:
      A: Nonviolent Communication (Rosenberg)
      B: Jobs To Be Done (JTBD)
      C: User Story Mapping
    correct: B

  differentiation:
    conflict_group: [problem-space-nvc, jobs-to-be-done, user-story-mapping]
    question: |
      A planning technique that arranges user activities along a horizontal
      "backbone" (the narrative flow of what users do) with vertical
      "walking skeleton" slices below, ordered by priority. The two-
      dimensional map helps teams identify release increments and ensures
      each release delivers end-to-end value. Which anchor is this?
    options:
      A: Nonviolent Communication (Rosenberg)
      B: Jobs To Be Done (JTBD)
      C: User Story Mapping
    correct: C

---

Summary

#	Conflict Group	Questions
1	TDD London vs. Chicago	2
2	Clean Architecture vs. Hexagonal	2
3	ADR Nygard vs. MADR	2
4	DRY vs. SPOT vs. SSOT	3
5	BLUF vs. Pyramid Principle	2
6	Socratic vs. Feynman vs. Five Whys	3
7	Chain of Thought (standalone vs. embedded)	1
8	NVC vs. JTBD vs. User Story Mapping	3
Total		18

Note: I expanded from the estimated ~8 to 18 questions because groups with 3+ members need one question per member to properly test differentiation in all directions.

Happy to submit this as a PR with the YAML files once the format/approach is confirmed.

---

Meta: Evaluation as part of the Anchor Lifecycle

This work surfaced a broader question: how do we ensure new anchors get evaluation specs? Currently, evaluation questions are authored retroactively. As the catalog grows, this creates an ever-growing backlog.

I'll open a separate issue proposing:

1. Evaluation spec (recognition + application) as Definition-of-Done for new anchors
2. Activation test result as a mandatory field in the Anchor Proposal template
3. Automated conflict-group detection (build script comparing related: and categories: to suggest missing differentiation questions)
4. Contract compliance tests as separate artifacts (builds on feat: Evaluation framework for Semantic Contracts (compliance testing) #370)

This keeps the current batch (#333, #334) unblocked while addressing the long-term scalability.

---

---

Beitrag: Level 3 (Differentiation) Fragen für alle 8 Conflict Groups

(Deutsche Version)

Hier sind die Level-3-Fragen für alle 8 im Issue definierten Conflict Groups. Jede Frage beschreibt Charakteristiken eines Ankers und fragt das Modell welcher Anker passt — mit den anderen Conflict-Group-Mitgliedern als Distraktoren.

---

Format

Jede Frage folgt dem Pattern der bestehenden Specs — wird als differentiation-Sektion in die jeweilige Anchor-YAML-Datei eingefügt.

---

Zusammenfassung

#	Conflict Group	Fragen
1	TDD London vs. Chicago	2
2	Clean Architecture vs. Hexagonal	2
3	ADR Nygard vs. MADR	2
4	DRY vs. SPOT vs. SSOT	3
5	BLUF vs. Pyramid Principle	2
6	Socratic vs. Feynman vs. Five Whys	3
7	Chain of Thought (standalone vs. embedded)	1
8	NVC vs. JTBD vs. User Story Mapping	3
Gesamt		18

Hinweis: Ich habe von den geschätzten ~8 auf 18 Fragen erweitert, weil Gruppen mit 3+ Mitgliedern eine Frage pro Mitglied brauchen um Differenzierung in alle Richtungen zu testen.

Kann das gerne als PR mit den YAML-Dateien einreichen sobald Format/Ansatz bestätigt ist.

[raifdmueller]

Review (automated triage) — ✅ Ready content exists, recommend merging with one correction

Differentiation (Level 3) questions for the conflict groups are valuable and JensGrote has already authored 18 full questions in-thread (expanded from the original ~8) and offered a PR. Good to merge.

Two things to fix on the way in, both already surfaced in your own #329 analysis:

1. Conflict group 7 (Chain-of-Thought) — flagged in EPIC: Semantic Anchor Evaluations across LLMs #329 as not a true differentiation pair (standalone CoT vs task-local isn't a genuine "which anchor" distinction). Drop or reframe that group rather than ship a question that tests a non-distinction.
2. Conflict group 6 is mislabeled "Rubber Duck" in the title but implemented as Five Whys — reconcile the label.

Recommendation: accept JensGrote's PR with those two edits; combine with #334. Part of EPIC #329; depends on #330. Also worth advancing his follow-up idea of making eval specs part of the anchor Definition of Done.

---

Generated by Claude Code