Status: Completed Focus: Rules-based logic, CLI-MVP with baseline v.0 dataset, standalone container.
Demo-First, Learning-Driven Execution Status: Current
This sprint plan is structured to prove architectural viability first, then extend capability, and only then invest in deeper internal polish where learning return is highest.
Key principles:
- Tracer bullets before polish
- Demos as validation artifacts
- Learning ROI over completeness
- CLI runtime retained intentionally
- Infrastructure changes only when justified
Deliver a playable Phase-2 demo that proves semantic answer validation works end-to-end and overcomes Phase-1 limitations using a minimal, controlled dataset.
This phase answers: “Does the semantic verification architecture work at all?”
Plan phase 2 sprints → Filter legacy → Normalize schema → Enrich & embed → Generate synthetic → Validate separately
→ Assemble Gold → Add context → Tiered validation → Release demo
Sprint Type: Planning / Guardrails
Lock the smallest viable slice required to demonstrate Phase-2 semantic intelligence and prevent scope creep.
- One dataset slice
- One SBERT model (off-the-shelf)
- One similarity strategy per question type
- CLI runtime retained
- Existing VM / container retained
- Performance optimization
- Pipeline generalization
- Monitoring / agents
- Runtime refactor (FastAPI)
- UX polish
There is zero ambiguity about:
- what is being demoed
- what is intentionally ignored
Prove that legacy data can be normalized into a truthful Phase-2 contract with shared reasoning scaffolding and type-aware validation.
Description:
- Create defined schema for datasets going forward with specified data types.
- Remove YN questions from legacy data prior to Phase-2 processing.
- Strip legacy dataset to match BaseQuestion schema (with out LLM enrichment features -> hint_1, explanation and the optional fields). These will be added in the tracer by the Context Refinery.
Acceptance Criteria:
- All YN questions excluded
- Remaining types: FR, EX, MCQ
- Removal count logged
- Aligned with ADR-P2-019
- standard Pyndantic schema for MCQ, EX, FR questions
- Legacy dataset ready for enrichment with LLM prompting
Description:
Upgrade the master prompts so both legacy and synthetic questions emit Phase-2 instructional scaffolding.
Scope:
- Freeze schemas per question type
- Add to all prompts:
hint_1— light nudge (directional) [required]hint_2— stronger cue (key concept) [optional]hint_3— near-solution framing [optional]explanation— full reasoning [required]
- Legacy run only:
- Generate
answer_variationsbased on existing question + answer text [required for FR / EX only]
- Generate
Validation policy
- Absence of these fields implicitly indicates unverified or unavailable source grounding.
- No attempt is made to backfill sources in Phase 2.
This policy ensures:
- schema consistency
- semantic readiness
- zero silent data corruption
Acceptance Criteria:
- Generated records conform to defined schemas
- No additional enrichment logic added
- Prompt updated once (no tuning loop)
Description:
The question type (FR / EX / MCQ) is determined prior to enrichment and governs which fields are generated.
Use LLM to enrich legacy questions without modifying their wording by filling in missing Phase-2 fields derived strictly from the existing question and correct answer. The LLM generates reasoning scaffolding (hint_1, optional hint_2 / hint_3), answer variations, and a high-level conceptual explanation. The LLM must not add new facts, rewrite the question, or fabricate source grounding; explanations must remain conceptual and source-agnostic. Fields that cannot be inferred safely are left empty.
- prevent cross contamination across hints and explanations with negative constraint to prompt: e.g. "Hints must provide directional guidance without using the keywords found in the
answeroranswer_variationsfields."
Schema mapping:
- FR / EX →
StandardQuestion - MCQ →
MCQQuestion
questionansweranswer_variationshint_1explanation
questionoptions[](structured list)- exactly one correct option
hint_1explanation
hint_2hint_3source_quotesource_reference
Absence of source fields implicitly indicates unverified grounding.
No attempt is made to hallucinate or backfill sources in Phase 2.
Acceptance Criteria:
- ≥95% of legacy FR / EX records pass Pydantic validation
- MCQs meet structural validity or are dropped
- Invalid records dropped (no repair loops)
[Postponed to after synthetic gen and to be done with qa_validation pipeline logic]
Description:
Generate SBERT embeddings required for offline deduplication and runtime semantic validation, with clearly separated responsibilities.
For semantic deduplication (offline only):
- All question types (FR, EX, MCQ):
- Question text embeddings are generated and used exclusively for deduplication.
- Question embeddings are not used for runtime answer validation.
For runtime semantic validation: generate in batches
- FR / EX:
- Embeddings generated for:
- correct answer
- all answer variations
- Embeddings generated for:
- MCQ:
- Embeddings generated for all option texts (correct + distractors)
- Question embeddings exist for all records and are usable for deduplication
- All required answer / option embeddings exist (no nulls)
- All records pass type-specific Pydantic validation
- Embeddings loadable by runtime (answer / option only)
- Publish Gold v0 (legacy only)
- Version artifacts (DVC)
Sprint 2.1 is DONE when:
- ✅ Gold v0 exists (legacy only)
- ✅ FR / EX records include answer variations, hints, explanation, and embeddings
- ✅ MCQs include structured options with option embeddings
- ✅ Optional fields do not block validation
- ✅ Prompt is reusable for synthetic generation
- ✅ Dataset loads cleanly via runtime loader
This sprint proves data integrity and shared reasoning scaffolding — not intelligence.
Demonstrate that the Content Factory can extend Gold safely without fragmenting schemas.
Scope:
- 1 book
- ~5 chapters (sequential)
- Use the same upgraded prompt
- Generate:
- FR: ~5–6 / chapter
- EX: ~5–6 / chapter
- MCQ: ~5–6 / chapter (structured options)
Acceptance Criteria:
- Grounded in source text
- Non-trivial
- FR / EX have meaningful answer variations
Scope:
- Remove semantic duplicates (within batch and vs Gold v0)
- Generate required embeddings
- Enforce strict schema validation
- MCQ distractor collision audit:
- For all MCQs, calculate the pairwise cosine similarity of all options.
- Constraint: If
$\Delta = Sim(Option_i, Option_j) > \text{Collision_Threshold}$ , the record fails validation -> question rejected. - Goal: Ensure the "Semantic Gap" between distractors is wide enough to avoid runtime ambiguity.
Acceptance Criteria:
- ≥95% pass Pydantic validation
- Failed records dropped (no manual fixes)
- No schema divergence
- Dataset loads cleanly
Acceptance Criteria:
- Gold v1 contains legacy + synthetic
- Synthetic records tagged via metadata
- Identical schemas across sources
Sprint 2.2 is DONE when:
- Gold v1 exists (legacy + synthetic)
- Hint and explanation schema is identical across sources
- Synthetic records are distinguishable via metadata
- Dataset still loads cleanly via runtime loader
This sprint proves extensibility — not semantic reasoning.
Make semantic intelligence visible and undeniable in gameplay.
Description: Implement validation tiers per question type.
FR / EX: Exact → Fuzzy → Semantic (answer + variations) MCQ: Exact → Fuzzy → Constrained semantic (option embeddings only)
MCQ semantic rules:
- Compare player input embedding to option embeddings only
- Accept only if:
- similarity ≥ τ_mcq
- best similarity exceeds second-best by margin Δ
- check for semantic collision between options: tie-breaker policy if two distractors are semantically similar (e.g. Regulas Black, Sirius Black) -> reject question in qa_validation stage.
- resolved option is correct
Acceptance Criteria:
- Container builds successfully
- No infra refactor
Acceptance Criteria:
- CLI demo runs end-to-end
- Gold v1 loads correctly
- Semantic improvement is visible in gameplay
- Phase-1 failure succeeds via semantic logic
- MCQs accept shorthand inputs (e.g. “sirius”)
- Ambiguous MCQ inputs rejected
- Logs show which tier accepted the answer
- Demo playable end-to-end
Phase 2 is DONE when:
- Gold dataset exists (legacy + synthetic)
- FR / EX semantic answers accepted correctly
- MCQs tolerate typos and partial input
- Phase-1 limitation is demonstrably resolved
- Demo is playable without explanation
Demonstrate that the architecture extends cleanly to support runtime intelligence without rewriting Phase-2 foundations.
This phase answers: “Does the design scale conceptually?”
Define Phase-3 strictly as a capability addition, not a system rewrite.
- CLI runtime retained
- Same Gold dataset
- Same embeddings
- Same VM / SSR deployment
- Production hardening
- Monitoring agents
- Scale optimizations
- FastAPI refactor
Sprint Type: Capability Expansion
Introduce a new intelligence layer that qualitatively changes gameplay.
- Persona layer
- Lightweight judge for ambiguous answers
- Explanation layer
- No refactor of Content Factory
- No refactor of Context Refinery
- Minimal logic only
A player experiences a clear qualitative difference in gameplay driven by runtime intelligence.
- Phase-3 MVP ADR
- README update:
- What Phase-3 adds
- Why the architecture enabled it
- Explicit “What Phase-3 proves” section
This phase is entered only after Phase-3 demo exists.
Duration: Open-ended, paced
Sprint Type: Data Science / Enrichment
Deepen semantic enrichment quality where learning value is highest.
- Feature engineering
- Semantic signal quality
- Evaluation metrics
- Schema discipline
- Toward Feature Store formalization
Notes: This is the main data science & NLP learning phase.
Description: Named Entity Recognition (NER) work is intentionally deferred to this phase -> after MVP developement. Implement NER to extract character names, locations, and magical objects to create a "Semantic Weighting" system (e.g., giving more importance to the 'Entity' than the 'Adjective' during answer validation).
- NER Feature Engineering: Leveraging spaCy or Transformers to tag Gold entities.
- Semantic signal quality: Tuning similarity thresholds based on entity presence. If two MCQ options share a Named Entity (e.g., both mention "Black"), the Refinery should flag it for a "disambiguation prompt" or a distractor rewrite.
- Toward Feature Store formalization: Mapping NER tags to the Context Refinery schema.
-
fine-tune sbert model
-
For FR/ EX: calibrate sbert score thresholds for ambiguous score range for answer checking (cosine similarity (tau) upper and lower bounds) with handoff to the SLM judge.
-
fine tune SLM judge role and feedback.
Filter the Gold dataset for semantic "Near-Misses" and factual consistency.
-
Method A (Semantic Similarity): Calculate the SBERT distance between the answer and each of the 3 distractors. If a distractor is too similar (e.g., > 0.95), flag it for manual review—it's either a hallucinated typo or a "trick" question that's too confusing for the player. Can allow for tricky MCQ: allow lower
$\Delta$ margin between options (instead of rejecting as done in phase 2). Flag these questions inqa_validationpipeline that they require SLM intervention if the player provides ambiguous answer. - Method B (Entity Consistency): Use NER model to check if the answer and options share the same label (e.g., if the answer is a PERSON, all options should be PERSON).
Duration: Optional
Sprint Type: Upstream Quality
Refine upstream generation only if demanded by Context Refinery improvements.
- Prompt robustness
- QA logic
- Sampling strategies
- Data quality scoring
| Decision | Rationale |
|---|---|
| CLI runtime retained | UX is not the core risk |
| VM / SSR retained | Minimize refactor tax |
| FastAPI deferred | Only justified by scale/time |
| Demo before polish | Architecture must earn optimization |
| DS polish postponed | Needs downstream demand |