test: scaffold prose retrieval golden-set schema and stratification plan

Spec-only measurement scaffold (no corpus yet, nothing runs in CI). phase1_plan.json: query_type x difficulty strata (fact_lookup/traceability/synthesis), 60 records + doc-type coverage constraints. golden_v0.schema.json: draft 2020-12 record schema extending the code GoldenLabel (query_type, expected_doc_spans, expected_trace_path, expected_doc_types) with conditional traceability/synthesis rules. examples.jsonl: three schema-validated placeholders. Mirrors tests/eval/pipeline + tests/eval/golden.

Author: deepakdgupta1

tests/eval/prose/README.md

@@ -0,0 +1,48 @@
+# Prose Retrieval Golden Set (SDLC document collateral)
+
+**Status: SPEC ONLY — awaiting corpus.** No prose corpus is present in this repo yet, so no records exist and **nothing here runs in CI** (this directory contains data + schema + docs, no `test_*.py` for pytest to collect). This is the measurement scaffold that must exist *before* any retrieval tuning, mirroring the code eval set.
+
+This is the prose sibling of the code eval pipeline (`tests/eval/golden/`, `tests/eval/pipeline/`, `tests/eval/harness/`). Full design and metrics: [`docs/research/document_collateral_retrieval_2026.md` §6](../../../docs/research/document_collateral_retrieval_2026.md).
+
+## Files
+
+| File | Role |
+|---|---|
+| `phase1_plan.json` | Committed stratification plan (machine-checked input). Mirror of `tests/eval/pipeline/phase1_plan.json`, keyed on `query_type × difficulty` with doc-type coverage constraints. |
+| `golden_v0.schema.json` | JSON Schema (draft 2020-12) for a final prose `GoldenLabel`. Extends the code record: `query_type` replaces `task_type`; `expected_doc_spans` replace `expected_entities`/`expected_files`; traceability/synthesis add `expected_trace_path` / `expected_doc_types`. |
+| `examples.jsonl` | Three placeholder records (one per query type) that validate against the schema. `method: "template_placeholder"`, `source_verified: false` — illustrative only, **not** labels. |
+
+## Query types & stratification (target: 60)
+
+| Query type | easy | medium | hard | total |
+|---|:--:|:--:|:--:|:--:|
+| `fact_lookup` | 8 | 8 | 4 | 20 |
+| `traceability` | 0 | 8 | 12 | 20 |
+| `synthesis` | 2 | 8 | 10 | 20 |
+
+- **fact_lookup** — answer in one passage of one doc; the local-first / precise-retrieval sweet spot.
+- **traceability** — multi-hop across the SDLC chain (Problem Statement → PRD → HLD → LLD → Release Notes → Support); the class flat RAG cannot answer and the document graph owns. No `easy` tier by definition.
+- **synthesis** — aggregate/compare across many docs (e.g. "summarize all healthcare case studies", "what changed v2→v3").
+
+## Cross-field gates (enforced by the harness, not the JSON Schema)
+
+- `line_range` is `[start, end]` with `start <= end` and both within the file at the recorded SHA.
+- Every `doc_id` + `heading_path` resolves against a fresh parse of the corpus.
+- traceability: `expected_trace_path` spans ≥ 3 distinct doc types; every node resolves in the document graph.
+- synthesis: `expected_doc_spans` cover ≥ 3 distinct documents.
+
+## Scoring (when the corpus exists)
+
+Reuse `tests/eval/harness/scorer.py` with prose extensions:
+- **Retrieval:** `recall@k` (primary — a missed span = a wrong generated answer), `nDCG@10`, `MRR`.
+- **Traceability:** path-completeness (fraction of `expected_trace_path` recovered).
+- **Synthesis:** document-coverage over `expected_doc_spans`.
+- **Generation:** faithfulness/groundedness + citation accuracy over `must_mention_facts` / `must_not_mention_facts`.
+- **Routing:** re-calibrate `sufficiency_score` against prose answer-correctness — do **not** port the code corpus's 0.8 threshold; re-measure (the code finding was over-confident).
+
+## Bootstrap procedure (blocked on corpus)
+
+1. Point the corpus at a sample doc set (the only missing input).
+2. Run the three-role Author / Oracle / Adversary pipeline; the Oracle reads documents instead of code.
+3. Mechanical gate resolves `doc_id` + `heading_path` + `line_range` and `feature_id`/`requirement_id` joins.
+4. Stratified human spot-check (15%), then bless `prose_golden_v1.0.json` with a `meta.json` recording per-doc SHAs (same drift guard as the code set).

tests/eval/prose/examples.jsonl

@@ -0,0 +1,3 @@
+{"query_id":"pq_example_fact","query_text":"What retry policy does the Atlas rate-limiting feature specify?","query_type":"fact_lookup","difficulty":"easy","expected_doc_spans":[{"doc_id":"prd/atlas-rate-limiting.md","doc_type":"prd","heading_path":"PRD > Requirements > Rate Limiting","line_range":[412,448],"role":"primary"}],"expected_trace_path":[],"expected_doc_types":["prd"],"must_mention_facts":["The retry policy is exponential backoff capped at 5 retries.","The base backoff interval is 200ms."],"must_not_mention_facts":["Retries are unbounded.","The policy is defined in the billing service."],"correct":true,"validation":{"source_verified":false,"human_reviewed":false,"method":"template_placeholder"},"schema_version":"0.1.0"}
+{"query_id":"pq_example_trace","query_text":"Trace the Atlas rate-limiting feature from its product requirement through to the customer-facing support article.","query_type":"traceability","difficulty":"hard","expected_doc_spans":[{"doc_id":"prd/atlas-rate-limiting.md","doc_type":"prd","heading_path":"PRD > Requirements > Rate Limiting","line_range":[412,448],"role":"primary"},{"doc_id":"hld/atlas-gateway.md","doc_type":"hld","heading_path":"HLD > Components > GatewayThrottler","line_range":[88,132],"role":"primary"},{"doc_id":"lld/atlas-rate-limiter.md","doc_type":"lld","heading_path":"LLD > Algorithm > Token Bucket","line_range":[40,96],"role":"primary"},{"doc_id":"release_notes/atlas-3.2.md","doc_type":"release_notes","heading_path":"Release Notes 3.2 > New Features","line_range":[12,20],"role":"supporting"},{"doc_id":"support_manual/atlas-429-troubleshooting.md","doc_type":"support_manual","heading_path":"Support > Errors > HTTP 429","line_range":[5,60],"role":"supporting"}],"expected_trace_path":["feature:atlas-rate-limiting","prd/atlas-rate-limiting.md","hld/atlas-gateway.md","lld/atlas-rate-limiter.md","release_notes/atlas-3.2.md","support_manual/atlas-429-troubleshooting.md"],"expected_doc_types":["prd","hld","lld","release_notes","support_manual"],"must_mention_facts":["Requirement REQ-RL-01 in the PRD is implemented by the GatewayThrottler component described in the HLD.","The LLD specifies a token-bucket algorithm for throttling.","Atlas 3.2 release notes announce the rate-limiting feature.","The support manual documents the HTTP 429 troubleshooting flow for end users."],"must_not_mention_facts":["The feature is implemented in the billing service.","Rate limiting shipped in Atlas 2.x."],"correct":true,"validation":{"source_verified":false,"human_reviewed":false,"method":"template_placeholder"},"schema_version":"0.1.0"}
+{"query_id":"pq_example_synth","query_text":"Summarize the common customer outcomes reported across the healthcare client case studies.","query_type":"synthesis","difficulty":"medium","expected_doc_spans":[{"doc_id":"case_study/northwind-health.md","doc_type":"case_study","heading_path":"Case Study > Results","line_range":[60,110],"role":"primary"},{"doc_id":"case_study/acme-care.md","doc_type":"case_study","heading_path":"Case Study > Results","line_range":[55,98],"role":"primary"},{"doc_id":"case_study/medix.md","doc_type":"case_study","heading_path":"Case Study > Outcomes","line_range":[70,120],"role":"primary"}],"expected_trace_path":[],"expected_doc_types":["case_study"],"must_mention_facts":["At least three healthcare case studies report reduced patient-onboarding time.","Two or more report improved compliance-audit pass rates."],"must_not_mention_facts":["The case studies are drawn from the fintech sector."],"correct":true,"validation":{"source_verified":false,"human_reviewed":false,"method":"template_placeholder"},"schema_version":"0.1.0"}

tests/eval/prose/golden_v0.schema.json

@@ -0,0 +1,128 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://knowcode.dev/schemas/prose/golden_v0.schema.json",
+  "title": "Prose GoldenLabel",
+  "description": "Final committed record for the SDLC document-collateral retrieval golden set. Extends the code GoldenLabel (tests/eval/golden/golden_v1.0.json): query_type replaces task_type; expected_doc_spans replace expected_entities/expected_files; traceability and synthesis add expected_trace_path / expected_doc_types. See docs/research/document_collateral_retrieval_2026.md section 6.",
+  "type": "object",
+  "additionalProperties": false,
+  "required": [
+    "query_id",
+    "query_text",
+    "query_type",
+    "difficulty",
+    "expected_doc_spans",
+    "must_mention_facts",
+    "must_not_mention_facts",
+    "validation",
+    "schema_version"
+  ],
+  "properties": {
+    "query_id": {
+      "type": "string",
+      "pattern": "^pq[0-9]{3,}$|^pq_[a-z0-9_]+$",
+      "description": "e.g. 'pq001' or 'pq_example_trace'."
+    },
+    "query_text": { "type": "string", "minLength": 8 },
+    "query_type": { "enum": ["fact_lookup", "traceability", "synthesis"] },
+    "difficulty": { "enum": ["easy", "medium", "hard"] },
+    "expected_doc_spans": {
+      "type": "array",
+      "minItems": 1,
+      "items": { "$ref": "#/$defs/docSpan" },
+      "description": "Prose analogue of expected_entities + expected_files: the passages a correct retrieval must surface."
+    },
+    "expected_trace_path": {
+      "type": "array",
+      "default": [],
+      "items": {
+        "type": "string",
+        "pattern": "^(feature|requirement|persona|component|release|decision|client):[a-z0-9._/-]+$|^[a-z0-9._/-]+\\.md$",
+        "description": "Ordered graph nodes for a traceability walk: typed entity refs (e.g. 'feature:atlas-rate-limiting') or document ids ('prd/atlas-rate-limiting.md')."
+      }
+    },
+    "expected_doc_types": {
+      "type": "array",
+      "default": [],
+      "items": { "$ref": "#/$defs/docType" },
+      "description": "For synthesis: the collection(s) in scope (e.g. ['case_study'])."
+    },
+    "must_mention_facts": { "type": "array", "items": { "type": "string" } },
+    "must_not_mention_facts": { "type": "array", "items": { "type": "string" } },
+    "correct": { "type": "boolean" },
+    "validation": { "$ref": "#/$defs/validation" },
+    "provenance": { "$ref": "#/$defs/provenance" },
+    "schema_version": { "const": "0.1.0" }
+  },
+  "allOf": [
+    {
+      "if": { "properties": { "query_type": { "const": "traceability" } } },
+      "then": {
+        "required": ["expected_trace_path"],
+        "properties": {
+          "expected_trace_path": { "minItems": 2 },
+          "expected_doc_spans": { "minItems": 2 }
+        }
+      }
+    },
+    {
+      "if": { "properties": { "query_type": { "const": "synthesis" } } },
+      "then": {
+        "required": ["expected_doc_types"],
+        "properties": {
+          "expected_doc_spans": { "minItems": 3 },
+          "expected_doc_types": { "minItems": 1 }
+        }
+      }
+    }
+  ],
+  "$defs": {
+    "docType": {
+      "enum": [
+        "problem_statement", "prd", "architecture", "hld", "lld",
+        "release_notes", "user_guide", "install_guide", "support_manual",
+        "brochure", "feature_catalog", "case_study", "whitepaper"
+      ]
+    },
+    "docSpan": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["doc_id", "doc_type", "heading_path", "line_range"],
+      "properties": {
+        "doc_id": { "type": "string", "description": "Repo-relative path of the document." },
+        "doc_type": { "$ref": "#/$defs/docType" },
+        "heading_path": {
+          "type": "string",
+          "description": "Breadcrumb of the section, e.g. 'PRD > Requirements > Rate Limiting'."
+        },
+        "line_range": {
+          "type": "array",
+          "items": { "type": "integer", "minimum": 1 },
+          "minItems": 2,
+          "maxItems": 2,
+          "description": "[start, end], 1-indexed, start <= end (cross-field check enforced by the mechanical gate)."
+        },
+        "role": { "enum": ["primary", "supporting", "contextual"] }
+      }
+    },
+    "validation": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["source_verified", "human_reviewed", "method"],
+      "properties": {
+        "source_verified": { "type": "boolean" },
+        "human_reviewed": { "type": "boolean" },
+        "method": { "type": "string" }
+      }
+    },
+    "provenance": {
+      "type": "object",
+      "properties": {
+        "query_author": { "type": "string" },
+        "oracle": { "type": "string" },
+        "adversary": { "type": "string" },
+        "human_reviewer": { "type": ["string", "null"] },
+        "reviewed_at": { "type": ["string", "null"] }
+      }
+    }
+  }
+}

tests/eval/prose/phase1_plan.json

@@ -0,0 +1,44 @@
+{
+  "schema_version": "0.1.0",
+  "dataset_version": "prose_golden_v0",
+  "target_records": 60,
+  "status": "spec_only_awaiting_corpus",
+  "query_types": ["fact_lookup", "traceability", "synthesis"],
+  "difficulty_definitions": {
+    "easy": "Answer lives in a single passage of a single document; keyword-matchable.",
+    "medium": "2-3 passages; one hop across sections or one cross-document reference; semantic retrieval matters.",
+    "hard": "3+ passages spanning multiple documents or doc types; multi-hop or a cross-cutting concern."
+  },
+  "strata": [
+    { "query_type": "fact_lookup",  "difficulty": "easy",   "target": 8 },
+    { "query_type": "fact_lookup",  "difficulty": "medium", "target": 8 },
+    { "query_type": "fact_lookup",  "difficulty": "hard",   "target": 4 },
+    { "query_type": "traceability", "difficulty": "easy",   "target": 0 },
+    { "query_type": "traceability", "difficulty": "medium", "target": 8 },
+    { "query_type": "traceability", "difficulty": "hard",   "target": 12 },
+    { "query_type": "synthesis",    "difficulty": "easy",   "target": 2 },
+    { "query_type": "synthesis",    "difficulty": "medium", "target": 8 },
+    { "query_type": "synthesis",    "difficulty": "hard",   "target": 10 }
+  ],
+  "doc_types": [
+    "problem_statement", "prd", "architecture", "hld", "lld",
+    "release_notes", "user_guide", "install_guide", "support_manual",
+    "brochure", "feature_catalog", "case_study", "whitepaper"
+  ],
+  "coverage_constraints": {
+    "min_queries_per_doc_type_as_primary": 2,
+    "min_distinct_doc_types_per_traceability_path": 3,
+    "min_distinct_docs_per_synthesis_query": 3
+  },
+  "rationale": [
+    "fact_lookup skews easy/medium: it is the class where precise chunking + reranking and the local-first answer have the strongest claim (mirrors LOCATE in the code set).",
+    "traceability has no 'easy' tier: a single-document lookup is not a trace. It skews hard because it is multi-hop across the SDLC chain and is the class flat RAG cannot answer.",
+    "synthesis skews medium/hard: aggregation/comparison over many documents; the 2 'easy' records are small single-collection summaries.",
+    "doc-type coverage is enforced as a secondary constraint instead of a full doc_type x query_type x difficulty cross-product (which would be 117 cells for a 60-record bootstrap)."
+  ],
+  "notes": [
+    "SPEC ONLY. No prose corpus is present in this repository yet, so no records are generated and nothing here runs in CI.",
+    "Mirrors tests/eval/pipeline/phase1_plan.json (the code eval plan). Reuses the three-role Author/Oracle/Adversary pipeline; the Oracle reads documents instead of code.",
+    "See docs/research/document_collateral_retrieval_2026.md section 6 for the full evaluation plan and metrics."
+  ]
+}