Socratic recovery over-flags arc42 Chapter 10 (Quality) as [OPEN] — split priorities from scenarios

[raifdmueller]

Observation

In the AIUP+Socratic experiment run "Contracts4+AIUP+Socratic" on raifdmueller/mistral-vibe, the recovered arc42 Chapter 10 (Quality Requirements) was reduced to two sentences:

Measurable quality scenarios and targets (startup-time SLA, per-turn latency, memory ceiling) are [OPEN — see Q3.10]. The quality-goal ranking is [OPEN — see Q4.9].

An independent review with arc42agentic graded the whole document 🔴 RED, driven almost entirely by this one chapter:

• [S10-01] critical — Chapter 10 is effectively empty; Maintainability and Security scenarios are code-derivable, so a fully blank mandatory chapter over-uses the [OPEN] logic.
• [KQS-01] critical — no quality goal is concretized by a measurable scenario anywhere, making the architecture rationale unfalsifiable.

Root cause

The Socratic process treats all of Quality as team-only knowledge. It is not. Two different things are conflated:

Aspect	Code-derivable?	Correct classification
Quality-goal priorities / ranking ("is security more important than speed?")	No	[OPEN] — Product Owner
Quality scenarios (what the system measurably does today)	Often yes	[ANSWERED] with file:line

Concretely, in the same run the recovery already had the evidence to populate scenarios:

• §8.1 produced a STRIDE threat catalogue (T-001…T-008) — Security scenarios trace straight to those.
• §8.3 documented a test pyramid — Maintainability scenarios (coverage, layering) derive from it.
• Performance probes, timeouts, and budgets exist in code (e.g. tool-output truncation, auto-compact thresholds).

All of that was discarded into a single [OPEN] pointer instead of being written up as scenarios.

Proposed fix

In the Socratic contract (#493), the brownfield page (#494) and the recovery skill (#495), make the Q4 / arc42-§10 handling explicit:

• Quality scenarios are [ANSWERED] when the code shows measurable behaviour — derive them, cite file:line, and reuse evidence already gathered in §8 (threats, test concept, performance probes).
• Only the quality-goal priority/ranking stays [OPEN] (Product Owner).
• arc42 Chapter 10 must never be just an [OPEN] pointer: it carries the derivable scenarios, with the ranking marked open above them.

Related

• Socratic Code-Theory Recovery: pin a fixed question-tree second level (contracts.json) #493 / Socratic Code-Theory Recovery: document the fixed question-tree second level (brownfield page) #494 / Socratic Code-Theory Recovery: enforce the fixed question-tree second level (recovery skill) #495 — the Socratic contract, brownfield page, and recovery skill. This refines the same Q4 branch.

[raifdmueller]

Analysis

The issue is valid and the root-cause diagnosis is correct. A few refinements after checking the current skill files (post-#497):

The split already half-exists — but it is not operative

references/iso-25010.md already states the principle:

Rule of thumb: the mechanism is in the code; the target and priority are in stakeholders' heads. Most Q4 leaves split into [ANSWERED] mechanism + [OPEN] target.

So the skill knows quality is not wholly team-only. The failure is that this lives in a reference file, not in the operative instruction. Neither the contract (#493 template), the Phase 1 prompt, nor phase-2-synthesize.md enforces it — so a run can still collapse the whole Q4/§10 branch into one [OPEN] pointer and nothing flags it. The fix is to make the split operative, not to discover it.

"Mechanism" is not "scenario" — that gap is the real defect

The reference says [ANSWERED] mechanism. arc42 Chapter 10 needs quality scenarios — a measurable stimulus/response, not just "retry logic exists". The missing step is turning a derived mechanism into a scenario. Two cases:

• The code carries a literal threshold (a timeout constant, a truncation limit, an auto-compact size) → a fully [ANSWERED] scenario, target included, cite file:line.
• The code carries only a mechanism, no number (retry exists, no SLO) → write the scenario's stimulus/response from code, but keep the target value [OPEN].

This must be explicit, or the fix risks the opposite failure: an LLM inventing plausible-looking thresholds to fill Chapter 10. "Derive scenarios" must never mean "fabricate numbers."

Q4 vs Q3.10 — the relationship must be defined

The fixed second level has both Q4.1–Q4.9 (ISO characteristics) and Q3.10 (arc42 Quality Requirements chapter). arc42.md only says Q3.10 "Connects to Q4" — too vague, and it is why the synthesis had nowhere to put scenarios. Define it: the Q4 branch derives the per-characteristic scenarios; arc42 Chapter 10 is synthesized from the [ANSWERED] Q4 scenarios plus the Q4.9 ranking. Stated that way, Chapter 10 is empty only if Q4 genuinely produced zero answered scenarios — which on any real codebase it will not.

Wire the §8 → §10 evidence reuse explicitly

The issue is right that the evidence is already gathered. Under the fixed level, §8 is Q3.8 (Crosscutting Concepts), whose Crosscutting Concepts contract mandates §8.1 Threat Model (STRIDE, T-IDs) and §8.3 Test Concept. Phase 2 should be told that Chapter 10 Security scenarios may cite the T-IDs from Q3.8, and Maintainability scenarios the Q3.8 test concept — instead of re-deriving or dropping them.

phase-2-synthesize.md needs a Chapter 10 carve-out

It currently says: "Mark chapters with no content as 'No information from Phase 1'." That rule is what licensed the empty Chapter 10. Add: Chapter 10 is never just an [OPEN] pointer — it carries every derivable scenario, with the goal ranking marked [OPEN] above them. "No information" is only legitimate when Q4 truly produced no answered leaf.

Files in scope

Beyond #493/#494/#495 as the issue notes, the concrete edits land in:

• references/iso-25010.md — promote the mechanism/target rule into an explicit "scenario" instruction; add the literal-threshold vs mechanism-only distinction.
• references/arc42.md — Q3.10 row: define the Q4 → Chapter 10 synthesis relationship.
• prompts/phase-2-synthesize.md — the Chapter 10 carve-out + §8 evidence reuse.
• prompts/phase-1-question-tree.md + the Socratic Code-Theory Recovery: pin a fixed question-tree second level (contracts.json) #493 contract — one line that Q4 scenarios are [ANSWERED] where code shows measurable behaviour, only ranking is [OPEN].
• examples.md — add an [ANSWERED] quality-scenario leaf (currently the Q4 example only shows a mechanism + an [OPEN] target).
• Plugin mirror auto-synced.

Net: this is a refinement of the Q4 branch and the §10 synthesis, consistent with the fixed-level model — no structural change to the tree.