fix(explain): stop honest narrations from false-failing the number-provenance gate (#80)

* test(ui): stop drylab narration test from making a live API call when creds are set

test_only_ok_status_ever_carries_text passed provider=None, which selects the
configured provider via get_provider(). On a machine with PERMEA_LLM_TOKEN,
PERMEA_LLM_ENDPOINT_ID and PERMEA_LLM_BASE_URL exported, running pytest would
issue a real billed request. Clear the three vars first, matching
test_missing_credentials_yields_unavailable, so the branch deterministically
takes the unavailable path. Assertions unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(explain): remove numeric literals from active warning descriptions so honest narrations stop tripping the number-provenance guard; lock with an invariant test (guardrails.py unchanged)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(explain): blank digit-first identifiers so a faithfully quoted data_sha256 stops false-failing the number-provenance gate

The identifier strip required an alnum run to START on a letter/underscore
([A-Za-z_][A-Za-z0-9_]*), so it removed trailing digits ("abc42" whole) but
not leading ones ("42abc" left "42"). A sha256 that begins with hex digits --
e.g. 41dba61b...8004d, which the demo fixture (_SEED = 20260721) emits -- left
its leading "41" behind as a bare number. collect_numbers deliberately does not
read numbers from string leaves, so that "41" traced to nothing and the gate
withheld a narration that faithfully quoted the hash (~1 in 4 live K-EXAONE runs,
Survey #29).

Blank any [A-Za-z0-9_] run WHOLE whenever it carries a letter or underscore, so
digits go whether they lead or trail. Pure-digit runs carry no letter and still
reach the number tokenizer unchanged. Only the identifier-blanking step changes:
the 3-gate composition, source.collect_numbers, _NUM_TOKEN, _THOUSANDS_GROUPED,
_strip_thousands_separators, and _forbidden_violations are untouched -- no
value-based allowlist, no re-prompting, no gate relaxation.

Known limitation, stated plainly in the tests: a fabricated identifier-shaped
token (a hallucinated hash) no longer trips the number gate via its leading
digits. That detection was accidental; hash fidelity is out of scope for the
number-provenance gate, which guards bare fabricated numbers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: Permea-lab-admin

permea_explain/guardrails.py

@@ -45,10 +45,20 @@
 # (e.g. delta_point = -0.05) is matched with its sign, not as its absolute value.
 _NUM_TOKEN = re.compile(r"-?\d+(?:\.\d+)?")
 
-# Identifier-like tokens (PERMEA-W101, ref_b58ee4, physchem_rf): digits inside them are part
-# of an opaque reference, not numeric facts, so strip identifiers before reading standalone
-# numbers. "PERMEA-W101" -> "PERMEA" and "W101" are both stripped, so 101 is never read.
-_IDENTIFIER_TOKEN = re.compile(r"[A-Za-z_][A-Za-z0-9_]*")
+# Identifier-like tokens (PERMEA-W101, ref_b58ee4, physchem_rf, the hex of a data_sha256):
+# digits inside them are part of an opaque reference, not numeric facts, so strip identifiers
+# before reading standalone numbers. A maximal run of [A-Za-z0-9_] is blanked WHOLE whenever
+# it carries a letter or underscore, so its digits go whether they LEAD or TRAIL the letters:
+# "PERMEA-W101" -> "PERMEA" and "W101" both vanish (101 never read), and a digit-first hash
+# "41dba61b...8004d" vanishes entire (the leading 41 never read). A pure-digit run carries no
+# letter and is left for the number tokenizer, unchanged.
+#
+# The earlier form, [A-Za-z_][A-Za-z0-9_]*, required the run to START on a letter/underscore.
+# That blanked trailing digits ("abc42" whole) but not leading ones ("42abc" left "42"), so a
+# sha256 that began with hex digits left those digits behind as a bare, untraceable number and
+# false-failed a narration that faithfully quoted the hash. Blanking is now symmetric.
+_ALNUM_RUN = re.compile(r"[A-Za-z0-9_]+")
+_HAS_LETTER_OR_UNDERSCORE = re.compile(r"[A-Za-z_]")
 
 _WCODE = re.compile(r"PERMEA-W[0-9]{3}")
 
@@ -78,14 +88,26 @@ def _strip_thousands_separators(text: str) -> str:
     return _THOUSANDS_GROUPED.sub(lambda m: m.group(1).replace(",", ""), text)
 
 
+def _blank_identifiers(text: str) -> str:
+    """Blank every [A-Za-z0-9_] run that carries a letter or underscore, whole.
+
+    A pure-digit run denotes a number and is left in place; a run that also holds a letter or
+    underscore is an opaque identifier (a warning code, a column name, a hash) and is removed
+    entirely -- leading digits included, so a digit-first hash leaves nothing behind."""
+    return _ALNUM_RUN.sub(
+        lambda m: " " if _HAS_LETTER_OR_UNDERSCORE.search(m.group(0)) else m.group(0),
+        text,
+    )
+
+
 def _standalone_number_tokens(text: str) -> list[str]:
     """Numeric tokens that stand alone -- NOT digit runs inside an identifier.
 
     Grouping commas are removed first, so a group-formatted count is assembled into the one
     number it denotes. This changes only how a token is ASSEMBLED from the prose; what counts
     as a match against the report is unchanged.
     """
-    without_ids = _IDENTIFIER_TOKEN.sub(" ", _strip_thousands_separators(text))
+    without_ids = _blank_identifiers(_strip_thousands_separators(text))
     return _NUM_TOKEN.findall(without_ids)
 
 

permea_explain/prompt.py

@@ -9,6 +9,11 @@
 the ``permea_core`` warning registry (a one-way import) so the model can explain what a code
 means. The numeric allowed-set is taken strictly from the report's typed fields, never from
 registry prose -- and only ACTIVE codes fire, whose descriptions carry no numeric literals.
+That last clause was an unchecked assertion and was false in practice (W001/W002 carried
+"permea-eval/1.0", W101 "Paper 1 P1", W501 a hardcoded "95%"), so faithful narrations copied
+numbers no report field could back and the provenance guard withheld them. The literals are
+gone and ``tests/test_explain_registry_numerals.py`` now enforces the invariant against the
+guardrail's own extractor.
 """
 from __future__ import annotations
 
@@ -30,7 +35,9 @@
    값과 문자 그대로 일치해야 합니다. 리포맷 금지 — 0.0432 를 "4.32%" 나 "4%" 로
    바꾸지 말고 그대로 쓰십시오. JSON 의 숫자 필드에 없는 숫자는 출력에 나올 수
    없습니다. 신뢰수준(예: "95% 신뢰구간"의 95)처럼 숫자 필드가 아니라 설명 문자열
-   안에만 있는 값은 숫자로 다시 쓰지 마십시오.
+   안에만 있는 값은 숫자로 다시 쓰지 마십시오. 아래의 레지스트리 설명(경고 코드의
+   의미)도 숫자의 출처가 아닙니다. 그 안에 숫자·버전·논문 참조가 보이더라도 출력에
+   옮기지 마십시오.
 
 2. fired 목록에 있는 경고만 서술하십시오. fired 에 없는 경고 코드(PERMEA-W###)를
    지어내지 마십시오.

src/permea_core/contracts/warnings.py

@@ -84,7 +84,7 @@ class WarningCode:
         fire_condition="EvalRun.identity_definition is None (or a built-in default was used, not an explicit declaration)",
         description=(
             "The identity definition governing the clustering was not explicitly declared. "
-            "Under permea-eval/1.0 it is required with no default: the entire identity-control "
+            "Under the permea-eval contract it is required with no default: the entire identity-control "
             "result depends on how identity is defined (alignment, denominator, gap treatment), "
             "so an undeclared or silently-defaulted definition makes the headline number "
             "uninterpretable and non-comparable across runs."
@@ -99,8 +99,8 @@ class WarningCode:
         severity=Severity.CRITICAL,
         fire_condition='EvalRun.headline_condition != "B"',
         description=(
-            "The reported headline is not the identity-controlled condition B. permea-eval/1.0 "
-            "fixes the headline to the identity-controlled evaluation; reporting the uncontrolled "
+            "The reported headline is not the identity-controlled condition B. The permea-eval "
+            "contract fixes the headline to the identity-controlled evaluation; reporting the uncontrolled "
             "condition A (or anything else) as the headline re-admits exactly the similarity "
             "leakage the protocol exists to remove."
         ),
@@ -132,7 +132,8 @@ class WarningCode:
         description=(
             "Identity-controlled evaluation (B) scores significantly below the uncontrolled "
             "evaluation (A), with the paired-bootstrap CI on the delta excluding zero. This is "
-            "the core Paper 1 P1 finding: a material fraction of the uncontrolled performance was "
+            "the core finding the identity-control protocol exists to expose: a material fraction "
+            "of the uncontrolled performance was "
             "similarity memorisation of near-duplicate train/test peptides, not generalisation. "
             "The magnitude that counts as 'material' is a diagnose-layer policy, not fixed here."
         ),
@@ -148,7 +149,7 @@ class WarningCode:
         severity=Severity.WARN,
         fire_condition="a reported delta has ci_excludes_zero == False (CI straddles zero)",
         description=(
-            "The paired-bootstrap 95% CI on the A-vs-B delta includes zero, so the two conditions "
+            "The paired-bootstrap CI on the A-vs-B delta includes zero, so the two conditions "
             "are not statistically distinguishable at that level. The point estimate should not be "
             "reported as a real effect; the result is inconclusive rather than null."
         ),

tests/test_explain_narrate.py

@@ -15,6 +15,7 @@
 from permea_explain import GuardrailViolation, guardrails, narrate, source
 from permea_explain.extract import extract
 from permea_explain.providers.base import Provider, ProviderResponse
+from permea_explain.violations import NUMERIC_UNTRACED
 
 GOOD_IDENTITY = {"alignment": "global", "denominator": "shorter_sequence", "gap_treatment": "free"}
 
@@ -307,6 +308,71 @@ def test_narrate_end_to_end_accepts_grouped_counts(tmp_path):
     assert result["numeric_provenance"]["all_numbers_traced"] is True
 
 
+# --------------------------------------------------------------------------------------
+# Symmetric identifier blanking (#0030)
+# --------------------------------------------------------------------------------------
+# The identifier strip once required a run to START on a letter/underscore, so a sha256 that
+# began with hex digits ("41dba61b...8004d") left its leading "41" behind as a bare number and
+# false-failed a narration that faithfully quoted the hash. Blanking now removes any
+# [A-Za-z0-9_] run carrying a letter/underscore WHOLE -- leading digits included -- while a
+# pure-digit run still reaches the number gate.
+#
+# KNOWN LIMITATION, stated plainly: after this change a FABRICATED identifier-shaped token (a
+# hallucinated hash, a made-up column name with digits) no longer trips the provenance gate
+# via its leading digits. That detection was ACCIDENTAL, never a guarantee -- the number gate
+# exists to catch bare fabricated NUMBERS, not fabricated identifiers, and hash fidelity is out
+# of scope for it. The tests below pin the intended behavior, not that lost side effect.
+
+# The demo fixture (permea_ui.fixtures, _SEED = 20260721) emits this exact digit-first sha256;
+# see tests/test_ui_drylab.py for the end-to-end regression over that fixture.
+_DIGIT_FIRST_SHA256 = "41dba61bf87bd33674c49388075707f5a0dade7060a2da6298bd7f91b1b8004d"
+
+
+def test_digit_first_identifier_leaves_no_bare_number():
+    """A digit-first hash is blanked whole; trailing-digit ids already were -- now symmetric."""
+    assert guardrails.numbers_in_narrative(f"데이터 해시는 {_DIGIT_FIRST_SHA256} 입니다.") == []
+    # "42abc" (leading digits) now matches "abc42" (trailing): both blank whole, neither leaks.
+    assert guardrails.numbers_in_narrative("식별자 42abc 와 abc42 를 봅니다.") == []
+
+
+def test_faithful_hash_quote_passes_provenance(tmp_path):
+    """(a) Quoting the report's data_sha256 no longer trips the number-provenance gate."""
+    _, report = _report_file(tmp_path)
+    report["context"]["data_sha256"] = _DIGIT_FIRST_SHA256  # digit-first, as the demo fixture emits
+    text = f"이 해설은 데이터 해시 {_DIGIT_FIRST_SHA256} 를 참조합니다. 조건 B 가 헤드라인입니다."
+    assert guardrails.render_checked(text, report) == text
+
+
+def test_fabricated_standalone_number_still_untraced(tmp_path):
+    """(b) LOAD-BEARING: a bare fabricated number absent from the report is STILL rejected.
+
+    Proves symmetric blanking did not weaken the gate: only alnum runs CARRYING a letter are
+    swallowed; a standalone "0.87" carries none, reaches the gate, and fails as NUMERIC_UNTRACED.
+    """
+    _, report = _report_file(tmp_path)
+    violations = guardrails.collect_violations(FAITHFUL + " 정확도는 0.87 입니다.", report)
+    assert [v.kind for v in violations] == [NUMERIC_UNTRACED]
+    assert "0.87" in violations[0].message
+
+
+def test_korean_adjacent_digits_still_reach_the_gate(tmp_path):
+    """(c) Hangul is outside [A-Za-z0-9_], so "180개" still yields "180" and is provenance-checked."""
+    assert guardrails.numbers_in_narrative("표본은 180개입니다.") == ["180"]
+    _, report = _report_file(tmp_path)  # this report has n=2959, not 180
+    with pytest.raises(GuardrailViolation, match="180"):
+        guardrails.render_checked("표본은 180개입니다.", report)
+
+
+def test_pure_digit_and_decimal_tokenization_unchanged():
+    """(d) Pure-digit, decimal, and thousands-grouped forms tokenize exactly as before."""
+    assert guardrails.numbers_in_narrative("총 1000 개, 95%, 임계값 0.02, 델타 -0.4344") == [
+        "1000", "95", "0.02", "-0.4344",
+    ]
+    assert guardrails.numbers_in_narrative("표본은 2,959개") == ["2959"]
+    # A warning code still loses its digits (W101 -> gone); the standalone 0.87 survives.
+    assert guardrails.numbers_in_narrative("PERMEA-W101 이 발화, 정확도 0.87") == ["0.87"]
+
+
 def test_extract_is_available_alongside_narrate():
     """Both task layers share one Provider and one verification discipline (#0023)."""
     import inspect

tests/test_explain_registry_numerals.py

@@ -0,0 +1,80 @@
+"""Cross-layer invariant: no ACTIVE warning description may carry a numeric literal.
+
+WHY THIS EXISTS. ``permea_explain.prompt._registry_context`` injects the title and
+description of every FIRED code into the NARRATE prompt as non-numeric context, and
+``prompt``'s module docstring asserts that the numeric allowed-set comes strictly from the
+report's typed fields, "never from registry prose". That assertion was FALSE in practice:
+W001/W002 carried "permea-eval/1.0", W101 carried "Paper 1 P1", and W501 hardcoded a "95%"
+CI level. A faithful narration that translated the context it was handed therefore emitted
+numbers with no report field behind them, and the number-provenance guard correctly -- but
+uselessly -- withheld it. Two live runs were lost to exactly this.
+
+The fix removed the literals; this test keeps them removed. It is deliberately a test and
+not a guardrail relaxation: the guard stays strict, and we remove the CAUSE instead.
+
+It reuses ``guardrails._standalone_number_tokens`` rather than reimplementing a number
+regex, so the invariant is stated in terms of the SAME extractor that will judge the
+narration at runtime. If that tokenizer changes, this test tracks it automatically and
+cannot drift into checking something subtly different.
+
+RESERVED codes are exempt: they have no firing logic, so they can never reach a prompt.
+W201/W301 ("Paper 1 P2"/"P3") and W403 ("n_jobs=-1", "~1e-7") still carry literals and must
+be sanitised before any of them is promoted to ACTIVE -- ``test_reserved_codes_are_the_only
+_exemption`` pins that list so a promotion cannot silently skip the cleanup.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from permea_core.contracts.warnings import WARNINGS, Status, active
+from permea_explain.guardrails import _standalone_number_tokens
+
+#: RESERVED codes known to still carry numerals. Not a permanent exemption -- an entry here
+#: is a debt to pay at promotion time, which is why the set is pinned rather than ignored.
+KNOWN_NUMERIC_RESERVED = {"PERMEA-W201", "PERMEA-W301", "PERMEA-W403"}
+
+
+@pytest.mark.parametrize("warning", active(), ids=lambda w: w.code)
+def test_active_description_has_no_numeric_literal(warning):
+    """An ACTIVE code's prompt-visible prose must contribute no numeric token.
+
+    ``title + description`` is exactly what ``_registry_context`` puts in the prompt, so a
+    token found here is a token the model may faithfully copy into a narration that then
+    fails the provenance gate for a number no report field can back.
+    """
+    prompt_visible = f"{warning.title} {warning.description}"
+    tokens = _standalone_number_tokens(prompt_visible)
+    assert tokens == [], (
+        f"{warning.code} is ACTIVE and its prompt-visible prose yields numeric token(s) "
+        f"{tokens}. A narration that copies one cannot trace it to any report field, so the "
+        f"number-provenance guard will withhold an otherwise honest narration. Rewrite the "
+        f"prose without the literal (drop the version tag / citation / hardcoded level); do "
+        f"not relax the guard."
+    )
+
+
+def test_reserved_codes_are_the_only_exemption():
+    """Pin which RESERVED codes still owe a cleanup, so promoting one cannot skip it."""
+    offenders = {
+        w.code
+        for w in WARNINGS
+        if w.status is Status.RESERVED
+        and _standalone_number_tokens(f"{w.title} {w.description}")
+    }
+    assert offenders == KNOWN_NUMERIC_RESERVED, (
+        f"RESERVED codes carrying numerals changed: {offenders}. If a code was promoted to "
+        f"ACTIVE, sanitise its prose first; if one was cleaned, drop it from "
+        f"KNOWN_NUMERIC_RESERVED."
+    )
+
+
+def test_extractor_actually_bites():
+    """Non-vacuity: the assertion must fail on the prose this fix removed.
+
+    Without this, a tokenizer that returned [] for everything would make the test above pass
+    silently. These are the exact literals that cost two live runs.
+    """
+    assert _standalone_number_tokens("Under permea-eval/1.0 it is required") == ["1.0"]
+    assert _standalone_number_tokens("This is the core Paper 1 P1 finding") == ["1"]
+    assert _standalone_number_tokens("The paired-bootstrap 95% CI on the delta") == ["95"]

tests/test_ui_drylab.py

@@ -17,6 +17,7 @@
 from fastapi.testclient import TestClient
 
 from permea_core.contracts.warnings import Status, WARNINGS
+from permea_explain import guardrails
 from permea_explain.providers.base import ProviderResponse
 from permea_ui import fixtures, runner
 from permea_ui.app import app
@@ -98,6 +99,24 @@ def test_guardrail_failure_yields_withheld_and_no_text(diagnosis):
     assert "offending output" not in (n.detail or ""), "detail must not leak the bad prose"
 
 
+def test_narration_quoting_the_data_sha256_is_not_withheld(diagnosis):
+    """A narration that faithfully quotes the report's data_sha256 must render, not be withheld.
+
+    Regression for #0030 over the demo fixture (fixtures._SEED = 20260721), whose sha256 is
+    digit-first -- 41dba61b...8004d. The old identifier strip left that leading "41" behind as
+    a bare, untraceable number, so the number-provenance gate withheld the narration ~1 in 4
+    live runs. Symmetric blanking removes the hash whole. No live API: the hash is the only
+    numeric-looking content and a FakeProvider supplies the prose.
+    """
+    sha = diagnosis["context"]["data_sha256"]
+    text = f"이 진단의 데이터 해시는 {sha} 입니다."
+    # The crux: the quoted hash contributes no standalone numeric token for the gate to check.
+    assert guardrails.numbers_in_narrative(text) == []
+    n = runner.narrate_diagnosis(diagnosis, provider=FakeProvider(text))
+    assert n.status == "ok", f"expected ok, got {n.status}: {n.detail}"
+    assert n.text == text
+
+
 def test_missing_credentials_yields_unavailable(diagnosis, monkeypatch):
     for var in ("PERMEA_LLM_TOKEN", "PERMEA_LLM_ENDPOINT_ID", "PERMEA_LLM_BASE_URL"):
         monkeypatch.delenv(var, raising=False)
@@ -119,8 +138,16 @@ def complete(self, *a, **kw):
     assert "TimeoutError" in (n.detail or "")
 
 
-def test_only_ok_status_ever_carries_text(diagnosis):
-    """Whatever the status, text is non-None only for `ok`. The UI keys off this."""
+def test_only_ok_status_ever_carries_text(diagnosis, monkeypatch):
+    """Whatever the status, text is non-None only for `ok`. The UI keys off this.
+
+    The `None` case must stay offline: with the three vars set it would select the
+    configured provider and bill a real call from the test suite. Cleared here so the
+    branch deterministically takes the `unavailable` path, as at
+    test_missing_credentials_yields_unavailable.
+    """
+    for var in ("PERMEA_LLM_TOKEN", "PERMEA_LLM_ENDPOINT_ID", "PERMEA_LLM_BASE_URL"):
+        monkeypatch.delenv(var, raising=False)
     for provider in (FakeProvider("정확도는 99.7% 입니다."), None):
         n = runner.narrate_diagnosis(diagnosis, provider=provider)
         assert (n.text is not None) == (n.status == "ok")