fix(delphi): address Copilot review on D10-D12 stack (consensus blob keys, priority serialization, defensive fixes)

Verified triage of all 83 Copilot review threads on PRs #2564-#2573:
1 real blocker + 4 escalations confirmed by direct code verification;
the rest nitpicks / already-fixed / intentional bug-mirrors. All fixes
TDD RED->GREEN.

- Consensus entry keys -> Clojure blob shape {tid, n-success, n-trials,
  p-success, p-test} (was Python-convention comment_id/n_success/...).
  Narrows the S1 deferral: these entries flow raw into
  result['consensus'], where server-helpers.ts:298-313 and client-report
  majorityStrict.jsx:23-27 pluck `tid` - the Python keys broke both
  consumers. Rep-comment entries keep comment_id until the deferred
  math-blob alignment PR.
- DynamoDB writer read D11 consensus from a key to_dynamo_dict never
  emits (repness.consensus_comments) -> always wrote the empty default;
  D11 data never reached Delphi_PCAResults. Now reads top-level
  result['consensus']. The round-trip test had stubbed to_dynamo_dict
  with the writer's wrong nested shape - stub corrected to the real
  producer shape.
- to_dynamo_dict comment priorities: int(value) -> Decimal-preserving.
  int() floors sub-1 priorities to 0 (real formula spans ~0.18-31.46 per
  D12.6), which the TS server's weighted routing reads as "no priority
  data". Harmless today (bug-mirror pins 49.0), landmine once #2571
  resolves. Legacy writer path Decimal-wrapped too.
- DynamoDB reader normalizes legacy list-shaped consensus to the dict
  shape (pre-D11 blobs).
- mod_out truthiness -> `is not None` x2 in repness.py (numpy
  array/Index-safe).
- _compute_comment_priorities fails closed (error log + empty dict) on
  PCA/columns desync instead of silently zip-truncating extremities.
- bench_repness.py imported the 14a-deleted comment_stats (ImportError
  on any benchmark run); benchmark import test added.
- Per-variant xfails replace blanket xfail(strict=False) on D11/D12
  real-data tests, so the variants that match Clojure gate again.
  DISCOVERY: scoping the blanket unmasked two previously-invisible
  incremental divergences (bg2018-incremental, pakistan-incremental
  consensus vs Clojure) that the blanket had silently absorbed -
  documented as known-bad incremental xfails, same family as
  biodiversity-incremental, deferred to the sequential-parity work.
  3 pre-existing CCR failures (verified identical on edge 722640e)
  marked with precise per-variant reasons. PGR regression tests skip
  with the 2026-06-11 goldens-deferral reason - the mark S3-5 claimed
  to add but never committed.
- test_repness_smoke consensus-entry structure assertion updated to the
  Clojure key shape (exact key-set check).
- ns docstrings corrected (ns includes PASS post ns-PASS fix; pa+pd <= 1
  reasoning in select_consensus_comments_df).
- PERF deferral comment on the _compute_group_votes scan in
  _compute_comment_priorities (follow-up issue to be filed).

Baseline (2026-07-04, stack top, --include-local): 13 failed / 476
passed / 18 skipped / 143 xfailed. All 13 accounted for: 10 stale-PGR
comparisons (now skipped per deferral), 3 pre-existing CCR (now precise
xfails).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013CiESAwcrZ6BcVfdkumnxa

commit-id:4cc93a23

Author: jucor

delphi/polismath/benchmarks/bench_repness.py

@@ -28,7 +28,6 @@
 from polismath.conversation import Conversation
 from polismath.pca_kmeans_rep.repness import (
     conv_repness,
-    comment_stats,
     compute_group_comment_stats_df,
     select_rep_comments_df,
     select_consensus_comments_df,

delphi/polismath/conversation/conversation.py

@@ -1132,13 +1132,34 @@ def _compute_comment_priorities(self) -> Dict[Any, float]:
         cmnt_proj = pca_project_cmnts(center, comps)
         extremity_arr = compute_comment_extremity(cmnt_proj)
 
+        # Fail closed on desync: if the PCA vectors were computed on a
+        # different column set than the current rating_mat (e.g. moderation
+        # changed between recomputes), zip() would silently truncate and
+        # assign E=0 to the overflow tids — wrong priorities with no
+        # signal. Empty priorities degrade the TS server to uniform
+        # routing, which is honest; silently wrong extremities are not.
+        # (Copilot review 2026-07-04, g4.)
+        n_cols = len(self.rating_mat.columns)
+        if len(extremity_arr) != n_cols:
+            logger.error(
+                f"comment_priorities: extremity length {len(extremity_arr)} "
+                f"!= rating_mat column count {n_cols} (stale PCA?); "
+                f"skipping priorities for this tick")
+            self.comment_priorities = {}
+            return self.comment_priorities
+
         # Column order of `center`/`comps`/`extremity_arr` matches
         # `self.rating_mat.columns` (PCA is computed on rating_mat).
         tid_extremity = dict(zip(self.rating_mat.columns, extremity_arr))
 
         # Per-group A/D/S aggregation. `_compute_group_votes` returns
         # {str(gid): {'n-members': N, 'votes': {tid: {A, D, S}}}}. S includes
         # PASS (line ~1222: `np.sum(~np.isnan(votes))`), matching Clojure.
+        # PERF (deferred, Copilot on PR #2568): this is an O(groups ×
+        # comments × members) scan on every recompute; vectorize or reuse
+        # the repness-stage aggregation — tracked in the follow-up issue
+        # "delphi: _compute_comment_priorities recomputes group votes on
+        # every tick".
         group_votes = self._compute_group_votes()
 
         priorities: Dict[Any, float] = {}
@@ -2454,10 +2475,18 @@ def float_to_decimal(obj):
             logger.info(f"[{time.time() - start_time:.2f}s] Processing comment priorities...")
             priorities = {}
             for cid, priority in self.comment_priorities.items():
+                # Preserve the float VALUE as Decimal (boto3 rejects raw
+                # floats). The previous int() truncation was harmless while
+                # the D12.6 bug-mirror pins every priority to 49.0, but the
+                # real formula (restored when issue #2571 resolves) spans
+                # ~0.18–31.46 on real data: int() floors sub-1 priorities
+                # to 0, which the TS server's weighted routing treats as
+                # "no priority data" — those comments would never be routed.
+                value = float_to_decimal(float(priority))
                 try:
-                    priorities[int(cid)] = int(priority)
+                    priorities[int(cid)] = value
                 except (ValueError, TypeError):
-                    priorities[cid] = int(priority)
+                    priorities[cid] = value
             result['comment_priorities'] = priorities
 
         # Process repness data efficiently

delphi/polismath/database/dynamodb.py

@@ -300,12 +300,16 @@ def write_conversation(self, conv) -> bool:
             if analysis_table:
                 if dynamo_data:
                     # Use pre-formatted data
-                    # D11 cascade fix (Investigation B, Site 1): source the FULL
-                    # consensus dict from repness.consensus_comments, preserving
-                    # both `agree` and `disagree` lists. The old code dropped
-                    # `disagree` entirely (`.get('consensus', {}).get('agree', [])`).
-                    consensus_comments = dynamo_data.get('repness', {}).get(
-                        'consensus_comments', {'agree': [], 'disagree': []}
+                    # D11 cascade fix (Investigation B, Site 1), corrected
+                    # 2026-07-04: `to_dynamo_dict()` surfaces consensus at
+                    # TOP-LEVEL `result['consensus']` — its `repness` dict
+                    # carries only `comment_repness`. The previous read of
+                    # `repness.consensus_comments` matched a key that never
+                    # exists, so the writer always stored the empty default
+                    # (the round-trip test masked this by stubbing
+                    # to_dynamo_dict with the wrong nested shape).
+                    consensus_comments = dynamo_data.get(
+                        'consensus', {'agree': [], 'disagree': []}
                     )
                     analysis_table.put_item(Item={
                         'zid': zid,
@@ -450,7 +454,13 @@ def write_conversation(self, conv) -> bool:
                             batch.put_item(Item={
                                 'zid_tick': zid_tick,
                                 'comment_id': str(comment_id),
-                                'priority': comment_priorities.get(comment_id, 0),
+                                # Legacy branch reads conv.comment_priorities
+                                # directly (raw floats) — convert like the
+                                # stats/consensus_score fields above, or
+                                # boto3 rejects the write (Copilot
+                                # 2026-07-04, e).
+                                'priority': self._replace_floats_with_decimals(
+                                    comment_priorities.get(comment_id, 0)),
                                 'stats': stats,
                                 'consensus_score': consensus_score,
                                 'zid': zid,
@@ -861,9 +871,16 @@ def read_math_by_tick(self, zid: str, math_tick: int) -> Dict[str, Any]:
                     # new dict shape `{'agree': [], 'disagree': []}` rather than
                     # the obsolete empty list `[]`, so downstream consumers
                     # always receive a uniformly-shaped value.
-                    result['consensus'] = analysis.get(
+                    stored_consensus = analysis.get(
                         'consensus_comments', {'agree': [], 'disagree': []}
                     )
+                    # Normalize legacy blobs (Copilot 2026-07-04, g3):
+                    # pre-D11 writers stored consensus as a (hardcoded-empty)
+                    # LIST. Map any list to the empty dict shape so readers
+                    # of old ticks never see a list.
+                    if isinstance(stored_consensus, list):
+                        stored_consensus = {'agree': [], 'disagree': []}
+                    result['consensus'] = stored_consensus
 
             # 2. Get groups data
             groups_table = self.tables.get('Delphi_KMeansClusters')

delphi/polismath/pca_kmeans_rep/repness.py

@@ -91,10 +91,12 @@ def prop_test(succ, n):
 
     Args:
         succ: Series of success counts (e.g. `na` or `nd` per row).
-        n: Series of trial counts. In all current callers this is `ns = na + nd`
-           (AGREE + DISAGREE per row) — PASS votes are NOT included, matching
-           what Clojure passes as `n-trials`. If you call this from elsewhere,
-           supply `na + nd` rather than a "total votes seen including pass" count.
+        n: Series of trial counts. In all current callers this is `ns` = the
+           count of ALL non-nil votes INCLUDING PASS (`notna().sum()`),
+           matching Clojure's `count-votes` with no vote arg
+           (`(count (filter identity votes))` — 0/PASS is truthy in Clojure,
+           repness.clj:56-61; ns-PASS fix 2026-06-11). If you call this from
+           elsewhere, supply the PASS-inclusive non-nil count, NOT `na + nd`.
 
     Returns:
         Series of z-scores. Positive when the smoothed proportion (succ+1)/(n+1)
@@ -519,7 +521,9 @@ def select_rep_comments_df(stats_df: pd.DataFrame,
     if stats_df.empty:
         return empty_df, None
 
-    mod_out_set = set(mod_out) if mod_out else set()
+    # `is not None`, not truthiness: mod_out may be a numpy array / pandas
+    # Index, whose bare truth value raises for len>1 (Copilot 2026-07-04).
+    mod_out_set = set(mod_out) if mod_out is not None else set()
     sufficient: List[Dict[str, Any]] = []
     best: Optional[Dict[str, Any]] = None
     # Track best's max(rat, rdt) as a sidecar scalar so we never have to mutate
@@ -685,7 +689,9 @@ def consensus_stats_df(vote_matrix_df: pd.DataFrame,
     df['pat'] = prop_test_vectorized(df['na'], df['ns'])
     df['pdt'] = prop_test_vectorized(df['nd'], df['ns'])
 
-    if mod_out:
+    # `is not None`, not truthiness: mod_out may be a numpy array / pandas
+    # Index, whose bare truth value raises for len>1 (Copilot 2026-07-04).
+    if mod_out is not None:
         mod_out_set = set(mod_out)
         df = df[~df.index.isin(mod_out_set)]
 
@@ -706,19 +712,25 @@ def select_consensus_comments_df(
       - Agree: `pa > 0.5 AND z-sig-90(pat)`, sorted desc by `am = pa * pat`.
       - Disagree: `pd > 0.5 AND z-sig-90(pdt)`, sorted desc by `dm = pd * pdt`.
 
-    With PSEUDO_COUNT smoothing the constraint `pa + pd = 1` is exact (na+nd=ns
-    after smoothing), so `pa > 0.5 ⟺ pd < 0.5` — the same tid cannot appear in
-    both lists.
+    Since `ns` counts all non-nil votes including PASS (ns ≥ na+nd),
+    `pa + pd = (na+nd+PSEUDO_COUNT)/(ns+PSEUDO_COUNT) ≤ 1`, so pa and pd
+    cannot both exceed 0.5 — the same tid cannot appear in both lists. (The
+    equality pa+pd=1 holds only for PASS-free comments.)
 
     Args:
         cons_stats: DataFrame indexed by tid with cols [na, nd, ns, pa, pd,
             pat, pdt], as produced by `consensus_stats_df`.
 
     Returns:
         Dict shape `{'agree': [entries], 'disagree': [entries]}`. Each entry
-        is `{comment_id, n_success, n_trials, p_success, p_test}` (Python
-        convention key naming per S1; math-blob alignment with Clojure's
-        hyphenated keys is a future PR).
+        is `{tid, n-success, n-trials, p-success, p-test}` — EXACTLY the
+        Clojure blob shape (repness.clj:181 + the ::consensus s/keys spec).
+        This narrows the S1 deferral (2026-07-04): consensus entries flow
+        raw into `result['consensus']` in to_dict / to_dynamo_dict, where
+        server-helpers.ts:298-313 and client-report's
+        majorityStrict.jsx:23-27 pluck `tid` — Python-convention keys broke
+        both. Rep-comment entries keep `comment_id` until the deferred
+        math-blob alignment PR.
     """
     if cons_stats.empty:
         return {'agree': [], 'disagree': []}
@@ -735,20 +747,20 @@ def select_consensus_comments_df(
 
     def _agree_entry(tid: Any, row: pd.Series) -> Dict[str, Any]:
         return {
-            'comment_id': int(tid),
-            'n_success': int(row['na']),
-            'n_trials': int(row['ns']),
-            'p_success': float(row['pa']),
-            'p_test': float(row['pat']),
+            'tid': int(tid),
+            'n-success': int(row['na']),
+            'n-trials': int(row['ns']),
+            'p-success': float(row['pa']),
+            'p-test': float(row['pat']),
         }
 
     def _disagree_entry(tid: Any, row: pd.Series) -> Dict[str, Any]:
         return {
-            'comment_id': int(tid),
-            'n_success': int(row['nd']),
-            'n_trials': int(row['ns']),
-            'p_success': float(row['pd']),
-            'p_test': float(row['pdt']),
+            'tid': int(tid),
+            'n-success': int(row['nd']),
+            'n-trials': int(row['ns']),
+            'p-success': float(row['pd']),
+            'p-test': float(row['pdt']),
         }
 
     return {

delphi/tests/test_benchmarks_importable.py

@@ -0,0 +1,21 @@
+"""Benchmarks must stay importable as the production API evolves.
+
+PR 14a deleted the scalar repness functions; `bench_repness.py` still
+imported `comment_stats`, so running any benchmark in that module crashed
+with ImportError. Plain import tests catch this class of drift at CI time
+(Copilot review 2026-07-04, g2).
+"""
+
+import importlib
+
+import pytest
+
+
+@pytest.mark.parametrize('module_name', [
+    'polismath.benchmarks.bench_repness',
+    'polismath.benchmarks.bench_pca',
+    'polismath.benchmarks.bench_update_votes',
+    'polismath.benchmarks.benchmark_utils',
+])
+def test_benchmark_module_imports(module_name):
+    importlib.import_module(module_name)