compdemocracy
diff --git a/‎delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md‎
Lines changed: 81 additions & 0 deletions b/‎delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md‎
Lines changed: 81 additions & 0 deletions
diff --git a/‎delphi/docs/PLAN_DISCREPANCY_FIXES.md‎
Lines changed: 18 additions & 3 deletions b/‎delphi/docs/PLAN_DISCREPANCY_FIXES.md‎
Lines changed: 18 additions & 3 deletions
diff --git a/‎delphi/polismath/conversation/conversation.py‎
Lines changed: 9 additions & 3 deletions b/‎delphi/polismath/conversation/conversation.py‎
Lines changed: 9 additions & 3 deletions
diff --git a/‎delphi/polismath/database/dynamodb.py‎
Lines changed: 29 additions & 6 deletions b/‎delphi/polismath/database/dynamodb.py‎
Lines changed: 29 additions & 6 deletions
@@ -1506,3 +1506,84 @@ See `~/polis/D10_D11_D12_GOLDENS_DECISIONS.md`. Highlights:
 ### What's Next
 
 PR 9 (D11) on top of D10 in the same spr stack.
+
+## Session: PR 9 — D11 consensus comment selection (2026-06-11)
+
+Landed in `/goal` mode. Decisions documented in
+`~/polis/D10_D11_D12_GOLDENS_DECISIONS.md` (D11.x section).
+
+### What landed
+
+**Production (`delphi/polismath/pca_kmeans_rep/repness.py`):**
+- New `consensus_stats_df(vote_matrix_df, mod_out=None) -> pd.DataFrame`:
+  whole-conversation per-comment stats (no group split, no `ra/rd/rat/rdt`).
+  Vectorized port of Clojure `consensus-stats` (repness.clj:284-290).
+- Rewrite `select_consensus_comments_df(cons_stats) -> Dict[str, List[Dict]]`:
+  matches Clojure `select-consensus-comments` (repness.clj:293-323).
+  Filters: agree `pa > 0.5 AND z-sig-90(pat)`, disagree
+  `pd > 0.5 AND z-sig-90(pdt)`. Ordering: descending `pa*pat` / `pd*pdt`.
+  Cap: top 5 each side. Output: `{'agree': [...], 'disagree': [...]}`.
+- `conv_repness` grows `mod_out: Optional[Iterable[int]] = None` kwarg.
+  Forwarded to both `select_rep_comments_df` and `consensus_stats_df`.
+- Consensus is now computed unconditionally (Clojure parity — pre-D11
+  Python had a `len(group_clusters) > 1` guard with no Clojure analog).
+- `_stats_row_to_dict` deleted (orphan after D11).
+
+**Caller (`conversation.py`):**
+- `_compute_repness` passes `mod_out=self.mod_out_tids` to `conv_repness`.
+
+**Downstream consumers updated** for the new dict shape:
+- `tests/test_repness_smoke.py::test_repness_structure` — iterates
+  `consensus['agree']` and `consensus['disagree']`.
+- `tests/test_pipeline_integrity.py::test_full_pipeline` — same.
+
+**Tests (12 new in `tests/test_discrepancy_fixes.py`):**
+- `TestD11ConsensusStatsDf` (4): basic counts, pseudocount pa/pd,
+  ns=0 fallback, mod_out filter.
+- `TestD11SelectConsensusBoundary` (8): empty input, clear agree
+  consensus, clear disagree consensus, divisive (no consensus), top-5
+  cap, entry keys (Python convention per S1), disagree entry key
+  mapping (n_success ← nd, p_success ← pd, p_test ← pdt),
+  mutually-exclusive agree/disagree lists.
+
+### Suite delta
+
+- Pre (post-D10): 313 passed, 12 skipped, 58 xfailed.
+- Post (this PR): 325 passed, 12 skipped, 58 xfailed.
+- Delta: +12 (the 12 new D11 synthetic tests). Zero regressions.
+
+### DISCOVERY: ns-PASS divergence
+
+The D11 real-data test (`test_consensus_matches_clojure`) showed 3-5/5
+overlap on cold_start — close but not exact. Investigation revealed a
+deeper bug:
+
+**Clojure's `:ns`** (via `count-votes` with `filter identity` —
+repness.clj:56-61) INCLUDES PASS votes (`0` is truthy in Clojure).
+
+**Python's `ns`** in BOTH `compute_group_comment_stats_df` and the new
+`consensus_stats_df` computes `ns = na + nd`, EXCLUDING PASS.
+
+This means every downstream metric (pa, pd, pat, pdt, ra, rd, rat, rdt,
+agree_metric, disagree_metric) is computed with the wrong denominator
+when PASS votes are present. The D5 PR #2519 journal claim that "PASS NOT
+included, matching Clojure" was a misreading of `count-votes`.
+
+**Impact:**
+- D5/D6/D7/D8 blob-comparison tests' "mismatches" were not (only)
+  upstream PCA/KMeans divergence — the ns-PASS divergence is at least
+  a contributing cause.
+- D11 consensus partial overlap is consistent with this divergence.
+- Fixing requires a separate PR affecting two production functions and
+  re-recording goldens.
+
+D11 real-data test xfailed with the right reason. Logic pinned by the
+12 synthetic tests (which never exercise PASS, so they don't show the
+divergence).
+
+This is now the top item under "Pending — needs team discussion" in
+PLAN.md, with a sketch of the fix.
+
+### What's Next
+
+PR 11 (D12) on top of D11 in the same spr stack.
@@ -29,9 +29,8 @@ This plan's "PR N" labels map to actual GitHub PRs as follows:
 | PR 12 (D15) | #2523 | Stack 16/17 | Fix D15: moderation handling |
 | (K-inv) | #2524 | Stack 17/17 | Fix K-means k divergence: preserve row order |
 | PR 14a (scalar deletion) | #2564 | — | Delete dead scalar paths in `repness.py`; migrate blob injection tests to vectorized |
-| PR ns-PASS fix | — (in flight) | — | Fix `ns` to include PASS votes in `compute_group_comment_stats_df` (Clojure `count-votes` parity) |
-| PR 8 (D10) | — (in flight) | — | Fix D10: rep comment selection — single-pass reduce matching Clojure |
-| PR 9 (D11) | — (WIP) | — | Fix D11: consensus selection — **NEEDS REWORK** |
+| PR 8 (D10) | #2566 | — | Fix D10: rep comment selection — single-pass reduce matching Clojure |
+| PR 9 (D11) | — (in flight) | — | Fix D11: consensus selection — whole-conv stats + per-side top-5 matching Clojure |
 | PR 10 (D3) | — (WIP) | — | Fix D3: k-smoother buffer — **NEEDS REWORK** |
 | PR 11 (D12) | — (WIP) | — | Fix D12: comment priorities — **NEEDS REWORK** |
 | PR 13 (D1) | — (WIP) | — | Fix D1: PCA sign flip prevention — **NEEDS REWORK** |
@@ -916,6 +915,22 @@ Tagging this as a follow-up. No code changes until we discuss.
 - **`to_dynamo_dict` parallel inline implementations** were refactored to
   route through the same helpers as `to_dict` in PR #2523 follow-up. No
   further action needed.
+- **`ns` includes-PASS-divergence** (DISCOVERED 2026-06-11 during D11). Clojure's
+  `:ns` (via `count-votes` with `filter identity` — repness.clj:56-61) INCLUDES
+  PASS votes. Python's `compute_group_comment_stats_df` and `consensus_stats_df`
+  both compute `ns = na + nd`, excluding PASS. This is a real divergence that
+  affects `pa, pd, pat, pdt, ra, rd, rat, rdt` everywhere — every downstream
+  metric and selection. The D5 PR #2519 journal claim ("PASS NOT included,
+  matching Clojure") was based on a misreading of `count-votes`. Currently
+  causing 3-5% divergence in pat values for tids with non-zero PASS counts;
+  visible at the consensus-selection margins (4/6 overlap on vw cold_start
+  agree, 1/3 on disagree). Needs a dedicated PR — affects:
+    - `compute_group_comment_stats_df` (line ~283: `ns = na + nd`).
+    - `consensus_stats_df` (line ~435: `ns = na + nd`).
+  Fix: `ns = (vote_matrix_df != 0).sum(axis=0)` no, actually we want to
+  count non-NaN: `ns = vote_matrix_df.notna().sum(axis=0)` for wide format;
+  for long-format `votes_long.groupby('comment').size()` after dropna.
+  Re-record goldens afterward.
 - **D10 take-5 eviction edge case** (2026-06-11). The Clojure-parity
   `select_rep_comments_df` introduced in PR 8 mirrors Clojure exactly:
   `take(5)` runs AFTER prepending the `best_agree` slot. When `best_agree`
 
@@ -753,16 +753,22 @@ def _compute_repness(self) -> None:
 
         # Check if we have groups
         if not self.group_clusters:
+            # B1 fix (D11 sub-agent review): consensus_comments must always be
+            # `{'agree': [], 'disagree': []}` (dict) post-D11, never `[]` (list).
             self.repness = {
                 'comment_ids': list(self.rating_mat.columns),
                 'group_repness': {},
-                'consensus_comments': []
+                'consensus_comments': {'agree': [], 'disagree': []}
             }
             logger.info(f"Representativeness completed in {time.time() - start_time:.2f}s (no groups)")
             return
 
-        # Compute representativeness (needs participant IDs, not base-cluster IDs)
-        self.repness = conv_repness(self.rating_mat, self._unfolded_group_clusters())
+        # Compute representativeness (needs participant IDs, not base-cluster IDs).
+        # `mod_out=self.mod_out_tids` forwards moderated-out tids to the rep + consensus
+        # selectors (Clojure parity per D11 / PR 9; matches repness.clj:222 and :296).
+        self.repness = conv_repness(self.rating_mat,
+                                    self._unfolded_group_clusters(),
+                                    mod_out=self.mod_out_tids)
         logger.info(f"Representativeness completed in {time.time() - start_time:.2f}s")
 
     def _compute_participant_info_optimized(self, vote_matrix: pd.DataFrame, group_clusters: List[Dict[str, Any]]) -> Dict[str, Any]:
 
@@ -300,6 +300,13 @@ def write_conversation(self, conv) -> bool:
             if analysis_table:
                 if dynamo_data:
                     # Use pre-formatted data
+                    # D11 cascade fix (Investigation B, Site 1): source the FULL
+                    # consensus dict from repness.consensus_comments, preserving
+                    # both `agree` and `disagree` lists. The old code dropped
+                    # `disagree` entirely (`.get('consensus', {}).get('agree', [])`).
+                    consensus_comments = dynamo_data.get('repness', {}).get(
+                        'consensus_comments', {'agree': [], 'disagree': []}
+                    )
                     analysis_table.put_item(Item={
                         'zid': zid,
                         'math_tick': math_tick,
@@ -308,7 +315,7 @@ def write_conversation(self, conv) -> bool:
                         'comment_count': dynamo_data.get('comment_count', 0),
                         'group_count': dynamo_data.get('group_count', 0),
                         'pca': dynamo_data.get('pca', {}),
-                        'consensus_comments': dynamo_data.get('consensus', {}).get('agree', [])
+                        'consensus_comments': consensus_comments
                     })
                 else:
                     # Legacy format
@@ -321,11 +328,21 @@ def write_conversation(self, conv) -> bool:
                         }
                         # Replace floats with Decimal for DynamoDB
                         pca_data = self._replace_floats_with_decimals(pca_data)
-                    
-                    # Create the analysis record with Decimal conversion
-                    consensus_comments = self._numpy_to_list(conv.consensus) if hasattr(conv, 'consensus') else []
+
+                    # D11 cascade fix (Investigation B, Site 2): the old code
+                    # sourced from `conv.consensus`, which is always `[]` post-D11
+                    # (the attribute was deprecated). Source from
+                    # `conv.repness['consensus_comments']` instead — the new shape
+                    # is `{'agree': [...], 'disagree': [...]}`.
+                    if hasattr(conv, 'repness') and conv.repness:
+                        consensus_comments = conv.repness.get(
+                            'consensus_comments', {'agree': [], 'disagree': []}
+                        )
+                    else:
+                        consensus_comments = {'agree': [], 'disagree': []}
+                    consensus_comments = self._numpy_to_list(consensus_comments)
                     consensus_comments = self._replace_floats_with_decimals(consensus_comments)
-                    
+
                     analysis_table.put_item(Item={
                         'zid': zid,
                         'math_tick': math_tick,
@@ -840,7 +857,13 @@ def read_math_by_tick(self, zid: str, math_tick: int) -> Dict[str, Any]:
                         }
 
                     # Set consensus
-                    result['consensus'] = analysis.get('consensus_comments', [])
+                    # D11 cascade fix (Investigation B, Site 3): default to the
+                    # new dict shape `{'agree': [], 'disagree': []}` rather than
+                    # the obsolete empty list `[]`, so downstream consumers
+                    # always receive a uniformly-shaped value.
+                    result['consensus'] = analysis.get(
+                        'consensus_comments', {'agree': [], 'disagree': []}
+                    )
 
             # 2. Get groups data
             groups_table = self.tables.get('Delphi_KMeansClusters')