Conversation
* Squashed commit of the following: commit 2024dde Author: Julien Cornebise <julien@cornebise.com> Date: Mon Nov 24 22:14:34 2025 +0000 Update test_powerit_pca_with_nans to expect ValueError The previous commit changed powerit_pca to raise ValueError when given NaN values, making NaN handling the caller responsibility. This test now verifies that behavior instead of testing graceful NaN handling. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> commit 9839770 Author: Julien Cornebise <julien@cornebise.com> Date: Mon Nov 24 22:14:34 2025 +0000 Fix PCA NaN handling to use column mean (matching Clojure) The Python PCA was using 0 to fill missing votes, while Clojure uses column means. This caused significant differences in PCA components (16 deg and 52 deg angle differences on VW dataset). Changes: - pca_project_dataframe: Replace NaN with column means instead of 0 - powerit_pca: Add ValueError if NaN values reach it (caller must preprocess) - test_regression.py: Enable ignore_pca_sign_flip for golden comparisons New tests: - test_pca_unit.py: test_nan_handling_uses_column_mean verifies column mean is used, test_nan_handling_differs_from_zero_fill confirms we are not using 0 - test_legacy_clojure_regression.py: test_pca_components_match_clojure compares Python vs Clojure PCA components (correlation and angle) Results after fix: - VW: PC1/PC2 correlation 1.0/-1.0, angle 0 deg (perfect match) - Biodiversity: PC1/PC2 correlation 0.99/-1.0, angle 7/4 deg (minor numerical differences from power iteration on 82% sparse data) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> commit 4b33297 Author: Julien Cornebise <julien@cornebise.com> Date: Mon Nov 24 22:14:34 2025 +0000 Test that sign flipping propagates If the PCA components flip sign, the projections must also flip sign. commit e307a52 Author: Julien Cornebise <julien@cornebise.com> Date: Fri Nov 21 09:36:22 2025 +0000 Flag an unneeded exception commit 91a1e9c Author: Julien Cornebise <julien@cornebise.com> Date: Wed Nov 19 15:49:34 2025 +0000 Report any scaling issues in PCA projections commit 395e636 Author: Julien Cornebise <julien@cornebise.com> Date: Wed Nov 19 15:25:46 2025 +0000 Turn off off the Clojure sign-flip for PCA Of course this breaks the comparison to the golden files, as those included the sign flip. So we're also giving the option in the regression test to ignore PCA sign-flips. That will be handy later when we work on improving the PCA implementation, as various PCA implementations have various sign conventions. commit 0dc1aa7 Author: Julien Cornebise <julien@cornebise.com> Date: Mon Nov 24 22:14:34 2025 +0000 Clean up unused variables and imports Address GitHub Copilot review comments: - Log superseded votes count in conversation.py instead of leaving unused - Remove unused p1_idx/p2_idx index lookups in corr.py - Remove unused all_passed variable in regression_comparer.py - Remove unused imports (numpy, Path, List, datetime, stats, pca/cluster functions) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add normalized angle computation to PCA regression test The original angle computation assumed unit-normalized vectors, which gave misleading results when comparing Python (exact PCA, unit vectors) with Clojure stochastic PCA for large conversations (non-unit vectors due to learning rate blending without re-normalization). Now outputs both the raw angle and the properly normalized angle, plus the norms of both vectors for diagnostics. The assertion now checks the normalized angle, which correctly identifies when vectors point in similar directions regardless of their magnitudes. Co-Authored-By: Claude <noreply@anthropic.com> * Remove madness "Align with clojure" and update golden records As discussed above, the alignment with clojure was an AI-hallucinated kludge to pass tests by customized scaling tailored to each dataset, hiding the problems with the Python PCA implementation (the handling of NaNs). Now that we have fixed those, we can remove that madness and update the golden records accordingly. * Address Copilot review feedback - Fix all-NaN column edge case in PCA: replace NaN col_means with 0.0 - Lower per-occurrence sign-flip log level to DEBUG (summary stays WARNING) - Fix dataset validation: explicit names always search local datasets Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use Optional[float] instead of float | None for typing consistency Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Log warning when all-NaN columns fall back to 0.0 in PCA Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove dead unnormalized angle code and misleading comment Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
* Make each test run their own Conversation object Until now, one shared object was getting reused across tests for a given dataset, and the cache kept on piling. * Actually reuse the fixtures... * Enable parallel test execution by dataset using xdist_group markers Use pytest-xdist's loadgroup distribution to run tests for different datasets on separate workers while keeping all tests for the same dataset on one worker (preserving fixture cache efficiency). Changes: - Add xdist_group marker to each dataset parameter via pytest.param() - Switch from --dist=loadscope to --dist=loadgroup in pytest config - Clean up conftest.py marker handling Performance: Tests now complete in ~2 minutes with 4 workers (-n4), down from ~9 minutes before deduplication and ~4 minutes sequential after deduplication. The xdist_group marker ensures each dataset's Conversation object is computed only once per worker. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Factorize xdist_group pattern across test suite for parallel execution Add reusable helpers in conftest.py for creating pytest.param objects with xdist_group markers, enabling efficient parallel test execution across all dataset-parametrized tests. Changes: - conftest.py: Add make_dataset_params() and get_available_dataset_params() helper functions for xdist_group marker creation - Update pytest_generate_tests to use xdist_group markers - Update test_pca_smoke.py, test_repness_smoke.py, test_conversation_smoke.py, test_legacy_repness_comparison.py, test_golden_data.py, test_pipeline_integrity.py to use the new helpers Pattern: Tests parametrized by dataset now use xdist_group markers via: - make_dataset_params(["biodiversity", "vw"]) for hardcoded lists - get_available_dataset_params() for dynamically discovered datasets With --dist=loadgroup (default in pyproject.toml), pytest-xdist groups tests by dataset, ensuring fixtures are computed once per worker. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add --datasets option to run tests on specific datasets New pytest option allows filtering tests to run on a subset of datasets: pytest --datasets=biodiversity # single dataset pytest --datasets=biodiversity,vw # multiple datasets Implementation: - pytest_generate_tests filters dynamic parametrization - pytest_collection_modifyitems deselects tests with static parametrization - Report header shows "Filtered to: ..." when active Works with both: - Dynamic parametrization (dataset fixture via pytest_generate_tests) - Static parametrization (@pytest.mark.parametrize with get_available_dataset_params()) Also updated tests/README.md with documentation for all pytest options including --include-local, --datasets, and parallel execution with -n. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix --datasets filter for test_legacy_clojure_regression.py The test file has its own pytest_generate_tests hook that was not respecting the --datasets option. Now imports _get_requested_datasets and make_dataset_params from conftest to properly filter datasets. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Document parallel test execution with pytest-xdist Add "Parallel Test Execution" section to regression_testing.md explaining: - How to use -n auto for parallel execution - How xdist_group markers keep dataset fixtures together - When NOT to use parallel execution (database tests, debugging) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Address Copilot review feedback on test infrastructure - Suppress RuntimeWarning from nanmean on all-NaN columns - Harden --datasets parsing: filter empty entries, error on empty set Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Replace dataset filtering with @pytest.mark.use_discovered_datasets marker - Remove substring-based _extract_dataset_from_test and pytest_collection_modifyitems filtering - Remove get_available_dataset_params() (import-time evaluation) - Add @pytest.mark.use_discovered_datasets marker for dynamic parametrization that respects --datasets and --include-local - Hardcoded @pytest.mark.parametrize tests are no longer affected by --datasets CLI flag - Unify parameter name to dataset_name across all test files - Add test_dataset_selection.py with unit and pytester integration tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
* Save difference log regardless of errors * Account for flip sign in numerical difference reporting * Reformat help message * Add cluster centers to PCA-related paths for sign flip handling Cluster centers are derived from PCA projections, so they inherit the sign ambiguity. This fix ensures sign flips are detected and corrected for `.center` paths in addition to `.pca.comps` and `.proj.` paths. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Allow for a few components with higher error tolerance * Fix per-component PCA sign flip detection Previously, sign flips were detected per-projection-vector, which fails when only some components are flipped (e.g., PC1 unchanged, PC2 flipped). Now detects flips at the component level (.pca.comps[N]) and stores them, then applies per-dimension correction to projections and cluster centers. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Replace element-wise tolerance with projection metrics for regression tests TLDR: it works! Differences were just numerical errors, artificially looking huge on small values. Looking at the whole cloud of projections confirmed the post sk-learn matches the pre-sklearn results perfectly well. Problem: -------- The regression comparer was failing on datasets like FLI and pakistan with thousands of "differences" showing relative errors up to 874%. Investigation revealed these were false positives: the high relative errors occurred on near-zero values (e.g., golden=4.54e-06 vs current=4.42e-05) where even tiny absolute differences (3e-04) produce huge relative errors. Diagnosis: ---------- Comparing sklearn SVD-based PCA against power iteration golden snapshots: - The projection point clouds are visually identical (see scatter plots) - Important values (Q70-Q100 percentile) match within 0.2% - Only near-zero values (Q0-Q7 percentile, ~0.0x median) show large rel errors - These near-zero values represent participants at the origin who don't affect visualization or clustering The element-wise (abs_tol, rel_tol) approach fundamentally cannot handle this case: it either fails on small values or is too loose for large values. Solution: --------- Added projection comparison metrics that measure what actually matters: | Metric | Threshold | What it measures | |-----------------------|-----------|-------------------------------------| | Max |error| / range | < 1% | Worst displacement as % of axis | | Mean |error| / range | < 0.1% | Average displacement as % of axis | | R² (all coordinates) | > 0.9999 | Variance explained (99.99%) | | R² (per dimension) | > 0.999 | Per-PC fit quality (99.9%) | | Procrustes disparity | < 1e-4 | Shape similarity after alignment | Results for FLI dataset: - Max |error| / range: 0.0617% (was flagging 28% rel error on Q1 values) - R²: 0.9999992 - Procrustes: 8.2e-07 All 7 local datasets now pass regression tests. Also added: - Quantile context in error reports (computed from ALL values, not just failures) - Explanation when element-wise diffs exist but metrics confirm match * Address Copilot review feedback on regression comparer - Exclude .pca.center from PCA sign-flip handling (not sign-ambiguous) - Use AND logic for overall_match (don't let projection metrics override stage failures) - Guard against division by zero when projection data_range is 0 - Fix return type annotation on _log_projection_metrics Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* Add PCA benchmark script * Replace handwritten PCA by SKLearn's implementation Note: we will update the golden records in a later commit. * Replace projection by Sklearn's fit_transform for better performance. I took care to keep the "sparsity-aware" scaling, computed in a vectorized way. * Remove unused start_vectors argument * Remove function wrapped_pca It's not needed anymore: the PCA is done by Sklearn, and the edge cases are already handled by wrapped_pca caller, i.e. pca_project_dataframe. Less code, less complexity, less maintenance. * Update biodiversity and vw golden snapshots with sklearn PCA Re-recorded golden snapshots using the new sklearn SVD-based PCA. These snapshots now serve as the baseline for regression tests. * Fix PCA warnings in test_conversation.py Tests that only verify sorting/moderation now use recompute=False to skip unnecessary PCA computation. Manager tests that need PCA but use minimal data have targeted filterwarnings with explanations. Changes: - Add recompute=False to sorting tests (no PCA needed for ID ordering) - Add recompute=False to test_moderation (only checks filtering) - Add @pytest.mark.filterwarnings to TestConversationManager tests that use minimal data through the manager interface 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Address Copilot review feedback on PCA implementation - Fix division by zero when participant has no votes (pca.py) Mirrors Clojure's (max n-votes 1) approach - Fix return type annotation for _log_projection_metrics (comparer.py) - Add explanatory comment for intentional except pass (comparer.py) - Remove unused variable in test (test_regression.py) - Add unit test for no-votes edge case (test_pca_unit.py) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Address Copilot review feedback on sklearn PCA migration - Use pca.mean_ instead of manual centering (sklearn handles centering internally) - Replace np.random.seed(42) with random_state=42 on PCA constructor - Update docstring to reflect sklearn PCA + column-mean imputation - Fix _pca_sign_flips type annotation to Dict[str, Dict[int, int]] - Scope outlier_fraction to PCA-related paths only (non-PCA data stays strict) - Add try/except around symlink creation for cross-platform robustness - Remove unused normalize_vector/proj_vec test helpers Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Address review feedback: update stale docstrings - pca.py: describe sklearn wrapper instead of deleted power iteration impl - bench_pca.py: use tool-agnostic `python` instead of hardcoded venv path Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
…ode (#2423) * refactor: remove legacy poller/system architecture Remove dead code that was replaced by scripts/job_poller.py + DynamoDB job queue: - polismath/__main__.py (legacy entry point) - polismath/system.py (legacy System/SystemManager) - polismath/poller.py (legacy Poller/PollerManager) - polismath/components/server.py (legacy Server/ServerManager) Updated __init__.py files to remove imports of deleted modules. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor: remove dead code and archive outdated docs - Remove polismath/utils/general.py (unused utility functions - 263 lines) - Remove unused squareform import from corr.py - Archive outdated docs to docs/archive/: - conversion_plan.md (conversion complete) - NEXT_STEPS.md (outdated) - project_structure.md (proposed, not actual) * docs: update RUNNING_THE_SYSTEM.md and archive more outdated docs - Update RUNNING_THE_SYSTEM.md: remove SystemManager references, add current CLI usage - Archive architecture_overview.md (describes Clojure, not Python) - Archive summary.md (references deleted poller/server/system components) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs: add dead code cleanup report with vulture analysis Detailed documentation of the dead code identification and removal process, including actual vulture output (98 findings) and recommendations for future cleanup. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: restore incorrectly deleted general.py and remove unused imports This commit corrects a critical false positive from the dead code cleanup: 1. RESTORED: polismath/utils/general.py (263 lines) - Was incorrectly marked as dead code - Actually has 4 active imports in production code: * postgres.py: postgres_vote_to_delphi() * run_math_pipeline.py: postgres_vote_to_delphi() * repness.py: AGREE, DISAGREE constants * clusters.py: (import existed but unused - see below) 2. REMOVED: Unused imports (5 total) - clusters.py: weighted_mean, weighted_means - postgres.py: JSON, QueuePool - job_poller.py: JSON, QueuePool 3. ADDED: docs/DEAD_CODE_CLEANUP_REPORT_CORRECTED.md - Documents the false positive - Provides root cause analysis - Recommends process improvements Tests verified: 211 passed, 7 skipped, 2 xfailed 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * docs: unify dead code cleanup reports Merge DEAD_CODE_CLEANUP_REPORT.md and DEAD_CODE_CLEANUP_REPORT_CORRECTED.md into a single comprehensive report documenting the entire cleanup series. Changes: - Combined both reports into unified DEAD_CODE_CLEANUP_REPORT.md - Documents false positive (general.py) and correction - Clarifies entry point is run_delphi (not delphi CLI) - Shows net impact: -730 lines from edge branch - All documentation files kept in docs/ (not archived) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * docs: add raw Vulture static analysis output Add complete output from Vulture dead code analyzer (147 findings): - 90%+ confidence: ~15 unused imports - 60%+ confidence: ~132 potentially unused functions/classes/methods This provides the raw data referenced in DEAD_CODE_CLEANUP_REPORT.md for future dead code cleanup efforts. Command used: vulture . --min-confidence 60 --exclude ".git,__pycache__,*.pyc,.venv,tests" 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: remove high-confidence unused imports (17 total) Remove all 90%+ confidence unused imports identified by Vulture: Files modified (10): - polismath/benchmarks/bench_repness.py (3 imports) - polismath/components/config.py (Set) - polismath/conversation/conversation.py (Set) - polismath/conversation/manager.py (Set) - polismath/database/postgres.py (Set) - polismath/utils/general.py (itertools, Set) - scripts/delphi_cli.py (Text, rprint) - umap_narrative/801_narrative_report_batch.py (csv, io) - umap_narrative/polismath_commentgraph/core/clustering.py (hdbscan, delayed, Parallel) - umap_narrative/polismath_commentgraph/utils/storage.py (QueuePool) All imports verified as unused by: 1. Vulture static analysis (90% confidence) 2. Manual grep verification 3. Test suite (45 tests passed) This addresses the high-priority cleanup items from vulture_analysis_output.txt. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* Consolidate Clojure comparison tests into test_legacy_clojure_regression.py - Delete test_golden_data.py: redundant with test_legacy_clojure_regression.py - Change @Skip to @xfail for test_group_clustering and test_comment_priorities so they run and document expected failures rather than being silently skipped 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Constrain xfail markers to AssertionError with strict=True Unconstrained xfail swallows any exception (KeyError, AttributeError, etc.), masking real regressions. Adding raises=AssertionError ensures only assertion failures are treated as expected, and strict=True alerts when tests start passing. Addresses GitHub Copilot review feedback on PR #2417. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * CI: run Delphi Python tests on PRs targeting jc/** branches Stacked PRs target jc/* branches, not edge/stable, so the Python CI was not running on them. Add jc/** to pull_request.branches so stack PRs get test coverage. Push triggers remain edge/stable only. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Stacked PRs target jc/* branches, not edge/stable, so server integration tests and Cypress E2E tests were not running on them. Add jc/** to pull_request.branches so stack PRs get full CI coverage (python-ci.yml already has this from a prior change). Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* FIx path of mock file for test run_math_pipeline_e2e * Clarify test name * Consolidate and reorganize Clojure comparison tests Consolidate three overlapping test files into a coherent suite: - Removed: tests/test_legacy_clojure_output.py (structure analysis) - Removed: tests/legacy_compare_with_clojure.py (direct comparison script) - Enhanced: tests/test_legacy_clojure_regression.py (main pytest suite) Created shared utilities and CLI tool: - polismath/regression/clojure_comparer.py: Comparison utilities with ClojureComparer class, cluster distribution/membership comparison, PCA projection comparison with transformations - scripts/clojure_comparer.py: Interactive CLI tool for detailed comparison (similar UX to regression_comparer.py) - docs/CLOJURE_COMPARISON.md: Comprehensive documentation Organized following codebase patterns: - Placed utilities in polismath/regression/ alongside other regression utilities (comparer.py, recorder.py, datasets.py) - Updated imports in consuming files - Added exports to polismath/regression/__init__.py for clean imports Added pytest marker: - @pytest.mark.clojure_comparison for selective test execution - Can exclude with: pytest -m "not clojure_comparison" Key discovery: - Clojure uses two-level clustering (participants→base-clusters→groups) - Python uses single-level clustering (participants→groups directly) - This architectural difference explains 0% Jaccard similarity Tests run and detect differences (as expected until initialization is aligned). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Remove legacy compare_implementations.py (replaced by clojure_comparer.py) This script originated as tests/test_real_data_comparison.py (March 2025), was renamed to scripts/compare_implementations.py in #2282 (Nov 2025), and has been broken since that rename (SyntaxError: global TOLERANCE declared after prior use, plus undefined variable on line 486). Its replacement, scripts/clojure_comparer.py, was introduced earlier in this PR. It uses the shared ClojureComparer library, auto-discovers datasets, supports --include-local, and does rigorous cluster comparison (Jaccard membership + L1 distribution). The one feature compare_implementations.py had that clojure_comparer.py lacks — multi- tolerance comment priority comparison — is moot until D12 is implemented, at which point it should be added to the shared ClojureComparer library. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Incorporate PR #2418 review feedback (Copilot + Christian) - Remove unused imports in test_legacy_clojure_regression.py - Add click as declared dependency in pyproject.toml - Fix redundant sys.exit with Click standalone mode - Fix circular import: import from .datasets directly, not package init - Remove SciPy hard dep: don't re-export clojure_comparer from __init__.py - Replace Wasserstein distance with L1 distance (Copilot: scipy was misused on probability vectors; L1 is well-defined for normalized distributions) - match_status now requires num_clusters_match and complete mapping (Copilot: could false-positive PASS with different cluster counts) - Add 23 unit tests for compare_cluster_distributions, compare_cluster_membership, compare_projections, and compute_distribution_similarity with synthetic inputs - Fix doc file paths in CLOJURE_COMPARISON.md ({report_id}_math_blob.json) - Fix float|None -> Optional style inconsistency (Christian's review) - CLI: try/except for test code import with clear error message - Also: remove stale PYTEST_ADDOPTS="-n auto" from ~/.exports Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
…tecture (#2431) * Implement two-level hierarchical clustering matching Clojure architecture Level 1: Participants → ~100 base clusters Level 2: Base clusters → 2-5 group clusters (silhouette-based k selection) Key changes: - Add sklearn-based kmeans with first-k initialization (matching Clojure) - Add silhouette coefficient calculation for optimal k selection - Implement participant filtering (in-conv threshold logic) - Add fold/unfold utilities for hierarchical cluster storage - Update tests to unfold and compare hierarchical structures - Document Clojure two-level architecture and non-determinism issue This implements the core two-level clustering but does not yet include incremental clustering (warm-start from previous state). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Fix two-level clustering edge cases and update tests Bug fixes from review: - clusters.py: adjust k down when fewer distinct points than requested - conversation.py: group_clusters members are base-cluster IDs (not participants) when only 1 base cluster; convert pandas Index to list for JSON serialization - pca.py: fallback comps shape uses min(n_comps, n_cols) not min(n_comps, 2) Test fixes: - test_clusters: init_clusters returns empty members with two-level clustering - test_conversation: use 10 comments to meet vote threshold - test_edge_cases: group_repness is {} (not {0: []}) when no clusters Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Mark known Clojure discrepancies as xfail and re-record golden snapshots - test_basic_outputs: xfail for D9/D5/D7 (wrong z-score thresholds and repness formulas produce empty comment_repness) - test_group_clustering: xfail for D2/D3 (wrong participant threshold and missing k-smoother produce different cluster counts) - test_comment_priorities: update xfail reason to reference D12 - Re-record golden snapshots after two-level clustering changes Test baseline: 11 passed, 6 xfailed, 0 failures. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Unfold group-cluster base-cluster IDs to participant IDs for downstream consumers Group clusters store base-cluster IDs in 'members' (matching Clojure's two-level clustering architecture), but downstream functions (conv_repness, participant_stats, group_votes) need participant IDs to join against the vote matrix. Add _unfolded_group_clusters() helper (equivalent to Clojure's clusters/group-members) and use it in all 5 call sites: - _compute_repness - _compute_participant_info - _compute_group_votes - to_dict group-votes - to_dynamo_dict group_votes Also re-record golden snapshots and remove xfail from test_repness_structure (now passes with correct unfolding). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Tighten Clojure comparison tests and fix outdated docstrings - test_group_clustering: unfold Python clusters via _unfolded_group_clusters() (both sides now use two-level clustering), tighten thresholds to 0.99 Jaccard / 0.01 distribution tolerance, require overall_match - test_comment_priorities: require exact match (1e-6 tolerance) for all comment IDs instead of 70% at 20% tolerance - clojure_comparer: fix docstring to reflect Python also uses two-level clustering Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix serialization outputting base-cluster IDs instead of participant IDs to_dict(), get_full_data(), and to_dynamo_dict() were writing self.group_clusters directly, whose 'members' contain base-cluster IDs (integers 0..N). Downstream consumers (group_data.py, Clojure compat, client apps) expect participant IDs. The internal helper _unfolded_group_clusters() already existed and was used for repness/participant_info/group_votes computation, but the five serialization sites were missed. Fix all five sites to unfold before serializing: - to_dict(): group-clusters, group_clusters, base-clusters - get_full_data(): group_clusters - to_dynamo_dict(): base_clusters / group_clusters Add test_serialization_unfolding.py (TDD: 6 tests fail on the bug, 8/8 pass after fix) using real recompute() pipeline output. Re-record golden snapshots to reflect the corrected output. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix missing Set import in conversation.py Added Set to the typing imports — it was used in the _get_in_conv_participants return type annotation but never imported, causing NameError on CI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
…alization (#2485) * Add script to generate cold-start Clojure math blobs for fair comparison Creates generate_cold_start_clojure.py to generate fresh cold-start Clojure reference data for fair Python vs Clojure comparison. The script: - Stops if math worker is running (prevents conflicts) - Backs up existing math_main row - Deletes row to force cold-start (load-or-init creates fresh new-conv) - Runs Clojure computation via Docker - Extracts cold-start math blob - Restores original row automatically Key features: - Support for --all flag to process all datasets - Support for --include-local flag for local datasets - Automatic zid lookup from report_id via reports table - Loads configuration from polis-kmeans/.env (DATABASE_URL) Test infrastructure updates: - datasets.py now prefers cold-start blobs when available - Added has_cold_start_blob field to DatasetInfo - get_dataset_files() uses cold-start blob by default Documentation updates: - Comprehensive usage guide in SESSION_HANDOFF_KMEANS.md - Commands to find and verify cold-start blobs - Configuration requirements and setup instructions Reference data: - Generated cold-start blobs for biodiversity and vw datasets - Tests will now use these for fair cold vs cold comparison Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Implement conversation replay approach for cold-start generation and add visualization with PCA sign flip detection Cold-start generation (generate_cold_start_clojure.py): - Rewrite using "conversation replay" approach that works with Clojure poller design - Creates temporary conversation with fresh zid, copies votes with fresh timestamps - Runs poller with MATH_ZID_ALLOWLIST to only process the replayed conversation - Automatically cleans up all temporary data (math tables, votes, conversation) - Add bash wrapper script (generate_cold_start.sh) that stops math containers first Visualization (visualize_cluster_comparison.py): - Add PCA sign flip detection by comparing component correlations - Apply sign corrections to base cluster centers before visualization - Fix convex hull rendering to show outlines for both datasets in overlay view - Include sign_flips in metrics JSON output Documentation: - Update SESSION_HANDOFF_KMEANS.md with new approach and remove "BROKEN" warnings - Document the conversation replay workflow and cleanup behavior Regenerate cold-start blobs for biodiversity and vw datasets. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Improve cold-start generator and cluster visualizer CLI Cold-start generator (generate_cold_start_clojure.py): - Add --pause-math to pause/resume workers instead of stopping - Add --verbose/-v for real-time Clojure poller output - Support multiple datasets as arguments - Use fast INSERT...SELECT for vote copying (was executemany) - Handle duplicate votes with DISTINCT ON - Remove shell wrapper (functionality now in Python script) Cluster visualizer (visualize_cluster_comparison.py): - Add --all option for processing all datasets - Synchronize X/Y axis limits in side-by-side plots - Print full absolute paths for generated PNGs - Support multiple datasets as arguments Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Compute cold-start math blobs for kmeans * Interrupt upon Clojure error * test_datasets: add cold_start blob fixture and has_cold_start arg Deferred from commit 18ad361 — the test_datasets.py changes depend on the has_cold_start_blob field introduced in datasets.py by the cold-start tooling. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Delphi Coverage Report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.