Add distribution aggregation and dual-write

[sanghoonio]

Summary

Adds bedset-level distribution aggregation, with the aggregation implemented primarily in SQL (leveraging gtars PR #248's reference-aligned bin widths).

Scope evolution

Initial scope was "add aggregation" (commit 300b45f). During review, two further changes landed:

• 0152f10: ruff import fix
• 7fe5c64: prune aggregation to meaningful fields + switch to SQL-side computation

The current PR reflects the full scope described below.

What's included

Schema changes

• Drop dead tssdist column (no model references, no writers, no readers)
• Add median_neighbor_distance scalar column to BedStats + model field
• Add distributions: dict | None field to BedStatsModel (per-file JSONB, from gtars backend)

New modules/aggregation.py

Computes BedSetDistributions from member files' per-file distributions:

• SQL aggregation (heavy lifting done in Postgres, not Python):
  • scalar_summaries: AVG/STDDEV on BedStats scalar columns + 25-bin histogram of per-file means
  • region_distribution: jsonb_each + jsonb_array_elements_text WITH ORDINALITY + GROUP BY (chrom, bin_idx) — valid now that gtars #248 gives reference-aligned bin widths per genome
  • tss_histogram: element-wise aggregation across fixed-axis 100-bin arrays
• Python aggregation (small nested JSONB):
  • partitions: mean ± sd of per-file partition percentages
  • composition: distinct value counts per metadata field (genome, assay, cell_type, tissue, target)

Aggregation fields pruned (not meaningful at collection level)

These stay in per-file distributions JSONB for single-file views, just aren't aggregated:

• widths_histogram: per-file variable-range bins aren't summable; use scalar_summaries.mean_region_width histogram instead
• neighbor_distances KDE: use new median_neighbor_distance scalar instead
• gc_content KDE: use scalar_summaries.gc_content mean instead
• chromosome_summaries: redundant with region_distribution
• expected_partitions: per-file null hypothesis, not a collection property

Dual-write in bedsets.create()

• Old SQL aggregation → bedset_means / bedset_standard_deviation columns (backward compat)
• New distribution aggregation → bedset_stats JSONB column (when members have distributions)

New retrieval methods

• BedAgentBedSet.get_distributions(): read bedset_stats JSONB with fallback to legacy scalar columns
• BedAgentBedFile.get_batch(): multi-ID bed metadata retrieval
• BedAgentBedFile.aggregate_collection(): ad-hoc collection stats on arbitrary bed ID list
• distributions: bool param on get_stats()

Performance impact (1000-file aggregation)

	Before	After
Wire transfer Postgres→worker	~40 MB	~150 KB
Latency	1-3 s	<500 ms
Worker memory	~40 MB of parsed JSONB	~few KB

The SQL path avoids pulling 1000 raw distribution blobs into the Python worker — Postgres does the element-wise summation and returns only the aggregate result.

Dependencies

• Depends on PR Add modular analysis backend config and distribution schema #112 (modular-backend-schema — adds distributions JSONB column and related models)
• Uses gtars PR #248's reference-aligned region_distribution bin widths (required for SQL aggregation correctness)

Test plan

• All 52 existing tests pass locally (5 skipped, 0 failed)
• CI lint + pytest
• Manual parity check: compare SQL aggregation output vs old Python output on a small test bedset (after gtars-processed data is available)

🤖 Generated with Claude Code