Expose scoped CES campaigns via MCP and docs; complete roadmap row

estimate_cost accepts the JSON scope shape; prepare_ces_campaign plans a
scoped CES campaign from a substrate handle (precomputed SIA via a result
handle); collect_campaign returns the assembled structure's summary and
scope report. The campaigns how-to and MCP reference gain scope and CES
sections.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PEAxNzhDCaTrntX3o1JqMV

Author: wmayner

ROADMAP.md

@@ -41,7 +41,7 @@ P0 · P1 · P2 · P3 · P4 · P5 · P6 · **P6a** (free-threaded 3.14t CI lane +
 | B15 `result.diff()` | ✅ landed | 2 | `a.diff(b)` on every top-level result (4.0/3.0 SIA, CES, RIA/MICE/Distinction, AcSIA/Account) returns a typed, `Displayable`, `to_pandas`-able `ResultDiff`: the signed Δφ (Δα for AC), whether the MIP genuinely changed vs a co-optimal tie-reshuffle (via `lex_key` over the tie set, which already encodes `EQUALITY_TOLERANCE`), distinctions/relations/account-links gained/lost/changed (keyed by mechanism/relata/direction+mechanism+purview), and config-diff attribution composing the landed `ConfigSnapshot.diff`. Cross-substrate diffs are allowed with a `substrate_note`; type mismatch raises `TypeError`; mechanism-level results and `Account` carry no `ConfigSnapshot` so their `config_diff` is `{}` by design. Pure comparison — no recompute, no φ/α change. `pyphi/models/diff.py`; `test/models/test_result_diff.py` (coverage invariant over every result type + MIP-reshuffle invariant). Pairs with B8 |
 | P11.8 Tier 2 | ✅ landed | 3 | Benchmark suite rebuilt on the golden harness (24 fixtures × {repertoires, mechanism_mips, phi_structure, sia} + parallel/cache/EMD/AC) driving an ASV nightly wall-time trend (results cached across runs; regression check is `asv continuous` HEAD~1 vs HEAD — adjacent commits only; issue alert, never blocks); a deterministic cProfile call-count gate (`test/integration/test_perf_counters.py`, exact pins in `test/data/perf/call_counts.json`, regen `scripts/gen_perf_counts.py`) **blocks PRs**; full-zoo counts tracked nightly as ASV `track_*`. Shared harness `test/golden/perf.py`; `.github/workflows/benchmark.yml`. Supersedes P15 "Layer D". |
 | P9.5 | ✅ landed | 4 | Math-fingerprint cache keys: label-free `blake2b-256` content fingerprints (`System`/`Substrate`/`MacroSystem`) key a refcounted `ContentCache` (`pyphi/cache/content.py`); the repertoire kernel cache keys on the System fingerprint and `potential_purviews` on the cm fingerprint, so equivalent/relabeled systems and same-topology weight sweeps reuse results. Refcounted GC eviction preserves prompt release. Surfaced + fixed a latent `MacroSystem` collision (overridden cause marginal was not in the inherited key). Byte-identical goldens; unblocks N4 |
-| P11 cluster backends | 🟡 partial | 4 | Dask backend landed 2026-07-20: `DaskScheduler` distributes map-reduce over a user-connected `distributed.Client` (snapshot propagation, cost-balanced chunks, deterministic short-circuit prefixes; nested dispatch runs in-task); `cluster` extra; `docs/howto/chtc.md` covers CHTC (plain condor sweeps + fat-node fully supported; Dask worker pools documented as a pilot pending CHTC port-access confirmation — most ports on CHTC nodes are closed, their first-party Dask/htmap glue is dead/archived, so the guide lists the facilitation questions). **Campaign surface cycle 1 landed 2026-07-20**: `pyphi.campaign` (tasks-as-data, no coordinator) — `prepare()` enumerates sweep cells (with the new substrates axis on `sweep()`), estimates each cell via `estimate_analysis` under its formalism preset, packs cost-balanced tasks (`jobs`/`units_per_job`, admission-control warnings above `infeasible_threshold`), and writes a self-contained directory (serialized tasks + substrates, manifest with all estimates, generated condor submit file); `python -m pyphi.campaign run` executes one task identically in the container, a shell, or tests (atomic outputs, per-cell ok/skipped/error entries, attempt-preserving re-runs); `status()`/`collect()` classify tasks purely from output files (done = exists and loads), drive resubmission via `remaining.txt`, and reassemble the exact local-sweep `SweepResult` (verified: campaign ≡ local sweep end-to-end through the real CLI). MCP tools `prepare_campaign`/`campaign_status`/`collect_campaign` + `campaigns` reference topic; `docs/howto/campaigns.md`. Remaining: scoped CES sharding + reconstruction — declarative `AxisScope`/`CESScope`, mechanism/purview-range/partition-stride planning ladder, tie-preserving exact merges, three SIA modes, certified bounds for scope exclusions — designed and verified in `docs/superpowers/specs/2026-07-20-ces-sharding-design.md` (companion: `2026-07-20-htcondor-campaign-design.md`). |
+| P11 cluster backends | ✅ landed | 4 | Both halves shipped. **Dask backend** (2026-07-20): `DaskScheduler` distributes map-reduce over a user-connected `distributed.Client` (snapshot propagation, cost-balanced chunks, deterministic short-circuit prefixes; nested dispatch runs in-task); `cluster` extra; `docs/howto/chtc.md` covers CHTC (the Dask worker-pool pattern remains documented as a pilot pending CHTC port-access confirmation — an external facilitation question, not code work). **Campaign surface** (cycles 1+2, 2026-07-20/21): `pyphi.campaign` tasks-as-data batch campaigns — sweep cells (substrates axis on `sweep()`, cost-balanced packing, admission control, condor emitter, campaign ≡ local sweep proven end-to-end) and scoped CES sharding for one system: declarative `AxisScope`/`CESScope` (explicit lists, order bounds, unit containment; serialized into tasks and provenance), scope-aware `estimate_analysis` + per-mechanism `mechanism_workloads`, a three-rung planner (mechanism → purview-range → partition-stride, `bottleneck_first` ordering for sparse reducibility), exact tie-preserving merges (sharded ≡ unsharded verified under both IIT 4.0 presets, winner identity included — per-shard winners restored to global enumeration order), three SIA modes (merged strides / precomputed / intrinsic-state with no Φₛ), version+scheme merge guards, and a certified scope report (exact Σφ_r lower bound + measured upper bounds from `iit4/bounds.py`). MCP: `prepare_campaign`, `prepare_ces_campaign`, `campaign_status`, `collect_campaign`, scope-aware `estimate_cost`, `campaigns` topic; `docs/howto/campaigns.md`. Specs: `docs/superpowers/specs/2026-07-20-htcondor-campaign-design.md` + `2026-07-20-ces-sharding-design.md`. |
 | Parallel engine unification | ✅ landed | 4 | Collapsed the two coexisting map-reduce implementations into one flat path: replaced the one-shot `MapReduce` class with a `pyphi.parallel.map_reduce()` function over the Scheduler Protocol, migrated all 10 call-sites, deleted the vestigial execution tree (`tree.py`/`test_tree.py` + dead `branch_factor`/`max_*` kwargs), and deduped the ~5 copied helpers. Backend selection (`config.parallel_backend`) is now honored from the public entry. Net ~900 lines removed; result-preserving (N2 + goldens + perf gate green). Prerequisite for B18. Spec/plan: `docs/superpowers/{specs,plans}/2026-06-23-unify-parallel-engine*` |
 | B18 adaptive chunking | ✅ landed | 4 | Activated `ChunkingPolicy.size_func`: a pure `cost_balanced_partition` LPT-packs items into cost-balanced chunks, and the chunk count is floored at `num_workers` (a general core-utilization win, de-risked empirically — 2.8–3× at every per-item cost, no overhead penalty). Cheap `size_func` proxies wired at the heterogeneous sites — relations (`overlap×degree`), purview eval (`2^|purview|`), concept eval (`2^|mechanism|`). Result-preserving (N2 incl. a forced-chunk concept test, goldens, perf gate). `pyphi/parallel/chunking.py`; spec/plan `docs/superpowers/{specs,plans}/2026-06-23-cost-balanced-chunking*`. **Dispatch-gate follow-up (2026-07-07)**: measured the below-chunksize regime (`benchmarks/b18_dispatch_gate.py` + `benchmarks/b18_dispatch_gate_results/`) and flipped the gate — `sequential_threshold` is now the sole dispatch gate; an explicit `chunksize` governs granularity only (a cost-sampled chunksize still gates, since it estimates ~1 s of work). Wins: purview eval 3–4× at 64–230 purviews, system partitions 2.5–4× at 64–2048, complexes 1.2–1.6× at 16–31 candidates; no regression on reducible short-circuiting systems. Per-level thresholds retuned to measured per-item costs (partition 1024→64, mech-partition & relations 1024→8192). The study also surfaced and fixed a per-item `hash(repr(snapshot))` (~1.3 ms) in the process-scheduler worker wrapper that dominated cheap-item workloads (~250× sequential relations speedup on its own) |
 | P15 | 🟡 partial | 5 | Import-surface cleanup landed (eager submodule walk → explicit registry imports + PEP-562 lazy `__getattr__`, guarded by a registry-contents test). jsonify→msgspec landed (`pyphi.serialize` replaces `pyphi.jsonify`; the per-class `to_json`/`from_json` hooks removed) **plus its loading surface** (path-based `serialize.save`/`load`, top-level `pyphi.save`/`pyphi.load`, `.save()`/`.load()` on the result types via a delegation-only `Serializable` mixin, deferred type registration, `substrate.from_json` removed, transparent gzip on `.gz` paths). pandas-extend done (display → B21). Project-stage markers stripped from `pyphi/` + a docstring narrative-removal slice landed. PR triage done (audit against `main`: nothing left to absorb; closes pending a maintainer). Test suite reorganized to mirror the `pyphi/` package layout (`test/<subpackage>/` + `test/integration/`). Remaining: the comprehensive docstring/docs sweep (now its own [Documentation overhaul](#documentation-overhaul-20--iit-40) project), changelog condense |

changelog.d/ces-sharding.feature.md

@@ -0,0 +1 @@
+Added scoped CES sharding to `pyphi.campaign`: declare the combinatorially feasible surface with `CESScope`/`AxisScope` (explicit lists, order bounds, unit containment), let `prepare_ces()` plan mechanism/purview-range/partition-stride shards to a per-job budget, and `collect()` reassembles the exact `CauseEffectStructure` — tie sets preserved, congruence and relations through the standard path — with a certified scope report (`Σφ_r` lower bound plus measured upper bounds). `estimate_analysis` accepts `scope=`; the `estimate_cost` and new `prepare_ces_campaign` MCP tools expose it.

docs/howto/campaigns.md

@@ -148,3 +148,80 @@ If any single cell's estimate exceeds `infeasible_threshold` (default 10⁹),
 within CHTC's 72-hour slot and needs narrower axes or a dedicated fat-node
 run ({doc}`chtc`). Pass `strict=True` to make the warning an error. Every
 per-cell estimate and packing decision is recorded in `manifest.json`.
+
+## Declare the feasible surface (scope)
+
+For large systems the full distinction computation is combinatorially out of
+reach; a **scope** declares which part of it you compute. A scope changes
+*what* is computed — with the exclusions recorded and certified — never a
+silent approximation: within the scope, every value is exact.
+
+```python
+from pyphi.campaign.scope import AxisScope, CESScope
+
+scope = CESScope(
+    # Mechanisms up to order 3 that involve unit A:
+    mechanisms=AxisScope(containing=("A",), max_order=3),
+    # Purviews of any order, but only within these units:
+    cause_purviews=AxisScope(within=("A", "B", "C")),
+)
+```
+
+Each axis (`mechanisms`, `cause_purviews`, `effect_purviews`) accepts an
+explicit list of unit sets (`explicit=...`, exclusive of the other fields)
+or any combination of `min_order`, `max_order`, `containing` (must include
+all these units), and `within` (must be a subset), combined by
+intersection. Units may be labels or indices. Purview constraints only
+narrow the connectivity-pruned candidates; partition sweeps cannot be
+scoped — a partial sweep would silently change φ.
+
+`pyphi.estimate_analysis(substrate, compute="ces", scope=scope)` prices the
+scoped workload before you commit to it.
+
+## Distribute one system's cause-effect structure
+
+`prepare_ces` turns one system's scoped analysis into a campaign:
+
+```python
+status = campaign.prepare_ces(
+    substrate,
+    state=(1, 0, 0),
+    scope=scope,
+    directory="ces-campaign",
+    units_per_job=1e6,
+    seed=42,
+)
+```
+
+The planner descends only as deep as the budget requires: whole mechanisms
+are cost-balanced into jobs; a mechanism over budget splits its purview
+list into ranges; a single (mechanism, purview) pair over budget splits its
+partition sweep into interleaved strides. System-partition strides for the
+SIA are planned the same way — unless you pass a precomputed `sia=`, or a
+`resolution_state=` (in which case the collected structure carries no Φₛ
+and congruence resolves against the given state, or the system's
+intrinsic-information state by default).
+
+On sparse substrates, pass `ordering="bottleneck_first"` so each stride
+evaluates partitions that sever the fewest present connections first —
+reducibility then short-circuits within the first evaluations. Ordering
+never affects results.
+
+Submission, monitoring, and resubmission work exactly as for sweep
+campaigns. `collect()` merges the shards exactly — tie sets preserved,
+identical to what a single machine would have produced over the same
+scope — and assembles the `CauseEffectStructure` through the standard
+analysis path:
+
+```python
+ces = campaign.collect("ces-campaign")
+report = campaign.scope_report("ces-campaign")
+print(report)
+```
+
+The **scope report** records what was computed and what the scope
+excluded, with certificates: the computed Σφ_r is an exact lower bound for
+the full structure (partial structures are exact substructures), and the
+measured upper bounds on Σφ_r and Φ come from the certified bound
+machinery. Missing shard groups (from failed or pending tasks collected
+with `partial=True`) are listed separately from scope exclusions.

pyphi/campaign/merge.py

@@ -116,6 +116,7 @@ def merge_purview_rias(
     if others:
         # numerics: exact — reported margin, not a selection.
         best_rival = max(float(m.phi) for m in others)
+        # numerics: exact — reported margin, not a selection.
         winner.purview_margin = max(0.0, float(winner.phi) - best_rival)
     return winner
 

pyphi/mcp/content/campaigns.md

@@ -69,3 +69,20 @@ re-run preserves the previous attempt under an `attempt-N` name.
   of one computation over a live worker pool you already have connected —
   requires open ports between scheduler and workers, which CHTC generally
   does not provide.
+
+## Scoped CES campaigns
+
+For one system too large to analyze whole, `prepare_ces_campaign`
+distributes the cause-effect structure computation itself. Declare the
+combinatorially feasible surface as a **scope** — per-axis constraint
+objects such as `{"mechanisms": {"max_order": 3, "containing": ["A"]},
+"cause_purviews": {"within": ["A", "B", "C"]}}` — and the planner shards
+the scoped work to a per-job budget, descending mechanism → purview-range →
+partition-stride only where needed. `estimate_cost` accepts the same
+`scope` to price the surface first. The SIA is sharded too, or supplied
+precomputed via `sia_ref`. Collection (`collect_campaign`) merges shards
+exactly (tie sets preserved), assembles the `CauseEffectStructure` through
+the standard path, and returns a **scope report**: the computed Σφ_r is an
+exact lower bound for the full structure, with certified measured upper
+bounds on Σφ_r and Φ for what the scope excluded. Within the scope every
+value is exact — a scope narrows the computation, never approximates it.