Expose pyphi.optimize; changelog, how-to, and ROADMAP updates

Lift optimize/OptimizationResult/weight_axes to the top-level namespace, add a
how-to section, a changelog fragment, and mark the Wave-7 landscape optimizer
driver landed in the ROADMAP dashboard.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y8rvjyrADD9obYiNKhxs3b

Author: wmayner

ROADMAP.md

@@ -63,7 +63,8 @@ P0 · P1 · P2 · P3 · P4 · P5 · P6 · **P6a** (free-threaded 3.14t CI lane +
 | Grain-search documentation | ✅ landed | — | The grain search across the Diátaxis framework: `docs/howto/grain-search.md` (pre-flight → run → micro history for τ>1 → read winners/margins/records/ties → parallelize → bound); `docs/theory/macro-units.md` (unit definition, intrinsic-unit criteria, recursive exclusion across grains, type map — first macro coverage in the theory section; citations verified against the 2024 PDF); blackboxing + temporal-grain tutorial sections in `docs/tutorials/macro.md` — the temporal specimen is a prospected three-unit substrate whose maximal complex is a τ=2 unit (φₛ 0.5083, margin 0.30, excluding the micro universe at 0.2075; provenance in `experiments/grain_tau_experiments/`, seeded, 23/120 hit rate; the exploration's 4-cycle demo does NOT survive the integration criterion and its 32,727 was a digit transposition of 2¹⁵−1); grain cost section in `docs/theory/computational-complexity.md` with a live `estimate()` cell. Also migrated `docs/tutorials/recursive-exclusion.md` off removed `pyphi.utils.eq` (docs builds on main were broken). Grain-exposure item 4. Spec: `2026-07-10-grain-search-docs-design.md` |
 | Macro-construction intermediate cache | ✅ landed | — | The macro TPM construction's mapping-independent Steps 1–2 intermediates (the discounted transition matrix and the sequence-class distributions, Marshall et al. 2024 Eqs. 26–31) are cached per substrate content fingerprint, keyed on (footprint, composed micro grain, apportionment structure) — grain-search candidates differing only in their mapping reuse the expensive construction prefix, so Step 1 runs once per distinct key instead of once per construction (the design measurement found 27× key redundancy on the default Example 1 sweep). Result-preserving: byte-identity pinned across the golden constructions with cache off/cold/warm, sweep records and complexes exactly equal, goldens and perf-counter pins unchanged. Module-level `ContentCache` beneath the search's per-run system cache; entries evicted with the substrate via GC; substrate isolation, nested-unit sharing, and thread-safety tested. `cache_macro_construction` infrastructure option (default on; off = no reads or writes). Grain-exposure item 5. Plan: `2026-07-08-grain-construction-cache.md` |
 | Complex unification | ✅ landed | 7 | One exclusion semantics, one winner type: the macro driver's literal Eq. 19 predicate computed only the first condensation layer (already-excluded candidates could veto others — complexes on chain topologies were missing), ruled a bug against the recursive reading (Marshall 2023 Alg. A1 / IIT 4.0 "assessed recursively"). Shared recursive cascade in `pyphi/condensation.py` (`Candidate` records: micro-footprint overlap currency + SIA/system providers) consumed by both `substrate.complexes` and `macro.complexes`; macro ties escalate through the S1 Composition cascade with Φ computed once per content fingerprint (single-fingerprint cliques skip CES entirely, certified by bit-identity) and compared as `PyPhiFloat` — fixing a latent raw-float argmax that resolved isomorphic-twin cliques on summation noise; `ComplexesResult.complexes` → `tuple[Complex]` (`units` + micro-footprint `node_indices` + `excluded`), `.ties` → Φ-tied cliques, `.maximal_complex` null-object; `Complex`/`ExcludedCandidate` gained `units` (serialized, recursive `MacroUnitSchema`); macro drivers reject IIT 3.0 eagerly. Chain regression + cross-door Hypothesis equivalence at `max_depth=0` + tie shadow-equality + parallel≡sequential. Fast-lane goldens byte-stable. Spec: `2026-07-09-complex-unification-design.md` |
-| `pyphi.landscape` | ✅ landed | — | Continuous-parameter analysis over substrate space (spec §7 item 1): `landscape_section` (1-D SIA sections as tidy DataFrames with selection identities, margins, and regime boundaries), `perturb` (central-difference derivatives, one-sided derivatives, margin derivatives, and `switch_distances` — linearized distance in parameter units to each selection switch; validated to ~1% against a bisected switch on the Fig-1A substrate), `weight_axis` (single-weight axis for `substrate_generator` substrates). Sequential-only. The n=5 roughness replication that gated the black-box optimizer driver has run (`experiments/substrate_landscape_experiments/FINDINGS.md`): signed normalized φₛ stays empirically smooth across MIP switches, so the driver is unblocked and ships population-first — now an open Wave-7 build (with the caveat that smoothness is of the objective value, not of selection identity). Docs: `docs/howto/landscape.md`. |
+| `pyphi.landscape` | ✅ landed | — | Continuous-parameter analysis over substrate space (spec §7 item 1): `landscape_section` (1-D SIA sections as tidy DataFrames with selection identities, margins, and regime boundaries), `perturb` (central-difference derivatives, one-sided derivatives, margin derivatives, and `switch_distances` — linearized distance in parameter units to each selection switch; validated to ~1% against a bisected switch on the Fig-1A substrate), `weight_axis` (single-weight axis for `substrate_generator` substrates). Sequential-only. The n=5 roughness replication that gated the black-box optimizer driver has run (`experiments/substrate_landscape_experiments/FINDINGS.md`): signed normalized φₛ stays empirically smooth across MIP switches, so the driver is unblocked and ships population-first — now landed as `pyphi.optimize` (row below). Docs: `docs/howto/landscape.md`. |
+| `pyphi.optimize` | ✅ landed | — | Wave-7 item 3 (spec §7 item 3): a seeded, population-first black-box optimizer over substrate weights. `optimize()` runs `scipy.optimize.differential_evolution` (vectorized, `polish=False`, seeded via `rng`) with the objective defaulting to the gate-validated signed normalized φₛ; `weight_axes` builds the vector search space (the N-D analogue of `weight_axis`); each generation's population fans out through `map_reduce`; the returned `OptimizationResult` carries a per-evaluation trajectory (params, objective, reachability, selection identity, margins) and `save()`. The four φₛ variants are selectable by name, plus an opt-in callable escape hatch for objectives the gate did not validate (CES-level Σφ_d, Φ). Reads only the scalar objective, never a selection identity (the gate caveat). Gate passed: `experiments/substrate_landscape_experiments/FINDINGS.md`. Docs: `docs/howto/landscape.md` §Optimizing over weights. Spec/plan: `2026-07-11-landscape-optimizer-driver-design.md`, `2026-07-11-landscape-optimizer-driver.md`. |
 | CES algebra operations | ✅ landed | 7 | Operations over cause-effect structures as first-class objects: `Distinctions.filter`; `CauseEffectStructure.induce`/`meet` (relation-closed `InducedSubstructure` views with a same-frame check; `project_ces`/`plot_ces` accept any relation-closed object); share-weighted fold contributions — `PhiFold.big_phi_contribution` weights each incident relation by the seeds' share `\|r ∩ F\|/\|r\|`, so the folds of **any** partition of the distinctions (multi-seed included) tile Φ exactly, with a closed-form analytical path; `pyphi.automorphism.structure_signature` + `are_structures_isomorphic` (structure equality up to unit relabeling); `CauseEffectStructure.relabel` through node-index bijections. Spec: `docs/superpowers/specs/2026-07-07-ces-algebra-exploration.md` |
 | B22 raw-float φ-comparison audit | ✅ landed | 1 | Eight raw-float φ/α comparison sites found and fixed. `PyPhiFloat` deleted — tolerant comparison (up to `config.numerics.precision`) consolidated at decision sites: the scalar predicates in the new `pyphi.numerics` module and `resolve_ties`' tolerant float-key clustering, so candidates tied up to precision co-select regardless of iteration order. `DistanceResult` is now a plain-float metadata carrier. Actual causation gains a system-level tie cascade (`resolve_ac_sia_tie`/`resolve_ac_nexus_tie`) plus tie bookkeeping (`AcSystemIrreducibilityAnalysis.ties`). An AST lint (`test/test_precision_lint.py`) gates recurrence of raw-float φ/α comparison. Spec: `docs/superpowers/specs/2026-07-10-precision-comparison-architecture-design.md` |
 | Relations query surface | ✅ landed | 7 | Tier 1–3 relation queries as a deterministic view over the linear-size relation summary, so the relation set need not be materialized (spec: `docs/superpowers/specs/2026-07-07-relations-without-materialization-design.md`). Tier 1 closed-form on `AnalyticalRelations` — moments (`sum_phi_moment`), degree spectrum (`degree_spectrum`/`num_relations_of_degree`), `phi_histogram`, `binding_matrix`, Möbius `num_faces`, `max_phi`; Tier 2 lazy exact-descending `strongest(k)` (best-first over the antitone φ_r bound); Tier 3 seeded Karp–Luby `sample()` → Horvitz–Thompson `RelationSample.estimate`; bounded `materialize()` escape hatch; fold-restricted overrides on `AnalyticalFoldRelations`; `CauseEffectStructure.distinction_importance`. Concrete/analytical parity tested at Fig-6D scale. Discharges N6 + N24. |
@@ -190,13 +191,15 @@ remainder, each with its gate verdict:
   `prune="certified"` is available only under
   `system_phi_measure="INTRINSIC_INFORMATION"`; any GID-mode gating is
   heuristic and must be labeled approximate.
-- **Landscape optimizer driver** *(open build)*. Gate **passed**: the n=5
-  roughness replication ran (`experiments/substrate_landscape_experiments/FINDINGS.md`) —
-  signed normalized φₛ stays empirically smooth across MIP switches, so the
-  optimizer ships population-first. Caveat: the smoothness is of the objective
-  value only; selection identity flips between near-tied set partitions at
-  single-grid-point frequency, so the driver must not depend on a stable MIP
-  identity.
+- **Landscape optimizer driver** *(landed as `pyphi.optimize`; dashboard row)*.
+  Gate **passed**: the n=5 roughness replication ran
+  (`experiments/substrate_landscape_experiments/FINDINGS.md`) — signed normalized
+  φₛ stays empirically smooth across MIP switches, so the optimizer ships
+  population-first (`scipy.optimize.differential_evolution`, seeded, per-evaluation
+  trajectory saved). The gate caveat — the smoothness is of the objective value
+  only, and selection identity flips between near-tied set partitions at
+  single-grid-point frequency — is honored by construction: the driver reads only
+  the scalar objective and never branches on a MIP identity.
 - **Relations follow-ups** *(open, small)*. (a) Switch visualization to
   `strongest(k)` — `pyphi/visualize/projection/__init__.py` still iterates
   `ces.relations` directly, so `AnalyticalRelations` structures cannot be

changelog.d/landscape-optimizer.feature.md

@@ -0,0 +1 @@
+`pyphi.optimize()`: a seeded, population-based black-box optimizer that searches over substrate connection weights for a maximizer of signed normalized φₛ (or another IIT quantity), with `weight_axes` building the search space and a saved per-evaluation trajectory.

docs/howto/landscape.md

@@ -126,6 +126,33 @@ after = pyphi.analyze(axis(0.704), (1, 0, 0), compute="sia")
 )
 ```
 
+## Optimizing over weights
+
+`landscape_section` shows $\varphi_s$ *along* an axis; `pyphi.optimize`
+*searches* over one. Give it a builder mapping a parameter vector to a
+substrate — `weight_axes` builds one, the vector analogue of `weight_axis` —
+a box of bounds, and a seed. The optimizer is population-based and
+gradient-free, so it is untroubled by the selection switches that make the
+landscape piecewise-smooth; it maximizes signed normalized $\varphi_s$ by
+default, the quantity that stays continuous across those switches.
+
+```{code-cell} python
+box_axis = pyphi.weight_axes(
+    [ising.probability] * 3, weights, [(0, 1), (1, 0)], temperature=0.25
+)
+result = pyphi.optimize(
+    box_axis, (1, 0, 0), [(0.1, 1.3), (0.1, 1.3)], seed=20260711, popsize=6, maxiter=8
+)
+float(result.best_objective), result.best_params
+```
+
+Every evaluation is logged to `result.to_pandas()`, and
+`result.save("run_seed20260711.json")` persists the trajectory and metadata.
+
+```{code-cell} python
+result.to_pandas().head()
+```
+
 ## Notes
 
 - Setting a weight to exactly 0 removes the connection from the derived

pyphi/__init__.py

@@ -110,6 +110,9 @@
 from .landscape import perturb as perturb
 from .landscape import weight_axis as weight_axis
 from .log import enable_logging as enable_logging
+from .optimize import OptimizationResult as OptimizationResult
+from .optimize import optimize as optimize
+from .optimize import weight_axes as weight_axes
 from .serialize import load
 from .serialize import save
 from .substrate import Substrate
@@ -155,6 +158,7 @@ def __getattr__(name: str) -> ModuleType:
     "Direction",
     "FactoredTPM",
     "JointTPM",
+    "OptimizationResult",
     "Substrate",
     "System",
     "Transition",
@@ -168,9 +172,11 @@ def __getattr__(name: str) -> ModuleType:
     "iit4_2026",
     "landscape_section",
     "load",
+    "optimize",
     "perturb",
     "phi_posterior",
     "save",
+    "weight_axes",
     "weight_axis",
 ] + [name for name in sorted(_SUBMODULE_NAMES) if not name.startswith("_")]