Merge provenance-writers: script-facing provenance writers

provenance.save_json/save_npz/save_dataframe encode parameters in the
filename, never overwrite existing files, and embed a full Provenance
record; read_metadata reads it back from any of the three formats. The
three benchmark scripts with hand-rolled copies now use the shared
helpers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PEAxNzhDCaTrntX3o1JqMV

Author: wmayner

ROADMAP.md

@@ -633,9 +633,13 @@ extending B1's bound-certificate assertions.
   is gone. Spec: `docs/superpowers/specs/2026-07-19-result-serialization-design.md`.
 - **Thread-backend progress (M13).** The thread backend discards the progress policy on entry
   (an in-code deferral note); hook the progress bar into its futures loop.
-- **Script-facing provenance writer (M14).** `provenance.save_json`/`save_npz` with git SHA,
-  parameters encoded in the filename, and no-clobber versioning — consolidating the pattern
-  that experiment scripts repeatedly re-implement.
+- **Script-facing provenance writer (M14).** *Landed 2026-07-20:*
+  `provenance.save_json`/`save_npz`/`save_dataframe` (parquet, per the DataFrame-output
+  convention) with `format_stem` filename encoding, `unique_path` no-clobber versioning, an
+  embedded `Provenance` record, and `read_metadata` to read it back; the three benchmark
+  scripts with hand-rolled copies now import the shared helpers (the cross-generation
+  harness keeps a fallback for the pre-refactor side). Spec:
+  `docs/superpowers/specs/2026-07-20-provenance-writers-design.md`.
 
 Already tracked elsewhere (verified during the triage): the Wave 7 ii-gate (M4), the
 intervention/lesion surface (N14 ← M8), the Φ-structure distance surface (N16 ← M6), the

benchmarks/b18_dispatch_gate.py

@@ -66,6 +66,8 @@
 
 import numpy as np
 
+from pyphi.provenance import unique_path
+
 RESULTS_DIR = Path(__file__).parent / "b18_dispatch_gate_results"
 
 # Workload sizes as fractions of the level's chunksize. Sizes at or below
@@ -642,15 +644,6 @@ def run_level(name: str, seed: int, trials: int) -> dict:
     }
 
 
-def unique_path(directory: Path, stem: str, suffix: str) -> Path:
-    path = directory / f"{stem}{suffix}"
-    version = 2
-    while path.exists():
-        path = directory / f"{stem}_v{version}{suffix}"
-        version += 1
-    return path
-
-
 def main() -> None:
     parser = argparse.ArgumentParser(description=__doc__)
     parser.add_argument("--seed", type=int, default=1810)

benchmarks/iit_3_vs_4/harness.py

@@ -507,18 +507,22 @@ def extract_phase_times(pstats_path: Path) -> dict[str, float]:
     return out
 
 
-def unique_path(directory: Path, stem: str, suffix: str) -> Path:
-    """Return a non-clobbering path: stem.suffix, then stem_v2.suffix, ..."""
-    directory.mkdir(parents=True, exist_ok=True)
-    base = directory / f"{stem}{suffix}"
-    if not base.exists():
-        return base
-    n = 2
-    while True:
-        candidate = directory / f"{stem}_v{n}{suffix}"
-        if not candidate.exists():
-            return candidate
-        n += 1
+try:
+    from pyphi.provenance import unique_path
+except ImportError:
+    # The pre-refactor generation has no pyphi.provenance; keep a local copy.
+    def unique_path(directory: Path, stem: str, suffix: str) -> Path:
+        """Return a non-clobbering path: stem.suffix, then stem_v2.suffix, ..."""
+        directory.mkdir(parents=True, exist_ok=True)
+        base = directory / f"{stem}{suffix}"
+        if not base.exists():
+            return base
+        n = 2
+        while True:
+            candidate = directory / f"{stem}_v{n}{suffix}"
+            if not candidate.exists():
+                return candidate
+            n += 1
 
 
 def write_record(record: dict[str, Any]) -> Path:

benchmarks/iit_3_vs_4/p18_inversion_share.py

@@ -35,6 +35,9 @@
 
 import numpy as np
 
+from pyphi.provenance import format_stem
+from pyphi.provenance import unique_path
+
 CORE_N = 6
 CORE_DENSITY = 0.35
 COUPLING_MEAN = 1.0
@@ -112,15 +115,11 @@ def _profile_sia(system) -> dict:
 
 def _output_path(seed: int, run_label: str | None) -> Path:
     results_dir = Path(__file__).parent / "results"
-    results_dir.mkdir(exist_ok=True)
-    label = f"_{run_label}" if run_label else ""
-    base = f"p18_inversion_share_seed{seed}{label}"
-    path = results_dir / f"{base}.json"
-    version = 2
-    while path.exists():
-        path = results_dir / f"{base}_v{version}.json"
-        version += 1
-    return path
+    return unique_path(
+        results_dir,
+        format_stem("p18_inversion_share", {"seed": seed}, run_label),
+        ".json",
+    )
 
 
 def main() -> None:

changelog.d/provenance-writers.feature.md

@@ -0,0 +1 @@
+Added script-facing provenance writers: `pyphi.provenance.save_json`, `save_npz`, and `save_dataframe` (parquet) encode parameters in the filename, never overwrite existing files (`_v2`/`_v3` versioning), and embed a full `Provenance` record (git SHA, seed, versions) in every output; `read_metadata` reads it back from any of the three formats.

docs/howto/save-load.md

@@ -133,6 +133,44 @@ except TypeError as error:
     print(error)
 ```
 
+## Experiment provenance writers
+
+Experiment scripts need two things beyond `pyphi.save`: output files that
+never overwrite earlier runs, and a record of how each file was produced.
+The writers in `pyphi.provenance` provide both. Parameters are encoded
+into the filename, a repeated save lands in a `_v2` file instead of
+clobbering the first, and every file embeds a full provenance record —
+pyphi version, git commit, timestamp, and seed.
+
+```{code-cell} python
+from pyphi import provenance
+
+path = provenance.save_json(
+    {"phi": 0.133873},
+    out,
+    "sweep_study",
+    params={"seed": 42, "trials": 60},
+)
+path.name
+```
+
+```{code-cell} python
+provenance.save_json(
+    {"phi": 0.5}, out, "sweep_study", params={"seed": 42, "trials": 60}
+).name
+```
+
+`save_npz` does the same for arrays of raw per-trial data, and
+`save_dataframe` writes a DataFrame as parquet — the format used for
+DataFrame outputs throughout PyPhi — with the metadata embedded in the
+parquet schema. `read_metadata` recovers the provenance and parameters
+from any of the three formats:
+
+```{code-cell} python
+metadata = provenance.read_metadata(path)
+{key: metadata["provenance"][key] for key in ("seed", "pyphi_version")}
+```
+
 ## Compatibility note
 
 This serializer is a deliberate format break from the `jsonify` layer used in