Bound a campaign shard's caches by its own memory request

A shard evaluates every mechanism it carries against one long-lived
System, and kernel repertoire entries are keyed on that System's
fingerprint and evicted only when it is collected, so cache occupancy
grows with the number of mechanisms packed into a shard under a bound
that is combinatorial in substrate size. The one brake,
cache_utils.memory_full(), compared resident memory to
maximum_cache_memory_percentage of total physical memory, which cannot
bound a process confined to an allocation smaller than the machine: on a
large execute node it permits tens of GB of cache while the scheduler
kills the job at its request. Real peak memory ran roughly ten times over
shard_memory_bytes, which models only the largest single repertoire and
so cannot see any of this.

Add infrastructure.maximum_cache_memory_bytes, an absolute
resident-memory ceiling that replaces the percentage when set. Grant
CACHE_HEADROOM_BYTES in shard_memory_bytes, derive the enforced ceiling
from a request with shard_cache_budget_bytes, and install it during shard
execution from the shard's own memory_bytes, deferring to a ceiling
configured at preparation time. The request and the enforced bound now
come from one figure, so the estimate cannot drift from what the job is
allowed.

An order-3 purview shard goes from a 1.5 GB request with no effective
ceiling to a 2.5 GB request with caching stopping at 2.25 GB resident.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014PYZcQZadLvUoodNM578Aa

Author: wmayner

CACHING.rst

@@ -10,7 +10,9 @@ Hamming matrices, ...) through a uniform process-local cache surface in
 - ``pyphi.cache.clear(name)``: clear one named cache.
 
 The total memory footprint of in-memory caches is bounded by the
-``config.infrastructure.maximum_cache_memory_percentage`` option.
+``config.infrastructure.maximum_cache_memory_percentage`` option, or, for a
+process allowed only part of the machine, by
+``config.infrastructure.maximum_cache_memory_bytes``.
 
 Setting ``config.infrastructure.disk_cache_results = True`` additionally
 persists whole SIA and cause-effect-structure results to a

ROADMAP.md

@@ -699,9 +699,11 @@ item), mutation testing (N3/N17 ← T2), and the matching exact-oracle/standard-
   `max_repertoire_cells`; `shard_memory_bytes`/`round_memory_bytes` in `pyphi/cost.py`;
   packing runs within memory classes; the submit file requests `$(memory)` from
   `task id, memory` rows in `remaining.txt` (uniform for whole-cell sweeps);
-  `request_memory` is now the floor under the estimates. Calibration constants
-  (`REPERTOIRE_FACTOR`, `BASE_MEMORY_BYTES`) to be validated against Condor's
-  `MemoryUsage` on the first real campaign.
+  `request_memory` is now the floor under the estimates. The first production
+  campaign found the estimate structurally incomplete rather than mis-calibrated —
+  see the 2026-07-24 section below. The calibration constants
+  (`REPERTOIRE_FACTOR`, `BASE_MEMORY_BYTES`, `CACHE_HEADROOM_BYTES`) still want
+  validation against Condor's `MemoryUsage`.
 - **✅ `limit=` on `prepare_ces`.** Threaded through with default `100_000_000` (10×
   the bare walk default), and the planning walk now runs once — `prepare_ces` passes
   its workloads into `plan_ces_shards` instead of walking twice.
@@ -799,6 +801,36 @@ item), mutation testing (N3/N17 ← T2), and the matching exact-oracle/standard-
 > is per abstraction — enumerable containers have lengths, closed forms have
 > counts, samples have estimates.
 
+### 2026-07-24 shard cache memory bound
+
+> A production campaign needed a 16 GB `request_memory` floor because real peak
+> memory ran roughly ten times over `shard_memory_bytes`. The reported cause — that
+> the estimate models only the largest single repertoire, so it degenerates to the
+> base overhead at low purview order — was true but secondary. A shard evaluates
+> every mechanism it carries against one long-lived `System`, and kernel repertoire
+> entries are keyed on that `System`'s fingerprint and evicted only when it is
+> collected, so cache occupancy grows with `units_per_job` under a bound that is
+> combinatorial in substrate size. The one brake, `cache_utils.memory_full()`,
+> compared resident memory to `maximum_cache_memory_percentage` of *total physical
+> memory*: on an execute node far larger than the job's allocation it permits tens
+> of GB of cache while the scheduler kills the job at its request. The brake existed
+> and was unreachable.
+
+- **✅ Cache ceiling tied to the shard's own request.** New
+  `infrastructure.maximum_cache_memory_bytes` (default `None`) gives an absolute
+  resident-memory ceiling and replaces the percentage when set — what any process
+  confined to less than the whole machine needs (a scheduler-managed job, a
+  container, a cgroup). `shard_memory_bytes` grants `CACHE_HEADROOM_BYTES` on top of
+  the working set, `shard_cache_budget_bytes` derives the enforced ceiling from a
+  request (less a reserve for allocations in flight), and shard execution installs
+  it from `spec.memory_bytes`, deferring to a ceiling configured at preparation
+  time. The request and the enforced bound come from one figure, so the estimate
+  cannot drift from what the job is actually allowed. An order-3 purview shard goes
+  from a 1.5 GB request with no effective ceiling to a 2.5 GB request with caching
+  stopping at 2.25 GB, and the 16 GB floor is no longer needed. The 1 GB headroom
+  is a design allowance, not yet a measured constant — peak `MemoryUsage` from a
+  real shard under this build would settle it.
+
 ## Context
 
 PyPhi is a scientific library implementing Integrated Information Theory (IIT). It has

changelog.d/shard-cache-budget.fix.md

@@ -0,0 +1,10 @@
+Bounded the in-memory caches of a campaign shard by the shard's own memory
+request. A shard evaluates every mechanism it carries against one long-lived
+`System`, whose cached repertoires are released only when that `System` is
+collected, so a shard packing many mechanisms accumulated cache entries with no
+effective ceiling: `maximum_cache_memory_percentage` measures against the
+machine's total RAM, which does not bound a job confined to a smaller
+allocation. The new `maximum_cache_memory_bytes` option gives an absolute
+ceiling, set automatically during shard execution, and `shard_memory_bytes` now
+includes the cache allowance it grants, so the request and the enforced ceiling
+come from the same figure.

docs/howto/cache.md

@@ -45,6 +45,18 @@ cached are still served.
 pyphi.config.maximum_cache_memory_percentage
 ```
 
+That percentage is measured against the machine's total RAM, which is the
+wrong quantity when the process may only use part of the machine — a batch job
+with a memory request, a container, or a cgroup. In that case set
+`maximum_cache_memory_bytes` to what the process is actually allowed, and the
+caches stop storing at that figure instead. Campaign shards do this for
+themselves, using the memory each shard requested.
+
+```{code-cell} python
+with pyphi.config.override(maximum_cache_memory_bytes=2 * 1024**3):
+    print(pyphi.config.maximum_cache_memory_bytes)
+```
+
 ### Inspecting the caches
 
 `pyphi.cache.info()` returns per-cache hit/miss/size statistics for every
@@ -142,6 +154,7 @@ pyphi.config.disk_cache_results = False
 | `cache_repertoires` | `True` | Memoize cause/effect repertoire computations. |
 | `cache_potential_purviews` | `True` | Memoize potential-purview enumeration. |
 | `maximum_cache_memory_percentage` | `50` | Upper bound on in-memory cache size, as a percentage of RAM. |
+| `maximum_cache_memory_bytes` | `None` | Upper bound in bytes, for a process confined to less than the whole machine. Replaces the percentage when set. |
 | `clear_system_caches_after_computing_sia` | `False` | Clear per-system caches after each SIA. |
 | `disk_cache_results` | `False` | Persist SIA and CES results to `__pyphi_cache__/`. |
 

pyphi/cache/cache_utils.py

@@ -12,8 +12,16 @@
 
 
 def memory_full():
-    """Check if the memory is too full for further caching."""
+    """Check if the memory is too full for further caching.
+
+    Measures resident memory against ``maximum_cache_memory_bytes`` when that
+    is set, and otherwise against ``maximum_cache_memory_percentage`` of total
+    physical memory.
+    """
     current_process = psutil.Process(os.getpid())
+    budget = config.infrastructure.maximum_cache_memory_bytes
+    if budget is not None:
+        return current_process.memory_info().rss > budget
     return (
         current_process.memory_percent()
         > config.infrastructure.maximum_cache_memory_percentage

pyphi/campaign/runner.py

@@ -23,6 +23,7 @@
 from pyphi.campaign import _resolve_compute_ref
 from pyphi.conf import config
 from pyphi.conf import presets
+from pyphi.cost import shard_cache_budget_bytes
 from pyphi.sweep import _run_cell
 from pyphi.sweep import _Skipped
 
@@ -96,12 +97,26 @@ def _run_sweep_task(task: Any, substrates: dict) -> tuple[list[CellOutput], bool
 
 
 def _shard_config(task: Any) -> dict[str, Any]:
-    return {
+    overrides = {
         **task.config_overrides,
         **presets.by_name[task.formalism],
         "parallel": False,
         "progress_bars": False,
     }
+    # Bound the shard's caches by its own memory request: entries accumulate
+    # across every mechanism the shard carries, and the default percentage of
+    # physical memory is no bound at all on a machine larger than the request.
+    # An explicit ceiling configured at preparation time is left alone.
+    spec = getattr(task, "spec", None)
+    if (
+        spec is not None
+        and spec.memory_bytes
+        and overrides.get("maximum_cache_memory_bytes") is None
+    ):
+        overrides["maximum_cache_memory_bytes"] = shard_cache_budget_bytes(
+            spec.memory_bytes
+        )
+    return overrides
 
 
 def _global_tie_indices(ties: Any, slice_parts: list, indices: list[int]) -> list[int]:

pyphi/conf/CLAUDE.md

@@ -41,7 +41,12 @@ option-by-option reference, loaded only when working with this directory.
 - **`cache_potential_purviews`**: Cache purviews (default: true)
 - **`cache_macro_construction`**: Cache mapping-independent macro-construction intermediates (default: true)
 - **`clear_system_caches_after_computing_sia`**: Clear after each SIA (default: false)
-- **`maximum_cache_memory_percentage`**: Memory limit for in-memory caches (default: 50)
+- **`maximum_cache_memory_percentage`**: Memory limit for in-memory caches, as a percentage of total physical memory (default: 50)
+- **`maximum_cache_memory_bytes`**: Absolute resident-memory ceiling above which
+  the caches stop storing (default: `None`). Replaces the percentage when set,
+  which is what a process confined to less than the whole machine needs — a
+  scheduler-managed job, a container, a cgroup. Campaign shard execution sets it
+  from the shard's own memory request.
 
 ## Measures (`config.formalism.iit`)
 

pyphi/conf/infrastructure.py

@@ -84,6 +84,13 @@ class InfrastructureConfig:
     parallel_backend: str = "local"
 
     maximum_cache_memory_percentage: int = 50
+    # An absolute ceiling on process resident memory, above which in-memory
+    # caches stop storing new entries (already-stored entries are still
+    # served). When set, it replaces `maximum_cache_memory_percentage`, which
+    # measures against total physical memory and so cannot bound a process
+    # confined to an allocation smaller than the machine — a scheduler-managed
+    # job, a container, or a cgroup.
+    maximum_cache_memory_bytes: int | None = None
     cache_repertoires: bool = True
     cache_potential_purviews: bool = True
     # When True (default), the macro TPM construction caches its
@@ -160,6 +167,8 @@ def __post_init__(self) -> None:
             "maximum_cache_memory_percentage",
             self.maximum_cache_memory_percentage,
         )
+        if self.maximum_cache_memory_bytes is not None:
+            _check_int("maximum_cache_memory_bytes", self.maximum_cache_memory_bytes)
         _check_bool("cache_repertoires", self.cache_repertoires)
         _check_bool("cache_potential_purviews", self.cache_potential_purviews)
         _check_bool("cache_macro_construction", self.cache_macro_construction)

pyphi/cost.py

@@ -35,6 +35,7 @@
     "mechanism_workloads",
     "partition_sweep_count",
     "round_memory_bytes",
+    "shard_cache_budget_bytes",
     "shard_memory_bytes",
 ]
 
@@ -46,6 +47,20 @@
 BASE_MEMORY_BYTES = 1 << 30
 """Per-task overhead: interpreter, imports, substrate TPM, task payload."""
 
+CACHE_HEADROOM_BYTES = 1 << 30
+"""Memory a shard's request grants its repertoire caches.
+
+A shard evaluates every mechanism it carries against one long-lived
+``System``, whose cached repertoires are released only when that ``System``
+is collected, so cache occupancy grows with the number of mechanisms packed
+into a shard rather than with the size of any one repertoire. Granting the
+allowance in the request and enforcing it during execution (see
+:func:`shard_cache_budget_bytes`) bounds that growth whatever the shard
+packs.
+"""
+
+_CACHE_RESERVE_BYTES = 256 * 1024**2
+
 _MEMORY_STEP_BYTES = 512 * 1024**2
 
 
@@ -67,11 +82,28 @@ def shard_memory_bytes(max_repertoire_cells: int) -> int:
     """Estimated peak memory of a shard from its largest repertoire.
 
     ``REPERTOIRE_FACTOR × 8 bytes × max_repertoire_cells +
-    BASE_MEMORY_BYTES``. The factor and base are calibration constants
-    validated against scheduler-reported memory usage; requests derived
-    from this estimate are rounded with :func:`round_memory_bytes`.
+    BASE_MEMORY_BYTES + CACHE_HEADROOM_BYTES``. The factor and base are
+    calibration constants validated against scheduler-reported memory
+    usage; the headroom is the cache allowance that
+    :func:`shard_cache_budget_bytes` enforces during execution. Requests
+    derived from this estimate are rounded with
+    :func:`round_memory_bytes`.
+    """
+    return (
+        REPERTOIRE_FACTOR * 8 * max_repertoire_cells
+        + BASE_MEMORY_BYTES
+        + CACHE_HEADROOM_BYTES
+    )
+
+
+def shard_cache_budget_bytes(memory_bytes: int) -> int:
+    """Cache ceiling for a shard whose memory request is ``memory_bytes``.
+
+    The request less a reserve for the allocations in flight when the
+    ceiling is reached, since the caches stop storing at the ceiling but
+    the computation continues to allocate above it.
     """
-    return REPERTOIRE_FACTOR * 8 * max_repertoire_cells + BASE_MEMORY_BYTES
+    return max(0, memory_bytes - _CACHE_RESERVE_BYTES)
 
 
 def round_memory_bytes(n: int) -> int:

pyphi/mcp/content/performance.md

@@ -29,7 +29,9 @@ combined footprint is bounded by `maximum_cache_memory_percentage` (default 50%
 of RAM); past that, no new entries are stored. Inspect with `pyphi.cache.info()`
 and clear with `pyphi.cache.clear_all()`. Analyzing many systems in a loop and
 watching memory climb? Set `clear_system_caches_after_computing_sia = True` to
-trade recomputation for a lower ceiling.
+trade recomputation for a lower ceiling. Under a batch scheduler, a container,
+or a cgroup, a percentage of the machine's RAM is no bound at all — set
+`maximum_cache_memory_bytes` to the process's actual allowance instead.
 
 **Disk-backed result cache (opt-in, off by default).** This is the one to reach
 for on expensive work: