Evict from in-memory caches instead of freezing at the memory ceiling

Once resident memory reached `maximum_cache_memory_bytes` (or
`maximum_cache_memory_percentage`), a cache refused every new entry for the
rest of the process, so which results stayed cached was decided by whichever
happened to be computed first and every later lookup missed. Entries now live
in a `ByteBoundedStore`, shared by `ContentCache` and the module-level `@cache`
decorator, which holds its byte weight at the level it had reached and admits
new entries by evicting least recently used ones.

Occupancy is measured in bytes rather than entries because neither argument
space is bounded by a count: the combinatorial index tables key on a sequence
length alone, giving one entry per system size but values growing as 2^N or
3^N, while `max_entropy_distribution` keys on a purview, giving one entry per
subset. An entry too large to fit the whole budget is refused rather than
allowed to flush the working set, and a bound latched during a transient spike
is re-checked and lifted when memory frees up.

Eviction does not lower resident memory — freeing Python objects returns their
memory to the process allocator, rarely to the operating system — so the policy
holds occupancy steady rather than trying to shrink it. What it changes is
which entries a fixed allocation is spent on. Measured on a scoped
cause-effect structure sweep with the ceiling binding over the last 58% of the
work: the ceiling cost 1.455x the unbounded run under the freeze and 1.037x
under eviction, with the hit rate rising from 72.6% to 95.5% against an
unbounded 95.6%, on fewer entries and lower peak memory than freezing used.

`memory_full()` built a fresh `psutil.Process` on every call, ten times the
cost of reading from an existing one, and it runs on every cache miss: 19s of
a 208s run at 1.3M misses. The handle is now reused and rebuilt after a fork.

The `@cache` decorator's `maxmem` branch is removed: every call site passed
`maxmem=None` and its default bound the config at import time, so it never ran.
Its `cache=` backing-store parameter goes too, since every call site passed a
fresh dict and no store was ever shared. `pyphi.cache.info()` now reports
`nbytes` and `evictions` alongside hits, misses, and entry count.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qi3MYsc3afQMxoWZZo2tEL

Author: wmayner

changelog.d/cache-eviction.change.md

@@ -0,0 +1,25 @@
+In-memory caches now evict under memory pressure instead of freezing. Once
+resident memory reaches `maximum_cache_memory_bytes` (or
+`maximum_cache_memory_percentage`), a cache used to refuse every new entry for
+the rest of the process, so which results stayed cached was decided by whichever
+happened to be computed first. It now holds its occupancy at the level it had
+reached and admits new entries by discarding the least recently used ones. On a
+scoped cause-effect structure sweep with the ceiling binding over the last 58%
+of the work, that cut the cost of the ceiling from 1.46× the unbounded run to
+1.04×, and raised the hit rate from 72.6% to 95.5% against an unbounded 95.6% —
+while holding fewer entries and less resident memory than freezing did.
+
+Occupancy is measured in bytes rather than entries, since the argument spaces
+differ in kind: the combinatorial index tables are keyed on a sequence length
+alone, giving one entry per system size but values growing as 2ᴺ or 3ᴺ, while
+`max_entropy_distribution` is keyed on a purview, giving one entry per subset.
+Neither is bounded by a count. An entry too large to fit the whole budget is
+skipped rather than allowed to displace everything else, and a ceiling reached
+during a transient spike is re-checked and lifted if memory frees up again.
+
+Eviction does not lower resident memory — freeing Python objects returns their
+memory to the process allocator, rarely to the operating system. What it changes
+is which entries a fixed allocation is spent on.
+
+`pyphi.cache.info()` now reports `nbytes` and `evictions` alongside hits, misses,
+and entry count.

changelog.d/cache-memory-check-cost.optimization.md

@@ -0,0 +1,6 @@
+The cache memory check no longer builds a new `psutil.Process` on every call.
+It runs on every cache miss, and constructing the handle cost about ten times as
+much as reading resident memory from an existing one: 14.5 µs per call against
+1.4 µs. On a scoped cause-effect structure sweep making 1.3 million misses, that
+was 19 seconds of a 208-second run, and the share grows the more the cache
+misses. The handle is now reused, and rebuilt after a fork.

pyphi/cache/__init__.py

@@ -28,41 +28,38 @@
 :mod:`pyphi.cache.registry` for the registry implementation.
 """
 
-import os
 from functools import update_wrapper
 
 import joblib
-import psutil
 
 from pyphi import constants
-from pyphi.conf import config
 
+from .cache_utils import ByteBoundedStore
 from .cache_utils import _CacheInfo
 from .cache_utils import _make_key
 
 # An on-disk cache for distributing pre-computed results with the PyPhi package
 joblib_memory = joblib.Memory(location=constants.DISK_CACHE_LOCATION, verbose=0)
 
 
-def cache(
-    cache=None,
-    maxmem: int | None = config.infrastructure.maximum_cache_memory_percentage,
-    typed: bool = False,
-):
-    """Memory-limited memoization decorator.
+def cache(typed: bool = False):
+    """Memoization decorator bounded by bytes.
 
-    Arguments to the cached function must be hashable.
+    Arguments to the cached function must be hashable. Entries are held in a
+    :class:`pyphi.cache.cache_utils.ByteBoundedStore`, so the cache grows
+    freely until resident memory reaches the configured ceiling and then holds
+    its occupancy steady, evicting least recently used entries to admit new
+    ones.
+
+    A byte bound rather than an entry bound because the argument spaces here
+    differ in kind: the combinatorial index tables are keyed on a sequence
+    length alone, giving at most one entry per system size but values that grow
+    as 2ᴺ or 3ᴺ, while :func:`pyphi.distribution.max_entropy_distribution` is
+    keyed on a purview, giving one entry per subset. Neither is bounded by a
+    count.
 
     Parameters
     ----------
-    cache : dict, optional
-        Backing store for cached results. A fresh empty dict is created when
-        omitted; passing one shares the store across decorated functions.
-    maxmem : float or None, optional
-        Maximum percentage of physical memory the cache may use, between 0 and
-        100 inclusive. ``None`` (or 0) means unlimited. Once the process
-        exceeds this fraction of memory, no new entries are stored, though
-        entries already cached are still served.
     typed : bool, optional
         If ``True``, arguments of different types are cached separately: for
         example, ``f(3.0)`` and ``f(3)`` are treated as distinct calls with
@@ -71,71 +68,46 @@ def cache(
     Notes
     -----
     The decorated function exposes ``cache_info()``, which returns a
-    ``(hits, misses, currsize)`` named tuple; ``cache_clear()``, which empties
-    the cache and resets its statistics; and ``__wrapped__``, the underlying
-    function.
+    ``(hits, misses, currsize, nbytes, evictions)`` named tuple;
+    ``cache_clear()``, which empties the cache and resets its statistics; and
+    ``__wrapped__``, the underlying function.
     """
-    # Constants shared by all lru cache instances:
+    store = ByteBoundedStore()
+    entries = store.data
     # Unique object used to signal cache misses.
-    if cache is None:
-        cache = {}
     sentinel = object()
     # Build a key from the function arguments.
     make_key = _make_key
 
     def decorating_function(user_function, hits=0, misses=0):
-        full = False
         # Bound method to look up a key or return None.
-        cache_get = cache.get
-
-        if not maxmem:
-
-            def wrapper(*args, **kwds):
-                # Simple caching without memory limit.
-                nonlocal hits, misses
-                key = make_key(args, kwds, typed)
-                result = cache_get(key, sentinel)
-                if result is not sentinel:
-                    hits += 1
-                    return result
-                result = user_function(*args, **kwds)
-                cache[key] = result
-                misses += 1
-                return result
-
-        else:
-            # Type narrowing: maxmem is not None in this branch
-            assert maxmem is not None, "maxmem should not be None in else branch"
-            maxmem_value = maxmem
-
-            def wrapper(*args, **kwds):
-                # Memory-limited caching.
-                nonlocal hits, misses, full
-                key = make_key(args, kwds, typed)
-                result = cache_get(key)
-                if result is not None:
-                    hits += 1
-                    return result
-                result = user_function(*args, **kwds)
-                if not full:
-                    cache[key] = result
-                    # Cache is full if the total recursive usage is greater
-                    # than the maximum allowed percentage.
-                    current_process = psutil.Process(os.getpid())
-                    full = current_process.memory_percent() > maxmem_value
-                misses += 1
+        cache_get = entries.get
+
+        def wrapper(*args, **kwds):
+            nonlocal hits, misses
+            key = make_key(args, kwds, typed)
+            result = cache_get(key, sentinel)
+            if result is not sentinel:
+                hits += 1
+                # Reinsert to move the entry to the recent end of the store's
+                # iteration order, which is the eviction order.
+                del entries[key]
+                entries[key] = result
                 return result
+            result = user_function(*args, **kwds)
+            store.admit(key, result)
+            misses += 1
+            return result
 
         def cache_info():
             """Report cache statistics."""
-            return _CacheInfo(hits, misses, len(cache))
+            return _CacheInfo(hits, misses, len(entries), store.nbytes, store.evictions)
 
         def cache_clear():
             """Clear the cache and cache statistics."""
-            nonlocal hits, misses, full
-            cache.clear()
+            nonlocal hits, misses
+            store.clear()
             hits = misses = 0
-            full = False
 
         wrapper.cache_info = cache_info  # type: ignore[attr-defined]
         wrapper.cache_clear = cache_clear  # type: ignore[attr-defined]
@@ -147,8 +119,9 @@ def cache_clear():
         _register_policy(
             _DictCacheAdapter(
                 name=f"{user_function.__module__}.{user_function.__qualname__}",
-                backing=cache,
-                stats=lambda: (cache_info().hits, cache_info().misses),
+                backing=entries,
+                stats=lambda: (hits, misses),
+                weigh=lambda: (store.nbytes, store.evictions),
             )
         )
 

pyphi/cache/cache_utils.py

@@ -3,12 +3,31 @@
 
 import os
 from collections import namedtuple
+from functools import lru_cache
+from typing import Any
 
+import numpy as np
 import psutil
 
 from pyphi.conf import config
 
-_CacheInfo = namedtuple("CacheInfo", ["hits", "misses", "currsize"])
+_CacheInfo = namedtuple(
+    "CacheInfo",
+    ["hits", "misses", "currsize", "nbytes", "evictions"],
+    defaults=(0, 0),
+)
+
+
+@lru_cache(maxsize=1)
+def _process_handle(pid: int) -> psutil.Process:
+    """The psutil handle for ``pid``, kept for reuse.
+
+    Constructing the handle costs ten times as much as reading resident
+    memory from an existing one, and :func:`memory_full` is called on every
+    cache miss. Holding only the newest handle means a forked child replaces
+    its parent's rather than reusing it.
+    """
+    return psutil.Process(pid)
 
 
 def memory_full():
@@ -18,7 +37,7 @@ def memory_full():
     is set, and otherwise against ``maximum_cache_memory_percentage`` of total
     physical memory.
     """
-    current_process = psutil.Process(os.getpid())
+    current_process = _process_handle(os.getpid())
     budget = config.infrastructure.maximum_cache_memory_bytes
     if budget is not None:
         return current_process.memory_info().rss > budget
@@ -28,6 +47,157 @@ def memory_full():
     )
 
 
+_ENTRY_OVERHEAD_BYTES = 512
+"""Non-payload cost of one entry: its key, the value object, and a dict slot.
+
+Calibrated against the resident-memory growth of a scoped cause-effect
+structure sweep, where entries averaged roughly 900 bytes by a walk of the live
+objects and roughly 540 bytes by the process's own resident growth. It is a
+single constant rather than a per-entry measurement because sizing each key
+exactly would cost more than the admission it informs.
+"""
+
+_ARRAY_DIM_BYTES = 16
+"""Per-dimension cost of an ndarray's shape and strides, one intp each."""
+
+_RECHECK_INTERVAL = 4096
+"""Admissions between ceiling re-checks once a store is bounded.
+
+Re-checking lets a bound that was latched during a transient spike be lifted,
+and keeps the cost of the check off all but one admission in this many.
+"""
+
+_MISSING = object()
+
+
+def entry_weight(value: Any) -> int:
+    """Estimated bytes one cached value occupies, including its key and slot.
+
+    An ndarray view's buffer belongs to the array it derives from, so only the
+    view object itself is charged. A sequence — the combinatorial index tables
+    are lists of tuples — is charged per element, since its cost is the
+    elements rather than any single buffer.
+    """
+    if isinstance(value, np.ndarray):
+        payload = value.nbytes if value.base is None else 0
+        return _ENTRY_OVERHEAD_BYTES + _ARRAY_DIM_BYTES * value.ndim + payload
+    if isinstance(value, list | tuple):
+        return _ENTRY_OVERHEAD_BYTES + sum(map(_element_weight, value))
+    return _ENTRY_OVERHEAD_BYTES
+
+
+_SEQUENCE_HEADER_BYTES = 56
+"""Object header of a tuple or list, before its element pointers."""
+
+_POINTER_BYTES = 8
+
+_SCALAR_BYTES = 32
+"""A small int or similar leaf, counted once rather than by identity.
+
+Small integers are interned, so charging each occurrence overstates a table of
+index tuples. The overstatement is uniform across entries, which is what the
+bound compares.
+"""
+
+_MAX_WEIGHT_DEPTH = 4
+
+
+def _element_weight(element: Any, depth: int = 0) -> int:
+    """Bytes one element of a cached sequence occupies.
+
+    Recurses through nested sequences to a fixed depth, since the index tables
+    nest two or three levels and a bound that stopped at the first would
+    undercount them by the width of every inner tuple.
+    """
+    if depth < _MAX_WEIGHT_DEPTH and isinstance(element, tuple | list):
+        return (
+            _SEQUENCE_HEADER_BYTES
+            + _POINTER_BYTES * len(element)
+            + sum(_element_weight(x, depth + 1) for x in element)
+        )
+    return _SCALAR_BYTES
+
+
+class ByteBoundedStore:
+    """A dict that holds its byte weight steady once memory reaches the ceiling.
+
+    Insertion order is the recency order, so the least recently used entry is
+    the first one iteration yields; a caller reinserts an entry on a hit to
+    move it to the recent end. Until resident memory reaches the cache ceiling
+    (see :func:`memory_full`) the store grows freely. From then on it admits an
+    entry by evicting least recently used ones, and refuses one too large to
+    fit an empty store rather than flushing everything to hold it.
+
+    Holding the weight steady, rather than shrinking it, is what the ceiling
+    can deliver: freeing Python objects returns their memory to the process
+    allocator for reuse but rarely to the operating system, so eviction does
+    not lower resident memory. What it buys is spending a fixed allocation on
+    recently used entries instead of on whichever were computed first.
+
+    Not internally synchronized. A caller sharing a store across threads holds
+    its own lock across :meth:`admit` and :meth:`discard`.
+    """
+
+    def __init__(self) -> None:
+        self.data: dict[Any, Any] = {}
+        self.evictions = 0
+        self._weight = 0
+        self._budget: int | None = None
+        self._admissions = 0
+
+    @property
+    def nbytes(self) -> int:
+        """Estimated bytes held by this store's entries."""
+        return self._weight
+
+    def admit(self, key: Any, value: Any) -> None:
+        """Store an entry, evicting least recently used ones to make room."""
+        weight = entry_weight(value)
+        self._admissions += 1
+        # Consult the ceiling while unbounded, periodically once bounded, and
+        # whenever the bound is about to refuse an entry outright, so a bound
+        # latched during a transient spike cannot persist.
+        if (
+            self._budget is None
+            or self._admissions % _RECHECK_INTERVAL == 0
+            or weight > self._budget
+        ):
+            if memory_full():
+                # Hold occupancy here and trade old entries for new ones.
+                if self._budget is None:
+                    self._budget = self._weight
+            else:
+                # Room again, whether because the spike that set the bound has
+                # passed or because memory was released elsewhere.
+                self._budget = None
+        if self._budget is not None:
+            while self.data and self._weight + weight > self._budget:
+                oldest, evicted = next(iter(self.data.items()))
+                del self.data[oldest]
+                self._weight -= entry_weight(evicted)
+                self.evictions += 1
+            if self._weight + weight > self._budget:
+                # Does not fit even in an empty store. Storing it would evict
+                # the whole working set for one entry.
+                return
+        self.discard(key)
+        self.data[key] = value
+        self._weight += weight
+
+    def discard(self, key: Any) -> None:
+        """Remove an entry if present, crediting back its weight."""
+        previous = self.data.pop(key, _MISSING)
+        if previous is not _MISSING:
+            self._weight -= entry_weight(previous)
+
+    def clear(self) -> None:
+        self.data.clear()
+        self.evictions = 0
+        self._weight = 0
+        self._budget = None
+        self._admissions = 0
+
+
 class _HashedSeq(list):
     """This class guarantees that ``hash()`` will be called no more than once
     per element.  This is important because the ``lru_cache()`` will hash the