oom-seq: add --oom-seq-randomize (per-sequence length + window variety)

`--oom-seq` previously emitted every sequence with a fixed length (--oom-seq-len)
and a single module-wide failure window (--oom-window), so covering a range of
sequence shapes meant running several fleet instances with different static
configs. New `--oom-seq-randomize` (default off) instead draws, independently for
each emitted sequence, the step count from [1, --oom-seq-len] and the failure
window from [1, --oom-window] -- the configured values become upper bounds. One
instance thus covers a spread of shapes; a fixed config is just the min==max case.

Implementation:
- oom_run() gains a `window` param (default _OOM_WINDOW) and uses it instead of
  the module constant; per-sequence windows are emitted as oom_run(..., window=k).
  The always-printed [OOM-SEQ] marker now includes window= so a crashing
  sequence's window is known on replay even without --oom-verbose.
- _oom_pick_seq_len()/_oom_pick_window() helpers; both seq generators (function
  and class-method) draw per sequence. Window 0 (legacy fail-forever) is left as a
  static default and never randomized into.
- Default off => generated output unchanged except oom_run's signature line.

Tests: TestOOMSeqGeneration gains randomize-on (per-sequence window in bounds,
lengths vary in bounds) and randomize-off (no per-call window kwarg) cases; the
fixture pins oom_seq_randomize=False (MagicMock auto-attrs are truthy). Full suite
326 passed / 32 skipped; ruff clean. Docs: CLAUDE.md + doc/oom-sequences.md
(which had already proposed this).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: devdanzin

CLAUDE.md

@@ -69,7 +69,8 @@ PYTHONPATH=$PWD python fuzzers/fusil-python-threaded --unsafe [options]
 #   --jit-mode synthesize|variational|legacy|all   (see JIT section)
 #   --oom-fuzz               OOM (allocation-failure) injection mode (see OOM section)
 #   --oom-seq                stateful call SEQUENCES (Phase 4): several calls per scan under
-#                            one failure window (found OOM-0036); --oom-seq-len/--oom-window
+#                            one failure window (found OOM-0036); --oom-seq-len/--oom-window;
+#                            --oom-seq-randomize varies len/window per sequence (those = maxes)
 #   --oom-dedup-catalog F    in-loop crash dedupe/labeling vs known_sites.tsv; add
 #                            --oom-dedup-prune to drop dups, --oom-dedup-resolve-segv
 #                            to resolve segvs via gdb so they dedupe too
@@ -164,7 +165,11 @@ str-subclass constructor bug).
 **bounded failure window** — `set_nomemory(start, start+k)` fails `k` allocations then
 *resumes* — so an allocation failure in one call can corrupt state a *later* call trips over
 (the cross-call "stale state" class the single-call sweep can't reach). `--oom-seq-len`
-(steps, default 3) / `--oom-window` (`k`, default 1; `0` = legacy fail-forever). Opt-in;
+(steps, default 3) / `--oom-window` (`k`, default 1; `0` = legacy fail-forever). Add
+`--oom-seq-randomize` to randomize each emitted sequence's length (in `[1, --oom-seq-len]`)
+and window (in `[1, --oom-window]`) independently, so one instance covers a range of
+sequence shapes (the configured values become upper bounds; per-sequence window is passed
+to `oom_run(..., window=k)`). Opt-in;
 default output unchanged without it. It found **OOM-0036** — a `list.append()` double-free
 under `MemoryError` in the `_CALL_LIST_APPEND` bytecode, filed as python/cpython#151818.
 Design + the windowed-`set_nomemory` semantics: **`doc/oom-sequences.md`**.

doc/oom-sequences.md

@@ -128,6 +128,14 @@ cheaper and more targeted, and it's the right first lever — fold it into Idea
 injection primitive (`--oom-window`, default 1), optionally randomizing `k` per sequence
 for extra spread. Frame it as coverage, not noise.
 
+> **Implemented (`--oom-seq-randomize`).** This per-sequence randomization now exists: with
+> the flag set, each emitted sequence draws its window uniformly from `[1, --oom-window]` and
+> its length from `[1, --oom-seq-len]` independently (the configured values become upper
+> bounds; the window is passed per call as `oom_run(..., window=k)`). One instance thus covers
+> a range of sequence shapes instead of a single static config — strictly more general (a
+> fixed config is just the `min == max` case). Default off; generated output is unchanged
+> without it.
+
 **Cost to be honest about:** auto-resume means the crash can occur *far* from the
 injection point (causal distance grows with `k` and sequence length), which weakens the
 "which allocation caused it" link and makes minimization harder. `k = 1` keeps it tight;

fusil/python/__init__.py

@@ -401,6 +401,15 @@ def createFuzzerOptions(self, parser: OptionParserWithSections) -> None:
             type="int",
             default=1,
         )
+        oom_options.add_option(
+            "--oom-seq-randomize",
+            help="Randomize each emitted sequence's length (in [1, --oom-seq-len]) and "
+            "failure window (in [1, --oom-window]) independently, so one instance covers a "
+            "range of sequence shapes instead of a single static config. The --oom-seq-len / "
+            "--oom-window values become the upper bounds (default: off)",
+            action="store_true",
+            default=False,
+        )
         oom_options.add_option(
             "--oom-verbose",
             help="In OOM mode, also print the sweep start index before each "

fusil/python/write_python_code.py

@@ -567,27 +567,29 @@ def oom_call(label, func, *args, **kwargs):
                 f"""
                 _OOM_WINDOW = {self.options.oom_window}
 
-                def oom_run(label, thunk):
+                def oom_run(label, thunk, window=_OOM_WINDOW):
                     # Stateful OOM sequence (Phase 4): sweep a bounded failure window
                     # across a multi-step thunk so a failure in one step can corrupt
-                    # state a later step trips over. set_nomemory(start, start+_OOM_WINDOW)
-                    # fails _OOM_WINDOW allocations then resumes succeeding, so steps after
-                    # the burst run on the damaged state (_OOM_WINDOW == 0 -> fail forever,
-                    # the legacy single-call semantics). The thunk guards each step
-                    # internally so the tail still runs after an earlier step raises; a real
-                    # crash (segfault/abort) terminates the process and is scored.
+                    # state a later step trips over. set_nomemory(start, start+window)
+                    # fails `window` allocations then resumes succeeding, so steps after
+                    # the burst run on the damaged state (window == 0 -> fail forever,
+                    # the legacy single-call semantics). `window` defaults to _OOM_WINDOW
+                    # but is passed per-sequence when --oom-seq-randomize is set. The thunk
+                    # guards each step internally so the tail still runs after an earlier
+                    # step raises; a real crash (segfault/abort) terminates the process and
+                    # is scored.
                     if not _OOM_AVAILABLE:
                         try:
                             thunk()
                         except BaseException:
                             pass
                         return
-                    print("[OOM-SEQ] " + label, file=stderr)
+                    print("[OOM-SEQ] " + label + " window=" + str(window), file=stderr)
                     for _start in range(_OOM_MAX_START):
                         if _OOM_VERBOSE:
-                            print("[OOM-SEQ]   start=" + str(_start) + " window=" + str(_OOM_WINDOW), file=stderr)
-                        if _OOM_WINDOW > 0:
-                            _set_nomemory(_start, _start + _OOM_WINDOW)
+                            print("[OOM-SEQ]   start=" + str(_start) + " window=" + str(window), file=stderr)
+                        if window > 0:
+                            _set_nomemory(_start, _start + window)
                         else:
                             _set_nomemory(_start, 0)
                         try:
@@ -1091,7 +1093,27 @@ def _generate_oom_function_call(
         self.write(0, ")")
         self.emptyLine()
 
-    def _write_oom_sequence(self, fn_name: str, seq_label: str, steps) -> None:
+    def _oom_seq_randomize(self) -> bool:
+        return bool(getattr(self.options, "oom_seq_randomize", False))
+
+    def _oom_pick_seq_len(self) -> int:
+        """Step count for ONE sequence. With --oom-seq-randomize, uniform in [1, oom_seq_len]
+        (the configured value is the upper bound); otherwise the fixed oom_seq_len."""
+        n = max(1, self.options.oom_seq_len)
+        if self._oom_seq_randomize() and n > 1:
+            return randint(1, n)
+        return n
+
+    def _oom_pick_window(self):
+        """Failure window for ONE sequence, or None to emit the harness default (_OOM_WINDOW).
+        With --oom-seq-randomize, uniform in [1, oom_window] (upper bound); otherwise None so
+        the generated oom_run() call is unchanged. window 0 (legacy fail-forever) is left as
+        the static default and never randomized into."""
+        if self._oom_seq_randomize() and self.options.oom_window > 1:
+            return randint(1, self.options.oom_window)
+        return None
+
+    def _write_oom_sequence(self, fn_name: str, seq_label: str, steps, window=None) -> None:
         """Emit a guarded multi-step thunk plus an oom_run() call (Phase 4 sequence).
 
         steps: list of (sublabel, target_expr, num_args), where target_expr is a string
@@ -1100,6 +1122,9 @@ def _write_oom_sequence(self, fn_name: str, seq_label: str, steps) -> None:
         the shared state (a live instance, or module/interpreter globals such as a pending
         exception) is what a later step may trip over. Result values are not threaded
         between steps yet (Phase 4b); the steps interact only through shared state.
+
+        window: per-sequence failure window (--oom-seq-randomize); None emits the default
+        oom_run(label, thunk) call (uses the module-level _OOM_WINDOW).
         """
         self.write(0, f"def {fn_name}():")
         saved = self.addLevel(1)
@@ -1121,7 +1146,10 @@ def _write_oom_sequence(self, fn_name: str, seq_label: str, steps) -> None:
         if not wrote:
             self.write(0, "pass")
         self.restoreLevel(saved)
-        self.write(0, f'oom_run("{seq_label}", {fn_name})')
+        if window is None:
+            self.write(0, f'oom_run("{seq_label}", {fn_name})')
+        else:
+            self.write(0, f'oom_run("{seq_label}", {fn_name}, window={window})')
         self.emptyLine()
 
     def _generate_oom_function_sequence(self, prefix: str) -> None:
@@ -1134,7 +1162,7 @@ def _generate_oom_function_sequence(self, prefix: str) -> None:
         """
         steps = []
         names = []
-        for j in range(max(1, self.options.oom_seq_len)):
+        for j in range(self._oom_pick_seq_len()):
             func_name = choice(self.module_functions)
             try:
                 func_obj = getattr(self.module, func_name)
@@ -1154,7 +1182,9 @@ def _generate_oom_function_sequence(self, prefix: str) -> None:
             return
         seq_label = f"{prefix}:{self.module_name}[" + ">".join(names) + "]"
         self.write(0, f"# OOM sequence: {' > '.join(names)}")
-        self._write_oom_sequence(f"_oom_seq_{prefix}", seq_label, steps)
+        self._write_oom_sequence(
+            f"_oom_seq_{prefix}", seq_label, steps, window=self._oom_pick_window()
+        )
 
     def _generate_oom_class_fuzzing(self, prefix: str, class_name: str, class_obj: type) -> None:
         """Emits an OOM sweep over a class constructor and, on a live instance, its methods.
@@ -1203,7 +1233,7 @@ def _generate_oom_class_fuzzing(self, prefix: str, class_name: str, class_obj: t
             # method B trips over (e.g. OOM-0035: write... then getvalue()).
             steps = []
             mnames = []
-            for j in range(max(1, self.options.oom_seq_len)):
+            for j in range(self._oom_pick_seq_len()):
                 m_name = choice(method_names)
                 m_obj = methods[m_name]
                 min_arg, max_arg = get_arg_number(m_obj, m_name, 0)
@@ -1218,7 +1248,9 @@ def _generate_oom_class_fuzzing(self, prefix: str, class_name: str, class_obj: t
                 mnames.append(m_name)
             seq_label = f"{prefix}:{self.module_name}.{class_name}[" + ">".join(mnames) + "]"
             self.write(0, f"# OOM sequence on {class_name}: {' > '.join(mnames)}")
-            self._write_oom_sequence(f"_oom_seq_{prefix}", seq_label, steps)
+            self._write_oom_sequence(
+                f"_oom_seq_{prefix}", seq_label, steps, window=self._oom_pick_window()
+            )
         else:
             for j in range(self.options.oom_methods):
                 m_name = choice(method_names)

tests/python/test_oom_fuzz.py

@@ -14,6 +14,8 @@
 import json
 import math
 import os
+import random
+import re
 import sys
 import tempfile
 import unittest
@@ -55,6 +57,7 @@ def _make_options(oom_fuzz, oom_verbose=False):
     o.oom_seq = False
     o.oom_seq_len = 3
     o.oom_window = 1
+    o.oom_seq_randomize = False
     # JIT options (WriteJITCode is constructed unconditionally); OOM mode never
     # dispatches to it, so legacy defaults are fine.
     o.jit_fuzz = False
@@ -81,6 +84,7 @@ def _generate(
     oom_seq=False,
     oom_seq_len=3,
     oom_window=1,
+    oom_seq_randomize=False,
 ):
     """Generate a fuzzing script against ``module`` and return its source."""
     parent = MagicMock()
@@ -91,6 +95,7 @@ def _generate(
     options.oom_seq = oom_seq
     options.oom_seq_len = oom_seq_len
     options.oom_window = oom_window
+    options.oom_seq_randomize = oom_seq_randomize
     parent.options = options
     parent.filenames = ["/bin/sh"]
     fd, path = tempfile.mkstemp(suffix="_oom_test.py")
@@ -218,9 +223,9 @@ def test_seq_emits_windowed_oom_run_harness(self):
         src = _generate(oom_fuzz=True, oom_seq=True, oom_seq_len=3, oom_window=2)
         ast.parse(src)
         # The oom_run harness + the bounded-window primitive (start .. start+k).
-        self.assertIn("def oom_run(label, thunk):", src)
+        self.assertIn("def oom_run(label, thunk, window=_OOM_WINDOW):", src)
         self.assertIn("_OOM_WINDOW = 2", src)
-        self.assertIn("_set_nomemory(_start, _start + _OOM_WINDOW)", src)
+        self.assertIn("_set_nomemory(_start, _start + window)", src)
         self.assertIn("_set_nomemory(_start, 0)", src)  # window==0 fallback branch
         self.assertIn('print("[OOM-SEQ] " + label', src)
         # Function sequences: a guarded multi-step thunk fed to oom_run.
@@ -260,6 +265,45 @@ def test_seq_method_chain_reuses_one_instance(self):
         self.assertRegex(src, r"getattr\(oom_inst_oc1_\w+, ")
         self.assertNotIn('oom_call("oc1m', src)  # single-call method sweep replaced
 
+    def test_seq_no_randomize_omits_per_call_window(self):
+        # Default (randomize off): oom_run() calls take no per-sequence window override,
+        # so the harness default (_OOM_WINDOW) applies -- output is unchanged.
+        src = _generate(oom_fuzz=True, oom_seq=True, oom_seq_len=3, oom_window=2)
+        calls = re.findall(r"oom_run\([^\n]*?, _oom_seq_f\d+\)", src)
+        self.assertTrue(calls, "expected default 2-arg oom_run() calls")
+        self.assertEqual(re.findall(r"oom_run\([^\n]*?, window=\d+\)", src), [])
+
+    def test_seq_randomize_emits_per_sequence_window_within_bounds(self):
+        random.seed(20240623)
+        src = _generate(
+            oom_fuzz=True, oom_seq=True, oom_seq_len=6, oom_window=8, oom_seq_randomize=True
+        )
+        ast.parse(src)
+        windows = [int(w) for w in re.findall(r"oom_run\([^\n]*?, window=(\d+)\)", src)]
+        self.assertTrue(windows, "randomize on should emit per-sequence window= kwargs")
+        self.assertTrue(all(1 <= w <= 8 for w in windows), windows)
+
+    def test_seq_randomize_varies_length_within_bounds(self):
+        # Across the per-session sequences, step counts stay in [1, oom_seq_len] and (with a
+        # wide bound + seed) are not all identical -> real per-sequence variety.
+        random.seed(42)
+        src = _generate(
+            oom_fuzz=True,
+            oom_seq=True,
+            oom_verbose=True,  # emits a "step sN:" marker per step so we can count
+            oom_seq_len=6,
+            oom_window=4,
+            oom_seq_randomize=True,
+        )
+        ast.parse(src)
+        lengths = [
+            len(re.findall(r"step s\d+:", body))
+            for body in re.split(r"def _oom_seq_f\d+\(\):", src)[1:]
+        ]
+        self.assertTrue(lengths, "expected function sequences")
+        self.assertTrue(all(1 <= n <= 6 for n in lengths), lengths)
+        self.assertGreater(len(set(lengths)), 1, f"lengths did not vary: {lengths}")
+
     def test_non_seq_oom_mode_has_no_seq_artifacts(self):
         src = _generate(oom_fuzz=True, oom_seq=False)
         for marker in ("oom_run", "_OOM_WINDOW", "[OOM-SEQ]", "_oom_seq_"):