feat(test-fill): enable phase-1-only pre-alloc generation (#2720)

* refactor(test-fill): allow phase-1-only pre-alloc generation

Re-order `FillCommand.create_executions` so that the execution plan
reflects the user's intent:

- `--use-pre-alloc-groups` now takes priority. The flag means pre-alloc
  groups already exist on disk from a previous run, so even alongside
  `--generate-all-formats` the run is single-phase.
- `--generate-pre-alloc-groups` without `--generate-all-formats` now
  runs phase 1 only; don't continue to phase 2.
- `--generate-all-formats` continues to trigger the full two-phase
  run.

Update `test_legacy_generate_pre_alloc_groups_still_works` to reflect
the new phase-1-only behaviour and add a test covering the
`--use-pre-alloc-groups` priority.

* test(test-fill): cover contradictory `--use-pre-alloc-groups` flag pairs

Add failing tests asserting that `FillCommand.create_executions` raises
`click.UsageError` when `--use-pre-alloc-groups` is combined with:

- `--generate-pre-alloc-groups`: the first asserts the groups already
  exist on disk, the second regenerates them: contradictory intents.
- `--clean`: `--clean` wipes the output directory that holds the
  pre-alloc groups the user is asserting exist.

Without a guard, `FillCommand.create_executions` silently returns a
single-phase plan and `FillingSession._load_pre_alloc_groups_from_folder`
then fails on a missing folder. Validation logic follows in the next
commit.

* fix(test-fill): reject contradictory `--use-pre-alloc-groups` flag pairs

Add `_validate_flag_combinations` to `FillCommand.create_executions`
so contradictory flag pairs fail fast with `click.UsageError` instead
of silently falling through to a broken single-phase execution.

Rejected combinations:

- `--use-pre-alloc-groups` + `--generate-pre-alloc-groups`: the first
  asserts the groups already exist, the second regenerates them.
- `--use-pre-alloc-groups` + `--clean`: `--clean` wipes the output
  directory that holds the pre-alloc groups.

`--use-pre-alloc-groups` + `--generate-all-formats` remains valid and
single-phase, per the intent stated in 116e5093e83.

* refactor(tooling): restore `bench-gas` two-phase fill

Swap `--generate-pre-alloc-groups` for `--generate-all-formats` in the
`bench-gas` recipe. After 116e5093e83, `--generate-pre-alloc-groups`
alone runs phase 1 only and produces no benchmark fixtures; the recipe
needs the full two-phase run to generate the `BlockchainEngineXFixture`
output it targets. Matches the flag used in `.github/configs/feature.yaml`.

* doc(test-fill): refresh pre-alloc flag docs

Update documentation to reflect the phase-1-only behaviour of
`--generate-pre-alloc-groups` introduced in 116e5093e83:

- Remove the "Alternative approach" note in
  `filling_tests_command_line.md` that recommended
  `--generate-pre-alloc-groups` as a single-command replacement for
  `--generate-all-formats`: it now produces no fixtures.
- Reframe the `blockchain_test_engine_x.md` usage notes to describe
  the two-invocation workflow (`--generate-pre-alloc-groups` then
  `--use-pre-alloc-groups`) as the explicit alternative to the
  single-command `--generate-all-formats` path.

Author: danceratopz

Justfile

@@ -220,7 +220,7 @@ bench-gas *args:
     uv run fill \
         --evm-bin="{{ evm_bin }}" \
         --gas-benchmark-values 1 \
-        --generate-pre-alloc-groups \
+        --generate-all-formats \
         --fork Osaka \
         -m "not slow" \
         -n auto --maxprocesses 10 --dist=loadgroup \

docs/filling_tests/filling_tests_command_line.md

@@ -107,14 +107,6 @@ This flag automatically performs a two-phase execution:
     uv run fill --generate-all-formats --output=fixtures.tar.gz tests/shanghai/
     ```
 
-!!! note "Alternative approach"
-    You can still use the legacy approach, but this will only generate the `BlockchainEngineXFixture` format:
-    ```console
-    # Single command that automatically does 2-phase execution
-    # but only generates BlockchainEngineXFixture
-    uv run fill --generate-pre-alloc-groups tests/shanghai/
-    ```
-
 ## Debugging the `t8n` Command
 
 The `--evm-dump-dir` flag can be used to dump the inputs and outputs of every call made to the `t8n` command for debugging purposes, see [Debugging Transition Tools](./debugging_t8n_tools.md).

docs/running_tests/test_formats/blockchain_test_engine_x.md

@@ -139,7 +139,7 @@ Engine API payload structure identical to the one defined in [Blockchain Engine
 ## Usage Notes
 
 - This format is generated when using:
-    - `--generate-pre-alloc-groups` flag (automatically triggers 2-phase execution, generates only `BlockchainEngineXFixture`)
+    - `--generate-pre-alloc-groups` followed by `--use-pre-alloc-groups` (two invocations: phase 1 populates the `pre_alloc` folder, phase 2 generates only `BlockchainEngineXFixture`)
     - `--generate-all-formats` flag (automatically triggers 2-phase execution, generates all fixture formats)
 - The `pre_alloc` folder is essential and must be distributed with the test fixtures
 - Tests are grouped by identical (fork + environment + pre-allocation) combinations

packages/testing/src/execution_testing/cli/pytest_commands/fill.py

@@ -33,33 +33,81 @@ def create_executions(
         self, pytest_args: List[str]
     ) -> List[PytestExecution]:
         """
-        Create execution plan that supports two-phase pre-allocation group
+        Create execution plan supporting two-phase pre-allocation group
         generation.
 
-        Returns single execution for normal filling, or two-phase execution
-        when --generate-pre-alloc-groups or --generate-all-formats is
-        specified.
+        Returns:
+        - Phase-1-only execution when `--generate-pre-alloc-groups` is
+          set without `--generate-all-formats`.
+        - Single-phase execution when `--use-pre-alloc-groups` is set,
+          regardless of `--generate-all-formats` (pre-alloc groups
+          already exist on disk from a previous run).
+        - Two-phase execution when `--generate-all-formats` is set.
+        - Normal single-phase execution otherwise.
+
         """
         processed_args = self.process_arguments(pytest_args)
         processed_args = self._add_default_ignores(processed_args)
+        self._validate_flag_combinations(processed_args)
 
-        # Check if we need two-phase execution
-        if self._should_use_two_phase_execution(processed_args):
-            return self._create_two_phase_executions(processed_args)
-        elif "--use-pre-alloc-groups" in processed_args:
-            # Only phase 2: using existing pre-allocation groups
+        if "--use-pre-alloc-groups" in processed_args:
+            # Pre-alloc groups already exist: single-phase fill only.
             return self._create_single_phase_with_pre_alloc_groups(
                 processed_args
             )
-        else:
-            # Normal single-phase execution
-            return [
-                PytestExecution(
-                    config_file=self.config_path,
-                    args=processed_args,
-                    allowed_exit_codes=self.allowed_exit_codes,
-                )
-            ]
+
+        has_phase_1_flag = "--generate-pre-alloc-groups" in processed_args
+        has_phase_2_flag = "--generate-all-formats" in processed_args
+
+        if has_phase_2_flag:
+            # --generate-all-formats always regenerates pre-alloc as phase 1.
+            return self._create_two_phase_executions(processed_args)
+        if has_phase_1_flag:
+            # Phase 1 only: generate pre-alloc groups without filling.
+            return [self._create_phase1_only_execution(processed_args)]
+
+        # Normal single-phase execution.
+        return [
+            PytestExecution(
+                config_file=self.config_path,
+                args=processed_args,
+                allowed_exit_codes=self.allowed_exit_codes,
+            )
+        ]
+
+    def _validate_flag_combinations(self, args: List[str]) -> None:
+        """
+        Reject contradictory flag combinations up front so phase 2 never
+        runs against a missing pre-alloc folder.
+        """
+        if "--use-pre-alloc-groups" not in args:
+            return
+        if "--generate-pre-alloc-groups" in args:
+            raise click.UsageError(
+                "--use-pre-alloc-groups and --generate-pre-alloc-groups "
+                "are mutually exclusive: the first asserts the groups "
+                "already exist on disk, the second regenerates them."
+            )
+        if "--clean" in args:
+            raise click.UsageError(
+                "--use-pre-alloc-groups cannot be combined with --clean: "
+                "--clean wipes the output directory that holds the "
+                "pre-alloc groups."
+            )
+
+    def _create_phase1_only_execution(
+        self, args: List[str]
+    ) -> PytestExecution:
+        """Create a phase-1 execution that generates pre-allocation groups."""
+        return PytestExecution(
+            config_file=self.config_path,
+            args=self._create_phase1_args(args),
+            description="generating pre-allocation groups",
+            allowed_exit_codes=[
+                *self.allowed_exit_codes,
+                pytest.ExitCode.NO_TESTS_COLLECTED,
+            ],
+        )
 
     def _create_two_phase_executions(
         self, args: List[str]
@@ -68,25 +116,11 @@ def _create_two_phase_executions(
         Create two-phase execution: pre-allocation group generation + fixture
         filling.
         """
-        # Phase 1: Pre-allocation group generation (clean and minimal output)
-        phase1_args = self._create_phase1_args(args)
-
-        # Phase 2: Main fixture generation (full user options)
-        phase2_args = self._create_phase2_args(args)
-
         return [
+            self._create_phase1_only_execution(args),
             PytestExecution(
                 config_file=self.config_path,
-                args=phase1_args,
-                description="generating pre-allocation groups",
-                allowed_exit_codes=[
-                    *self.allowed_exit_codes,
-                    pytest.ExitCode.NO_TESTS_COLLECTED,
-                ],
-            ),
-            PytestExecution(
-                config_file=self.config_path,
-                args=phase2_args,
+                args=self._create_phase2_args(args),
                 description="filling test fixtures",
             ),
         ]
@@ -195,13 +229,6 @@ def _add_use_pre_alloc_groups_flag(self, args: List[str]) -> List[str]:
         """Add --use-pre-alloc-groups flag to argument list."""
         return args + ["--use-pre-alloc-groups"]
 
-    def _should_use_two_phase_execution(self, args: List[str]) -> bool:
-        """Determine if two-phase execution is needed."""
-        return (
-            "--generate-pre-alloc-groups" in args
-            or "--generate-all-formats" in args
-        )
-
     def _is_watch_mode(self, args: List[str]) -> bool:
         """Check if any watch flag is present in arguments."""
         return any(flag in args for flag in ["--watch", "--watcherfall"])

packages/testing/src/execution_testing/cli/tests/test_generate_all_formats.py

@@ -2,6 +2,9 @@
 
 from unittest.mock import patch
 
+import click
+import pytest
+
 from execution_testing.cli.pytest_commands.fill import (
     FillCommand,
 )
@@ -76,26 +79,79 @@ def test_generate_all_formats_removes_clean_from_phase2() -> None:
     assert "--clean" not in phase2_args
 
 
-def test_legacy_generate_pre_alloc_groups_still_works() -> None:
-    """Test that the legacy --generate-pre-alloc-groups flag still works."""
+def test_generate_pre_alloc_groups_alone_is_phase_1_only() -> None:
+    """
+    Test that --generate-pre-alloc-groups without --generate-all-formats
+    runs phase 1 only, so CI can populate pre-alloc groups on a
+    dedicated runner without wasting time on phase 2.
+    """
     command = FillCommand()
 
     with patch.object(command, "process_arguments", side_effect=lambda x: x):
         pytest_args = ["--generate-pre-alloc-groups", "tests/somedir/"]
         executions = command.create_executions(pytest_args)
 
-    assert len(executions) == 2
+    assert len(executions) == 1
 
-    # Phase 1: Should have --generate-pre-alloc-groups
     phase1_args = executions[0].args
     assert "--generate-pre-alloc-groups" in phase1_args
+    assert "--use-pre-alloc-groups" not in phase1_args
+    assert "--generate-all-formats" not in phase1_args
 
-    # Phase 2: Should have --use-pre-alloc-groups but NOT --generate-all-
-    # formats
-    phase2_args = executions[1].args
-    assert "--use-pre-alloc-groups" in phase2_args
-    assert "--generate-all-formats" not in phase2_args
-    assert "--generate-pre-alloc-groups" not in phase2_args
+
+def test_use_pre_alloc_groups_forces_single_phase() -> None:
+    """
+    Test that --use-pre-alloc-groups always runs a single phase, even
+    alongside --generate-all-formats (pre-alloc groups already exist on
+    disk from a previous run).
+    """
+    command = FillCommand()
+
+    with patch.object(command, "process_arguments", side_effect=lambda x: x):
+        pytest_args = [
+            "--use-pre-alloc-groups",
+            "--generate-all-formats",
+            "tests/somedir/",
+        ]
+        executions = command.create_executions(pytest_args)
+
+    assert len(executions) == 1
+    assert "--use-pre-alloc-groups" in executions[0].args
+    assert "--generate-all-formats" in executions[0].args
+
+
+def test_use_and_generate_pre_alloc_groups_together_is_rejected() -> None:
+    """
+    --use-pre-alloc-groups + --generate-pre-alloc-groups are contradictory:
+    the first asserts the groups exist, the second regenerates them.
+    """
+    command = FillCommand()
+
+    with patch.object(command, "process_arguments", side_effect=lambda x: x):
+        pytest_args = [
+            "--use-pre-alloc-groups",
+            "--generate-pre-alloc-groups",
+            "tests/somedir/",
+        ]
+        with pytest.raises(click.UsageError, match="mutually exclusive"):
+            command.create_executions(pytest_args)
+
+
+def test_use_pre_alloc_groups_with_clean_is_rejected() -> None:
+    """
+    --use-pre-alloc-groups + --clean is contradictory: --clean wipes the
+    output directory that holds the pre-alloc groups.
+    """
+    command = FillCommand()
+
+    with patch.object(command, "process_arguments", side_effect=lambda x: x):
+        pytest_args = [
+            "--use-pre-alloc-groups",
+            "--clean",
+            "tests/somedir/",
+        ]
+        with pytest.raises(click.UsageError, match="--clean"):
+            command.create_executions(pytest_args)
 
 
 def test_single_phase_without_flags() -> None: