Allow exclude with playlist-file; update docstrings throughout

exclude is no longer restricted to the inline include form — it now applies
to both base-set forms (include and playlist-file). The exclude logic was
lifted out of _resolve_inline (now removed) into _compute_scenario_ids so
it runs uniformly after the base set is built.

- _check_constraints: 'exclude' now requires include OR playlist-file
- _load_playlist: returns Set[int] instead of List[int]; sorting deferred
  to _compute_scenario_ids after exclude is applied
- _compute_scenario_ids: single unified method handles both forms + exclude
- Warning message updated: 'include set' → 'base set'

Docstrings updated:
- ScenarioScopeConfig: full class docstring documenting both forms,
  exclude compatibility, caching behaviour, and index convention
- _expand_entries: clarified entry formats and boolean handling note
- validate_optim_config: enumerated bullet-point list of all checks
- ScenarioBuilder.validate_mc_scenarios: documents return contract

Four new tests cover playlist-file + exclude (basic, range, all-excluded,
orphan warning). Updated test for the relaxed exclude-without-base error.

https://claude.ai/code/session_01EfjU9tBfdyZ9nZmQFCe5uU

Author: claude

docs/CHANGELOG.md

@@ -37,6 +37,9 @@ scenario-scope:
 
 Other changes:
 
+- `exclude` is now compatible with both `include` and `playlist-file`.
+  Use it to subtract a few scenarios at run time without modifying the
+  playlist file.
 - `validate_optim_config()` now accepts an optional `scenario_builder`
   argument and cross-checks all playlist indices against every scenario group,
   raising a `ValueError` for out-of-bounds indices.

docs/user-guide/optim-config.md

@@ -70,7 +70,9 @@ The total number of timesteps solved is `last-time-step − first-time-step + 1`
 Selects which Monte-Carlo scenarios to simulate.  Indices are **0-based**,
 consistent with the `modeler-scenariobuilder.dat` file convention.
 
-Two mutually exclusive forms are supported:
+The base scenario set is defined by exactly one of two mutually exclusive
+keys: `include` (inline) or `playlist-file` (from a JSON file).  `exclude`
+is optional and applies to **either** form.
 
 ### Inline form (`include` / `exclude`)
 
@@ -80,7 +82,7 @@ integers, and inclusive `"a-b"` range strings.
 | Key | Type | Default | Description |
 |---|---|---|---|
 | `include` | list | — | Scenarios to run (required in inline form) |
-| `exclude` | list | — | Scenarios to remove from the include set (optional) |
+| `exclude` | list | — | Scenarios to remove from the base set (optional) |
 
 Each entry in `include` or `exclude` may be:
 
@@ -115,9 +117,9 @@ scenario-scope:
 
 - All indices must be ≥ 0.
 - Overlapping entries in `include` are deduplicated automatically.
-- Excludes that do not appear in `include` produce a warning and have no effect.
+- Excludes that do not appear in the base set produce a warning and have no effect.
 - Output is always sorted in ascending order.
-- `exclude` cannot be used without `include`.
+- `exclude` cannot be used without `include` or `playlist-file`.
 
 **Default behaviour** (no `scenario-scope` key at all, or an empty block):
 runs scenario 0 only.
@@ -126,7 +128,7 @@ runs scenario 0 only.
 
 ### Playlist-file form (`playlist-file`)
 
-Point to a JSON file containing a flat array of 1-based integer scenario
+Point to a JSON file containing a flat array of 0-based integer scenario
 indices.  Useful when the list of scenarios is generated programmatically or
 is too large to embed in YAML.
 
@@ -139,11 +141,21 @@ scenario-scope:
   playlist-file: mc_playlist.json   # resolved relative to optim-config.yml
 ~~~
 
-The referenced file must contain a flat JSON array of positive integers, for
-example:
+The referenced file must contain a flat JSON array of non-negative integers:
 
 ~~~ json
-[1, 3, 5, 7, 9, 11, 13]
+[0, 2, 4, 6, 8, 10, 12]
+~~~
+
+`exclude` can be combined with `playlist-file` to subtract specific scenarios
+at run time without modifying the file:
+
+~~~ yaml
+scenario-scope:
+  playlist-file: mc_playlist.json
+  exclude:
+    - 4
+    - "8-10"
 ~~~
 
 GemsPy reads and validates the playlist eagerly when `load_optim_config()` is
@@ -154,7 +166,7 @@ called, so any I/O or format errors surface immediately at load time.
 - The file must be a flat JSON array of integers (no booleans, strings, or objects).
 - All indices must be ≥ 0.
 - Duplicates are silently removed; the result is sorted ascending.
-- `include` / `exclude` and `playlist-file` are mutually exclusive.
+- `include` and `playlist-file` are mutually exclusive.
 
 ---
 

src/gems/optim_config/parsing.py

@@ -136,7 +136,16 @@ def parsed_parameters(self) -> Dict[str, Any]:
 
 
 def _expand_entries(entries: List[Union[int, str]]) -> Set[int]:
-    """Expand 0-based ints and inclusive 'a-b' range strings into a set of 0-based indices."""
+    """Expand a list of scenario specifiers into a set of 0-based indices.
+
+    Each entry may be:
+    - a non-negative integer (e.g. ``5``),
+    - a string integer (e.g. ``"5"``), or
+    - an inclusive range string (e.g. ``"0-9"``).
+
+    Booleans are rejected upstream by the Pydantic field validator and will
+    never reach this function.
+    """
     result: Set[int] = set()
     for entry in entries:
         if isinstance(entry, int):
@@ -163,6 +172,33 @@ def _expand_entries(entries: List[Union[int, str]]) -> Set[int]:
 
 
 class ScenarioScopeConfig(ModifiedBaseModel):
+    """Declares which Monte-Carlo scenarios to simulate.
+
+    Two mutually exclusive ways to define the base scenario set:
+
+    - **Inline** (``include``): a list of 0-based integers, string-integers,
+      and/or ``"a-b"`` range strings.
+    - **File** (``playlist_file``): path to a flat JSON array of 0-based
+      integers, resolved relative to ``optim-config.yml`` by
+      ``load_optim_config()``.
+
+    ``exclude`` is optional and compatible with *both* forms.  It subtracts a
+    set of scenarios from the base set using the same entry format.  Entries
+    in ``exclude`` that are not in the base set are silently ignored (a
+    ``UserWarning`` is emitted).
+
+    ``include`` and ``playlist_file`` are mutually exclusive.
+    ``exclude`` without any base set raises ``ValueError``.
+
+    All indices are 0-based, consistent with
+    ``modeler-scenariobuilder.dat``.
+
+    The resolved list is computed lazily on first access to
+    ``scenario_ids`` and cached for the lifetime of the object.
+    ``load_optim_config()`` triggers eager resolution so that file I/O
+    errors surface at load time.
+    """
+
     include: Optional[List[Union[int, str]]] = None
     exclude: Optional[List[Union[int, str]]] = None
     playlist_file: Optional[Path] = None
@@ -186,8 +222,8 @@ def _check_constraints(self) -> "ScenarioScopeConfig":
         has_file = self.playlist_file is not None
         if has_inline and has_file:
             raise ValueError("'include' and 'playlist-file' are mutually exclusive")
-        if self.exclude is not None and not has_inline:
-            raise ValueError("'exclude' requires 'include'")
+        if self.exclude is not None and not has_inline and not has_file:
+            raise ValueError("'exclude' requires 'include' or 'playlist-file'")
         return self
 
     @property
@@ -198,12 +234,27 @@ def scenario_ids(self) -> List[int]:
 
     def _compute_scenario_ids(self) -> List[int]:
         if self.playlist_file is not None:
-            return self._load_playlist()
-        if self.include is None:
+            included = self._load_playlist()
+        elif self.include is not None:
+            included = _expand_entries(self.include)
+        else:
             return [0]
-        return self._resolve_inline()
 
-    def _load_playlist(self) -> List[int]:
+        if self.exclude is not None:
+            excluded = _expand_entries(self.exclude)
+            orphans = excluded - included
+            if orphans:
+                warnings.warn(
+                    f"Excluded scenario indices {sorted(orphans)} "
+                    "are not in the base set and have no effect",
+                    UserWarning,
+                    stacklevel=2,
+                )
+            included -= excluded
+
+        return sorted(included)
+
+    def _load_playlist(self) -> Set[int]:
         try:
             with self.playlist_file.open() as f:  # type: ignore[union-attr]
                 data = json.load(f)
@@ -223,20 +274,7 @@ def _load_playlist(self) -> List[int]:
             raise ValueError(
                 f"'{self.playlist_file}': all scenario indices must be >= 0"
             )
-        return sorted(set(data))
-
-    def _resolve_inline(self) -> List[int]:
-        included = _expand_entries(self.include or [])
-        excluded = _expand_entries(self.exclude or [])
-        orphans = excluded - included
-        if orphans:
-            warnings.warn(
-                f"Excluded scenario indices {sorted(orphans)} "
-                "are not in the include set and have no effect",
-                UserWarning,
-                stacklevel=2,
-            )
-        return sorted(included - excluded)
+        return set(data)
 
 
 class OptimConfig(ModifiedBaseModel):
@@ -416,12 +454,17 @@ def validate_optim_config(
 ) -> None:
     """Cross-validate optim-config entries against the resolved system.
 
-    Checks that every referenced ID exists, that master variables do not
-    depend on time, and that master constraints and objectives only reference
-    variables assigned to master or master-and-subproblems.
-    When scenario_builder is provided, also verifies that all scenario indices
-    from the playlist are defined in every scenario group.
-    Raises ValueError listing all violations.
+    Performs the following checks:
+
+    - Every model ID referenced in ``config.models`` exists in the system.
+    - Master variables are time-independent.
+    - Master constraints and objective contributions only reference variables
+      assigned to ``master`` or ``master-and-subproblems``.
+    - If ``scenario_builder`` is provided, every scenario index in
+      ``config.scenario_scope.scenario_ids`` is defined for every scenario
+      group in the builder.
+
+    Raises ``ValueError`` listing all violations found.
     """
     models_in_system = {c.model.id: c.model for c in system.all_components}
     errors: List[str] = []

src/gems/study/scenario_builder.py

@@ -68,7 +68,14 @@ def resolve_vectorized(
         return arr[mc_scenarios]
 
     def validate_mc_scenarios(self, scenario_ids: List[int]) -> List[str]:
-        """Return error messages for scenario indices not defined in any group."""
+        """Return error messages for scenario indices out of range in any group.
+
+        For each scenario group, checks that every index in ``scenario_ids``
+        is within ``[0, len(group_array) - 1]``.  Returns a list of
+        human-readable error strings (one per offending group); an empty list
+        means all indices are valid.  Does not raise — callers collect errors
+        and raise at the end.
+        """
         errors = []
         for group, arr in self._group_arrays.items():
             out_of_bounds = [idx for idx in scenario_ids if idx >= len(arr)]

tests/unittests/optim_config/test_scenario_scope_config.py

@@ -88,11 +88,51 @@ def test_exclude_orphan_raises_warning() -> None:
     assert "4" in str(caught[0].message)
 
 
-def test_exclude_without_include_raises() -> None:
-    with pytest.raises(ValueError, match="'exclude' requires 'include'"):
+def test_exclude_without_any_base_raises() -> None:
+    with pytest.raises(
+        ValueError, match="'exclude' requires 'include' or 'playlist-file'"
+    ):
         ScenarioScopeConfig(exclude=[0])
 
 
+# ---------------------------------------------------------------------------
+# playlist-file + exclude
+# ---------------------------------------------------------------------------
+
+
+def test_playlist_file_with_exclude(tmp_path: Path) -> None:
+    playlist = tmp_path / "playlist.json"
+    playlist.write_text(json.dumps([0, 1, 2, 3, 4]))
+    cfg = ScenarioScopeConfig(playlist_file=playlist, exclude=[2, 4])
+    assert cfg.scenario_ids == [0, 1, 3]
+
+
+def test_playlist_file_with_exclude_range(tmp_path: Path) -> None:
+    playlist = tmp_path / "playlist.json"
+    playlist.write_text(json.dumps([0, 1, 2, 3, 4, 5, 6]))
+    cfg = ScenarioScopeConfig(playlist_file=playlist, exclude=["2-4"])
+    assert cfg.scenario_ids == [0, 1, 5, 6]
+
+
+def test_playlist_file_with_exclude_all_leaves_empty(tmp_path: Path) -> None:
+    playlist = tmp_path / "playlist.json"
+    playlist.write_text(json.dumps([0, 1, 2]))
+    cfg = ScenarioScopeConfig(playlist_file=playlist, exclude=["0-2"])
+    assert cfg.scenario_ids == []
+
+
+def test_playlist_file_with_exclude_orphan_warns(tmp_path: Path) -> None:
+    playlist = tmp_path / "playlist.json"
+    playlist.write_text(json.dumps([0, 1, 2]))
+    cfg = ScenarioScopeConfig(playlist_file=playlist, exclude=[5])
+    with warnings.catch_warnings(record=True) as caught:
+        warnings.simplefilter("always")
+        result = cfg.scenario_ids
+    assert result == [0, 1, 2]
+    assert len(caught) == 1
+    assert "5" in str(caught[0].message)
+
+
 # ---------------------------------------------------------------------------
 # Inline form — validation errors
 # ---------------------------------------------------------------------------