Tighten parser boundaries for deterministic inputs

Author: stacknil

tools/sbom-diff-and-risk/README.md

@@ -9,6 +9,7 @@ It uses conservative heuristics for change intelligence. By default it does not
 - Normalize two local inputs into a shared component schema.
 - Diff components as `added`, `removed`, and `changed`.
 - Apply conservative, heuristic risk buckets to newly added and changed components.
+- Apply optional local policy enforcement over those findings.
 - Produce machine-friendly JSON and reviewer-friendly Markdown reports.
 - Stay fully local-file based by default.
 
@@ -42,13 +43,15 @@ When a `purl` includes a version, the tool keeps the full value in `Component.pu
 - No reputation scoring or malware verdicts.
 - No hidden enrichment or implicit network access.
 - No web UI.
+- No packaged GitHub Marketplace Action.
 
 ## Supported Formats
 
 - CycloneDX JSON
 - SPDX JSON
 - `requirements.txt`
-- `pyproject.toml`
+- `pyproject.toml` via PEP 621 `[project]` metadata
+- `pyproject.toml` dependency groups via PEP 735 `[dependency-groups]` with explicit selection
 
 ## Risk Bucket Semantics
 
@@ -123,13 +126,26 @@ sbom-diff-risk compare \
   --out-md outputs/pyproject-report.md
 ```
 
+Generate reports for a specific PEP 735 dependency group:
+
+```bash
+sbom-diff-risk compare \
+  --before examples/pyproject_groups_before.toml \
+  --after examples/pyproject_groups_after.toml \
+  --format pyproject-toml \
+  --pyproject-group dev \
+  --out-json outputs/pyproject-groups-report.json \
+  --out-md outputs/pyproject-groups-report.md
+```
+
 ## CLI Flags
 
 - `--before path`
 - `--after path`
 - `--format auto|cyclonedx-json|spdx-json|requirements-txt|pyproject-toml`
 - `--before-format cyclonedx-json|spdx-json|requirements-txt|pyproject-toml`
 - `--after-format cyclonedx-json|spdx-json|requirements-txt|pyproject-toml`
+- `--pyproject-group name`
 - `--out-json path`
 - `--out-md path`
 - `--out-sarif path`
@@ -144,18 +160,19 @@ sbom-diff-risk compare \
 
 ## Examples
 
-The [`examples/`](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/examples) directory includes:
+The [examples/](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/examples) directory includes:
 
 - before/after inputs for CycloneDX JSON, SPDX JSON, `requirements.txt`, and `pyproject.toml`
+- dependency-group examples at `examples/pyproject_groups_before.toml` and `examples/pyproject_groups_after.toml`
 - example policies at `examples/policy-minimal.yml` and `examples/policy-strict.yml`
-- a sample CycloneDX-based JSON report at [`sample-report.json`](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/examples/sample-report.json)
-- a sample CycloneDX-based Markdown report at [`sample-report.md`](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/examples/sample-report.md)
-- sample policy-warn reports at [`sample-policy-warn-report.json`](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/examples/sample-policy-warn-report.json) and [`sample-policy-warn-report.md`](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/examples/sample-policy-warn-report.md)
-- sample policy-fail reports at [`sample-policy-fail-report.json`](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/examples/sample-policy-fail-report.json) and [`sample-policy-fail-report.md`](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/examples/sample-policy-fail-report.md)
-- a sample SARIF export at [`sample-sarif.sarif`](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/examples/sample-sarif.sarif)
-- requirements-based sample reports at [`sample-requirements-report.json`](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/examples/sample-requirements-report.json) and [`sample-requirements-report.md`](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/examples/sample-requirements-report.md)
+- a sample pass JSON report at [sample-report.json](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/examples/sample-report.json)
+- a sample pass Markdown report at [sample-report.md](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/examples/sample-report.md)
+- sample policy-warn reports at [sample-policy-warn-report.json](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/examples/sample-policy-warn-report.json) and [sample-policy-warn-report.md](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/examples/sample-policy-warn-report.md)
+- sample policy-fail reports at [sample-policy-fail-report.json](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/examples/sample-policy-fail-report.json) and [sample-policy-fail-report.md](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/examples/sample-policy-fail-report.md)
+- a sample SARIF export at [sample-sarif.sarif](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/examples/sample-sarif.sarif)
+- requirements-based sample reports at [sample-requirements-report.json](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/examples/sample-requirements-report.json) and [sample-requirements-report.md](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/examples/sample-requirements-report.md)
 
-## Enforcement
+## Enforcement Mode
 
 Policy enforcement is optional and deterministic. Exit codes are stable:
 
@@ -207,23 +224,47 @@ sbom-diff-risk compare \
   --out-sarif outputs/report.sarif
 ```
 
-For GitHub code scanning integration guidance and a minimal upload workflow, see [docs/github-code-scanning.md](D:/OneDrive/Code/scientific-computing-toolkit-real/tools/sbom-diff-and-risk/docs/github-code-scanning.md).
+For GitHub code scanning integration guidance and a minimal upload workflow, see [docs/github-code-scanning.md](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/docs/github-code-scanning.md).
+
+## Parser Boundaries
+
+Deterministic local mode intentionally supports a conservative subset of packaging syntax. The detailed matrix lives in [docs/parser-boundaries.md](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/docs/parser-boundaries.md).
+
+### requirements.txt subset
+
+| Syntax | Status | Notes |
+| --- | --- | --- |
+| Plain PEP 508 requirement entries | Supported | Names, specifiers, extras, and markers |
+| Comments, blank lines, line continuations | Supported | Normalized locally without installer behavior |
+| `-r`, `--requirement` | Unsupported | Include chains fail closed |
+| `-c`, `--constraint` | Unsupported | Constraint files fail closed |
+| Editable installs | Unsupported | `-e` and `--editable` are rejected |
+| Direct URL, VCS, and local path refs | Unsupported | Includes `pkg @ https://...`, `git+...`, wheels, archives, and local paths |
+| Index and source options | Unsupported | Includes `--index-url`, `--extra-index-url`, `--find-links`, and related flags |
+
+### pyproject.toml subset
+
+- default parsing supports PEP 621 `[project.dependencies]` and `[project.optional-dependencies]`
+- dependency groups are supported through PEP 735 `[dependency-groups]`
+- dependency groups must be selected explicitly with `--pyproject-group <name>`
+- dependency groups are not treated as aliases for `[project.optional-dependencies]`
+- tool-specific layouts such as Poetry, Hatch, and PDM remain out of scope in v0.2
 
 ## Limitations
 
-- v0.1 is local-file based only.
+- default mode is local-file based only.
 - `generated_at` remains `null` to preserve deterministic report output.
 - `stale_package` is not resolved offline. The report emits `not_evaluated` instead.
-- SARIF export intentionally covers only a conservative subset of findings in v0.1.
+- SARIF export intentionally covers only a conservative subset of findings in v0.2.
 - No vulnerability database integration, CVE matching, or advisory enrichment.
-- `requirements.txt` support intentionally covers a conservative subset: plain PEP 508 requirement entries, comments, direct URL requirements, and line continuations.
-- `requirements.txt` intentionally does not support pip include/constraint directives such as `-r`, `-c`, or arbitrary install flags in v0.1.
-- `pyproject.toml` support intentionally covers a conservative subset: PEP 621 `[project.dependencies]` and `[project.optional-dependencies]`.
-- `pyproject.toml` intentionally does not support tool-specific layouts such as Poetry, Hatch, or PDM sections in v0.1.
+- `requirements.txt` support intentionally covers a conservative subset: plain PEP 508 requirement entries, comments, extras, markers, and line continuations.
+- `requirements.txt` intentionally rejects include/constraint directives, editable installs, direct URL/path refs, index/source options, and other pip-only install flags in deterministic mode.
+- `pyproject.toml` support intentionally covers a conservative subset: PEP 621 `[project.dependencies]`, `[project.optional-dependencies]`, and explicit PEP 735 `[dependency-groups]` selection.
+- `pyproject.toml` intentionally does not support tool-specific layouts such as Poetry, Hatch, or PDM sections in v0.2.
 - Risk buckets are heuristics, not security verdicts.
 - Runtime-generated `outputs/` artifacts are ignored; tracked examples live in `examples/`.
-- Policy files are YAML-only in v0.1 and unknown rule ids fail closed.
+- Policy files are YAML-only in v0.2 and unknown rule ids fail closed.
 
 ## Current Status
 
-The project now normalizes local CycloneDX JSON, SPDX JSON, `requirements.txt`, and PEP 621 `pyproject.toml` inputs into the shared component model, diffs them deterministically, and generates stable JSON/Markdown/SARIF reports with golden tests and optional policy enforcement.
+The project now normalizes local CycloneDX JSON, SPDX JSON, `requirements.txt`, and conservative `pyproject.toml` inputs, including explicit PEP 735 dependency-group selection, into the shared component model, diffs them deterministically, and generates stable JSON/Markdown/SARIF reports with tests and optional policy enforcement.

tools/sbom-diff-and-risk/docs/parser-boundaries.md

@@ -0,0 +1,49 @@
+# Parser boundaries
+
+`sbom-diff-and-risk` intentionally supports a conservative parser subset so local runs remain deterministic, auditable, and CI-friendly.
+
+The project does not try to emulate a package installer. When syntax would require resolver behavior, implicit includes, index lookups, or environment-specific side effects, the parser fails closed with an explicit error.
+
+## requirements.txt
+
+`requirements.txt` is treated as a narrow manifest format, not as "everything pip can do in a file".
+
+| Syntax | Status | Notes |
+| --- | --- | --- |
+| Plain PEP 508 names and version specifiers | Supported | Example: `requests==2.31.0` |
+| Extras and markers | Supported | Example: `pytest[testing]>=8.0 ; python_version >= "3.11"` |
+| Comments and blank lines | Supported | Stripped before parsing |
+| Line continuations | Supported | Continued lines are joined deterministically |
+| `-r`, `--requirement` | Unsupported | Include chains are rejected |
+| `-c`, `--constraint` | Unsupported | Constraint files are rejected |
+| `-e`, `--editable` | Unsupported | Editable installs are rejected |
+| Direct URL, VCS, or local path references | Unsupported | Includes `pkg @ https://...`, `git+...`, `file://...`, wheels, and local archives |
+| Index and source options | Unsupported | Includes `--index-url`, `--extra-index-url`, `--find-links`, `--trusted-host`, `--no-index` |
+| Other pip-only install flags | Unsupported | Includes hash flags, binary toggles, prerelease flags, and related installer controls |
+
+When unsupported syntax appears, the parser raises `UnsupportedInputError` and the CLI returns exit code `2`.
+
+## pyproject.toml
+
+`pyproject.toml` support is also intentionally narrow:
+
+| Section | Status | Notes |
+| --- | --- | --- |
+| `[project.dependencies]` | Supported | Parsed by default |
+| `[project.optional-dependencies]` | Supported | Parsed by default and kept distinct from dependency groups |
+| `[dependency-groups]` | Supported | Requires explicit `--pyproject-group <name>` selection |
+| `{ include-group = "name" }` inside dependency groups | Supported | Includes are resolved locally and deterministically |
+| Missing requested dependency group | Explicit error | Reported as `InputSelectionError` |
+| Poetry, Hatch, PDM, or other tool-specific dependency sections | Unsupported | Not parsed in v0.2 |
+
+Dependency groups are not merged automatically with `[project.optional-dependencies]`. They solve different problems and are kept separate on purpose.
+
+## Error taxonomy
+
+The parser uses explicit error classes so CI logs are understandable:
+
+- `MalformedInputError`: the file is syntactically malformed.
+- `UnsupportedInputError`: the file is valid enough to read, but deterministic mode intentionally does not support the construct.
+- `InputSelectionError`: the user asked for a parser selection the input cannot satisfy, such as a missing dependency group.
+
+The CLI maps these parser failures to exit code `2`.

tools/sbom-diff-and-risk/docs/sbom-basics.md

@@ -2,7 +2,7 @@
 
 This project treats SBOMs as one possible source of dependency inventory data.
 
-For v0.1, the tool is intentionally limited to local-file parsing, normalization, diffing, and heuristic reporting.
+For v0.2, the tool is intentionally limited to local-file parsing, normalization, diffing, and conservative heuristic reporting.
 
 ## Supported local inputs
 
@@ -17,18 +17,20 @@ For v0.1, the tool is intentionally limited to local-file parsing, normalization
 
 - supported: plain PEP 508 requirement entries
 - supported: comments and blank lines
-- supported: direct URL requirements
 - supported: line continuations
-- not supported: `-r`, `--requirement`, `-c`, `--constraint`, or arbitrary pip install flags
+- not supported: `-r`, `--requirement`, `-c`, `--constraint`, editable installs, direct URL/path refs, or pip index/options
 
-`pyproject.toml` support is intentionally conservative in v0.1:
+`pyproject.toml` support is intentionally conservative in v0.2:
 
 - supported: PEP 621 `[project.dependencies]`
 - supported: PEP 621 `[project.optional-dependencies]`
+- supported: PEP 735 `[dependency-groups]` with explicit `--pyproject-group` selection
 - not supported: Poetry, Hatch, PDM, or other tool-specific dependency sections
 
 These boundaries are deliberate so the tool can stay deterministic and explicit about what it does and does not parse.
 
+For the detailed supported/unsupported matrix, see [parser-boundaries.md](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/docs/parser-boundaries.md).
+
 ## Normalization goals
 
 - keep one internal `Component` model

tools/sbom-diff-and-risk/examples/pyproject_groups_after.toml

@@ -0,0 +1,25 @@
+[project]
+name = "grouped-project"
+version = "0.2.0"
+dependencies = [
+  "requests==2.32.0",
+]
+
+[project.optional-dependencies]
+docs = [
+  "mkdocs>=1.6",
+]
+
+[dependency-groups]
+lint = [
+  "ruff==0.6.3",
+]
+dev = [
+  "pytest==8.3.1",
+  "mypy==1.11.2",
+  { include-group = "lint" },
+]
+test = [
+  "pytest-cov==5.0.0",
+  { include-group = "dev" },
+]

tools/sbom-diff-and-risk/examples/pyproject_groups_before.toml

@@ -0,0 +1,24 @@
+[project]
+name = "grouped-project"
+version = "0.1.0"
+dependencies = [
+  "requests==2.31.0",
+]
+
+[project.optional-dependencies]
+docs = [
+  "mkdocs>=1.6",
+]
+
+[dependency-groups]
+lint = [
+  "ruff==0.5.0",
+]
+dev = [
+  "pytest==8.2.0",
+  { include-group = "lint" },
+]
+test = [
+  "pytest-cov==5.0.0",
+  { include-group = "dev" },
+]

tools/sbom-diff-and-risk/src/sbom_diff_risk/cli.py

@@ -8,7 +8,7 @@
 from .diffing import diff_components
 from .errors import ParseError, PolicyError
 from .models import CompareReport, ReportComponents, ReportMetadata, ReportSummary
-from .normalize import SUPPORTED_FORMATS, normalize_input
+from .normalize import SUPPORTED_FORMATS, normalize_input_with_options
 from .policy_evaluator import evaluate_policy
 from .policy_parser import build_policy
 from .presentation import effective_policy_evaluation, summarize_violations_by_rule
@@ -51,6 +51,11 @@ def build_parser() -> argparse.ArgumentParser:
         default=None,
         help="Explicit format for the after input.",
     )
+    compare.add_argument(
+        "--pyproject-group",
+        default=None,
+        help="Select a PEP 735 [dependency-groups] group when a compared input is pyproject.toml.",
+    )
     compare.add_argument("--out-json", type=Path, default=None, help="Write a JSON report to this path.")
     compare.add_argument("--out-md", type=Path, default=None, help="Write a Markdown report to this path.")
     compare.add_argument(
@@ -107,6 +112,8 @@ def main(argv: Sequence[str] | None = None) -> int:
 
 
 def run_compare(args: argparse.Namespace) -> int:
+    pyproject_group = getattr(args, "pyproject_group", None)
+
     if args.enrich_pypi:
         raise NotImplementedError("--enrich-pypi is reserved for a later network-enabled release.")
 
@@ -122,8 +129,18 @@ def run_compare(args: argparse.Namespace) -> int:
 
     before_declared = _resolve_declared_format(args.format, args.before_format)
     after_declared = _resolve_declared_format(args.format, args.after_format)
-    before_format, before_components, before_notes = normalize_input(before_path, before_declared)
-    after_format, after_components, after_notes = normalize_input(after_path, after_declared)
+    before_format, before_components, before_notes = normalize_input_with_options(
+        before_path,
+        declared_format=before_declared,
+        pyproject_group=pyproject_group,
+    )
+    after_format, after_components, after_notes = normalize_input_with_options(
+        after_path,
+        declared_format=after_declared,
+        pyproject_group=pyproject_group,
+    )
+    if pyproject_group and before_format != "pyproject-toml" and after_format != "pyproject-toml":
+        raise ValueError("--pyproject-group requires at least one pyproject.toml input.")
     policy, policy_path = build_policy(policy_path=args.policy, fail_on=args.fail_on, warn_on=args.warn_on)
 
     added, removed, changed = diff_components(before_components, after_components)

tools/sbom-diff-and-risk/src/sbom_diff_risk/errors.py

@@ -5,5 +5,17 @@ class ParseError(ValueError):
     """Raised when an input file cannot be parsed into normalized components."""
 
 
+class MalformedInputError(ParseError):
+    """Raised when an input is syntactically malformed."""
+
+
+class UnsupportedInputError(ParseError):
+    """Raised when deterministic mode rejects otherwise valid input syntax."""
+
+
+class InputSelectionError(ParseError):
+    """Raised when an explicit parser selection cannot be satisfied."""
+
+
 class PolicyError(ValueError):
-    """Raised when a policy file or policy override is invalid."""
+    """Raised when policy parsing or evaluation inputs are invalid."""

tools/sbom-diff-and-risk/src/sbom_diff_risk/normalize.py

@@ -47,11 +47,23 @@ def detect_format(path: Path) -> str:
 
 
 def normalize_input(path: Path, declared_format: str | None = None) -> tuple[str, list[Component], list[str]]:
+    return normalize_input_with_options(path, declared_format=declared_format, pyproject_group=None)
+
+
+def normalize_input_with_options(
+    path: Path,
+    *,
+    declared_format: str | None = None,
+    pyproject_group: str | None = None,
+) -> tuple[str, list[Component], list[str]]:
     selected_format = declared_format or detect_format(path)
     if selected_format not in SUPPORTED_FORMATS:
         raise ValueError(
             f"Unsupported input format {selected_format!r}. Supported formats: {', '.join(SUPPORTED_FORMATS)}."
         )
 
+    if selected_format == "pyproject-toml":
+        return selected_format, pyproject_toml.parse(path, dependency_group=pyproject_group), []
+
     parser = _FORMAT_PARSERS[selected_format]
     return selected_format, parser(path), []