Add benchmark scenario guidance (#3531)

Author: koxudaxi

docs/performance-benchmarks.md

@@ -6,6 +6,24 @@ This page tracks datamodel-code-generator release and main-branch benchmark resu
 
 datamodel-code-generator supports many schema styles and production use cases, so it includes a broad set of useful options. As releases add more capabilities, these benchmarks help keep the implementation measured, managed, and tuned so code generation stays fast in everyday use.
 
+## Scenario Guide
+
+Each scenario combines an input type with a case size. This guide is generated from the scenario keys in the benchmark JSON and the collector case definitions, so it changes when the benchmark matrix changes.
+
+| Scenario | Input fixture | Formatters | Represents |
+| --- | --- | --- | --- |
+| Small / JSON Schema | `tests/data/jsonschema/person.json` | `default` (black/isort default), `builtin` (Built-in), `ruff` (Ruff) | Compact fixture that emphasizes CLI startup, parsing, and formatter overhead for JSON Schema to Pydantic v2 model generation. |
+| Small / OpenAPI | `tests/data/openapi/api.yaml` | `default` (black/isort default), `builtin` (Built-in), `ruff` (Ruff) | Compact fixture that emphasizes CLI startup, parsing, and formatter overhead for OpenAPI component resolution and Pydantic v2 model generation. |
+| Large / JSON Schema | `tests/data/performance/large_models.json` | `default` (black/isort default), `builtin` (Built-in), `ruff` (Ruff) | Larger fixture that emphasizes parser and model graph throughput for JSON Schema to Pydantic v2 model generation. |
+| Large / OpenAPI | `tests/data/performance/openapi_large.yaml` | `default` (black/isort default), `builtin` (Built-in), `ruff` (Ruff) | Larger fixture that emphasizes parser and model graph throughput for OpenAPI component resolution and Pydantic v2 model generation. |
+
+## Interpreting Metrics
+
+- `median_ms` is the primary comparison value: it is the median generation duration after warmup runs, and lower is faster.
+- `min_ms`, `max_ms`, and `stdev_ms` describe the measured spread for the same row; wide ranges usually mean CI runner noise rather than a deliberate code change.
+- Formatter comparisons are scoped to the same scenario and version. `default` is the black/isort default baseline, while `builtin` and `ruff` ratios compare their medians to that baseline.
+- `ok` rows have timing data. `unsupported` means the formatter or option was unavailable in that release. `failed` means installation or command execution failed, so timing cells are intentionally empty.
+
 ## Historical Trends
 
 The benchmark data on this page is loaded from `docs/data/release-benchmarks.json` and rendered in the browser. Release runs update that JSON file directly, so the published page reflects new release measurements after the docs deployment without rewriting this Markdown file.

scripts/build_release_benchmark_docs.py

@@ -15,13 +15,15 @@
 from typing import Any
 
 try:
+    from scripts.collect_release_benchmarks import BENCHMARK_CASES, DEFAULT_FORMATTERS
     from scripts.release_benchmark_safety import (
         SAFE_FIELD_PATTERN,
         safe_optional_field,
         safe_release_version,
         validated_benchmark_entry,
     )
 except ModuleNotFoundError:  # pragma: no cover - direct script execution
+    from collect_release_benchmarks import BENCHMARK_CASES, DEFAULT_FORMATTERS
     from release_benchmark_safety import (
         SAFE_FIELD_PATTERN,
         safe_optional_field,
@@ -33,6 +35,28 @@
 DATA_PATH = ROOT / "docs" / "data" / "release-benchmarks.json"
 NOTES_PATH = ROOT / "docs" / "data" / "release-benchmark-notes.json"
 DOCS_PATH = ROOT / "docs" / "performance-benchmarks.md"
+CASE_ORDER = ("small", "large")
+CASE_LABELS = {
+    "small": "Small",
+    "large": "Large",
+}
+CASE_GUIDANCE = {
+    "small": "Compact fixture that emphasizes CLI startup, parsing, and formatter overhead",
+    "large": "Larger fixture that emphasizes parser and model graph throughput",
+}
+INPUT_TYPE_LABELS = {
+    "jsonschema": "JSON Schema",
+    "openapi": "OpenAPI",
+}
+INPUT_TYPE_GUIDANCE = {
+    "jsonschema": "JSON Schema to Pydantic v2 model generation",
+    "openapi": "OpenAPI component resolution and Pydantic v2 model generation",
+}
+FORMATTER_LABELS = {
+    "default": "black/isort default",
+    "builtin": "Built-in",
+    "ruff": "Ruff",
+}
 
 
 @dataclass(frozen=True, slots=True)
@@ -89,6 +113,40 @@ def _integer(value: object, *, default: int = 0) -> int:
             return default
 
 
+def _label(value: str) -> str:
+    return " ".join(part.capitalize() for part in value.replace("_", "-").split("-") if part) or value
+
+
+def _case_label(case: str) -> str:
+    return CASE_LABELS.get(case, _label(case))
+
+
+def _input_type_label(input_type: str) -> str:
+    return INPUT_TYPE_LABELS.get(input_type, _label(input_type))
+
+
+def _formatter_label(formatter: str) -> str:
+    return FORMATTER_LABELS.get(formatter, _label(formatter))
+
+
+def _order_key(value: str, order: tuple[str, ...]) -> tuple[int, str]:
+    try:
+        return order.index(value), value
+    except ValueError:
+        return len(order), value
+
+
+def _benchmark_case_paths() -> dict[tuple[str, str], str]:
+    paths: dict[tuple[str, str], str] = {}
+    for benchmark_case in BENCHMARK_CASES:
+        try:
+            input_path = benchmark_case.path.relative_to(ROOT)
+        except ValueError:
+            input_path = benchmark_case.path
+        paths[benchmark_case.input_type, benchmark_case.name] = input_path.as_posix()
+    return paths
+
+
 def _entry_from_raw(raw: object, *, path: Path, index: int) -> BenchmarkEntry:
     if not isinstance(raw, dict):
         msg = f"Benchmark entry #{index} must be an object"
@@ -156,6 +214,100 @@ def load_benchmark_data(path: Path = DATA_PATH, *, notes_path: Path | None = NOT
     )
 
 
+def _scenario_sort_key(scenario: tuple[str, str]) -> tuple[tuple[int, str], str]:
+    input_type, case = scenario
+    return _order_key(case, CASE_ORDER), input_type
+
+
+def _scenario_keys(entries: tuple[BenchmarkEntry, ...]) -> tuple[tuple[str, str], ...]:
+    scenarios = {(entry.input_type, entry.case) for entry in entries}
+    return tuple(sorted(scenarios, key=_scenario_sort_key))
+
+
+def _scenario_formatters(data: BenchmarkData, *, input_type: str, case: str) -> tuple[str, ...]:
+    formatters = {entry.formatter for entry in data.entries if entry.input_type == input_type and entry.case == case}
+    return tuple(sorted(formatters, key=lambda formatter: _order_key(formatter, DEFAULT_FORMATTERS)))
+
+
+def _scenario_description(input_type: str, case: str) -> str:
+    case_guidance = CASE_GUIDANCE.get(case, f"{_case_label(case)} fixture")
+    input_guidance = INPUT_TYPE_GUIDANCE.get(input_type, f"{_input_type_label(input_type)} generation")
+    return f"{case_guidance} for {input_guidance}."
+
+
+def _scenario_fixture(input_type: str, case: str, benchmark_case_paths: dict[tuple[str, str], str]) -> str:
+    if input_path := benchmark_case_paths.get((input_type, case)):
+        return f"`{input_path}`"
+    return "not declared by `scripts/collect_release_benchmarks.py`"
+
+
+def _markdown_table_cell(value: str) -> str:
+    return value.replace("|", "\\|")
+
+
+def _scenario_table_lines(data: BenchmarkData) -> tuple[str, ...]:
+    benchmark_case_paths = _benchmark_case_paths()
+    lines = [
+        "| Scenario | Input fixture | Formatters | Represents |",
+        "| --- | --- | --- | --- |",
+    ]
+    for input_type, case in _scenario_keys(data.entries):
+        formatter_labels = ", ".join(
+            f"`{formatter}` ({_formatter_label(formatter)})"
+            for formatter in _scenario_formatters(data, input_type=input_type, case=case)
+        )
+        row = (
+            f"{_case_label(case)} / {_input_type_label(input_type)}",
+            _scenario_fixture(input_type, case, benchmark_case_paths),
+            formatter_labels,
+            _scenario_description(input_type, case),
+        )
+        lines.append("| " + " | ".join(_markdown_table_cell(cell) for cell in row) + " |")
+    return tuple(lines)
+
+
+def render_scenario_guidance(data: BenchmarkData) -> tuple[str, ...]:
+    """Render static benchmark scenario and metric guidance from the loaded data."""
+    lines = [
+        "## Scenario Guide",
+        "",
+        (
+            "Each scenario combines an input type with a case size. This guide is generated from the scenario keys "
+            "in the benchmark JSON and the collector case definitions, so it changes when the benchmark matrix "
+            "changes."
+        ),
+        "",
+    ]
+    if data.entries:
+        lines.extend(_scenario_table_lines(data))
+    else:
+        lines.append("No benchmark scenarios have been collected yet.")
+    lines.extend((
+        "",
+        "## Interpreting Metrics",
+        "",
+        (
+            "- `median_ms` is the primary comparison value: it is the median generation duration after warmup runs, "
+            "and lower is faster."
+        ),
+        (
+            "- `min_ms`, `max_ms`, and `stdev_ms` describe the measured spread for the same row; wide ranges usually "
+            "mean CI runner noise rather than a deliberate code change."
+        ),
+        (
+            "- Formatter comparisons are scoped to the same scenario and version. `default` is the "
+            "black/isort default baseline, while `builtin` and `ruff` ratios compare their medians to that baseline."
+        ),
+        (
+            "- `ok` rows have timing data. `unsupported` means the formatter or option was unavailable in that "
+            "release. `failed` means installation or command execution failed, so timing cells are intentionally "
+            "empty."
+        ),
+        "",
+    ))
+    return tuple(lines)
+
+
 def render_release_benchmark_markdown(data: BenchmarkData) -> str:
     """Render the release benchmark documentation shell."""
     data_state = "available" if data.entries else "empty"
@@ -178,6 +330,7 @@ def render_release_benchmark_markdown(data: BenchmarkData) -> str:
             "measured, managed, and tuned so code generation stays fast in everyday use."
         ),
         "",
+        *render_scenario_guidance(data),
         "## Historical Trends",
         "",
         (

tests/data/expected/release_benchmark_docs/release_benchmark_cli_outputs.txt

@@ -10,6 +10,23 @@ This page tracks datamodel-code-generator release and main-branch benchmark resu
 
 datamodel-code-generator supports many schema styles and production use cases, so it includes a broad set of useful options. As releases add more capabilities, these benchmarks help keep the implementation measured, managed, and tuned so code generation stays fast in everyday use.
 
+## Scenario Guide
+
+Each scenario combines an input type with a case size. This guide is generated from the scenario keys in the benchmark JSON and the collector case definitions, so it changes when the benchmark matrix changes.
+
+| Scenario | Input fixture | Formatters | Represents |
+| --- | --- | --- | --- |
+| Small / JSON Schema | `tests/data/jsonschema/person.json` | `default` (black/isort default), `builtin` (Built-in), `ruff` (Ruff) | Compact fixture that emphasizes CLI startup, parsing, and formatter overhead for JSON Schema to Pydantic v2 model generation. |
+| Large / JSON Schema | `tests/data/performance/large_models.json` | `builtin` (Built-in) | Larger fixture that emphasizes parser and model graph throughput for JSON Schema to Pydantic v2 model generation. |
+| Large / OpenAPI | `tests/data/performance/openapi_large.yaml` | `builtin` (Built-in), `ruff` (Ruff) | Larger fixture that emphasizes parser and model graph throughput for OpenAPI component resolution and Pydantic v2 model generation. |
+
+## Interpreting Metrics
+
+- `median_ms` is the primary comparison value: it is the median generation duration after warmup runs, and lower is faster.
+- `min_ms`, `max_ms`, and `stdev_ms` describe the measured spread for the same row; wide ranges usually mean CI runner noise rather than a deliberate code change.
+- Formatter comparisons are scoped to the same scenario and version. `default` is the black/isort default baseline, while `builtin` and `ruff` ratios compare their medians to that baseline.
+- `ok` rows have timing data. `unsupported` means the formatter or option was unavailable in that release. `failed` means installation or command execution failed, so timing cells are intentionally empty.
+
 ## Historical Trends
 
 The benchmark data on this page is loaded from `docs/data/release-benchmarks.json` and rendered in the browser. Release runs update that JSON file directly, so the published page reflects new release measurements after the docs deployment without rewriting this Markdown file.

tests/data/expected/release_benchmark_docs/release_benchmark_escaped_hostile_text.txt

@@ -6,6 +6,21 @@ This page tracks datamodel-code-generator release and main-branch benchmark resu
 
 datamodel-code-generator supports many schema styles and production use cases, so it includes a broad set of useful options. As releases add more capabilities, these benchmarks help keep the implementation measured, managed, and tuned so code generation stays fast in everyday use.
 
+## Scenario Guide
+
+Each scenario combines an input type with a case size. This guide is generated from the scenario keys in the benchmark JSON and the collector case definitions, so it changes when the benchmark matrix changes.
+
+| Scenario | Input fixture | Formatters | Represents |
+| --- | --- | --- | --- |
+| Small / OpenAPI | `tests/data/openapi/api.yaml` | `default` (black/isort default) | Compact fixture that emphasizes CLI startup, parsing, and formatter overhead for OpenAPI component resolution and Pydantic v2 model generation. |
+
+## Interpreting Metrics
+
+- `median_ms` is the primary comparison value: it is the median generation duration after warmup runs, and lower is faster.
+- `min_ms`, `max_ms`, and `stdev_ms` describe the measured spread for the same row; wide ranges usually mean CI runner noise rather than a deliberate code change.
+- Formatter comparisons are scoped to the same scenario and version. `default` is the black/isort default baseline, while `builtin` and `ruff` ratios compare their medians to that baseline.
+- `ok` rows have timing data. `unsupported` means the formatter or option was unavailable in that release. `failed` means installation or command execution failed, so timing cells are intentionally empty.
+
 ## Historical Trends
 
 The benchmark data on this page is loaded from `docs/data/release-benchmarks.json` and rendered in the browser. Release runs update that JSON file directly, so the published page reflects new release measurements after the docs deployment without rewriting this Markdown file.

tests/data/expected/release_benchmark_docs/release_benchmark_rendered.txt

@@ -8,6 +8,23 @@ This page tracks datamodel-code-generator release and main-branch benchmark resu
 
 datamodel-code-generator supports many schema styles and production use cases, so it includes a broad set of useful options. As releases add more capabilities, these benchmarks help keep the implementation measured, managed, and tuned so code generation stays fast in everyday use.
 
+## Scenario Guide
+
+Each scenario combines an input type with a case size. This guide is generated from the scenario keys in the benchmark JSON and the collector case definitions, so it changes when the benchmark matrix changes.
+
+| Scenario | Input fixture | Formatters | Represents |
+| --- | --- | --- | --- |
+| Small / JSON Schema | `tests/data/jsonschema/person.json` | `default` (black/isort default), `builtin` (Built-in), `ruff` (Ruff) | Compact fixture that emphasizes CLI startup, parsing, and formatter overhead for JSON Schema to Pydantic v2 model generation. |
+| Large / JSON Schema | `tests/data/performance/large_models.json` | `builtin` (Built-in) | Larger fixture that emphasizes parser and model graph throughput for JSON Schema to Pydantic v2 model generation. |
+| Large / OpenAPI | `tests/data/performance/openapi_large.yaml` | `builtin` (Built-in), `ruff` (Ruff) | Larger fixture that emphasizes parser and model graph throughput for OpenAPI component resolution and Pydantic v2 model generation. |
+
+## Interpreting Metrics
+
+- `median_ms` is the primary comparison value: it is the median generation duration after warmup runs, and lower is faster.
+- `min_ms`, `max_ms`, and `stdev_ms` describe the measured spread for the same row; wide ranges usually mean CI runner noise rather than a deliberate code change.
+- Formatter comparisons are scoped to the same scenario and version. `default` is the black/isort default baseline, while `builtin` and `ruff` ratios compare their medians to that baseline.
+- `ok` rows have timing data. `unsupported` means the formatter or option was unavailable in that release. `failed` means installation or command execution failed, so timing cells are intentionally empty.
+
 ## Historical Trends
 
 The benchmark data on this page is loaded from `docs/data/release-benchmarks.json` and rendered in the browser. Release runs update that JSON file directly, so the published page reflects new release measurements after the docs deployment without rewriting this Markdown file.