feat: auto-generate YAML config templates from schema, CLI bug fix

Bug fix:
- CLI crash on --load-pattern + --target-qps (IndexError) — LoadPattern.type
  used alias= instead of name= on cyclopts.Parameter

Template generation:
- regenerate_templates.py generates minimal + full YAML templates from
  Pydantic schema field defaults (no hardcoded values)
- Full templates include inline # comments auto-discovered from
  Field(description=) and Enum/Literal annotations
- Minimal templates: required fields + placeholders only
- Pre-commit hook auto-regenerates on schema changes; CI validates via
  CI env var detection (check-only mode)
- init command uses model_dump(exclude_none=True) for offline/online/
  concurrency; falls back to handwritten templates for eval/submission

Schema improvements:
- Added Field(description=) to all HTTPClientConfig, AccuracyConfig fields
- New ScorerMethod enum (pass_at_1, string_match, rouge, code_bench_scorer,
  shopify_category_f1) for AccuracyConfig.eval_method

Tests:
- Hypothesis fuzz: 4000 random CLI flag combinations
- Unit: all templates validate via from_yaml_file (auto-discovered glob)
- Integration: generated templates run e2e against echo server
- Init command: all 5 types tested

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: viraatc

.github/workflows/test.yml

@@ -53,3 +53,44 @@ jobs:
           python -m pip install pip==26.0.1
           pip install -e ".[dev,test,performance]"
           pip-audit
+
+  schema-updated:
+    runs-on: ubuntu-latest
+    if: github.event_name == 'pull_request'
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          fetch-depth: 0
+
+      - name: Check for schema changes
+        id: schema
+        run: |
+          CHANGED=$(git diff --name-only origin/${{ github.base_ref }}...HEAD -- \
+            'src/inference_endpoint/config/schema.py' \
+            'src/inference_endpoint/endpoint_client/config.py' \
+            'src/inference_endpoint/commands/benchmark/cli.py' \
+            'scripts/regenerate_templates.py' \
+            'src/inference_endpoint/config/templates/*.yaml')
+          echo "changed=$([[ -n "$CHANGED" ]] && echo true || echo false)" >> "$GITHUB_OUTPUT"
+
+      - name: Set up Python 3.12
+        if: steps.schema.outputs.changed == 'true'
+        uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        if: steps.schema.outputs.changed == 'true'
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[test]"
+
+      - name: Run schema fuzz tests
+        if: steps.schema.outputs.changed == 'true'
+        run: |
+          pytest -xv -m schema_fuzz
+
+      - name: Check YAML templates are up to date
+        if: steps.schema.outputs.changed == 'true'
+        run: |
+          python scripts/regenerate_templates.py --check

.pre-commit-config.yaml

@@ -35,7 +35,7 @@ repos:
     hooks:
       - id: prettier
         types_or: [yaml, json, markdown]
-        exclude: ^(src/inference_endpoint/openai/openai_types_gen.py|src/inference_endpoint/openai/openapi.yaml)$
+        exclude: ^(src/inference_endpoint/openai/openai_types_gen.py|src/inference_endpoint/openai/openapi.yaml|src/inference_endpoint/config/templates/)
 
   - repo: local
     hooks:
@@ -48,12 +48,12 @@ repos:
         args: ["--tb=short", "--strict-markers"]
         stages: [manual]
 
-      - id: validate-templates
-        name: Validate YAML templates against schema
-        entry: python -c "from pathlib import Path; from inference_endpoint.config.schema import BenchmarkConfig; [BenchmarkConfig.from_yaml_file(f) for f in sorted(Path('src/inference_endpoint/config/templates').glob('*.yaml'))]"
+      - id: regenerate-templates
+        name: Regenerate YAML templates from schema defaults
+        entry: python scripts/regenerate_templates.py
         language: system
         pass_filenames: false
-        files: ^src/inference_endpoint/config/(schema\.py|templates/)
+        files: ^(src/inference_endpoint/config/(schema\.py|templates/.*)|src/inference_endpoint/endpoint_client/config\.py|scripts/regenerate_templates\.py)$
 
       - id: add-license-header
         name: Add license headers

AGENTS.md

@@ -158,7 +158,7 @@ src/inference_endpoint/
 │   ├── ruleset_registry.py    # Ruleset registry
 │   ├── user_config.py         # UserConfig dataclass for ruleset user overrides
 │   ├── rulesets/mlcommons/    # MLCommons-specific rules, datasets, models
-│   └── templates/             # YAML config templates (offline, online, eval, etc.)
+│   └── templates/             # YAML config templates (_template.yaml minimal, _template_full.yaml all defaults)
 ├── openai/                    # OpenAI-compatible API types and adapters
 │   ├── types.py               # OpenAI response types
 │   ├── openai_adapter.py      # Request/response adapter
@@ -209,6 +209,7 @@ All of these run automatically on commit:
 - `mypy` type checking
 - `prettier` for YAML/JSON/Markdown
 - License header enforcement
+- `regenerate-templates`: auto-regenerates YAML config templates from schema defaults when `schema.py`, `config.py`, or `regenerate_templates.py` change
 
 **Always run `pre-commit run --all-files` before committing.**
 

docs/CLI_QUICK_REFERENCE.md

@@ -79,9 +79,8 @@ inference-endpoint validate-yaml -c test.yaml
 ### Utilities
 
 ```bash
-# Generate config templates
-inference-endpoint init offline        # or: online, eval, submission
-inference-endpoint init submission
+# Generate config template (all fields with defaults)
+inference-endpoint init offline        # or: online, concurrency
 
 # Show system info
 inference-endpoint info
@@ -238,14 +237,14 @@ inference-endpoint benchmark online \
 
 ```bash
 # 1. Generate template
-inference-endpoint init submission
+inference-endpoint init offline  # or: online, concurrency
 
-# 2. Edit submission_template.yaml (set model, datasets, ruleset, endpoint)
+# 2. Edit offline_template.yaml (set model, datasets, endpoint)
 
 # 3. Run (YAML mode)
 inference-endpoint benchmark from-config \
-  --config submission_template.yaml \
-  --report-dir official_results
+  --config offline_template.yaml \
+  --report-dir results
 ```
 
 ### Validate First

docs/DEVELOPMENT.md

@@ -276,6 +276,21 @@ pytest -s -v
 python -m pdb -m pytest test_file.py
 ```
 
+## 📄 YAML Config Templates
+
+Config templates in `src/inference_endpoint/config/templates/` are auto-generated from schema defaults. When you change `config/schema.py`, regenerate them:
+
+```bash
+python scripts/regenerate_templates.py
+```
+
+The pre-commit hook auto-regenerates templates when `schema.py`, `config.py`, or `regenerate_templates.py` change. CI validates templates are up to date via `--check` mode.
+
+Two variants are generated per mode (offline, online, concurrency):
+
+- `_template.yaml` — minimal: only required fields + placeholders
+- `_template_full.yaml` — all fields with schema defaults + inline `# options:` comments
+
 ## 📦 Package Management
 
 ### Adding Dependencies

docs/LOCAL_TESTING.md

@@ -141,7 +141,7 @@ Cleaning up...
 inference-endpoint -v info
 
 # Generate template
-inference-endpoint init --template offline
+inference-endpoint init offline
 
 # Validate config
 inference-endpoint validate-yaml --config offline_template.yaml

pyproject.toml

@@ -97,6 +97,8 @@ test = [
     "aiohttp==3.13.5",
     # Plotting for benchmark sweep mode
     "matplotlib==3.10.8",
+    # Property-based testing (CLI fuzz)
+    "hypothesis==6.151.10",
 ]
 performance = [
     "pytest-benchmark==5.2.3",
@@ -184,6 +186,7 @@ markers = [
     "integration: marks tests as integration tests",
     "unit: marks tests as unit tests",
     "run_explicitly: mark test to only run explicitly",
+    "schema_fuzz: hypothesis CLI fuzz tests (run in CI on schema changes)",
 ]
 filterwarnings = [
     "ignore:Session timeout reached:RuntimeWarning",