feat(metrics): consolidate gpt-oss/deepseek accuracy, dedup reporting, surface TPS

AGENTS.md

@@ -99,7 +99,7 @@ Dataset Manager --> Load Generator --> Endpoint Client --> External Endpoint
 | **TensorRT-LLM**                    | `src/inference_endpoint/trtllm/`                                                                                   | Adapter for TensorRT-LLM endpoints. `TRTLLMAdapter` sends requests; `TRTLLMSSEAccumulator` handles SSE streaming responses.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
 | **DeepSeek-R1 (MLPerf)**            | `src/inference_endpoint/evaluation/scoring.py` (`LegacyMLPerfDeepSeekR1Scorer`), `examples/07_DeepSeekR1_Example/` | MLPerf DeepSeek-R1 accuracy. TensorRT-LLM is OpenAI-compatible, so it is served via `api_type: openai` / `openai_completions` (no dedicated trtllm adapter). The combined multi-subset eval (`math500`/`aime`/`gpqa`/`mmlu_pro`/`livecodebench`) is the official MLCommons `eval_accuracy.py`, run out-of-process via `uv run --project` against the isolated subproject at `src/inference_endpoint/evaluation/legacy_mlperf_deepseek_r1/` (a uv subproject excluded from the parent wheel; mirrors the VBench pattern). The example feeds the exact MLPerf prompt via pre-tokenized `input_tokens` to `/v1/completions`.                                                                                                                                                      |
 | **VideoGen**                        | `src/inference_endpoint/videogen/`                                                                                 | Adapter for video-generation endpoints (e.g. trtllm-serve `POST /v1/videos/generations`, used by MLPerf WAN2.2-T2V-A14B). Defaults to `response_format=video_path` (server saves video to shared storage and returns path) to avoid large byte payloads. Accuracy mode also runs on `video_path`: the adapter mirrors the path into `response_output` so the event log carries it to `VBenchScorer` (see `evaluation/scoring.py`), which scores videos via VBench from a sibling `uv` subproject at `examples/09_Wan22_VideoGen_Example/accuracy/` (vbench's `transformers==4.33.2` + `numpy<2` pins are incompatible with the parent env, so it runs out-of-process via `uv run --project`). Dataset is ingested via the generic JSONL loader.                                |
-| **Compliance (submission checker)** | `src/inference_endpoint/compliance/checker.py`, `scripts/check_compliance.py`                                      | Validates a completed run's report directory against a registered ruleset. `check_submission(report_dir, ruleset, model)` reads the resolved `config.yaml` plus scorer output (`results.json` for accuracy, `scores.json` for the agentic perf run) and runs config-lock (deterministic + single-stream), the accuracy gate (`score >= factor x reference`, factor 0.97 for Edge-Agentic), and run validity (0 dropped turns). Server-side launch flags (`--reasoning off`, `--ctx-size`) aren't in client artifacts, so they're surfaced as manual attestations. CLI: `scripts/check_compliance.py REPORT_DIR` (exit 0 = pass).                                                                                                                                               |
+| **Compliance (submission checker)** | `src/inference_endpoint/compliance/checker.py`, `scripts/check_compliance.py`                                      | Validates a completed run's report directory against a registered ruleset. `check_submission(report_dir, ruleset, model)` reads the resolved `config.yaml` plus scorer output (`accuracy/accuracy_results.json` for accuracy, `scores.json` for the agentic perf run) and runs config-lock (deterministic + single-stream), the accuracy gate (`score >= factor x reference`, factor 0.97 for Edge-Agentic), and run validity (0 dropped turns). Server-side launch flags (`--reasoning off`, `--ctx-size`) aren't in client artifacts, so they're surfaced as manual attestations. CLI: `scripts/check_compliance.py REPORT_DIR` (exit 0 = pass).                                                                                                                             |
 | **Compliance (audit tests)**        | `src/inference_endpoint/compliance/`, `commands/audit.py`                                                          | MLPerf compliance audits. `AuditTest` protocol + `AuditRunSpec`/`AuditRunArtifacts` + registry (`compliance/__init__.py`); `OutputCachingAudit` (`compliance/audit_test/output_caching_test.py`, which also owns the QPS-specific `AuditRunStats`) implements MLPerf **TEST04** output-caching detection — reference phase (distinct samples) vs. fixed-sample audit phase, comparing QPS against `threshold`. `commands/audit.py:run_audit` runs phases via `AuditTest.plan_runs`/`validate`, writing `audit_result.json`/`verify_<TEST>.txt` atomically via `compliance/result.py`. Enabled by the `audit:` YAML block; `cli._run` runs it after the main benchmark (upstream MLPerf order: perf run, then TEST04), or standalone with `audit.only: true`. Performance-only. |
 
 ### Hot-Path Architecture

docs/compliance_audit_plan.md

@@ -84,7 +84,7 @@ _intent_ over this repo's own artifacts.
 This repo names MLPerf **TEST04** the **output-caching test**: id `output_caching_test`
 (`AuditTestId.OUTPUT_CACHING_TEST`), config class `OutputCachingTestConfig`, audit
 `OutputCachingAudit`, and artifacts (under `<report_dir>/audit/`)
-`audit_result.json` + `verify_OUTPUT_CACHING_TEST.txt`. Where this doc writes
+`audit_output_caching_test.json` + `verify_OUTPUT_CACHING_TEST.txt`. Where this doc writes
 "TEST04" it means the upstream MLPerf test the output-caching audit re-implements.
 
 ---
@@ -134,7 +134,7 @@ benchmark from-config
             │
             │ 3. verify(runs, cfg) ; 4. write_result (atomic)
             ▼
-   <report_dir>/audit/ : audit_result.json  +  verify_OUTPUT_CACHING_TEST.txt
+   <report_dir>/audit/ : audit_output_caching_test.json  +  verify_OUTPUT_CACHING_TEST.txt
 ```
 
 ### Program flow (output-caching audit / MLPerf TEST04, two phases)
@@ -227,7 +227,7 @@ report is already written).
                            ▼
             ┌──────────────────────────────────────────┐
             │ write_result → <report_dir>/audit/ [atomic]│
-            │   audit_result.json        (durable first) │
+            │   audit_output_caching_test.json        (durable first) │
             │   verify_OUTPUT_CACHING_TEST.txt  (marker) │
             └──────────────┬───────────────────────────┘
                            ▼
@@ -404,7 +404,7 @@ The generic loop never names a specific test:
    `sys.exit` — `0` (PASS) / `1` (FAIL). Errors are not flattened to a single code:
    they propagate to `main.py`'s handler, which uses the repo-wide scheme
    (`InputValidationError` → `2`, `SetupError` → `3`, `ExecutionError` → `4`). The on-disk
-   `audit_result.json` is the durable record; the exit code is the automation signal.
+   `audit_output_caching_test.json` is the durable record; the exit code is the automation signal.
 
 ### Verifier — one core + in-process adapter
 
@@ -440,7 +440,7 @@ re-check-from-disk adapter — the audit runs only via `benchmark from-config`.
 ```
 src/inference_endpoint/compliance/
 ├── __init__.py        # AuditTest protocol, AuditRunSpec/Stats/Artifacts, AUDIT_TESTS map, get_audit_test()
-├── result.py          # AuditResult + atomic write → audit_result.json + verify_<TEST>.txt
+├── result.py          # AuditResult + atomic write → audit_<test_id>.json + verify_<TEST>.txt
 └── audit_test/
     ├── __init__.py     # package marker for the AuditTest implementations
     ├── output_caching_test.py  # OutputCachingAudit: plan_runs (reference + audit specs) + validate + verify_output_caching core
@@ -478,7 +478,7 @@ report_dir/
     ├── reference/               # audit reference phase    (samples=64)
     ├── output_caching/          # audit fixed-sample phase (samples=64)
     ├── verify_OUTPUT_CACHING_TEST.txt
-    └── audit_result.json
+    └── audit_output_caching_test.json
 ```
 
 ### WAN2.2-T2V — the first target
@@ -598,7 +598,7 @@ run; `verify` reads `events.jsonl` and checks mean OSL within `[ref × 0.9, ref
 ## 8. Success criteria (goal-driven; verify before done)
 
 1. **Integration** — `benchmark from-config` with an `audit:` block runs both phases
-   back-to-back and writes `audit_result.json` + `verify_OUTPUT_CACHING_TEST.txt`; PASS against a
+   back-to-back and writes `audit_output_caching_test.json` + `verify_OUTPUT_CACHING_TEST.txt`; PASS against a
    no-caching `mock_http_echo_server`, FAIL against a caching mock.
 2. **Completion guard** — a phase that completes far fewer than its _requested_ count fails
    the result (`completed < requested × (1 − threshold)` → FAIL), independent of the other

docs/evaluation/DESIGN.md

@@ -88,9 +88,9 @@ documented prominently in the dataset README.
 
 ## Integration Points
 
-| Component                          | Role                                                   |
-| ---------------------------------- | ------------------------------------------------------ |
-| `commands/benchmark/execute.py`    | Builds scorer/extractor configs and runs scoring       |
-| `dataset_manager/predefined/`      | Provides ground truth labels alongside prompts         |
-| `evaluation/livecodebench/`        | Provides external execution path for LiveCodeBench     |
-| `results.json` / benchmark reports | Receives computed accuracy summary during finalization |
+| Component                        | Role                                                   |
+| -------------------------------- | ------------------------------------------------------ |
+| `commands/benchmark/execute.py`  | Builds scorer/extractor configs and runs scoring       |
+| `dataset_manager/predefined/`    | Provides ground truth labels alongside prompts         |
+| `evaluation/livecodebench/`      | Provides external execution path for LiveCodeBench     |
+| `accuracy/accuracy_results.json` | Receives computed accuracy summary during finalization |

examples/03_BenchmarkComparison/compare_with_vllm.py

@@ -349,7 +349,7 @@ def run_inference_endpoint(
     results = parse_inference_endpoint_output(full_output)
 
     # Load report JSON to enrich or backfill metrics from stdout (e.g., detailed TTFT/TPOT stats, output lengths)
-    report_json_path = report_dir / "result_summary.json"
+    report_json_path = report_dir / "performance" / "result_summary.json"
     if report_json_path.exists():
         try:
             with open(report_json_path) as f:

examples/11_Edge_Agentic_Example/README.md

@@ -241,30 +241,26 @@ Results are written to `results/bfcl_v4_multi_turn/`.
 # Single-turn overall accuracy
 python3 -c "
 import json, pathlib
-r = json.loads(pathlib.Path('results/edge_agentic_full_run/results.json').read_text())
-print('Overall ST accuracy:',
-      r['accuracy_scores']['bfcl_v4::function_calling']['breakdown']['overall_accuracy'], '%')
+r = json.loads(pathlib.Path('results/edge_agentic_full_run/accuracy/accuracy_results.json').read_text())
+e = next(x for x in r['accuracy_scores'] if x['dataset_name'] == 'bfcl_v4::function_calling')
+print('Overall ST accuracy:', e['breakdown']['overall_accuracy'], '%')
 "
 
 # Multi-turn overall accuracy (only if you ran the optional Step 3)
 python3 -c "
 import json, pathlib
-r = json.loads(pathlib.Path('results/bfcl_v4_multi_turn/results.json').read_text())
-print('Overall MT accuracy:',
-      r['accuracy_scores']['bfcl_v4::multi_turn']['score']['overall_accuracy'], '%')
+r = json.loads(pathlib.Path('results/bfcl_v4_multi_turn/accuracy/accuracy_results.json').read_text())
+e = next(x for x in r['accuracy_scores'] if x['dataset_name'] == 'bfcl_v4::multi_turn')
+print('Overall MT accuracy:', e['breakdown']['overall_accuracy'], '%')
 "
 ```
 
-> Note: the single-turn pipeline writes `results.json` (accuracy nested under
-> `accuracy_scores['bfcl_v4::function_calling']`). There is no separate
-> `accuracy_scores.json`. A human-readable summary is also written to
+> Note: accuracy is written to `accuracy/accuracy_results.json`, whose
+> `accuracy_scores` is a **list** of per-dataset entries — index it by
+> `dataset_name` (as above). Each entry has a scalar `score` and, for
+> multi-subset scorers, a `breakdown` block with per-subset detail. A
+> human-readable summary is also written to
 > `results/edge_agentic_full_run/report.txt`.
->
-> Result shape differs between the two pipelines: the **single-turn** entry
-> stores `score` as a scalar fraction (0–1) with the per-subset dict under a
-> separate `breakdown` key (hence `['breakdown']['overall_accuracy']` above),
-> whereas the **multi-turn** CLI writes `score` as a dict (hence
-> `['score']['overall_accuracy']` in the multi-turn snippet).
 
 ---
 
@@ -325,26 +321,27 @@ print('Inline accuracy score:', s['score'], '| valid run:', missing == 0, '| mis
 # BFCL v4 single-turn accuracy (the gated metric)
 python3 -c "
 import json, pathlib
-r = json.loads(pathlib.Path('results/edge_agentic_full_run/results.json').read_text())
-print('Overall ST accuracy:',
-      r['accuracy_scores']['bfcl_v4::function_calling']['breakdown']['overall_accuracy'], '%')
+r = json.loads(pathlib.Path('results/edge_agentic_full_run/accuracy/accuracy_results.json').read_text())
+e = next(x for x in r['accuracy_scores'] if x['dataset_name'] == 'bfcl_v4::function_calling')
+print('Overall ST accuracy:', e['breakdown']['overall_accuracy'], '%')
 "
 ```
 
 Both scores land in `results/edge_agentic_full_run/`: the BFCL gate under
-`results.json` (`accuracy_scores['bfcl_v4::function_calling']`) and the inline
-performance checker in `scores.json`, alongside the performance metrics
-(throughput, TTFT, TPOT, per-turn latency, ISL/OSL).
+`accuracy/accuracy_results.json` (the `bfcl_v4::function_calling` entry in the
+`accuracy_scores` list) and the inline performance checker in `scores.json`,
+alongside the performance metrics (throughput, TTFT, TPOT, per-turn latency,
+ISL/OSL) in `performance/result_summary.json`.
 
 ### Publish for the MLPerf submission checker
 
 The MLPerf Inference submission checker (`tools/submission/submission_checker`
 in `mlcommons/inference`, v5.0+) reads endpoints results directly from the
-artifacts a run already writes — `result_summary.json`, `results.json`, and
-`config.yaml` — so no separate "log" format is needed. `scripts/publish_submission.py`
-copies that trio into the directory layout the checker walks and self-verifies
-the fields it reads (primary-metric QPS, p99 latency, TTFT/TPOT p99, and the
-accuracy score):
+artifacts a run already writes — `performance/result_summary.json`,
+`accuracy/accuracy_results.json`, and `config.yaml` — so no separate "log" format
+is needed. `scripts/publish_submission.py` copies those into the directory layout
+the checker walks and self-verifies the fields it reads (primary-metric QPS, p99
+latency, TTFT/TPOT p99, and the accuracy score):
 
 ```bash
 python scripts/publish_submission.py \

pyproject.toml

@@ -21,7 +21,9 @@ conflicts = [
 ]
 # CVE floors for transitive deps we don't declare directly. click is pulled in
 # via transformers -> typer; 8.3.2 carries PYSEC-2026-2132 (fixed in 8.3.3).
-constraint-dependencies = ["click>=8.3.3"]
+# setuptools is pulled in via pbr/torch; 82.0.1 carries PYSEC-2026-3447 (fixed
+# in 83.0.0).
+constraint-dependencies = ["click>=8.3.3", "setuptools>=83.0.0"]
 
 [tool.uv.build-backend]
 module-root = "src"

scripts/plot_results.py

@@ -39,8 +39,8 @@ def main() -> int:
         "report_dir",
         nargs="+",
         type=Path,
-        help="Run report directory (contains results.json / scores.json / "
-        "result_summary.json).",
+        help="Run report directory (contains accuracy/accuracy_results.json / "
+        "scores.json / performance/result_summary.json).",
     )
     parser.add_argument(
         "--out-dir",