feat(metrics): consolidate gpt-oss/deepseek accuracy, dedup reporting, surface TPS

[arekay-nv]

Summary

Reworks accuracy reporting into a uniform per-dataset list and aligns the run's report directory with the MLPerf endpoints submission layout (mlcommons/inference#2628).

Note: an earlier revision of this PR added a composite gpt-oss consolidation scorer; that was removed in favor of the per-dataset list below.

Per-dataset accuracy list

• Report.accuracy and results.json's accuracy_scores are now an always-present list (empty for perf-only), one entry per accuracy dataset. Each entry: dataset_name, extractor, ground_truth_column, score, unit_samples, num_repeats, total_samples (= unit × repeats), duration_s (the dataset's issue-phase wall-clock), complete, and — for multi-subset scorers (DeepSeek-R1, BFCL) — a BFCL-shaped breakdown.
• The single breakdown reader find_accuracy_breakdown iterates the list; compliance/checker.py + metrics/results_plots.py are unchanged behind it.
• Rendered per-dataset in report.txt / results_summary.json: <name>: <score> (unit=U, repeats=R, total=T, duration=Ds).

Removed the composite gpt-oss path

• Deleted the composite gptoss_120b_accuracy predefined dataset, GptOss120bAccuracyScorer, and _weighted_breakdown; the per-subset aime25::gptoss / gpqa::gptoss / livecodebench::gptoss datasets are the sole path. DeepSeek-R1 keeps its combined-subset breakdown as one list entry.

Report directory conforms to #2628

<report_dir>/
  config.yaml
  performance/results_summary.json   # the perf-metrics Report
  accuracy/accuracy_results.json     # the accuracy list
  audit/audit_<test_id>.json         # e.g. audit_output_caching_test.json (+ verify_<TEST>.txt)
  results.json                       # full run: config + perf + accuracy_scores + responses + errors
  report.txt · events.jsonl · metrics/ · sample_idx_map.json

• scripts/publish_submission.py still copies the artifact trio flat into the MLPerf performance/run_1/ submission layout.

Fixes surfaced during validation

• Coerce numpy scalar scores to native float: the base Scorer.score() returned np.mean(...) (numpy.float64), which crashed Report.to_json (msgspec) and results.json (json) serialization — caught on a live gpt-oss run.
• Updated scripts/publish_submission.py and evaluation/bfcl_v4_multi_turn_cli.py to the list shape.

End-to-end validation (GB200 NVL72)

• gpt-oss-120b (trtllm-serve): aime25 0.808 (8×) / gpqa 0.723 (5×) / livecodebench 0.849 (3×) @ 32768 OSL.
• DeepSeek-R1-NVFP4 (trtllm-serve DEP4): overall 81.04 on the full 4388-sample eval (math500 95.8 / aime 83.3 / lcb 87.7 / mmlu_pro 77.3 / gpqa 66.7) — matches the MLPerf golden (81.36).

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[gemini-code-assist]

Code Review

This pull request introduces support for consolidating multiple accuracy datasets into a single combined score using a consolidation group tag, which is particularly useful for rolling up subsets like gpt-oss (e.g., AIME25, GPQA, and LiveCodeBench) into a sample-weighted overall score. It extracts shared accuracy breakdown helpers into a new module, refactors compliance and plotting utilities to use them, and updates the benchmark finalization process to attach consolidated accuracy breakdowns to the report. The feedback suggests a cleaner and safer approach to retrieving the dataset sample count during per-subset audit entry generation by avoiding fragile assert statements and direct internal attribute access in favor of the public num_samples() API.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[nvzhihanj]

Thanks for the change. Can you help add some examples of the accuracy report (e.g. for gpt-oss) after the change?

[nvzhihanj]

Review Council — architecture review (Concern-driven)

Reviewed by Claude + Codex (independent), thorough. Two maintainer concerns; both confirmed by both reviewers. Proposing the concrete change for each.

🏗️ Concern 1 — execute.py is doing too much (~1656 lines)

The accuracy path was grown inside the orchestrator. Concrete decomposition (both reviewers agree; removes ~350 lines from execute.py):

• Accuracy scoring/OSL — _score_accuracy (~140 lines), _phase_osl_stats, _phase_response_counts, _load_osl_tokenizer, _write_scoring_artifacts → new evaluation/accuracy_scoring.py (sibling of the new accuracy_results.py), taking a small typed input (eval_configs + tokenizer_name + phase timings) instead of BenchmarkContext. finalize_benchmark becomes a one-line score = accuracy_scoring.score(...).
• Profiling — _derive_profile_urls / _post_profile / _render_profile_status / _write_profiling_section + _PROFILE_PATHS (~95 self-contained lines, zero BenchmarkContext coupling) → new commands/benchmark/profiling.py (a ProfilerController).
• Dataset loading (_load_datasets, _resolve_accuracy_components) → commands/benchmark/datasets.py (lower priority).

Good: the new accuracy_results.py actually reduces duplication (dedups helpers previously copy-pasted in checker.py + results_plots.py). But report.py now imports aggregator-internal privates (SeriesSampler, _DEFAULT_PERCENTILES, _metric_to_dict) — leaky cross-layer coupling (see the report.py:46 thread).

⚙️ Concern 2 — accuracy OSL re-tokenizes on the main process (duplicated tokenizer)

Correct that it's a second AutoTokenizer in the main process + a different implementation from the aggregator's sharded BatchTokenizer. It is not duplicate work on the same samples (the aggregator only tokenizes the perf window — is_tracking is off for accuracy phases), but it is a real finalize-time CPU cost, and it re-tokenizes scorer-transformed text (BFCL's get_outputs() serializes merged tool-calls → wrong OSL).

Concrete change (the right architecture): the OpenAI/sglang responses already carry usage.completion_tokens/prompt_tokens, but it's dropped in accumulator.get_final_output / session.py before it reaches events.jsonl. Persist server usage into QueryResult/the event log → OSL/ISL become a direct read that eliminates both client re-tokenizers, with the aggregator's BatchTokenizer as the single shared fallback. Caveat (both reviewers): MLPerf OSL wants the reference tokenizer for determinism, and some servers omit usage (or need stream_options.include_usage), so the shared reference-tokenizer fallback stays mandatory — "delete all client tokenization" is too strong. The key point: never a bespoke main-thread AutoTokenizer; one shared path.

🔎 Additional findings (posted inline)

• scoring.py:224 — base Scorer.score() excludes truly-missing samples from the denominator → inflated accuracy; all-missing → NaN written as non-standard JSON.
• accuracy_results.py:93 — average_accuracy is a plain mean of heterogeneous scorer scales (0–1 vs 0–100).
• execute.py:1377 — isinstance(score, float) misses np.float32.
• tests/unit/compliance/test_checker.py:203 — fixture false-confidence (overall_accuracy that the real DeepSeek breakdown omits).

Concrete change suggestions are also left inline on the existing open threads (execute.py:1247/1255/1323/1352/1417/1438, accuracy_results.py:43, report.py:46, publish_submission.py:145).

[nvzhihanj]

Review Council — net-new findings

Inline findings not already covered by the open threads (architecture proposal is in the top-level summary). Reviewed by Claude + Codex.

[viraatc]

Review Council — local pass (Codex + Grok + manual verification)

Scope: integration + the report-layout / accuracy-list migration. The core migration is sound — I verified the BFCL and DeepSeek-R1 compliance/breakdown paths and that the list-shape consumers (checker.py, results_plots.py, publish_submission.py, bfcl_v4_multi_turn_cli.py) are all migrated. Most issues below are user-facing artifacts the diff didn't touch. (BFCL/dataset coupling inside the accuracy module is intentionally out of scope — tracked separately.)

Inline comments: gpt-oss plots empty · average_accuracy scale + dataset_type · numpy coercion · the accuracy-write site (stale artifacts / dead collector.responses / non-atomic).

🔴 Not inline (files unchanged by this PR) — broken examples

The layout change breaks committed examples users follow verbatim:

• examples/11_Edge_Agentic_Example/README.md:244,252 — reads results.json + dict accuracy_scores['bfcl_v4::function_calling']; it's now accuracy/accuracy_results.json + a list → FileNotFoundError / TypeError.
• examples/03_BenchmarkComparison/compare_with_vllm.py:352 — reads report_dir / 'result_summary.json'; it's now under performance/.

Fix: update both to the new paths + list shape (mirror bfcl_v4_multi_turn_cli.py:229-247).

Notes (low)

• Accuracy-write failure is swallowed (except Exception: logger.error) → exit-0 with no accuracy artifact; consider hard-failing when scoring produced entries.
• find_accuracy_breakdown returns the first subset_scores entry — fine for BFCL-only / DSR1-only; a hypothetical mixed run could grab the wrong block by ordering.

Gemini errored out; Opus didn't return in time — this is Codex + Grok + manual verification.

[viraatc]

LGTM, thanks!