WIP - Accuracy Checkers based on proposal by nv-alicheng · Pull Request #36 · mlcommons/endpoints

nv-alicheng · 2025-12-01T22:34:37Z

What does this PR do?

Type of change

Bug fix
New feature
Documentation update
Refactor/cleanup

Related issues

Testing

Tests added/updated
All tests pass locally
Manual testing completed

Checklist

Code follows project style
Pre-commit hooks pass
Documentation updated (if needed)

github-actions · 2025-12-01T22:34:46Z

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

gemini-code-assist · 2025-12-01T22:35:03Z

Summary of Changes

Hello @nv-alicheng, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request lays the groundwork for comprehensive accuracy checking within the inference endpoint. It introduces a modular system for defining and managing datasets and evaluators, enabling the integration of various benchmarks like GPQA. The changes also adapt the existing load generation and session management to support accuracy tests, ensuring that model outputs can be correctly linked back to their source data for evaluation and reporting.

Highlights

Accuracy Dataset Framework: Introduction of AccuracyDataset base class and DatasetFormat enum to standardize how datasets are defined, generated, and loaded for accuracy evaluations.
GPQA Dataset Implementation: Added a concrete implementation for the GPQA (Graduate-Level Google-Proof Q&A) benchmark, including data loading from HuggingFace, processing, and saving.
Accuracy Evaluator Framework: Established Evaluator and Extractor base classes, along with ABCDExtractor for robust extraction of multiple-choice answers from model outputs using regex patterns.
Integration with Load Generator: Modified the load generation system to support accuracy testing by tracking sample UUIDs to original dataset indices and allowing for multiple load generators (performance and accuracy) within a session.
Report Generation for Accuracy: Enhanced session reporting to save the mapping between sample UUIDs and their dataset indices, facilitating post-inference accuracy evaluation.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces the foundational components for accuracy checking, including base classes for datasets and evaluators, along with a concrete implementation for the GPQA dataset. The changes are a good step towards building out the accuracy evaluation framework.

My review includes a few key points:

A critical bug fix in session.py for iterating over accuracy test generators.
A high-severity bug fix in gpqa.py related to loading dataset variants from Hugging Face.
Suggestions to improve the design of dataset loading by making automatic generation more accessible through the base classes.
Minor fixes for logging and a typo.

Overall, the structure is well-thought-out, and with these fixes, it will be a solid foundation.

gemini-code-assist · 2025-12-01T22:37:11Z

+                    for _, generator in accuracy_test_generators:
+                        for _ in generator:
+                            pass


There is a bug in how you iterate over accuracy_test_generators. The expression for _, generator in accuracy_test_generators: attempts to iterate over the keys of the dictionary and unpack each key. This will likely fail or produce incorrect behavior. To iterate over the LoadGenerator instances, you should iterate over the dictionary's values.

Suggested change

for _, generator in accuracy_test_generators:

for _ in generator:

pass

for generator in accuracy_test_generators.values():

for _ in generator:

pass

gemini-code-assist · 2025-12-01T22:37:12Z

+    def generate(self, datasets_dir: Path):
+        # Load the variant from HuggingFace
+        try:
+            raw_ds = hf_datasets.load_dataset("Idavidrein/gpqa", f"gpqa_{self.variant}")


There's a bug when loading the dataset from Hugging Face. When max_samples is provided, self.variant is modified to include the sample count (e.g., "diamond_100"). This modified variant name is then used to load the dataset, which will likely fail as the Hugging Face dataset variant does not include the sample count. You should use self._variant_name, which stores the original variant name ("diamond", "extended", or "main").

Suggested change

raw_ds = hf_datasets.load_dataset("Idavidrein/gpqa", f"gpqa_{self.variant}")

raw_ds = hf_datasets.load_dataset("Idavidrein/gpqa", f"gpqa_{self._variant_name}")

gemini-code-assist · 2025-12-01T22:37:12Z

+        variable.
+
+        If <variant> is not specified or not applicable, the default value is 'full'. The
+        variant should be specied as an instance variable: .variant.


There is a typo in the docstring.

Suggested change

variant should be specied as an instance variable: .variant.

variant should be specified as an instance variable: .variant.

gemini-code-assist · 2025-12-01T22:37:12Z

+        raise NotImplementedError
+
+    @abstractmethod
+    def load(self, datasets_dir: Path) -> Any:


The signature of the abstract method load should include the create_if_not_exists parameter to match the implementation in subclasses like GPQA. This ensures consistency and allows callers using the AccuracyDataset interface to leverage this functionality.

Suggested change

def load(self, datasets_dir: Path) -> Any:

def load(self, datasets_dir: Path, create_if_not_exists: bool = False) -> Any:

gemini-code-assist · 2025-12-01T22:37:12Z

+            print(f"Error loading dataset: {e}")
+            print("Note: This dataset may require HuggingFace authentication.")
+            print("Run: huggingface-cli login")


It's better to use the configured logger (logger.error or logger.warning) instead of print() for outputting error messages and instructions. This provides more consistent and controllable logging within the application.

Suggested change

print(f"Error loading dataset: {e}")

print("Note: This dataset may require HuggingFace authentication.")

print("Run: huggingface-cli login")

logger.error(f"Error loading dataset: {e}")

logger.error("Note: This dataset may require HuggingFace authentication.")

logger.error("Run: huggingface-cli login")

gemini-code-assist · 2025-12-01T22:37:12Z

+        self.accuracy_dataset = accuracy_dataset
+        self.dataset_dir = dataset_dir
+
+        self.ds = self.accuracy_dataset.load(dataset_dir)


To improve usability, consider allowing the Evaluator to create datasets that don't exist. You can do this by adding a create_if_not_exists: bool = False parameter to Evaluator.__init__ and passing it to the load method here. This change assumes that the AccuracyDataset.load abstract method is also updated to accept this parameter.

Suggested change

self.ds = self.accuracy_dataset.load(dataset_dir)

self.ds = self.accuracy_dataset.load(dataset_dir, create_if_not_exists=create_if_not_exists)

nvzhihanj · 2025-12-04T01:08:01Z

+from typing import Any, ClassVar
+
+
+class DatasetFormat(Enum):


I would recommend that we consolidate the dataset format to some standard, and we provide conversion script to convert to the standard format (otherwise maintaining the behavior of so many formats is tough)

Pickle / parquet / jsonl seem like the most popolar ones.

nvzhihanj · 2025-12-04T04:22:38Z

+            correct_answer = f"choice{correct_index + 1}"
+
+            # Create processed row
+            processed_row = {


Seems missing the common f-string for question formatting - plan to add it somewhere else?

…ABCDExtractor for GPQA

…ile with support for reasoning sequences

+        evaluator = Evaluator(
+            extractor=ABCDExtractor,
+            scorer=PassAt1Scorer(),
+            accuracy_dataset=mock_dataset,
+            dataset_dir=Path("/fake/path"),
+        )


+        evaluator = Evaluator(
+            extractor=ABCDExtractor,
+            scorer=PassAt1Scorer(),
+            accuracy_dataset=mock_dataset,
+            dataset_dir=Path("/fake/path"),
+        )


+        evaluator = Evaluator(
+            extractor=ABCDExtractor,
+            scorer=PassAt1Scorer(),
+            accuracy_dataset=mock_dataset,
+            dataset_dir=Path("/fake/path"),
+        )


+        evaluator = Evaluator(
+            extractor=ABCDExtractor,
+            scorer=PassAt1Scorer(),
+            accuracy_dataset=mock_dataset,
+            dataset_dir=Path("/fake/path"),
+        )


+        evaluator = Evaluator(
+            extractor=ABCDExtractor,
+            scorer=PassAt1Scorer(),
+            accuracy_dataset=mock_dataset,
+            dataset_dir=Path("/fake/path"),
+        )


+        evaluator = Evaluator(
+            extractor=Mock(extract=lambda x: x),  # No-op extractor
+            scorer=RougeScorer(use_stemmer=True),
+            accuracy_dataset=mock_dataset,
+            dataset_dir=Path("/fake/path"),
+        )


+        evaluator = Evaluator(
+            extractor=Mock(extract=lambda x: x),
+            scorer=RougeScorer(use_stemmer=True),
+            accuracy_dataset=mock_dataset,
+            dataset_dir=Path("/fake/path"),
+        )


+        evaluator = Evaluator(
+            extractor=ABCDExtractor,
+            scorer=PassAt1Scorer(),
+            accuracy_dataset=mock_dataset,
+            dataset_dir=Path("/fake/path"),
+        )


+        evaluator = Evaluator(
+            extractor=ABCDExtractor,
+            scorer=PassAt1Scorer(),
+            accuracy_dataset=mock_dataset,
+            dataset_dir=Path("/fake/path"),
+        )


+        evaluator = Evaluator(
+            extractor=ABCDExtractor,
+            scorer=PassAt1Scorer(),
+            accuracy_dataset=mock_dataset,
+            dataset_dir=Path("/fake/path"),
+        )


…cess function to generate v6.0 compatible dataset file

nv-alicheng · 2026-02-14T00:16:50Z

Closing because this was merged in a separate PR.

nv-alicheng requested a review from a team as a code owner December 1, 2025 22:34

github-actions Bot requested review from arekay-nv and nvzhihanj December 1, 2025 22:34

gemini-code-assist Bot reviewed Dec 1, 2025

View reviewed changes

viraatc self-requested a review December 3, 2025 00:00

nvzhihanj reviewed Dec 4, 2025

View reviewed changes

nv-alicheng force-pushed the feature/alicheng-accuracyrun branch 3 times, most recently from aa7c46d to 067fddd Compare December 11, 2025 21:44

tianmu-li mentioned this pull request Dec 17, 2025

llama2-70b dataset and model enabling #62

Closed

nv-alicheng added 8 commits December 18, 2025 10:17

Add stubs and base classes for Accuracy checkers

a5b7fcd

Run accuracy datasets in separate load generators in succession, add …

800b17e

…ABCDExtractor for GPQA

Update Evaluator interface, start MCQ Evaluator

4781898

Add ABCDExtractor, scorers, tests

3579bc5

Move outputs recording into the EventDB instead of a separate JSONL f…

621777d

…ile with support for reasoning sequences

Fixes for artifacts from rebase

b40c79d

pre-commit linter fixes

c85952f

Partial update for datasets integration into DSManager

189ebd3

nv-alicheng force-pushed the feature/alicheng-accuracyrun branch from d8352d1 to 189ebd3 Compare December 19, 2025 00:13

github-code-quality Bot found potential problems Dec 19, 2025

View reviewed changes

nv-alicheng added 4 commits December 19, 2025 19:16

Remove broken code that Rashid will add later, add converter post-pro…

11999c9

…cess function to generate v6.0 compatible dataset file

Add generic OpenAI harmony formatter utility file

13a4a9e

Add SGLang adapter and SSEChunk support to worker

6e12bbd

Add flag to CLI and endpoint config to determine the API type

46e243f

nv-alicheng closed this Feb 14, 2026

github-actions Bot locked and limited conversation to collaborators Feb 14, 2026

arekay-nv deleted the feature/alicheng-accuracyrun branch April 2, 2026 03:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP - Accuracy Checkers based on proposal#36

WIP - Accuracy Checkers based on proposal#36
nv-alicheng wants to merge 12 commits intomainfrom
feature/alicheng-accuracyrun

nv-alicheng commented Dec 1, 2025

Uh oh!

github-actions Bot commented Dec 1, 2025 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Dec 1, 2025

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Uh oh!

nvzhihanj Dec 4, 2025

Uh oh!

nvzhihanj Dec 4, 2025

Uh oh!

nv-alicheng commented Feb 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	raw_ds = hf_datasets.load_dataset("Idavidrein/gpqa", f"gpqa_{self.variant}")
	raw_ds = hf_datasets.load_dataset("Idavidrein/gpqa", f"gpqa_{self._variant_name}")

	variant should be specied as an instance variable: .variant.
	variant should be specified as an instance variable: .variant.

	def load(self, datasets_dir: Path) -> Any:
	def load(self, datasets_dir: Path, create_if_not_exists: bool = False) -> Any:

	self.ds = self.accuracy_dataset.load(dataset_dir)
	self.ds = self.accuracy_dataset.load(dataset_dir, create_if_not_exists=create_if_not_exists)

Conversation

nv-alicheng commented Dec 1, 2025

What does this PR do?

Type of change

Related issues

Testing

Checklist

Uh oh!

github-actions Bot commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot commented Dec 1, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

nvzhihanj Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

nvzhihanj Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

nv-alicheng commented Feb 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Dec 1, 2025 •

edited

Loading