Skip to content

feat: support evaluating multiple input configurations per kernel task#69

Merged
shangkunwang01 merged 4 commits into
mainfrom
shangkun-eval-multi-inputs
Jul 21, 2026
Merged

feat: support evaluating multiple input configurations per kernel task#69
shangkunwang01 merged 4 commits into
mainfrom
shangkun-eval-multi-inputs

Conversation

@shangkunwang01

Copy link
Copy Markdown
Collaborator

This PR upgrades the JAX evaluation module to support running and benchmarking multiple input data configurations sequentially for a single kernel task.

Key Changes:

  • Evaluation Data Structures (EvaluationResult):
    • Transitioned all metric fields (compiled_successfully, numerically_correct, max_abs_diff, reference_time_ms, error_trace, etc.) from scalar types to List types.
    • speedup properties now dynamically compute lists of speedups matching the input configurations.
  • Harness Execution (harness_code.py):
    • get_inputs() can now return either a single tuple or a list of tuples (dynamic_args, static_args).
    • The evaluation now loops over all provided input configurations.
    • Added explicit shape matching checks to prevent jnp.allclose from crashing on shape mismatches.
  • Evaluator & Utils (jax_kernel_evaluator.py, evaluation_utils.py):
    • Geometric mean speedup calculation computes the task-level speedup average.

@shangkunwang01
shangkunwang01 requested a review from NinaCai July 13, 2026 23:43

@NinaCai NinaCai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you put an example (e.g. DSv4) with multiple input? So that we could follow the format for other test cases?

@shangkunwang01

Copy link
Copy Markdown
Collaborator Author

Can you put an example (e.g. DSv4) with multiple input? So that we could follow the format for other test cases?

Sure, add one commit to remove outdated files and add a dsv4 test case.

@shangkunwang01
shangkunwang01 force-pushed the shangkun-eval-multi-inputs branch from 1d2cf39 to b002d80 Compare July 21, 2026 17:51

@NinaCai NinaCai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like you removed a couple of ops in dataset? Is it intended?

@shangkunwang01

Copy link
Copy Markdown
Collaborator Author

Looks like you removed a couple of ops in dataset? Is it intended?

Intended. They are outdated.

@shangkunwang01
shangkunwang01 merged commit 3f62918 into main Jul 21, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants