Skip to content

shapiq Benchmark#521

Open
IsaH57 wants to merge 21 commits into
mainfrom
benchmark_ep
Open

shapiq Benchmark#521
IsaH57 wants to merge 21 commits into
mainfrom
benchmark_ep

Conversation

@IsaH57
Copy link
Copy Markdown
Collaborator

@IsaH57 IsaH57 commented May 18, 2026

Motivation and Context

This PR consolidates and validates a full shapiq_benchmark layer, ensuring all benchmark types (Interventional, Pathdependent, Local XAI, Image, and TabPFN) share a consistent interface for exact value computation and game access. It also strengthens the string-based dataset/model loading flow and the known-parameter lookup path, so benchmarks can be reproduced reliably across datasets and model families while still supporting custom inputs.

What’s included

  • Benchmark protocol and ground-truth computer abstractions
  • Dataset/model loading and validation, including string-based loaders and known-parameter lookups
  • Benchmark implementations:
    -- Interventional tree explanations in InterventionalBench()
    -- Pathdependent tree explanations in PathdependentBench()
    -- Local XAI explanations in LocalXAIBench()
    -- Image explanations in ImageBench()
    -- TabPFN explanations in TabPFNBench()
  • Metrics for approximation quality and ranking comparison
  • Configs holding optimized parameters for 6 model - dataset combinations

Public API Changes

  • No Public API changes
  • Yes, Public API changes (Details below)

How Has This Been Tested?

  • Manual test runs for all benchmarks
  • Each test covers both string-based loading and custom model/data paths where applicable

Checklist

  • The changes have been tested locally.
  • Documentation has been updated (if the public API or usage changes).
  • An entry has been added to CHANGELOG.md (if relevant for users).
  • The code follows the project's style guidelines.
  • I have considered the impact of these changes on the public API.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 20, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants