shapiq.vision Sub-package for Image Classification

[S2k-1]

Motivation and Context

shapiq currently has no first-class support for explaining vision models. The only image-handling code lives inside the benchmark games (shapiq_games.benchmark.local_xai.ImageClassifier) where masking logic is hard-coded per model, which is not suitable for end-user pipelines.

This PR introduces shapiq.vision, a new subpackage that plugs cleanly into the existing Explainer → Game → Imputer pipeline and supports both CNN and Vision Transformer (ViT) architectures. The design follows the pattern of shapiq.tree / shapiq.graph by separating concerns into dedicated modules:

• players.py — pluggable player-definition strategies (superpixel, fixed grid, custom masks, ViT patch tokens)
• masking.py — pluggable masking strategies (mean-color, zero/constant, token-based attention masking for ViTs)
• architecture.py — model-coupling layer that binds a player strategy and a masking strategy to a concrete model type (CNNArchitecture, TransformerArchitecture)
• imputer.py — ImageImputer(Imputer) that implements batched value_function evaluation over coalitions
• explainer.py — ImageExplainer(Explainer) that wires the imputer into the approximator pipeline
• utils.py — format-agnostic image conversion helpers (as_hwc_array, to_tensor_chw, etc.)

---

Public API Changes

• No Public API changes
• Yes, Public API changes (Details below)

No existing public APIs are modified or removed. We just implemented new ones for the shapiq.vision subpackage. See example:

        from shapiq.vision.architecture import CNNArchitecture, TransformerArchitecture
        from shapiq.vision.explainer import ImageExplainer

        # --- CNN (ResNet-style) ---
        arch = CNNArchitecture(model=my_resnet)
        explainer = ImageExplainer(model_architecture=arch, data=my_image)
        iv = explainer.explain_function(x=None, budget=256)

        # --- ViT ---
        arch = TransformerArchitecture(model=my_vit, vit_processor=processor)
        explainer = ImageExplainer(model_architecture=arch, data=my_image,
                                   index="SII", max_order=2)
        iv = explainer.explain_function(x=None, budget=512)

---

How Has This Been Tested?

Introduced 135 unit tests.
Manually verified end to end with ResNet-18 (torchvision) and ViT (google/vit-base-patch32-384) on a sample on ImageNet images.

---

Checklist

• The changes have been tested locally.
• Documentation has been updated (if the public API or usage changes).
• An entry has been added to CHANGELOG.md (if relevant for users).
• The code follows the project's style guidelines.
• I have considered the impact of these changes on the public API.

---

Notes

Draft PR for LMU Practical: Toolbox for Machine Learning (Group Vision subpackage)

We have tested different types of CNNs and ViTs for our provided architecture setup (ViT & CNN). More detailed information can be found in: S2k-1#26
We have tested 28 CNNs and 15 ViTs, out of those we achieved 23/28 for CNN (We already have the fix for the 5 missing ones) and for ViTs we achieved 2/15. We already planned on how to make the architecture more robust so we increase the amount of supported ViTs.

Due to missing testing, not all masking strategies that we experimented with are merged into this PR version. We are working on that.

[mmschlk]

Hello, thank you for your PR already! Please ping me here as soon as you want a PR review. :)

Note however, that for a review, the CI pipeline should really be green first (all tests pass, the code-quality checks are okay, and the docs building pipeline compiles). Otherwise it's hard for me to give you good feedback. :)

I can also approve the workflows from time to time if you ping me.

[S2k-1]

@mmschlk ty/ruff errors fixed. CI workflow on our fork only got errors for codecov upload. (expected due to missing token)
Please approve the workflow here :)

[t-muras]

We also added two examples to the documentation to showcase how to interact with the vision subpackage and how to define different player strategies. These are kept rather simple until now, we'll extend them after we decided how to refactor the architectures (after the feedback).

We also now copied the behavior from ShaplEIG and SPEX approximators in defining optional dependencies for the vision package.

[t-muras]

@mmschlk Could you approve the workflow, so we can verify the full pipeline runs successfully?

[S2k-1]

@Advueu963 could you please approve the workflow or ping @mmschlk

[codecov]

Codecov Report

❌ Patch coverage is 85.78947% with 81 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/shapiq/plot/vision.py	13.20%	46 Missing ⚠️
src/shapiq/vision/utils.py	83.51%	15 Missing ⚠️
src/shapiq/vision/players.py	94.31%	10 Missing ⚠️
src/shapiq/vision/__init__.py	58.33%	5 Missing ⚠️
src/shapiq/interaction_values.py	33.33%	2 Missing ⚠️
src/shapiq/vision/architecture.py	99.01%	1 Missing ⚠️
src/shapiq/vision/explainer.py	96.87%	1 Missing ⚠️
src/shapiq/vision/imputer.py	97.95%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[mmschlk]

Thank you already for your pull request and all of your work already on this matter!! :) I only had some minor comments (see individually). It's already pretty nice!

The only "major-ish" thing I have is also a bit of a question. Did we decided not to wirre ImageExplainer into the Explainer dispatching logic? So, we said we do not expect users to do

explainer = Explainer(image_model, ...)

and expect explainer to be ImageExplainer? Because I think this is not possible, yet. I would however, like to know weather it would be easy to allow users to call

explainer = Explainer(CNNArchitecture(model), ...)

and then get ImageExplainer. But then only these special architectures would work in the dispatching. Is this doable?

Thank you already!

[mmschlk]

we actually went to "SV" as default in v1.5.0 and order 1.

[mmschlk]

Decouple InteractionValues from the optional vision subpackage. InteractionValues is the central output type of the whole library and has to stay importable without torch/scikit-image. This method makes it depend on the optional vision subpackage (the ImageExplainer import up at line 32 even in the Type block), and unlike every other plot_* here it asks the caller to pass the explainer back in and then reaches into explainer.imputer.player_masks. Please move this out of the core type. We already have shapiq.plot.vision.image_attribution_plot; expose it there and have it take image + player_masks directly. The core class shouldn't know vision types exist.

[mmschlk]

A few parameters are out of step with our other explainers and I'd like parity before we can merge:

• budget is read from kwargs here instead of being an explicit argument (compare TabularExplainer.explain_function(x, budget=...)). Please make it a real signature parameter so it shows up in the docs/IDE.
• no random_state is this not also necessary here?

[mmschlk]

A few parameters are out of step with our other explainers and I'd like parity before we can merge:

• approximator (here in init): TabularExplainer lets users pass one; here it's hard-coded to "auto". Superpixel player counts get large, so users will want to swap it.
• class_index (here in init): base and TabularExplainer accept it; here the explained class is hard-coded to argmax inside the architecture, so a multi-class model can't choose what to explain. We need that toggle however. I would like to be able to do this in the init as well.

[mmschlk]

This hard-codes a 3x3 grid, but PatchStrategy.init requires grid_size % sqrt(n_players) == 0. For one of the most common ViT, vit-base-patch16-224, the grid_size is 224/16 = 14 and 14 % 3 != 0, so this raises ValueError on construction and the default path crashes for the standard model. The examples only work because they use patch32-384 (grid 12). Please make the default adapt to grid_size (pick a perfect square whose root divides the grid), and add a regression test against a 14x14 grid so this can't silently come back.

[mmschlk]

Heads-up, and this one isn't really your fault as we already have the same pattern in ShaplEIG. This eager from .architecture import ... pulls architecture.py (and its top-level import torch) at import-shapiq time, so even a tabular-only user pays ~0.5s of torch import. The try/except guards graceful failure but not laziness. Please defer this with a module-level getattr (PEP 562) so torch is only imported when ImageExplainer is actually used. Soon I will open a matching issue against ShaplEIG so we fix both consistently then.

[mmschlk]

Our Imputer is a Game, and the approximator is built once for a fixed n_players. fit() recomputes self.n_features from the new image — and for SuperpixelStrategy the SLIC count varies image to image — but the approximator built back in explainer.py:106 is never rebuilt. So explaining a second image with one explainer desyncs the game and approximator (shape mismatch or silently wrong values). For this we need to rebuild the approximator when n_features changes. Right now reusing an explainer across images is unsafe.