feat(metrics): add Expected and Maximum Calibration Error metrics#3799
Open
rpp5501 wants to merge 3 commits into
Open
feat(metrics): add Expected and Maximum Calibration Error metrics#3799rpp5501 wants to merge 3 commits into
rpp5501 wants to merge 3 commits into
Conversation
Add ExpectedCalibrationError and MaximumCalibrationError to ignite.metrics, implementing the model-calibration metrics requested in pytorch#1009 ("as coded in Baal"). Both metrics share a fixed-size, per-bin accumulator (sum of confidences, correct count, and sample count per bin) updated in place with index_add_, so the retained state is O(num_bins) regardless of dataset size. This avoids the O(N) torch.cat growth of the prior attempt (pytorch#3132), which OOMs on large datasets. compute() is fully vectorized (no Python loop over bins) and uses @reinit__is_reduced / @sync_all_reduce for correct reduction under distributed training. y_pred is expected to be probabilities; binary (B,)/(B, 1) and multiclass (B, C)/(B, C, ...) inputs are supported. Register both metrics in ignite.metrics, document them in docs/source/metrics.rst, and add unit and distributed tests validated against a NumPy reference of the Baal binning. Closes pytorch#1009
Contributor
There was a problem hiding this comment.
Pull request overview
Adds model calibration metrics to ignite.metrics by introducing Expected Calibration Error (ECE) and Maximum Calibration Error (MCE), with an O(num_bins) accumulator design suitable for large datasets and distributed training.
Changes:
- Introduces
ExpectedCalibrationErrorandMaximumCalibrationErrormetrics with per-bin accumulators and distributed reduction support. - Adds unit and distributed tests validated against a NumPy reference implementation.
- Registers the metrics in
ignite.metricsand includes them in the metrics documentation list.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| tests/ignite/metrics/test_calibration_error.py | Adds reference-based unit tests and distributed integration tests for ECE/MCE. |
| ignite/metrics/calibration_error.py | Implements shared bin-accumulation logic and the ECE/MCE metric computations. |
| ignite/metrics/init.py | Exposes the new metrics at the ignite.metrics package level. |
| docs/source/metrics.rst | Lists the new metrics in the documentation’s “Complete list of metrics”. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| Examples: | ||
| To use with ``Engine`` and ``process_function``, simply attach the metric instance to the engine. | ||
| The output of the engine's ``process_function`` needs to be in the format of | ||
| ``(y_pred, y)`` or ``{'y_pred': y_pred, 'y': y, ...}``. If not, ``output_tranform`` can be added |
Comment on lines
+109
to
+112
| n_iters = y.shape[0] // batch_size + 1 | ||
| for i in range(n_iters): | ||
| idx = i * batch_size | ||
| metric.update((y_pred[idx : idx + batch_size], y[idx : idx + batch_size])) |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1009
Description
Adds
ExpectedCalibrationErrorandMaximumCalibrationErrortoignite.metrics,implementing the model-calibration metrics requested in #1009 ("as coded in Baal").
Both metrics share a fixed-size, per-bin accumulator (sum of confidences, correct
count, and sample count per bin) updated in place with
index_add_, so the retainedstate is O(num_bins) regardless of dataset size. This avoids the O(N)
torch.catgrowth of the prior attempt (#3132), which OOMs on large datasets.compute()is fully vectorized (no Python loop over bins) and uses@reinit__is_reduced/@sync_all_reducefor correct reduction under distributedtraining.
y_predis expected to be probabilities; binary(B,)/(B, 1)andmulticlass
(B, C)/(B, C, ...)inputs are supported.Registers both metrics in
ignite.metrics, documents them indocs/source/metrics.rst, and adds unit and distributed tests validated against aNumPy reference of the Baal binning.
Check list: