Skip to content

Fix MetricGroup requiring (y_pred, y) keys on mapping outputs - #3808

Open
ShreyanshGoyal wants to merge 2 commits into
pytorch:masterfrom
ShreyanshGoyal:fix/metricgroup-output-unrolling
Open

Fix MetricGroup requiring (y_pred, y) keys on mapping outputs#3808
ShreyanshGoyal wants to merge 2 commits into
pytorch:masterfrom
ShreyanshGoyal:fix/metricgroup-output-unrolling

Conversation

@ShreyanshGoyal

Copy link
Copy Markdown

Summary

Fixes #3806.

MetricGroup.update() already applies each child metric's own output_transform to pull what it needs out of the group's output, per the documented contract:

output_transform of each metric in the group is also called upon its update.

But MetricGroup inherited Metric.iteration_completed unchanged, which enforces required_output_keys == ("y_pred", "y") whenever the (group-transformed) engine output is a mapping — and unconditionally raises if the group's own required_output_keys is None. That check only makes sense for a single "leaf" metric consuming (y_pred, y) directly; a MetricGroup wrapping heterogeneous child metrics with their own output_transforms has no business requiring specific keys on the raw output.

As a result, any engine returning a dict with keys other than 'y_pred'/'y' (e.g. a multi-output model returning {"outputs_1": ..., "masks": ...}) broke as soon as it was wrapped in a MetricGroup, even though attaching the same child metric directly worked fine:

loss = Loss(MSE, output_transform=lambda o: o["outputs_1"])
loss.attach(train_validator, "loss")  # works

train_metrics = MetricGroup({"loss": loss})
train_metrics.attach(train_validator, "train_metrics")  # ValueError: ... should contain ('y_pred', 'y') keys, but given ['outputs_1', 'masks']

Manually setting required_output_keys on the group didn't help either — it triggered MetricGroup's inherited multi-output "unrolling" logic, which reinterpreted the mapping as a (y_pred, y)-shaped tuple and unrolled it into mismatched per-metric updates (each child's own output_transform then received a plain tensor pair instead of the original mapping).

Fix

MetricGroup.iteration_completed is now overridden to pass a mapping output straight through to update() without the required_output_keys check, while preserving the existing multi-output unrolling behavior (skip_unrolling) for non-mapping outputs — this is unchanged and still covered by the distributed integration test.

Test plan

  • Added test_mapping_output_with_custom_keys to tests/ignite/metrics/test_metric_group.py, which attaches the same Loss metric directly and via a MetricGroup on an engine returning a dict with custom keys, and asserts both produce identical results. Confirmed this test fails with the original ValueError on the pre-fix code and passes after the fix.
  • pytest tests/ignite/metrics/test_metric_group.py -v — all non-distributed tests pass, plus the gloo_cpu distributed integration test (verifying the group-level unrolling path is untouched)
  • pytest tests/ignite/metrics/ (full suite, excluding modules needing optional deps I didn't have installed) — 4882 passed, 1 pre-existing unrelated failure (missing pynvml, unrelated to this change)
  • mypy ignite/metrics/metric_group.py and ruff check/ruff format --check — clean

MetricGroup.update() already applies each child metric's own
output_transform to pull what it needs out of the group's output, per the
documented contract ("output_transform of each metric in the group is also
called upon its update"). But MetricGroup inherited Metric.iteration_completed
unchanged, which enforces required_output_keys == ("y_pred", "y") whenever
the (group-transformed) engine output is a mapping -- and unconditionally
raises if the group's own required_output_keys is None. That check only
makes sense for a single "leaf" metric consuming (y_pred, y) directly; a
MetricGroup wrapping heterogeneous child metrics with their own
output_transforms has no business requiring specific keys on the raw output.

As a result, any engine returning a dict with keys other than 'y_pred'/'y'
(e.g. a multi-output model returning {"outputs_1": ..., "masks": ...}) broke
as soon as it was wrapped in a MetricGroup, even though attaching the same
child metric directly worked fine. Manually setting required_output_keys on
the group didn't help either: it triggered MetricGroup's inherited
multi-output "unrolling" logic, which reinterpreted the mapping as a
(y_pred, y)-shaped tuple and unrolled it into mismatched per-metric updates.

Fixes pytorch#3806.

MetricGroup.iteration_completed is now overridden to pass a mapping output
straight through to update() without the required_output_keys check, while
preserving the existing multi-output unrolling behavior (skip_unrolling) for
non-mapping outputs. Added a regression test that attaches the same Loss
metric directly and via a MetricGroup and asserts they produce identical
results, which fails with the original ValueError on the pre-fix code.
@github-actions github-actions Bot added the module: metrics Metrics module label Jul 13, 2026
loss_fn = torch.nn.MSELoss()

direct_engine = Engine(step)
Loss(loss_fn, output_transform=lambda o: o["outputs_1"]).attach(direct_engine, "loss")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to be consistent either don't save metric group, or save both metric object

Keep the directly attached and grouped Loss instances so the regression test can compare both their outputs and accumulated states, as requested in review.
@ShreyanshGoyal

Copy link
Copy Markdown
Author

Thanks — addressed in 18e8dcc. The regression test now keeps both the directly attached and grouped Loss instances and compares their state dictionaries as well as their computed outputs. The full MetricGroup test file passes (5 passed, 5 skipped for unavailable distributed backends).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module: metrics Metrics module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Attaching every loss separately is different from attaching them with MetricGroup

2 participants