Fix MetricGroup requiring (y_pred, y) keys on mapping outputs - #3808
Open
ShreyanshGoyal wants to merge 2 commits into
Open
Fix MetricGroup requiring (y_pred, y) keys on mapping outputs#3808ShreyanshGoyal wants to merge 2 commits into
ShreyanshGoyal wants to merge 2 commits into
Conversation
MetricGroup.update() already applies each child metric's own
output_transform to pull what it needs out of the group's output, per the
documented contract ("output_transform of each metric in the group is also
called upon its update"). But MetricGroup inherited Metric.iteration_completed
unchanged, which enforces required_output_keys == ("y_pred", "y") whenever
the (group-transformed) engine output is a mapping -- and unconditionally
raises if the group's own required_output_keys is None. That check only
makes sense for a single "leaf" metric consuming (y_pred, y) directly; a
MetricGroup wrapping heterogeneous child metrics with their own
output_transforms has no business requiring specific keys on the raw output.
As a result, any engine returning a dict with keys other than 'y_pred'/'y'
(e.g. a multi-output model returning {"outputs_1": ..., "masks": ...}) broke
as soon as it was wrapped in a MetricGroup, even though attaching the same
child metric directly worked fine. Manually setting required_output_keys on
the group didn't help either: it triggered MetricGroup's inherited
multi-output "unrolling" logic, which reinterpreted the mapping as a
(y_pred, y)-shaped tuple and unrolled it into mismatched per-metric updates.
Fixes pytorch#3806.
MetricGroup.iteration_completed is now overridden to pass a mapping output
straight through to update() without the required_output_keys check, while
preserving the existing multi-output unrolling behavior (skip_unrolling) for
non-mapping outputs. Added a regression test that attaches the same Loss
metric directly and via a MetricGroup and asserts they produce identical
results, which fails with the original ValueError on the pre-fix code.
| loss_fn = torch.nn.MSELoss() | ||
|
|
||
| direct_engine = Engine(step) | ||
| Loss(loss_fn, output_transform=lambda o: o["outputs_1"]).attach(direct_engine, "loss") |
Collaborator
There was a problem hiding this comment.
to be consistent either don't save metric group, or save both metric object
Keep the directly attached and grouped Loss instances so the regression test can compare both their outputs and accumulated states, as requested in review.
Author
|
Thanks — addressed in 18e8dcc. The regression test now keeps both the directly attached and grouped |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #3806.
MetricGroup.update()already applies each child metric's ownoutput_transformto pull what it needs out of the group's output, per the documented contract:But
MetricGroupinheritedMetric.iteration_completedunchanged, which enforcesrequired_output_keys == ("y_pred", "y")whenever the (group-transformed) engine output is a mapping — and unconditionally raises if the group's ownrequired_output_keysisNone. That check only makes sense for a single "leaf" metric consuming(y_pred, y)directly; aMetricGroupwrapping heterogeneous child metrics with their ownoutput_transforms has no business requiring specific keys on the raw output.As a result, any engine returning a dict with keys other than
'y_pred'/'y'(e.g. a multi-output model returning{"outputs_1": ..., "masks": ...}) broke as soon as it was wrapped in aMetricGroup, even though attaching the same child metric directly worked fine:Manually setting
required_output_keyson the group didn't help either — it triggeredMetricGroup's inherited multi-output "unrolling" logic, which reinterpreted the mapping as a(y_pred, y)-shaped tuple and unrolled it into mismatched per-metric updates (each child's ownoutput_transformthen received a plain tensor pair instead of the original mapping).Fix
MetricGroup.iteration_completedis now overridden to pass a mapping output straight through toupdate()without therequired_output_keyscheck, while preserving the existing multi-output unrolling behavior (skip_unrolling) for non-mapping outputs — this is unchanged and still covered by the distributed integration test.Test plan
test_mapping_output_with_custom_keystotests/ignite/metrics/test_metric_group.py, which attaches the sameLossmetric directly and via aMetricGroupon an engine returning a dict with custom keys, and asserts both produce identical results. Confirmed this test fails with the originalValueErroron the pre-fix code and passes after the fix.pytest tests/ignite/metrics/test_metric_group.py -v— all non-distributed tests pass, plus thegloo_cpudistributed integration test (verifying the group-level unrolling path is untouched)pytest tests/ignite/metrics/(full suite, excluding modules needing optional deps I didn't have installed) — 4882 passed, 1 pre-existing unrelated failure (missingpynvml, unrelated to this change)mypy ignite/metrics/metric_group.pyandruff check/ruff format --check— clean