Commit be87334
fix(loss): exclude mixed_type padding atoms from the training loss (#5738)
## Problem
In the `mixed_type` data format, short frames are padded with `type =
-1` ghost atoms up to a fixed `nloc`, and the real atom count varies per
frame within a batch. The training loss normalized by the padded scalar
`natoms` and took unmasked or cross-frame-pooled means, so ghost atoms
diluted the force/atomic denominators and mis-normalized the extensive
energy/virial/property terms. As a result a padded `[3-atom + 5-atom]`
batch did not produce the same loss/gradient as processing the 3-atom
and 5-atom frames separately. Only `mixed_type` batches are affected;
non-mixed training was already exact.
## Fix
The per-atom mask (`atype >= 0`) now reaches the loss under the existing
`model_dict["mask"]` convention: pt_expt already propagates it; the pt
(torch.jit) backend recovers it from `atype` via a new
`TaskLoss._inject_atom_mask` helper called from every pt loss `forward`
(the exported forward drops the model's per-atom mask, so it is
recovered training-side only — the exported artifact is untouched).
Every loss term is then normalized per frame so that a padded batch
equals the grad-accumulation of the individual frames at their real
sizes: per-atom terms (force, atom_ener, atom_pref, atomic dos/tensor,
spin real-force, generalized force) use a per-frame masked mean;
extensive terms (energy, virial, property) divide by the per-frame real
atom count; global already-reduced terms (global dos/tensor) use a plain
mean with the previous atom-count weighting dropped. Every change
reduces exactly to the previous formula when the mask is all-ones, so
non-mixed training is numerically identical (no-op to rounding). Covered
across five shared loss types in both backends:
`deepmd/dpmodel/loss/{ener,ener_spin,dos,tensor,property}.py` (which
serve pt_expt) and
`deepmd/pt/loss/{ener,ener_spin,dos,tensor,property}.py`.
Two additional fixes surfaced during the work: the extensive property
normalization called `xp.sum(mask, -1)` with a positional axis, which
raises `TypeError` under the array_api_compat torch namespace (every
pt_expt extensive-property run) — now `axis=-1`; and `ener_spin`'s MAE
energy and real-force terms were pre-existingly inconsistent with
`ener.py` (they summed over frames without per-atom normalization) —
they are now aligned with `ener.py`, which changes their non-mixed MAE
loss values (a deliberate bug fix, see Known Limitations).
## Test
New `source/tests/common/dpmodel/test_loss_padding.py` and
`source/tests/pt/test_loss_padding.py` assert, for every per-atom and
extensive term of all five loss types in both backends, that a padded
`[3+5]` batch loss equals the mean of the two frames processed
separately, plus an all-ones-mask non-mixed no-op guard per term, and a
torch-tensor path through the dpmodel property loss (which reproduces
the positional-axis crash on the old form). An audit added invariant
coverage for the generalized-force and spin magnetic-force terms and
confirmed they are free of padding artifacts.
## Known limitations
The tf backend loss is unchanged and retains the same mixed_type
behavior (follow-up). The pt-only losses `dens`, `population`, `denoise`
are not covered (follow-up). `ener_spin`'s magnetic-force (`force_mag`)
MAE term uses a sum over frames rather than a mean, so it does not
satisfy the frame-average invariant — this is not a padding artifact
(ghost atoms are correctly excluded via `mask_mag`), but a separate
pre-existing MAE frame-normalization inconsistency, left for a follow-up
decision. The `enable_atom_ener_coeff` path sums ghost atomic energies
before the energy reduction (pre-existing; ghost atom_ener is ~0 by
convention). Existing `mixed_type` trainings will not reproduce
numerically — the new values are the correct ones. Ghost label forces
are assumed ~0 by the dpdata convention; the mask makes the loss robust
even if they are not.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added automatic atom-mask injection so padded/multi-frame inputs with
ghost atoms are handled consistently.
* **Bug Fixes**
* Reworked masked loss normalization to use per-frame masked-mean
reductions for energy, forces, virials, atom/property terms, DOS/CDF,
tensor L2, and spin losses.
* Standardized masked global DOS/CDF and global tensor L2 to use
unweighted mean squared error.
* Improved masking behavior for generalized-force projection and updated
RMSE/MAE reporting accordingly.
* **Tests**
* Added/extended gradient-accumulation and padding-mask invariance
suites across dpmodel and pt backends, including atom-mask injection
coverage.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Han Wang <wang_han@iapcm.ac.cn>1 parent a152b3b commit be87334
13 files changed
Lines changed: 5691 additions & 476 deletions
File tree
- deepmd
- dpmodel/loss
- pt/loss
- source/tests
- common/dpmodel
- pt
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
130 | 130 | | |
131 | 131 | | |
132 | 132 | | |
133 | | - | |
| 133 | + | |
134 | 134 | | |
135 | | - | |
136 | | - | |
137 | | - | |
138 | | - | |
139 | | - | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
140 | 142 | | |
141 | | - | |
| 143 | + | |
142 | 144 | | |
143 | 145 | | |
144 | 146 | | |
| |||
155 | 157 | | |
156 | 158 | | |
157 | 159 | | |
158 | | - | |
| 160 | + | |
159 | 161 | | |
160 | | - | |
161 | | - | |
162 | | - | |
163 | | - | |
164 | | - | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
165 | 169 | | |
166 | | - | |
| 170 | + | |
167 | 171 | | |
168 | 172 | | |
169 | 173 | | |
| |||
175 | 179 | | |
176 | 180 | | |
177 | 181 | | |
| 182 | + | |
| 183 | + | |
178 | 184 | | |
179 | | - | |
180 | | - | |
181 | | - | |
| 185 | + | |
| 186 | + | |
182 | 187 | | |
183 | | - | |
184 | 188 | | |
185 | 189 | | |
186 | | - | |
187 | 190 | | |
188 | 191 | | |
189 | 192 | | |
| |||
199 | 202 | | |
200 | 203 | | |
201 | 204 | | |
| 205 | + | |
| 206 | + | |
202 | 207 | | |
203 | | - | |
204 | | - | |
205 | | - | |
| 208 | + | |
| 209 | + | |
206 | 210 | | |
207 | | - | |
208 | 211 | | |
209 | 212 | | |
210 | | - | |
211 | 213 | | |
212 | 214 | | |
213 | 215 | | |
| |||
0 commit comments