…ers (deepmodeling#5455)

## Summary

- Adds an explicit `del lammps` before `MPI.Finalize()` in all four
mpirun-driven LAMMPS test runners (`run_mpi_pair_deepmd.py`,
`run_mpi_pair_deepmd_spin.py`, `run_mpi_pair_deepmd_dpa3_pt2.py`,
`run_mpi_pair_deepmd_spin_dpa3_pt2.py`).
- Fixes a teardown-order race that intermittently manifests as
**subprocess exit code 136 (SIGFPE)** for
`test_pair_deepmd_mpi_dpa3_spin_empty_subdomain` on the GitHub Actions
CUDA runner image.

## Background

Recent CI runs on multiple unrelated PRs (deepmodeling#5446, deepmodeling#5450) hit the
identical failure signature:

```
short test summary info ============================
FAILED source/lmp/tests/test_lammps_spin_dpa3_pt2.py::test_pair_deepmd_mpi_dpa3_spin_empty_subdomain
  - subprocess.CalledProcessError: ... returned non-zero exit status 136.
```

- Reproduces ~1 in 5 runs on the GitHub Actions CUDA image
(`nvidia/cuda:12.9.1-cudnn-devel-ubuntu22.04`).
- **Does not reproduce on a V100 Bohrium dev box** — 60/60 consecutive
passes.

So it's a pre-existing flake, not caused by either of the recent PRs.

## Root cause (empirically confirmed)

The runner ends with:

```python
forces_global = lammps.lmp.gather_atoms(...)
...
MPI.Finalize()
```

`lammps` is still alive when `MPI.Finalize()` returns. Python then
garbage-collects it during interpreter shutdown, which triggers
`LAMMPS::~LAMMPS` → `Finish::end()` → **`MPI_Allreduce`** for timing
aggregation. By that time, MPI has already been finalized, which is
undefined behavior.

I instrumented the runner with timestamped prints to verify the order
directly. Without the fix:

```
t=3311.770  R1: BEFORE MPI.Finalize
t=3311.778  R0/R1: AFTER MPI.Finalize     ← MPI is finalized
t=3311.778  R0/R1: PY ATEXIT
… process exit, LAMMPS destructor runs HERE
```

With the fix:

```
t=3423.100  R1: AFTER del lammps (LAMMPS destructor done)   ← MPI still up
t=3423.108  R0/R1: BEFORE MPI.Finalize
t=3423.108  R0/R1: AFTER MPI.Finalize
```

So the LAMMPS destructor now runs while MPI is still up, which is what
its `MPI_Allreduce`/`MPI_Gather` calls require.

The reason this manifests as SIGFPE only on the CUDA CI image (not on
V100) is most likely that the CI image (or one of its preloaded
libraries) enables FP-exception trapping; on V100 the same
MPI-after-Finalize errors return silently. The flake is
environment-specific, but the underlying antipattern is unconditional
and worth fixing in any environment.

## Test plan

- [x] Local CPU: 29/29 LAMMPS tests pass (`test_lammps_dpa3_pt2.py`,
`test_lammps_spin_dpa3_pt2.py`)
- [x] Remote V100: 50/50 stress runs of the previously-failing test
- [x] Empirical confirmation that the fix flips the
LAMMPS-destructor-vs-MPI.Finalize ordering (see Background)
- [ ] CI: re-run the spin LAMMPS suite multiple times to confirm the
SIGFPE no longer appears

## Known limitations

- Cannot directly observe the SIGFPE on V100, so the fix has not been
observed *preventing* the actual crash — only correcting the antipattern
that we have strong reason to believe causes it.
- If the failure persists after merge, the next candidate root cause is
CUDA stream destruction order, and we should revisit.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved MPI cleanup sequence in multiple test runners to prevent
finalization-related crashes when executing tests in distributed MPI
environments.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/deepmodeling/deepmd-kit/pull/5455?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Han Wang <wang_han@iapcm.ac.cn>

…ling#5467)

## Summary

`TestChangeBias` is the dominant memory hog in `Test Python` shard `(10,
3.13)` of the CI matrix — by itself it peaks at **~5 GB RSS**, leaving
so little headroom under the 7 GB GitHub-hosted runner that the shard
intermittently loses communication with the GitHub Actions server. This
causes the recurring `runner lost communication` failure that has
affected many recent PRs (deepmodeling#5446, deepmodeling#5448, deepmodeling#5450, deepmodeling#5455, deepmodeling#5456, …).

This PR shrinks the change-bias test dataset from 80 frames to 5 frames,
dropping the class's peak RSS to **~1.7 GB** while keeping all 9 tests
passing — including the strict `atol=1e-10` `pt2_pte_consistency` check.

## How I located it

Local reproduction of shard `(10, 3.13)` (using the same
`.test_durations` cache CI uses, so identical test partitioning):

| Test profiled in isolation | Peak RSS |
|---|---|
| **`test_change_bias_frozen_pte`** | **5.04 GB** ← outlier |
| `TestDeepEvalEnerPt2` class | 1.43 GB |
| `TestDeepEvalEnerAparamPt2` class | 1.44 GB |
| `TestSpinInference::test_get_use_spin` | 1.41 GB |
| `test_finetune_from_pt2_use_pretrain_script` | 1.41 GB |
| `test_training_loop_compiled` | 1.32 GB |
| `test_export_pipeline` | 1.57 GB |
| `test_descriptor_shape_dpa1` | 1.34 GB |

Then phase-by-phase RSS profiling inside `dp change-bias` showed the 4.3
GB jump happens entirely inside `compute_output_stats` →
`_compute_model_predict`. Scaling experiment confirms it: peak grows
**linearly at ~50 MB per frame** of input data.

| nbatches | Peak RSS | per-frame |
|---|---|---|
| 1  | 567 MB | — |
| 5  | 781 MB | +43 MB |
| 20 | 1583 MB | +53 MB |
| 80 | 4797 MB | +53 MB |

That's a leak in the `torch.no_grad()`-wrapped `forward_common_atomic`
somewhere — separate from autograd. The water example has 80 frames at
batch_size=1, so the CLI default `nbatches = min(data.get_nbatches()) =
80` triggers all 80 forwards in one go.

## Why I can't just pass `-n 5`

`_load_batch_set` shuffles when it loads the set. If `nbatches <
total_frames`, the loop samples a random subset — and the two calls in
`test_change_bias_pt2_pte_consistency` (running in the **same Python
process** via `main(cmds)`, with `dp_random`'s state advancing between
calls) would see **different** subsets → different biases → the
`atol=1e-10` assertion fails.

`nbatches == total_frames` makes the forward enumerate **every** frame
regardless of shuffle order, so the aggregate bias is invariant under
shuffle. Determinism is preserved.

## The fix

Build a 5-frame subset of `examples/water/data/data_0` in
`TestChangeBias.setUpClass` and point both the trainer config and the
change-bias `-s` argument at it. `nbatches` then resolves to 5 (= the
new dataset size = full enumeration), and all 9 tests pass with peak RSS
at ~1.7 GB.

## Test plan

- [x] All 9 tests in `TestChangeBias` pass locally (CPU fp64):
  - `test_change_bias_with_data`
  - `test_change_bias_with_data_sys_file`
  - `test_change_bias_with_user_defined`
  - `test_change_bias_frozen_pte`
  - `test_change_bias_frozen_pt2`
  - `test_change_bias_frozen_pt2_user_defined`
- `test_change_bias_pt2_pte_consistency` (atol=1e-10 — the
determinism-sensitive one)
  - `test_change_bias_pte_preserves_model_def_script`
  - `test_change_bias_pt2_preserves_model_def_script`
- [x] Peak RSS measurement (kernel `ru_maxrss`):
  - Before: 5.66 GB
  - After: **1.62 GB** (single test) / **1.75 GB** (whole class)
- [ ] CI shard `(10, 3.13)` confirms no more `runner lost communication`
on this branch (pending CI run)

## Known limitations

- **The underlying ~50 MB/frame leak in `forward_common_atomic` remains
as a production bug.** Users running `dp change-bias` on real datasets
with thousands of frames will see multi-GB RSS growth. Worth a separate
follow-up to find and patch the leak.
- The 5-frame number is somewhat arbitrary. It's chosen as the smallest
value that (a) keeps the assertion logic working, (b) leaves room for
the least-squares regression (need ≥ ntypes = 2 frames).
- `_make_subset_dataset` only handles `set.000`; multi-set datasets
would need extension. Not needed for water/data_0.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Reduced the dataset used by a change-bias test to a small, fixed
subset (5 frames) so test runs use far less disk space and complete
faster.
* Test setup now builds and points to the truncated dataset for all
related invocations, lowering resource overhead during CI and local
testing.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/deepmodeling/deepmd-kit/pull/5467?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Han Wang <wang_han@iapcm.ac.cn>