Bug: pppm/dplr + fix dplr hangs or crashes with multi-MPI (except ntasks=1), including inconsistent behavior with restart and read_data

[Ziyang-You]

When using pair_style deepmd + fix dplr + kspace_style pppm/dplr, the simulation only runs reliably with 1 MPI process (ntasks=1). With multiple MPI processes (ntasks ≥ 2), it frequently hangs indefinitely during PPPM initialization or crashes with MPI collective errors. The behavior is inconsistent — some systems (from read_data) can run with multi-MPI, while others (especially from read_restart) fail.

This makes DPLR simulations impractical on HPC clusters, as single-MPI runs are extremely slow.

Environment

• DeepMD-kit version: v3.1.3 (commit b2c8511, 2026-03-18)
• LAMMPS version: 22 Jul 2025 - Update 2 (stable_22Jul2025_update2)
• Backend: CPU (TensorFlow + PyTorch)
• Architecture: aarch64 (Linux)
• Installation: pip install deepmd-kit[cpu,lmp]

Input script, sbatch inputs and out
in.zip

[njzjz-bot]

Thanks for the detailed report. I had a first look at the DPLR PPPM code path, and there is a suspicious mismatch for recent LAMMPS versions (LAMMPS_VERSION_NUMBER >= 20221222) in source/lmp/pppm_dplr.cpp.

After poisson(), the AD-differentiation branch currently does a reverse communication with REVERSE_RHO:

if (differentiation_flag == 1)
#if LAMMPS_VERSION_NUMBER >= 20221222
    gc->reverse_comm(Grid3d::KSPACE, this, REVERSE_RHO, 1, sizeof(FFT_SCALAR),
                     gc_buf1, gc_buf2, MPI_FFT_SCALAR);

For the older GridComm path this is a forward communication with FORWARD_AD, and the IK branch for Grid3d is also a forward communication. So this looks like a typo introduced when GridComm was migrated to Grid3d. The likely fix is:

-    gc->reverse_comm(Grid3d::KSPACE, this, REVERSE_RHO, 1, sizeof(FFT_SCALAR),
-                     gc_buf1, gc_buf2, MPI_FFT_SCALAR);
+    gc->forward_comm(Grid3d::KSPACE, this, FORWARD_AD, 1, sizeof(FFT_SCALAR),
+                     gc_buf1, gc_buf2, MPI_FFT_SCALAR);

This kind of wrong collective/communication direction is consistent with multi-MPI hangs or MPI collective errors while still being invisible in ntasks=1.

Could you try the patch above on your failing input? As a temporary workaround, if it is acceptable for your calculation, you may also try switching away from the AD branch, e.g. kspace_modify diff ik, and compare results carefully.

If the case still fails after this patch, there may be a second issue around DPLR bonded pairs after read_restart/domain decomposition, so a minimal reproducer that contains both the read_data and read_restart variants would be very helpful.

Authored by OpenClaw (model: custom-chat-jinzhezeng-group/gpt-5.5)

[njzjz]

@Yi-FanLi please take a look here. It was made by you in #2300.

deepmd-kit/source/lmp/pppm_dplr.cpp

Lines 145 to 154 in 4b6506d

	#if LAMMPS_VERSION_NUMBER >= 20221222
	gc->reverse_comm(Grid3d::KSPACE, this, REVERSE_RHO, 1, sizeof(FFT_SCALAR),
	gc_buf1, gc_buf2, MPI_FFT_SCALAR);
	#elif LAMMPS_VERSION_NUMBER >= 20210831 && LAMMPS_VERSION_NUMBER < 20221222
	gc->forward_comm(GridComm::KSPACE, this, 1, sizeof(FFT_SCALAR), FORWARD_AD,
	gc_buf1, gc_buf2, MPI_FFT_SCALAR);
	#else
	gc->forward_comm_kspace(this, 1, sizeof(FFT_SCALAR), FORWARD_AD, gc_buf1,
	gc_buf2, MPI_FFT_SCALAR);
	#endif

[Yi-FanLi]

I truely made a mistake there. Fixed it in #5567 .