Found during a Codex global scan of deepmodeling/deepmd-kit at commit 73de44b1f94471b2e3bdb6b11f57b34d7bc791bb.
Problem
merge_lmdb() copies raw frame bytes from all source LMDBs but keeps only the first source type_map in merged metadata.
Evidence:
- Normal LMDB reading can remap atom-type indices when an LMDB
type_map differs from the model type map:
|
# Build type remapping if LMDB's type_map differs from model's type_map |
|
lmdb_type_map = meta.get("type_map") |
|
self._lmdb_type_map = lmdb_type_map |
|
self._type_remap: np.ndarray | None = None |
|
if lmdb_type_map is not None and list(lmdb_type_map) != list(type_map): |
|
# Build remap: lmdb_type_idx -> model_type_idx |
|
remap = np.empty(len(lmdb_type_map), dtype=np.int32) |
|
for i, name in enumerate(lmdb_type_map): |
|
if name not in type_map: |
|
raise ValueError( |
|
f"Element '{name}' in LMDB type_map {lmdb_type_map} " |
|
f"not found in model type_map {type_map}" |
|
) |
|
remap[i] = type_map.index(name) |
|
self._type_remap = remap |
|
log.info( |
|
f"Type remapping: LMDB {lmdb_type_map} -> model {type_map}, " |
|
f"remap={remap}" |
|
) |
merge_lmdb() records the first source type_map:
|
for src_path in src_paths: |
|
src_env = _open_lmdb(src_path) |
|
with src_env.begin() as txn: |
|
meta = _read_metadata(txn) |
|
nframes, src_fmt, natoms_per_type = _parse_metadata(meta) |
|
fallback_natoms = sum(natoms_per_type) |
|
|
|
if first_system_info is None: |
|
first_system_info = meta.get("system_info", {}) |
|
if first_type_map is None: |
|
first_type_map = meta.get("type_map") |
|
|
- It then copies each frame's raw bytes without decoding or remapping atom-type indices:
|
with src_env.begin() as src_txn, dst_env.begin(write=True) as dst_txn: |
|
for i in range(nframes): |
|
src_key = format(i, src_fmt).encode() |
|
raw = src_txn.get(src_key) |
|
if raw is None: |
|
continue |
|
dst_key = format(frame_idx, fmt).encode() |
|
dst_txn.put(dst_key, raw) |
- The merged metadata writes only the first
type_map:
|
# Write merged metadata with frame_nlocs for fast init |
|
merged_meta = { |
|
"nframes": frame_idx, |
|
"frame_idx_fmt": fmt, |
|
"system_info": first_system_info or {}, |
|
"frame_nlocs": frame_nlocs, |
|
"frame_system_ids": frame_system_ids, |
|
} |
|
if first_type_map is not None: |
|
merged_meta["type_map"] = first_type_map |
|
with dst_env.begin(write=True) as txn: |
|
txn.put(b"__metadata__", msgpack.packb(merged_meta, use_bin_type=True)) |
Impact
If source LMDBs contain the same elements in different orders, frames from later sources are interpreted under the first source's type order. This silently swaps species after merging.
Suggested Fix
Reject source LMDBs with incompatible type_map values, or decode and remap atom_types while copying frames. Add a merge test with one source using ["O", "H"] and another using ["H", "O"], asserting that element identities survive the merge.
Found during a Codex global scan of
deepmodeling/deepmd-kitat commit73de44b1f94471b2e3bdb6b11f57b34d7bc791bb.Problem
merge_lmdb()copies raw frame bytes from all source LMDBs but keeps only the first sourcetype_mapin merged metadata.Evidence:
type_mapdiffers from the model type map:deepmd-kit/deepmd/dpmodel/utils/lmdb_data.py
Lines 301 to 319 in 73de44b
merge_lmdb()records the first sourcetype_map:deepmd-kit/deepmd/dpmodel/utils/lmdb_data.py
Lines 1767 to 1778 in 73de44b
deepmd-kit/deepmd/dpmodel/utils/lmdb_data.py
Lines 1784 to 1791 in 73de44b
type_map:deepmd-kit/deepmd/dpmodel/utils/lmdb_data.py
Lines 1824 to 1835 in 73de44b
Impact
If source LMDBs contain the same elements in different orders, frames from later sources are interpreted under the first source's type order. This silently swaps species after merging.
Suggested Fix
Reject source LMDBs with incompatible
type_mapvalues, or decode and remapatom_typeswhile copying frames. Add a merge test with one source using["O", "H"]and another using["H", "O"], asserting that element identities survive the merge.