Skip to content

[Code scan] Reject or remap incompatible type maps when merging LMDB datasets #5634

Description

@njzjz

Found during a Codex global scan of deepmodeling/deepmd-kit at commit 73de44b1f94471b2e3bdb6b11f57b34d7bc791bb.

Problem

merge_lmdb() copies raw frame bytes from all source LMDBs but keeps only the first source type_map in merged metadata.

Evidence:

  • Normal LMDB reading can remap atom-type indices when an LMDB type_map differs from the model type map:
    # Build type remapping if LMDB's type_map differs from model's type_map
    lmdb_type_map = meta.get("type_map")
    self._lmdb_type_map = lmdb_type_map
    self._type_remap: np.ndarray | None = None
    if lmdb_type_map is not None and list(lmdb_type_map) != list(type_map):
    # Build remap: lmdb_type_idx -> model_type_idx
    remap = np.empty(len(lmdb_type_map), dtype=np.int32)
    for i, name in enumerate(lmdb_type_map):
    if name not in type_map:
    raise ValueError(
    f"Element '{name}' in LMDB type_map {lmdb_type_map} "
    f"not found in model type_map {type_map}"
    )
    remap[i] = type_map.index(name)
    self._type_remap = remap
    log.info(
    f"Type remapping: LMDB {lmdb_type_map} -> model {type_map}, "
    f"remap={remap}"
    )
  • merge_lmdb() records the first source type_map:
    for src_path in src_paths:
    src_env = _open_lmdb(src_path)
    with src_env.begin() as txn:
    meta = _read_metadata(txn)
    nframes, src_fmt, natoms_per_type = _parse_metadata(meta)
    fallback_natoms = sum(natoms_per_type)
    if first_system_info is None:
    first_system_info = meta.get("system_info", {})
    if first_type_map is None:
    first_type_map = meta.get("type_map")
  • It then copies each frame's raw bytes without decoding or remapping atom-type indices:
    with src_env.begin() as src_txn, dst_env.begin(write=True) as dst_txn:
    for i in range(nframes):
    src_key = format(i, src_fmt).encode()
    raw = src_txn.get(src_key)
    if raw is None:
    continue
    dst_key = format(frame_idx, fmt).encode()
    dst_txn.put(dst_key, raw)
  • The merged metadata writes only the first type_map:
    # Write merged metadata with frame_nlocs for fast init
    merged_meta = {
    "nframes": frame_idx,
    "frame_idx_fmt": fmt,
    "system_info": first_system_info or {},
    "frame_nlocs": frame_nlocs,
    "frame_system_ids": frame_system_ids,
    }
    if first_type_map is not None:
    merged_meta["type_map"] = first_type_map
    with dst_env.begin(write=True) as txn:
    txn.put(b"__metadata__", msgpack.packb(merged_meta, use_bin_type=True))

Impact

If source LMDBs contain the same elements in different orders, frames from later sources are interpreted under the first source's type order. This silently swaps species after merging.

Suggested Fix

Reject source LMDBs with incompatible type_map values, or decode and remap atom_types while copying frames. Add a merge test with one source using ["O", "H"] and another using ["H", "O"], asserting that element identities survive the merge.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    In Progress

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions