Skip to content

FragmentMetadata.to_json() fails with stable row IDs enabled #6403

Description

@pengw0048

FragmentMetadata.to_json() raises NotImplementedError when the fragment has stable row ID metadata (row_id_meta is not None). This makes it impossible to serialize fragment metadata via JSON when enable_stable_row_ids=True.

Minimal repro

import lance
import pyarrow as pa

uri = "/tmp/repro_to_json.lance"
ds = lance.write_dataset(pa.table({"x": [1, 2, 3]}), uri, enable_stable_row_ids=True)
ds.get_fragments()[0].metadata.to_json()  # raises NotImplementedError

Error

NotImplementedError: PyRowIdMeta.asdict is not yet supported.

The traceback points to lance/fragment.py:122:

self.row_id_meta.asdict() if self.row_id_meta is not None else None

Impact

Any workflow that round-trips fragment metadata through JSON breaks when stable row IDs are enabled. For example, serializing FragmentMetadata via json.dumps(frag.to_json()) after fragment.update_columns() for later reconstruction with FragmentMetadata.from_json() no longer works. This is needed for distributed workflows where fragment metadata must be serialized for transport between processes.

Environment

  • lance 5.0.0-beta.4
  • Python 3.11
  • Linux x86_64

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions