fix(table): fall back on invalid global index metadata

[XiaoHongbo-Hope]

Purpose

Linked issue: close #577

Missing or unparseable sorted global-index metadata was converted to an empty BTreeIndexMeta. That value is interpreted as an only-null index, so file-level pruning could return no matching ranges without opening an index file that actually contains matching rows.

This was reproduced on main at a9a8f6b: the regression test returned an indexed result instead of falling back to the normal table scan. Malformed key lengths could also panic during metadata deserialization before fallback was possible.

A sorted entry without its outer GlobalIndexMeta cannot be attributed to a field or row range, so scanner creation must fall back entirely. For entries with outer metadata, one unavailable shard makes that field fall back; simply skipping the shard would still omit its rows.

Brief change log

• Fall back entirely when a sorted index entry lacks its outer GlobalIndexMeta.
• Preserve unavailable sorted-index metadata instead of substituting only-null metadata.
• Fall back for a field when any of its index shards has unavailable metadata.
• Validate serialized first-key and last-key lengths before slicing.
• Reject a one-byte, truncated versioned metadata trailer.
• Cover mixed valid/invalid shards, missing outer metadata, malformed lengths, and truncated trailers.

Tests

• cargo fmt --all -- --check
• cargo clippy -p paimon --all-targets -- -D warnings
• cargo test -p paimon --all-targets
  • 1726 passed, 1 ignored

API and Format

No API or storage-format changes. Malformed or truncated BTree metadata now returns InvalidData instead of panicking or being interpreted as legacy metadata.

Documentation

No documentation changes are required for this correctness fix.

[JingsongLi]

Blocking: malformed global-index metadata is currently treated as an unavailable optimization and silently falls back to a normal table scan.

BTreeIndexMeta::deserialize(bytes).ok() discards the InvalidData error, and evaluate_leaf later returns Ok(None) when any shard has unavailable metadata. This makes a corrupted or incomplete declared BTree/Bitmap index indistinguishable from an unsupported predicate or an absent index. It also hides storage corruption and may unexpectedly turn an indexed query into a full scan without any warning.

The bounds checks added to BTreeIndexMeta::deserialize are correct: malformed external data must return a structured error instead of panicking. However, that error should be propagated as Error::DataInvalid, consistent with failures while opening or querying the physical index file.

Please consider changing GlobalIndexScanner::create to return Result<Option<Self>>, reserving Ok(None) for cases where no applicable index exists or the predicate cannot be evaluated. Missing required metadata and deserialization failures for a declared sorted index should return Err. The regression tests should assert an error rather than a successful fallback.