Skip to content

Parquet: Compute geometry bounding box metrics#17161

Open
huan233usc wants to merge 1 commit into
apache:mainfrom
huan233usc:geo-parquet-bbox
Open

Parquet: Compute geometry bounding box metrics#17161
huan233usc wants to merge 1 commit into
apache:mainfrom
huan233usc:geo-parquet-bbox

Conversation

@huan233usc

Copy link
Copy Markdown
Contributor

Computes a bounding box for geometry columns in Parquet so they can carry
lower_bounds/upper_bounds and participate in data skipping. This is the first
slice of the geo bounds work (Phase 2), scoped to the clean, unambiguous case.

Problem

The Parquet footer's lexicographic min/max over WKB bytes is not meaningful for
spatial values, so ParquetMetrics diverts geometry/geography columns to
counts-only metrics. As a result geometry columns get no bounds and cannot be
data-skipped.

Approach

Compute the box at write time, since the footer cannot provide it. This reuses
the existing writer-side value-scanning metrics channel — the same path float
and double already use to track NaN counts the footer can't give
(ParquetValueWriter.metrics()ParquetWriter.metrics()):

  • A new GeometryWriter writes byte-identical WKB and folds each value's XY
    coordinates into a running box, emitting a FieldMetrics<GeospatialBound> whose
    lower/upper corners are the box. These serialize into the existing
    lower_bounds/upper_bounds maps through the geometry Conversions case. No
    changes to ParquetMetrics, ParquetWriter, or Conversions.
  • The value-scanned metrics take precedence over the counts-only footer branch, and
    the optional-field writer reconciles the null count, so the builder tracks only
    non-null values.

WKBBoundingBox (new, in api/geospatial next to GeospatialBound/BoundingBox)
is a pure-Java WKB coordinate scanner — no JTS dependency. It walks all OGC
geometry types (point, linestring, polygon, multi*, collection), handles both byte
orders (per-geometry, so mixed-endian collections work), reads past Z/M (XY-only
box), skips NaN coordinates (so POINT EMPTY contributes nothing, per spec), and
validates the buffer defensively (truncation, bad byte order, unknown type, and
oversized counts all raise IllegalArgumentException rather than reading out of
bounds).

Scope

GEOMETRY only, 2D (XY), planar — the case the spec calls straightforward and
unambiguous. Deliberately left as follow-ups:

  • GEOGRAPHY bounds (longitude periodicity, latitude extrema via edges, numeric
    correction, out-of-range coordinate policy) — needs its own numeric design.
  • Higher dimensions (XYZM), the content_stats geo_lower/geo_upper struct write
    bridge, ORC/Avro geo bounds, and CRS validation.

Tests

  • TestWKBBoundingBox — the scanner across all geometry types, Z/M skip, both/mixed
    endianness, nested collections, empties, NaN skip, ±Inf, and malformed inputs.
  • TestGeometryFieldMetrics — the builder's bounds, counts, and empty/no-value cases.
  • TestMetrics.testMetricsForGeospatialTypes (new, exercised via TestParquetMetrics)
    — end-to-end: two geometry points produce the expected bbox in lower_bounds/
    upper_bounds; geography stays bounds-less; ORC is skipped (geo is Parquet-only).
  • TestParquetDataWriter.testGeospatialRoundTrip still passes unchanged, confirming
    the WKB bytes written are byte-identical.

The Parquet footer's lexicographic min/max over WKB bytes is not meaningful for
geometry, so geometry columns previously wrote counts only and could not be data
skipped. Scan each value's coordinates as it is written and accumulate a 2D (XY)
bounding box, emitting it as a FieldMetrics through the writer-side metrics
channel (the same path float and double use for NaN counts). The box's lower and
upper corners serialize into the existing lower_bounds and upper_bounds maps via
the geometry conversion.

A new pure-Java WKB coordinate scanner (WKBBoundingBox, in api/geospatial, no JTS
dependency) walks all OGC geometry types, skips Z and M, ignores NaN coordinates,
and validates the buffer defensively. Geography, higher dimensions, and the
content_stats geo struct bounds are left as follow-ups.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant