Commit df38aab
committed
[analytics-backend-datafusion] Apply schema coercion on indexed-executor placeholder
The indexed-execution path inferred the parquet schema via build_segments
(which calls FileFormat::infer_schema) and registered it directly on the
PlaceholderProvider that from_substrait_plan binds against. The non-indexed
paths (session_context.rs, query_executor.rs, api.rs) all routed their
inferred schemas through schema_coerce::coerce_inferred_schema before
registering, but the indexed path skipped this step.
Result: an OpenSearch `ip` column lands as parquet `BinaryView` on disk;
isthmus on the Java side serializes the Substrait base schema as plain
`Binary` (Substrait has no view types). The placeholder reports `BinaryView`
while the plan declares `Binary` — DataFusion's substrait consumer rejects
the bind with:
Substrait error: Field '<x>' in Substrait schema has a different type
(Binary) than the corresponding field in the table schema (BinaryView).
Every analytics-engine query against an index that includes an `ip` column
(every OTEL-logs query in CalciteSearchCommandIT, for example) fails at
fragment start.
Apply coerce_inferred_schema right after build_segments, before
PlaceholderProvider construction. The placeholder, the
expr_to_bool_tree analysis, and the downstream IndexedTableProvider all
see the same Substrait-compatible (Binary / Int64 / Float32) schema, so
the bind succeeds and the parquet reader's SchemaAdapter handles the
per-batch BinaryView→Binary relabeling at scan time.
This restores parity with the other infer_schema sites; no behavior
change for non-IP columns.
Signed-off-by: Kai Huang <ahkcs@amazon.com>1 parent 8fdff04 commit df38aab
1 file changed
Lines changed: 1 addition & 0 deletions
Lines changed: 1 addition & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
450 | 450 | | |
451 | 451 | | |
452 | 452 | | |
| 453 | + | |
453 | 454 | | |
454 | 455 | | |
455 | 456 | | |
| |||
0 commit comments