Commit f8fee15
committed
[analytics-backend-datafusion] Apply schema coercion on indexed-executor placeholder
The indexed-execution path inferred the parquet schema via build_segments
(which calls FileFormat::infer_schema) and registered it directly on the
PlaceholderProvider that from_substrait_plan binds against. The non-indexed
paths (session_context.rs, query_executor.rs, api.rs) all routed their
inferred schemas through schema_coerce::coerce_inferred_schema before
registering, but the indexed path skipped this step.
Result: an OpenSearch `ip` column lands as parquet `BinaryView` on disk;
isthmus on the Java side serializes the Substrait base schema as plain
`Binary` (Substrait has no view types). The placeholder reports `BinaryView`
while the plan declares `Binary` — DataFusion's substrait consumer rejects
the bind with:
Substrait error: Field '<x>' in Substrait schema has a different type
(Binary) than the corresponding field in the table schema (BinaryView).
Every analytics-engine query against an index that includes an `ip` column
(every OTEL-logs query in CalciteSearchCommandIT, for example) fails at
fragment start.
Apply coerce_inferred_schema right after build_segments, before
PlaceholderProvider construction. The placeholder, the
expr_to_bool_tree analysis, and the downstream IndexedTableProvider all
see the same Substrait-compatible (Binary / Int64 / Float32) schema, so
the bind succeeds and the parquet reader's SchemaAdapter handles the
per-batch BinaryView→Binary relabeling at scan time.
This restores parity with the other infer_schema sites; no behavior
change for non-IP columns.
Signed-off-by: Kai Huang <ahkcs@amazon.com>1 parent ae47f33 commit f8fee15
1 file changed
Lines changed: 1 addition & 0 deletions
Lines changed: 1 addition & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
444 | 444 | | |
445 | 445 | | |
446 | 446 | | |
| 447 | + | |
447 | 448 | | |
448 | 449 | | |
449 | 450 | | |
| |||
0 commit comments