Skip to content

Commit df38aab

Browse files
committed
[analytics-backend-datafusion] Apply schema coercion on indexed-executor placeholder
The indexed-execution path inferred the parquet schema via build_segments (which calls FileFormat::infer_schema) and registered it directly on the PlaceholderProvider that from_substrait_plan binds against. The non-indexed paths (session_context.rs, query_executor.rs, api.rs) all routed their inferred schemas through schema_coerce::coerce_inferred_schema before registering, but the indexed path skipped this step. Result: an OpenSearch `ip` column lands as parquet `BinaryView` on disk; isthmus on the Java side serializes the Substrait base schema as plain `Binary` (Substrait has no view types). The placeholder reports `BinaryView` while the plan declares `Binary` — DataFusion's substrait consumer rejects the bind with: Substrait error: Field '<x>' in Substrait schema has a different type (Binary) than the corresponding field in the table schema (BinaryView). Every analytics-engine query against an index that includes an `ip` column (every OTEL-logs query in CalciteSearchCommandIT, for example) fails at fragment start. Apply coerce_inferred_schema right after build_segments, before PlaceholderProvider construction. The placeholder, the expr_to_bool_tree analysis, and the downstream IndexedTableProvider all see the same Substrait-compatible (Binary / Int64 / Float32) schema, so the bind succeeds and the parquet reader's SchemaAdapter handles the per-batch BinaryView→Binary relabeling at scan time. This restores parity with the other infer_schema sites; no behavior change for non-IP columns. Signed-off-by: Kai Huang <ahkcs@amazon.com>
1 parent 8fdff04 commit df38aab

1 file changed

Lines changed: 1 addition & 0 deletions

File tree

sandbox/plugins/analytics-backend-datafusion/rust/src/indexed_executor.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -450,6 +450,7 @@ pub async unsafe fn execute_indexed_with_context(
450450
let (segments, schema) = build_segments(&state, Arc::clone(&store), object_metas.as_ref(), writer_generations.as_ref())
451451
.await
452452
.map_err(DataFusionError::Execution)?;
453+
let schema = crate::schema_coerce::coerce_inferred_schema(schema);
453454
for (i, seg) in segments.iter().enumerate() {
454455
}
455456

0 commit comments

Comments
 (0)