Extend the connector to support Arrow Flight endpoints.
Requirements:
- Connect to Arrow Flight servers using FlightClient
- Schema discovery via FlightInfo/getSchema
- Stream FlightData directly into VectorSchemaRoot (skip JSON→Arrow conversion)
- Support Flight SQL for JDBC-like sources exposed via Flight
Design considerations:
- Most efficient path: Arrow Flight already speaks Arrow, so no conversion needed
- Reuse Spark integration layer (RESTScan, PartitionReader) but bypass JSON/HTTP layers
- Config:
type = flight, endpoint = "grpc://host:port", command = "..."
- Could support ColumnarBatch directly for zero-copy (unlike REST which uses InternalRow)
Blocked by: Manual REST config workflow complete
Extend the connector to support Arrow Flight endpoints.
Requirements:
Design considerations:
type = flight,endpoint = "grpc://host:port",command = "..."Blocked by: Manual REST config workflow complete