Skip to content

Arrow Flight data source support #21

@Neutrinic

Description

@Neutrinic

Extend the connector to support Arrow Flight endpoints.

Requirements:

  • Connect to Arrow Flight servers using FlightClient
  • Schema discovery via FlightInfo/getSchema
  • Stream FlightData directly into VectorSchemaRoot (skip JSON→Arrow conversion)
  • Support Flight SQL for JDBC-like sources exposed via Flight

Design considerations:

  • Most efficient path: Arrow Flight already speaks Arrow, so no conversion needed
  • Reuse Spark integration layer (RESTScan, PartitionReader) but bypass JSON/HTTP layers
  • Config: type = flight, endpoint = "grpc://host:port", command = "..."
  • Could support ColumnarBatch directly for zero-copy (unlike REST which uses InternalRow)

Blocked by: Manual REST config workflow complete

Metadata

Metadata

Assignees

No one assigned

    Labels

    roadmapFuture roadmap items

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions