Add Apache Arrow result streaming to DuckDBCommand

[luisquintanilla]

What's Changed

Adds a managed Apache Arrow read surface to DuckDB.NET.Data, addressing #26. It is built on DuckDB's current Arrow C Data Interface (duckdb_to_arrow_schema / duckdb_data_chunk_to_arrow), not the deprecated duckdb_query_arrow family, so it should keep working as the legacy API is removed.

Two new APIs on DuckDBCommand:

// Friendly async path
await using var command = connection.CreateCommand();
command.CommandText = "select 42 as answer, 'duckdb' as name";

await foreach (RecordBatch batch in command.ExecuteArrowBatchesAsync(cancellationToken))
{
    // consume Apache.Arrow RecordBatch values
}

// Lower-level interop path
using IArrowArrayStream stream = command.ExecuteArrowStream();

How it works

• Bindings (DuckDB.NET.Bindings): adds duckdb_result_get_arrow_options, duckdb_to_arrow_schema, duckdb_data_chunk_to_arrow, the duckdb_error_data accessors, and a DuckDBArrowOptions safe handle.
• Data (DuckDB.NET.Data): a DuckDBArrowArrayStream : IArrowArrayStream builds the ArrowSchema once from the result column types/names, then loops duckdb_fetch_chunk and converts each chunk into an Arrow RecordBatch. The native ArrowSchema / ArrowArray structs are handed straight to Apache.Arrow.C.CArrowSchemaImporter.ImportSchema and CArrowArrayImporter.ImportRecordBatch, so all type mapping is delegated to DuckDB's own converter (no per-type C# code).
• The Apache.Arrow package dependency lives in DuckDB.NET.Data only; DuckDB.NET.Bindings stays free of it.

Open question

The main thing to confirm is where the Apache.Arrow dependency should sit. This PR puts it in DuckDB.NET.Data, with Bindings only exposing the native structs/functions. Happy to move it if you would prefer a separate package (e.g. DuckDB.NET.Data.Arrow) so that callers who do not need Arrow do not take the dependency.

Scope

In scope (this PR):

• Read path, DuckDB result -> Arrow, for the first result set of a command.

Deferred (follow-ups if the direction is welcome):

• Arrow ingest/scan (Arrow -> DuckDB).
• Multiple result sets from a single multi-statement command.

Tests

Adds ArrowResultTests covering schema shape, scalar values, nulls, multi-chunk streaming (range(5000) yields multiple batches), and the ExecuteArrowStream() path. The full existing suite (6886 tests) still passes locally.

Opening as a draft pending your call on the dependency-location question above.

[codecov]

Codecov Report

❌ Patch coverage is 82.97872% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.46%. Comparing base (fd47bca) to head (0037674).
⚠️ Report is 6 commits behind head on develop.

Files with missing lines	Patch %	Lines
DuckDB.NET.Data/Arrow/DuckDBArrowArrayStream.cs	84.84%	6 Missing and 4 partials ⚠️
DuckDB.NET.Bindings/DuckDBWrapperObjects.cs	50.00%	3 Missing and 1 partial ⚠️
...B.NET.Data/Extensions/DuckDBErrorDataExtensions.cs	60.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #334      +/-   ##
===========================================
- Coverage    87.59%   87.46%   -0.14%     
===========================================
  Files           75       77       +2     
  Lines         3038     3134      +96     
  Branches       445      463      +18     
===========================================
+ Hits          2661     2741      +80     
- Misses         268      278      +10     
- Partials       109      115       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[coveralls]

Coverage Report for CI Build 28102732334

Warning

Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes.
Quick fix: rebase this PR. Learn more →

Coverage decreased (-0.1%) to 89.358%

Details

• Coverage decreased (-0.1%) from the base build.
• Patch coverage: 10 uncovered changes across 3 files (84 of 94 lines covered, 89.36%).
• 6 coverage regressions across 2 files.

Uncovered Changes

File	Changed	Covered	%
DuckDB.NET.Data/Arrow/DuckDBArrowArrayStream.cs	66	60	90.91%
DuckDB.NET.Bindings/DuckDBWrapperObjects.cs	8	5	62.5%
DuckDB.NET.Data/Extensions/DuckDBErrorDataExtensions.cs	5	4	80.0%
Total (4 files)	94	84	89.36%

Coverage Regressions

6 previously-covered lines in 2 files lost coverage.

File	Lines Losing Coverage	Coverage
DuckDB.NET.Data/Mapping/DuckDBAppenderMap.cs	5	57.89%
DuckDB.NET.Data/Extensions/TypeExtensions.cs	1	97.0%

---

Coverage Stats

Relevant Lines:	3134
Covered Lines:	2856
Line Coverage:	91.13%
Relevant Branches:	1583
Covered Branches:	1359
Branch Coverage:	85.85%
Branches in Coverage %:	Yes
Coverage Strength:	421088.62 hits per line

---

💛 - Coveralls

[Copilot]

Pull request overview

Adds an Apache Arrow read/streaming surface to DuckDB.NET.Data by exposing DuckDB’s Arrow C Data Interface through new DuckDBCommand APIs, plus tests to validate batch/schema/value behavior.

Changes:

• Added DuckDBCommand.ExecuteArrowStream() and ExecuteArrowBatchesAsync() to stream the first result set as Arrow schema/record batches.
• Implemented DuckDBArrowArrayStream : IArrowArrayStream to convert DuckDB data chunks to Arrow record batches via DuckDB’s Arrow C API.
• Added new bindings/safe handles for DuckDB Arrow functions/options and introduced Arrow-focused test coverage.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
DuckDB.NET.Test/Arrow/ArrowResultTests.cs	New tests validating Arrow schema, scalar values, null handling, multi-chunk streaming, and stream semantics.
DuckDB.NET.Data/DuckDBCommand.cs	Adds public Arrow streaming APIs on DuckDBCommand.
DuckDB.NET.Data/Data.csproj	Adds Apache.Arrow NuGet dependency to DuckDB.NET.Data.
DuckDB.NET.Data/Arrow/DuckDBArrowArrayStream.cs	New Arrow array stream implementation bridging DuckDB chunks to Arrow RecordBatch.
DuckDB.NET.Bindings/NativeMethods/NativeMethods.Arrow.cs	Adds P/Invoke declarations for DuckDB Arrow C Data Interface functions and error-data accessors.
DuckDB.NET.Bindings/DuckDBWrapperObjects.cs	Adds DuckDBArrowOptions safe handle wrapper.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[luisquintanilla]

@Giorgi, I've addressed the feedback. Let me know if there's any other changes or suggestions you have here. Thanks!

[Giorgi]

Thanks for the PR @luisquintanilla Added some comments. One question: If DbCommand has several statements, we can retrieve ArrowStream or RecordBatch only for the first one, unlike DbDataReader which has NextResult method. Should we do anything about it? Does it make sense to move the new methods to DuckDBDataReader or will that be error-prone for the callers?

[Giorgi]

@luisquintanilla Thanks for implementing such a long-standing feature request! It would be great if you could add a page about it to https://github.com/Giorgi/DuckDB.NET-Docs, as I'm not very familiar with Arrow.

[luisquintanilla]

Thanks for merging this @Giorgi

I was doing some research into your question

One question: If DbCommand has several statements, we can retrieve ArrowStream or RecordBatch only for the first one, unlike DbDataReader which has NextResult method. Should we do anything about it? Does it make sense to move the new methods to DuckDBDataReader or will that be error-prone for the callers?

I dug into how the rest of the Arrow ecosystem handles this before answering.

Short version: I'd keep the Arrow methods on DuckDBCommand and not move them to DuckDBDataReader. Moving them there would be error-prone, and "one Arrow stream per result set" is the pattern every Arrow database client follows.

Why single-stream is the norm. The Arrow C Stream Interface defines ArrowArrayStream.get_schema to return a schema that "is the same for all data chunks in the stream." Two result sets usually have different schemas, so they can't share one ArrowArrayStream. Every client lands in the same place, and the links below point at the exact lines: DuckDB C API(duckdb_to_arrow_schema / duckdb_data_chunk_to_arrow work per result, with duckdb_extract_statements for multi-statement input), JDBC arrowExportStream(), ADBC (C StatementExecuteQuery,C# ExecuteQuery()), and Go all return one Arrow stream per result set, with no NextResult equivalent. go-duckdb is the closest precedent: its QueryContext runs every statement and returns a reader for the last one.

Why not DuckDBDataReader. The reader pre-fetches chunk 0 of each result set in
InitChunkData() to compute HasRows, before the caller reads anything. An Arrow accessor hosted on
the reader would either have to unwind that prefetch or silently drop chunk 0 from the stream, which
is a data-loss bug. The reader also already owns the native DuckDBResult and its chunks. An Arrow
stream needs to own the result, the arrow options, and the per-chunk arrays too, so hosting both on
one reader gives two owners of the same native result and ambiguous disposal. So to your question,
yes, it would be error-prone for callers.

What I'd propose:

1. Keep the methods on DuckDBCommand, returning the first result set (current behavior), and
   document that clearly.
2. Track multi-result-set parity as a follow-up. If we want it, I'd skip the reader and add something
   like IEnumerable<IArrowArrayStream> ExecuteArrowStreams() that yields one stream per statement.
   It maps directly onto the existing lazy PrepareMultiple iterator, so each MoveNext runs the
   next statement, a natural NextResult-style cadence, without entangling the data reader.

One behavior detail worth deciding now, separate from the bigger question. Because the current method
returns the first result set and the statement iterator is lazy, statements after the first result
set don't run at all. That's fine for SELECT 1; SELECT 2, but for something like
SELECT ...; CREATE TABLE ... the trailing side effect gets skipped. Options: (a) document it as
first-statement-only, (b) drain the remaining statements after the returned one so side effects still
run, or (c) go with the ExecuteArrowStreams() shape above. I lean (a) for this PR with (b) or (c)
as a follow-up, but I'm happy to do whatever you prefer.

Related: Apache.Arrow placement. Right now the dependency sits in DuckDB.NET.Data, and
Bindings stays native-only. That's the simplest option and keeps one package, at the cost of an
Apache.Arrow transitive dependency for every DuckDB.NET.Data consumer. If you'd rather isolate
it, a separate DuckDB.NET.Data.Arrow package works too. It would need InternalsVisibleTo, since
the stream type is now internal. I'm happy either way. Your call on which you'd prefer to maintain.

Re: documentation I'll add to it and as mentioned above, I'll document the first result set behavior. Let me know if you need anything else.

[Giorgi]

We could make ExecuteArrowStream return IEnumerable<IArrowArrayStream> and ExecuteArrowBatchesAsync return IEnumerable<IAsyncEnumerable<RecordBatch>>. That way, every IArrowArrayStream will have its own schema, or we can wait for feedback first.