Skip to content

fix(go): align buffers before C Data export#129

Open
dataders wants to merge 1 commit into
mainfrom
codex-align-c-data-buffers
Open

fix(go): align buffers before C Data export#129
dataders wants to merge 1 commit into
mainfrom
codex-align-c-data-buffers

Conversation

@dataders
Copy link
Copy Markdown

What changed

This adds an internal Go helper that prepares record batches for Arrow C Data export by copying only buffers whose base pointer is not aligned to 64 bytes.

The c-shared driver template now runs record batches through that helper before calling cdata.ExportArrowRecordBatch, and the generated c-shared driver packages have been regenerated from the template.

Why

Some consumers of Arrow C Data import primitive buffers as typed native buffers and require the pointer to satisfy the scalar type alignment. Decimal128 is backed by 16-byte values, so a Decimal128 value buffer exported through C Data with an 8-byte-aligned pointer can fail in strict consumers.

The Go drivers can receive Arrow IPC-backed buffers whose byte storage is usable inside Go but not suitable to hand out unchanged across the C Data FFI boundary. The c-shared driver owns that boundary, so it should ensure the buffers it exports are aligned.

Validation

  • go test ./pkg/internal/cdataalign -count=1
  • go test -tags driverlib ./pkg/flightsql ./pkg/panicdummy ./pkg/snowflake ./pkg/bigquery ./pkg/databricks ./pkg/salesforce -count=1
  • Built the Snowflake c-shared driver locally with a forced-misaligned allocator repro setup.

@dataders dataders force-pushed the codex-align-c-data-buffers branch from 692eaf7 to e31ec4c Compare May 11, 2026 01:48
@dataders dataders marked this pull request as ready for review May 11, 2026 02:07
@dataders dataders force-pushed the codex-align-c-data-buffers branch from e31ec4c to 3b409ae Compare May 11, 2026 02:18
Copy link
Copy Markdown
Collaborator

@felipecrv felipecrv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the latest versions of arrow-go already guarantee alignment that satisfies arrow-rs requirements. All the drivers built from this repo are safe already.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants