fix: Ensure target file size is properly respected in vortex sink by peasee · Pull Request #33 · spiceai/vortex

peasee · 2026-03-11T10:54:43Z

Replaces the DataFusion demuxer with our own handling implementation to ensure we track files across streams
Ensures target file size is respected in cases with no partitioning columns as well as with partitioning columns
Adds a comprehensive test suite to check different data insertion scenarios
Ensures all error cases are properly handled and returned to the caller, so that an invalid data state can never be entered

Copilot

Pull request overview

This PR refactors the Vortex DataFusion sink write path to better respect target_file_size across both partitioned and non-partitioned outputs, replacing reliance on DataFusion’s demux behavior with sink-managed routing and file rotation. It also adds an expanded test suite covering file splitting and multi-partition write scenarios.

Changes:

Implement custom file rotation and partition routing in VortexSink, including a compression-ratio-based estimate to approximate target compressed size.
Add a concurrency cap for partition writers and new tests for partitioned writes, file splitting, and write-id consistency.
Force a single sink input stream via a repartition step to ensure one coordinated write per statement.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 8 comments.

File	Description
`vortex-datafusion/src/tests/nested_projection.rs`	Marks nested projection tests as ignored.
`vortex-datafusion/src/persistent/sink.rs`	Replaces the prior sink writing/demux approach with custom per-file/per-partition writers, rotation logic, and extensive tests.
`vortex-datafusion/src/persistent/format.rs`	Adds a repartition-to-1 step before `DataSinkExec` to ensure a single coordinated sink write.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

This reverts commit 6c3be0f.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

* fix: Ensure target file sizes are respected * fix: Improve robustness for streams split over multiple cores * wip * wip * wip * wip * refactor: Improve robustness * refactor: Improve concurrent writer robustness for memory bounds * Update vortex-datafusion/src/persistent/sink.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update vortex-datafusion/src/persistent/format.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update vortex-datafusion/src/persistent/sink.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update vortex-datafusion/src/persistent/sink.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update vortex-datafusion/src/tests/nested_projection.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * review: Address comments * Update vortex-datafusion/src/persistent/sink.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * review: Address comments * Revert "Update vortex-datafusion/src/persistent/sink.rs" This reverts commit 6c3be0f. * refactor: Simplify * test: Add file writer finish notify test * review: Address comments --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: Ensure target file size is properly respected in vortex sink (#33) * fix: Ensure target file sizes are respected * fix: Improve robustness for streams split over multiple cores * wip * wip * wip * wip * refactor: Improve robustness * refactor: Improve concurrent writer robustness for memory bounds * Update vortex-datafusion/src/persistent/sink.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update vortex-datafusion/src/persistent/format.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update vortex-datafusion/src/persistent/sink.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update vortex-datafusion/src/persistent/sink.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update vortex-datafusion/src/tests/nested_projection.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * review: Address comments * Update vortex-datafusion/src/persistent/sink.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * review: Address comments * Revert "Update vortex-datafusion/src/persistent/sink.rs" This reverts commit 6c3be0f. * refactor: Simplify * test: Add file writer finish notify test * review: Address comments --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update cargo.lock --------- Co-authored-by: William <98815791+peasee@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

peasee added 8 commits March 10, 2026 11:37

fix: Ensure target file sizes are respected

d104dd7

fix: Improve robustness for streams split over multiple cores

7042358

wip

e0302f0

wip

6827c4d

wip

54ec115

wip

ce57587

refactor: Improve robustness

2ca0698

refactor: Improve concurrent writer robustness for memory bounds

ca1874c

peasee self-assigned this Mar 11, 2026

peasee added the bug Something isn't working label Mar 11, 2026

Copilot AI review requested due to automatic review settings March 11, 2026 10:54

Copilot started reviewing on behalf of peasee March 11, 2026 10:55 View session

Copilot AI reviewed Mar 11, 2026

View reviewed changes

Update vortex-datafusion/src/persistent/sink.rs

90784ab

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings March 11, 2026 11:03

peasee and others added 2 commits March 11, 2026 21:03

Update vortex-datafusion/src/persistent/format.rs

6b5bdf5

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update vortex-datafusion/src/persistent/sink.rs

6d49750

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot started reviewing on behalf of peasee March 11, 2026 11:03 View session

peasee and others added 3 commits March 11, 2026 21:03

Update vortex-datafusion/src/persistent/sink.rs

cdec6e0

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update vortex-datafusion/src/tests/nested_projection.rs

1cf9c7c

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

review: Address comments

e778016

Copilot AI reviewed Mar 11, 2026

View reviewed changes

Comment thread vortex-datafusion/src/persistent/sink.rs Outdated

Comment thread vortex-datafusion/src/persistent/sink.rs Outdated

Comment thread vortex-datafusion/src/persistent/format.rs Outdated

Comment thread vortex-datafusion/src/persistent/sink.rs

Update vortex-datafusion/src/persistent/sink.rs

6c3be0f

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings March 11, 2026 11:08

Copilot started reviewing on behalf of peasee March 11, 2026 11:08 View session

review: Address comments

3b92cec

Copilot AI reviewed Mar 11, 2026

View reviewed changes

Comment thread vortex-datafusion/src/persistent/sink.rs Outdated

Comment thread vortex-datafusion/src/persistent/format.rs Outdated

peasee added 2 commits March 11, 2026 21:27

Revert "Update vortex-datafusion/src/persistent/sink.rs"

3199cee

This reverts commit 6c3be0f.

refactor: Simplify

eae5565

Copilot AI review requested due to automatic review settings March 11, 2026 22:50

Copilot started reviewing on behalf of peasee March 11, 2026 22:50 View session

Copilot AI reviewed Mar 11, 2026

View reviewed changes

Comment thread vortex-datafusion/src/persistent/sink.rs

test: Add file writer finish notify test

d03fbd9

phillipleblanc reviewed Mar 12, 2026

View reviewed changes

Comment thread vortex-datafusion/src/persistent/sink.rs

phillipleblanc reviewed Mar 12, 2026

View reviewed changes

Comment thread vortex-datafusion/src/persistent/sink.rs Outdated

phillipleblanc reviewed Mar 12, 2026

View reviewed changes

Comment thread vortex-datafusion/src/persistent/sink.rs Outdated

phillipleblanc approved these changes Mar 12, 2026

View reviewed changes

review: Address comments

4843fcf

Copilot AI review requested due to automatic review settings March 12, 2026 04:44

Copilot started reviewing on behalf of peasee March 12, 2026 04:44 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

Comment thread vortex-datafusion/src/persistent/sink.rs

peasee merged commit 8a09bc7 into spiceai-52 Mar 12, 2026
17 of 46 checks passed

peasee deleted the peasee/260310-partitioned-file-split-target-size branch March 12, 2026 05:05

krinart mentioned this pull request Mar 31, 2026

Backport file size fix #35

Merged

Conversation

peasee commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

peasee commented Mar 11, 2026 •

edited

Loading