Skip to content

chore: Move writer-related logic to "writers" module#3385

Merged
andygrove merged 3 commits into
apache:mainfrom
EmilyMatt:writers-module
Feb 4, 2026
Merged

chore: Move writer-related logic to "writers" module#3385
andygrove merged 3 commits into
apache:mainfrom
EmilyMatt:writers-module

Conversation

@EmilyMatt
Copy link
Copy Markdown
Contributor

@EmilyMatt EmilyMatt commented Feb 4, 2026

Which issue does this PR close?

Towards #3554

Rationale for this change

Separate the actual writing logic, so it can be extended/modified later, rather than be tangled within the shuffle writer.

What changes are included in this PR?

  1. Created a writers module
  2. Moved BufBatchWriter and PartitionWriter in said module
  3. Exposed specific functionality required by other code, rather than have everything available as before.
  4. Removed the to_df_err function, as DatafusionError already has a #[from] clause for all these errors, added a map_err() on the outer function calls so we add a backtrace and know the error is from the shuffle(the to_df_err function essentially just added a "error in shuffle write" prefix to each error)

How are these changes tested?

The regular tests should cover these changes.

Copy link
Copy Markdown
Member

@andygrove andygrove left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, Thanks @EmilyMatt

@andygrove andygrove merged commit 0b3329c into apache:main Feb 4, 2026
126 of 127 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants