Skip to content

fix(query): rotate parquet spill row groups#19990

Open
dqhl76 wants to merge 2 commits into
databendlabs:mainfrom
dqhl76:fix-row-group-rotate
Open

fix(query): rotate parquet spill row groups#19990
dqhl76 wants to merge 2 commits into
databendlabs:mainfrom
dqhl76:fix-row-group-rotate

Conversation

@dqhl76

@dqhl76 dqhl76 commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

  • Rotate parquet spill files before a single file reaches parquet's row group ordinal limit.
  • Support multiple spill files per aggregate bucket / grace hash join partition when rotation happens.
  • Add unit coverage for rolling spill writer file rotation.

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions Bot added the pr-bugfix this PR patches a bug in codebase label Jun 10, 2026
@dqhl76 dqhl76 requested a review from zhang2014 June 10, 2026 07:11
@dqhl76 dqhl76 force-pushed the fix-row-group-rotate branch from 463b407 to f50d39f Compare June 17, 2026 09:35
@dqhl76 dqhl76 marked this pull request as draft June 17, 2026 09:56
@dqhl76 dqhl76 force-pushed the fix-row-group-rotate branch from f50d39f to a5f25f7 Compare June 17, 2026 11:42
@dqhl76 dqhl76 force-pushed the fix-row-group-rotate branch from a5f25f7 to 04a39ac Compare June 17, 2026 11:51
@dqhl76 dqhl76 marked this pull request as ready for review June 17, 2026 11:53

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c0e49dda65

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/query/service/src/spillers/async_buffer.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-bugfix this PR patches a bug in codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant