feat: Add stripe stats pruning - #970
Open
apurva-meta wants to merge 1 commit into
Open
Conversation
meta-cla
Bot
added
the
CLA Signed
|
@apurva-meta has exported this pull request. If you are a Meta employee, you can view the originating Diff in D112143149. |
meta-codesync
Bot
changed the title
apurva-meta
force-pushed
the
export-D112143149
branch
2 times, most recently
from
July 15, 2026 18:52
7f68015 to
da20511
Compare
meta-codesync
Bot
changed the title
apurva-meta
added a commit
to apurva-meta/nimble
that referenced
this pull request
Jul 15, 2026
Summary: Add a stripe-level vectorized stats optional section for Nimble files and use it in the selective reader to conservatively skip stripes for integral filters when min/max proves no rows can match. The writer snapshots per-stripe column stats and reconstructs file-level stats by merging those snapshots, preserving a fallback path for files that do not emit stripe snapshots. The reader preloads and deserializes the new optional stats section while treating absent stats as a normal full-scan fallback. Deduplicated columns are now handled correctly across stripes. mergeColumnStats merges into the wrapped base statistics (which back the delegating getters) and accumulates the dedup counters, instead of writing to unused fields. The per-stripe ancestor deduplication wrapping applied during finalization is undone between stripes (releasing the base collector) so every stripe finalizes from an identical collector structure; without this, stripe 1+ skipped the child logical-size rollup and the reconstructed file-level stats were wrong (tripping the raw-size consistency check). Genuine deduplicated nodes configured at init stay wrapped. Differential Revision: D112143149
apurva-meta
force-pushed
the
export-D112143149
branch
from
July 15, 2026 19:19
da20511 to
f807f9b
Compare
apurva-meta
added a commit
to apurva-meta/nimble
that referenced
this pull request
Jul 15, 2026
Summary: Add a stripe-level vectorized stats optional section for Nimble files and use it in the selective reader to conservatively skip stripes for integral filters when min/max proves no rows can match. The writer snapshots per-stripe column stats and reconstructs file-level stats by merging those snapshots, preserving a fallback path for files that do not emit stripe snapshots. The reader preloads and deserializes the new optional stats section while treating absent stats as a normal full-scan fallback. Deduplicated columns are now handled correctly across stripes. mergeColumnStats merges into the wrapped base statistics (which back the delegating getters) and accumulates the dedup counters, instead of writing to unused fields. The per-stripe ancestor deduplication wrapping applied during finalization is undone between stripes (releasing the base collector) so every stripe finalizes from an identical collector structure; without this, stripe 1+ skipped the child logical-size rollup and the reconstructed file-level stats were wrong (tripping the raw-size consistency check). Genuine deduplicated nodes configured at init stay wrapped. Differential Revision: D112143149
apurva-meta
force-pushed
the
export-D112143149
branch
from
July 15, 2026 19:57
f807f9b to
0eea530
Compare
apurva-meta
added a commit
to apurva-meta/nimble
that referenced
this pull request
Jul 15, 2026
Summary: Add a stripe-level vectorized stats optional section for Nimble files and use it in the selective reader to conservatively skip stripes for integral filters when min/max proves no rows can match. The writer snapshots per-stripe column stats and reconstructs file-level stats by merging those snapshots, preserving a fallback path for files that do not emit stripe snapshots. The reader preloads and deserializes the new optional stats section while treating absent stats as a normal full-scan fallback. Deduplicated columns are now handled correctly across stripes. mergeColumnStats merges into the wrapped base statistics (which back the delegating getters) and accumulates the dedup counters, instead of writing to unused fields. The per-stripe ancestor deduplication wrapping applied during finalization is undone between stripes (releasing the base collector) so every stripe finalizes from an identical collector structure; without this, stripe 1+ skipped the child logical-size rollup and the reconstructed file-level stats were wrong (tripping the raw-size consistency check). Genuine deduplicated nodes configured at init stay wrapped. Differential Revision: D112143149
apurva-meta
force-pushed
the
export-D112143149
branch
from
July 15, 2026 23:00
0eea530 to
60d2c83
Compare
meta-codesync
Bot
changed the title
apurva-meta
force-pushed
the
export-D112143149
branch
from
July 16, 2026 18:17
60d2c83 to
6f6e177
Compare
meta-codesync
Bot
changed the title
apurva-meta
added a commit
to apurva-meta/nimble
that referenced
this pull request
Jul 16, 2026
Summary: Add a stripe-level vectorized stats optional section for Nimble files and use it in the selective reader to conservatively skip stripes for integral filters when min/max proves no rows can match. The writer snapshots per-stripe column stats and reconstructs file-level stats by merging those snapshots, preserving a fallback path for files that do not emit stripe snapshots. The reader preloads and deserializes the new optional stats section while treating absent stats as a normal full-scan fallback. Deduplicated columns are now handled correctly across stripes. mergeColumnStats merges into the wrapped base statistics (which back the delegating getters) and accumulates the dedup counters, instead of writing to unused fields. The per-stripe ancestor deduplication wrapping applied during finalization is undone between stripes (releasing the base collector) so every stripe finalizes from an identical collector structure; without this, stripe 1+ skipped the child logical-size rollup and the reconstructed file-level stats were wrong (tripping the raw-size consistency check). Genuine deduplicated nodes configured at init stay wrapped. Differential Revision: D112143149
apurva-meta
force-pushed
the
export-D112143149
branch
from
July 16, 2026 18:27
6f6e177 to
3971cf4
Compare
meta-codesync
Bot
changed the title
apurva-meta
force-pushed
the
export-D112143149
branch
from
July 16, 2026 19:50
3971cf4 to
83461cc
Compare
meta-codesync
Bot
changed the title
apurva-meta
force-pushed
the
export-D112143149
branch
from
July 16, 2026 20:18
83461cc to
2c6ea6d
Compare
meta-codesync
Bot
changed the title
apurva-meta
force-pushed
the
export-D112143149
branch
from
July 16, 2026 20:50
2c6ea6d to
e488f06
Compare
Summary: Add a stripe-level vectorized stats optional section for Nimble files and use it in the selective reader to conservatively skip stripes for integral filters when min/max proves no rows can match. The writer snapshots per-stripe column stats and reconstructs file-level stats by merging those snapshots, preserving a fallback path for files that do not emit stripe snapshots. The reader preloads and deserializes the new optional stats section while treating absent stats as a normal full-scan fallback. Deduplicated columns are now handled correctly across stripes. mergeColumnStats merges into the wrapped base statistics (which back the delegating getters) and accumulates the dedup counters, instead of writing to unused fields. The per-stripe ancestor deduplication wrapping applied during finalization is undone between stripes (releasing the base collector) so every stripe finalizes from an identical collector structure; without this, stripe 1+ skipped the child logical-size rollup and the reconstructed file-level stats were wrong (tripping the raw-size consistency check). Genuine deduplicated nodes configured at init stay wrapped. Differential Revision: D112143149
apurva-meta
force-pushed
the
export-D112143149
branch
from
July 16, 2026 20:57
e488f06 to
949180a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Add a stripe-level vectorized stats optional section for Nimble files and use it in the selective reader to conservatively skip stripes for integral filters when min/max proves no rows can match.
The writer snapshots per-stripe column stats and reconstructs file-level stats by merging those snapshots, preserving a fallback path for files that do not emit stripe snapshots. The reader preloads and deserializes the new optional stats section while treating absent stats as a normal full-scan fallback.
Deduplicated columns are now handled correctly across stripes. mergeColumnStats merges into the wrapped base statistics (which back the delegating getters) and accumulates the dedup counters, instead of writing to unused fields. The per-stripe ancestor deduplication wrapping applied during finalization is undone between stripes (releasing the base collector) so every stripe finalizes from an identical collector structure; without this, stripe 1+ skipped the child logical-size rollup and the reconstructed file-level stats were wrong (tripping the raw-size consistency check). Genuine deduplicated nodes configured at init stay wrapped.
Differential Revision: D112143149