Skip to content

feat: support cache mode for prefetch#113

Merged
lucasfang merged 4 commits into
alibaba:mainfrom
lucasfang:prefetch
Feb 5, 2026
Merged

feat: support cache mode for prefetch#113
lucasfang merged 4 commits into
alibaba:mainfrom
lucasfang:prefetch

Conversation

@lucasfang

@lucasfang lucasfang commented Feb 4, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Adds configurable “prefetch cache mode” support so prefetching can enable/disable read-ahead cache initialization depending on whether predicates and/or bitmap selection are used.

Tests

API and Format

PrefetchCacheMode ReadContext::GetPrefetchCacheMode() const;
ReadContextBuilder& SetPrefetchCacheMode(PrefetchCacheMode mode);

Documentation

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds configurable “prefetch cache mode” support so prefetching can enable/disable read-ahead cache initialization depending on whether predicates and/or bitmap selection are used.

Changes:

  • Replaces the boolean prefetch-cache switch with a new PrefetchCacheMode enum and threads it through ReadContext / ReadContextBuilder.
  • Updates prefetch reader creation to accept the new cache mode and conditionally initialize the cache via NeedInitCache().
  • Updates and extends existing unit tests to use the new API and validate defaults.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
include/paimon/utils/read_ahead_cache.h Introduces PrefetchCacheMode enum and documents intended behavior.
include/paimon/read_context.h Public API change: ReadContext now exposes cache mode; builder uses SetPrefetchCacheMode.
src/paimon/core/operation/read_context.cpp Implements builder storage/defaults and passes cache mode into ReadContext.
src/paimon/core/operation/read_context_test.cpp Validates default cache mode and builder-set cache mode.
src/paimon/core/operation/internal_read_context.h Exposes cache mode through internal context wrapper.
src/paimon/core/operation/abstract_split_read.cpp Passes cache mode to PrefetchFileBatchReaderImpl::Create.
src/paimon/common/reader/prefetch_file_batch_reader_impl.h Updates create/ctor signatures and stores cache mode + predicate.
src/paimon/common/reader/prefetch_file_batch_reader_impl.cpp Implements conditional cache initialization based on cache mode/predicate/bitmap.
src/paimon/common/reader/prefetch_file_batch_reader_impl_test.cpp Updates tests to new create signature.
src/paimon/core/deletionvectors/apply_deletion_vector_batch_reader_test.cpp Updates prefetch reader creation signature usage.
src/paimon/common/file_index/bitmap/apply_bitmap_index_batch_reader_test.cpp Updates prefetch reader creation signature usage.
Comments suppressed due to low confidence (1)

src/paimon/common/reader/prefetch_file_batch_reader_impl.cpp:303

  • New cache-mode behavior is implemented via NeedInitCache() and the if (cache_ && NeedInitCache()) gate, but there are no unit tests exercising the non-ALWAYS modes (e.g. ensuring cache init is skipped when a predicate and/or bitmap is present). Adding targeted tests for EXCLUDE_PREDICATE, EXCLUDE_BITMAP, and EXCLUDE_BITMAP_OR_PREDICATE would help prevent regressions in this new feature.
bool PrefetchFileBatchReaderImpl::NeedInitCache() const {
    switch (cache_mode_) {
        case PrefetchCacheMode::NEVER:
            return false;
        case PrefetchCacheMode::EXCLUDE_PREDICATE:
            return predicate_ == nullptr;
        case PrefetchCacheMode::EXCLUDE_BITMAP:
            return selection_bitmap_ == std::nullopt;
        case PrefetchCacheMode::EXCLUDE_BITMAP_OR_PREDICATE:
            return predicate_ == nullptr && selection_bitmap_ == std::nullopt;
        case PrefetchCacheMode::ALWAYS:
            return true;
        default:
            return true;
    }
}

void PrefetchFileBatchReaderImpl::Workloop() {
    std::vector<std::future<void>> futures;
    futures.resize(readers_.size());
    if (cache_ && NeedInitCache()) {
        auto read_ranges = readers_[0]->PreBufferRange();
        if (read_ranges.ok()) {
            std::vector<ByteRange> ranges;
            for (const auto& read_range : read_ranges.value()) {

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread include/paimon/read_context.h
Comment thread include/paimon/read_context.h
Comment thread include/paimon/utils/read_ahead_cache.h
Comment thread src/paimon/common/reader/prefetch_file_batch_reader_impl.cpp
Comment thread src/paimon/common/reader/prefetch_file_batch_reader_impl.cpp
Comment thread src/paimon/common/reader/prefetch_file_batch_reader_impl.cpp
Comment thread include/paimon/utils/read_ahead_cache.h

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lucasfang
lucasfang merged commit d1afa9b into alibaba:main Feb 5, 2026
8 checks passed
@lucasfang
lucasfang deleted the prefetch branch April 20, 2026 07:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants