Skip to content

Support dynamic expansion of RDMA block pool#3155

Merged
chenBright merged 1 commit intoapache:masterfrom
chenBright:auto_extend_block_pool
Nov 29, 2025
Merged

Support dynamic expansion of RDMA block pool#3155
chenBright merged 1 commit intoapache:masterfrom
chenBright:auto_extend_block_pool

Conversation

@chenBright
Copy link
Copy Markdown
Contributor

@chenBright chenBright commented Nov 18, 2025

What problem does this PR solve?

Issue Number:

Problem Summary:

When FLAGS_rdma_memory_pool_buckets > 1, the block pool cannot be dynamically expanded.

What is changed and the side effects?

Changed:

Dynamically expanded memory is stored in an expansion_list protected by extend_lock, and each bucket only retrieves memory from the expansion_list at the corresponding index.

Side effects:

  • Performance effects:

  • Breaking backward compatibility:


Check List:

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR enhances the RDMA block pool to support dynamic memory expansion when multiple buckets are configured. Previously, dynamic expansion was restricted when FLAGS_rdma_memory_pool_buckets > 1, limiting scalability.

Key Changes:

  • Introduces expansion_list and expansion_size arrays to stage newly allocated memory before distributing it to buckets
  • Removes the restriction preventing dynamic expansion with multiple buckets
  • Refactors memory management to use scope guards for cleaner lock handling
  • Deprecates FLAGS_rdma_memory_pool_user_specified_memory flag while maintaining backward compatibility

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File Description
src/butil/memory/scope_guard.h Adds BUTIL_SCOPE_EXIT macro as an alias to BRPC_SCOPE_EXIT for consistent naming across the codebase
src/brpc/rdma/block_pool.cpp Implements expansion_list mechanism for thread-safe dynamic memory expansion, refactors lock management using scope guards, and updates memory pool extension logic

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/brpc/rdma/block_pool.cpp
Comment thread src/brpc/rdma/block_pool.cpp Outdated
Comment thread src/brpc/rdma/block_pool.cpp Outdated
Comment thread src/brpc/rdma/block_pool.cpp Outdated
Comment thread src/brpc/rdma/block_pool.cpp Outdated
@yanglimingcn
Copy link
Copy Markdown
Contributor

When FLAGS_rdma_memory_pool_buckets is greater than 1, a bug may occur due to lock granularity issues.

@chenBright
Copy link
Copy Markdown
Contributor Author

When FLAGS_rdma_memory_pool_buckets is greater than 1, a bug may occur due to lock granularity issues.

This PR can solve this problem.

@chenBright chenBright force-pushed the auto_extend_block_pool branch from 5eb557b to 48a20bb Compare November 18, 2025 03:14
@yanglimingcn
Copy link
Copy Markdown
Contributor

FLAGS_rdma_memory_pool_user_specified_memory is used to allow users to register specific memory, such as SPDK. In these scenarios, memory needs to be allocated by SPDK and then registered in RDMA, which can achieve zero copy.

@chenBright
Copy link
Copy Markdown
Contributor Author

chenBright commented Nov 18, 2025

FLAGS_rdma_memory_pool_user_specified_memory is used to allow users to register specific memory, such as SPDK. In these scenarios, memory needs to be allocated by SPDK and then registered in RDMA, which can achieve zero copy.

Ok, I will revert it.

When the idle_list is out of memory in AllocBlockFrom function, it seems that the user cannot immediately dynamically expand specific memory. Is that right?

@yanglimingcn
Copy link
Copy Markdown
Contributor

In this scenario, the memory should be pre-registered, and dynamic allocation requires the user to provide memory too.

@chenBright chenBright force-pushed the auto_extend_block_pool branch from 48a20bb to 3a03c02 Compare November 18, 2025 13:03
@chenBright
Copy link
Copy Markdown
Contributor Author

FLAGS_rdma_memory_pool_user_specified_memory is used to allow users to register specific memory, such as SPDK. In these scenarios, memory needs to be allocated by SPDK and then registered in RDMA, which can achieve zero copy.

Ok, I will revert it.

@yanglimingcn done

@yanglimingcn
Copy link
Copy Markdown
Contributor

LGTM

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 10 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/brpc/rdma/block_pool.cpp
Comment thread src/brpc/rdma/block_pool.cpp
Comment thread src/brpc/rdma/block_pool.cpp
Comment thread src/brpc/rdma/block_pool.cpp
Comment thread src/brpc/rdma/block_pool.cpp
Comment thread src/brpc/rdma/block_pool.cpp
Comment thread src/brpc/rdma/block_pool.cpp
Comment thread src/brpc/rdma/block_pool.cpp
Comment thread src/brpc/rdma/block_pool.cpp
Comment thread src/brpc/rdma/block_pool.cpp
@chenBright chenBright merged commit 3e40e3f into apache:master Nov 29, 2025
23 checks passed
@chenBright chenBright deleted the auto_extend_block_pool branch January 11, 2026 04:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants