distsql, kv: add query-level per-store cop request limiter - #69360
distsql, kv: add query-level per-store cop request limiter#69360solotzg wants to merge 23 commits into
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughIntroduces a ChangesPer-store cop request concurrency limit within a query
Sequence Diagram(s)sequenceDiagram
participant Select
participant DistSQLContext
participant copIteratorWorker
participant acquireQueryCopStoreLimiter
participant QueryCopStoreLimiter
participant CoprRequestLimiter
participant TiKV
Select->>DistSQLContext: read QueryCopStoreLimiter
alt QueryCopStoreLimiter != nil for TiKV
Select->>Select: kvReq.QueryCopStoreLimiter = dctx.QueryCopStoreLimiter
end
Select->>copIteratorWorker: dispatch kvReq
loop for each region
copIteratorWorker->>acquireQueryCopStoreLimiter: resolve RPC context + acquire per-store limiter
acquireQueryCopStoreLimiter->>QueryCopStoreLimiter: GetStoreLimiter(storeID)
QueryCopStoreLimiter-->>acquireQueryCopStoreLimiter: per-store CoprRequestLimiter
acquireQueryCopStoreLimiter->>CoprRequestLimiter: Acquire(finishCh)
alt per-store admission succeeds
CoprRequestLimiter-->>copIteratorWorker: releaseFn
copIteratorWorker->>CoprRequestLimiter: Acquire(finishCh) for request-rate
copIteratorWorker->>TiKV: send cop request
TiKV-->>copIteratorWorker: response
copIteratorWorker->>CoprRequestLimiter: Release() request-rate
copIteratorWorker->>releaseFn: call Release per-store
else early exit or error
copIteratorWorker-->>copIteratorWorker: return early
end
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
pkg/kv/kv.go (1)
657-667: 🧹 Nitpick | 🔵 Trivial | 💤 Low valueVariable shadowing:
cshadows the receiver.The inner variable
c := limiter.Capacity()shadows the receiverc *compositeCoprRequestLimiter. Consider renaming tocapfor clarity.Suggested fix
func (c *compositeCoprRequestLimiter) Capacity() int { capacity := 0 for _, limiter := range c.limiters { - c := limiter.Capacity() - if capacity == 0 || c < capacity { - capacity = c + cap := limiter.Capacity() + if capacity == 0 || cap < capacity { + capacity = cap } } return capacity }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/kv/kv.go` around lines 657 - 667, In the Capacity method of the compositeCoprRequestLimiter receiver, the inner loop variable `c := limiter.Capacity()` is shadowing the receiver `c *compositeCoprRequestLimiter`, creating confusion and potential bugs. Rename the inner variable from `c` to `cap` and update all references to this variable within the loop (including the comparison `c < capacity`) to use the new name instead.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/kv/kv_test.go`:
- Around line 210-213: Replace the time.Sleep call with a deterministic
synchronization mechanism to ensure the precondition is met before proceeding.
Instead of relying on a fixed sleep duration to assume Acquire has taken
limiter1 and is blocked on limiter2, use a channel-based signal to confirm that
the token has actually been acquired from limiter1 before closing the done
channel. This ensures the test validates the rollback path reliably without
depending on scheduler timing.
---
Nitpick comments:
In `@pkg/kv/kv.go`:
- Around line 657-667: In the Capacity method of the compositeCoprRequestLimiter
receiver, the inner loop variable `c := limiter.Capacity()` is shadowing the
receiver `c *compositeCoprRequestLimiter`, creating confusion and potential
bugs. Rename the inner variable from `c` to `cap` and update all references to
this variable within the loop (including the comparison `c < capacity`) to use
the new name instead.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: c3bec8a8-f539-4ec2-a2ec-7117fecd8272
📒 Files selected for processing (20)
pkg/distsql/BUILD.bazelpkg/distsql/context/context.gopkg/distsql/context/context_test.gopkg/distsql/distsql.gopkg/distsql/distsql_test.gopkg/distsql/request_builder.gopkg/executor/distsql.gopkg/executor/set_test.gopkg/kv/BUILD.bazelpkg/kv/kv.gopkg/kv/kv_test.gopkg/session/session.gopkg/sessionctx/vardef/tidb_vars.gopkg/sessionctx/variable/session.gopkg/sessionctx/variable/setvar_affect.gopkg/sessionctx/variable/sysvar.gopkg/store/copr/copr_test/BUILD.bazelpkg/store/copr/copr_test/coprocessor_test.gopkg/store/copr/coprocessor.gopkg/util/mock/context.go
💤 Files with no reviewable changes (1)
- pkg/store/copr/copr_test/BUILD.bazel
Codecov Report❌ Patch coverage is Please upload reports for the commit 629ab78 to get more accurate results. Additional details and impacted files@@ Coverage Diff @@
## master #69360 +/- ##
================================================
- Coverage 76.3242% 73.9114% -2.4129%
================================================
Files 2041 2058 +17
Lines 559285 579135 +19850
================================================
+ Hits 426870 428047 +1177
- Misses 131515 150735 +19220
+ Partials 900 353 -547
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
pkg/store/copr/coprocessor.go (1)
1806-1835:⚠️ Potential issue | 🟠 Major | ⚡ Quick winAcquire the per-store limiter before taking the request-local token.
Line 1806 takes the aggregate request-local token before
acquireQueryCopStoreLimiter, so a task blocked behind a hot store still consumes a request-local slot and can throttle unrelated stores without an in-flight send. Move the request-local acquire after the store limiter and keep the new pre-resolution error on the samederr.ToTiDBErrpath asSendReqCtxerrors.Proposed fix
- acquiredRequestRateLimit := false - if worker.requestRateLimit != nil { - if worker.requestRateLimit.Acquire(worker.finishCh) { - return nil, nil - } - acquiredRequestRateLimit = true - } releaseQueryCopStoreLimiter, exit, err := worker.acquireQueryCopStoreLimiter(bo, task, ops) if err != nil { - if acquiredRequestRateLimit { - worker.requestRateLimit.Release() - } - return nil, errors.Trace(err) + return nil, errors.Trace(derr.ToTiDBErr(err)) } if exit { - if acquiredRequestRateLimit { - worker.requestRateLimit.Release() - } return nil, nil } + acquiredRequestRateLimit := false + if worker.requestRateLimit != nil { + if worker.requestRateLimit.Acquire(worker.finishCh) { + if releaseQueryCopStoreLimiter != nil { + releaseQueryCopStoreLimiter() + } + return nil, nil + } + acquiredRequestRateLimit = true + } // Keep the request-rate token and send attempt in a small scope so the // token is released immediately after the send attempt returns, while // still remaining panic-safe.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/store/copr/coprocessor.go` around lines 1806 - 1835, The request-local token from worker.requestRateLimit is being acquired before the per-store limiter via acquireQueryCopStoreLimiter, which allows tasks blocked on hot stores to still consume request-local slots and throttle unrelated stores. Reorder the acquisition logic so that acquireQueryCopStoreLimiter is called first, then only acquire worker.requestRateLimit after successfully acquiring the store limiter. Update the error handling paths to release the request-local token only when it has been acquired, and ensure any pre-resolution errors follow the same derr.ToTiDBErr path as SendReqCtx errors.
🧹 Nitpick comments (1)
pkg/store/copr/coprocessor.go (1)
1896-1904: 🧹 Nitpick | 🔵 TrivialTrack the best-effort store-selection caveat.
The comment says
SendReqCtxmay send to a different store than the pre-resolvedrpcCtx, so the limiter can charge the wrong per-store bucket during retries, stale cache, or leader changes. Please link a tracking issue or add a targeted test documenting that this sysvar is intentionally best-effort until the acquisition moves into the client-go send path.Do you want me to help draft the follow-up issue or a focused test case for this caveat?
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/store/copr/coprocessor.go` around lines 1896 - 1904, The comment block discussing the pre-resolution caveat in SendReqCtx needs to document the intentional best-effort nature of this store-selection approach. Either add a reference to an existing tracking issue that addresses moving the per-store limiter acquisition into the client-go send path, or add a targeted test case that explicitly documents and validates this best-effort behavior when the actual store differs from the pre-resolved rpcCtx during retries, stale cache lookups, leader changes, or store unavailability. This ensures the caveat is tracked and not accidentally "fixed" without proper coordination.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@pkg/store/copr/coprocessor.go`:
- Around line 1806-1835: The request-local token from worker.requestRateLimit is
being acquired before the per-store limiter via acquireQueryCopStoreLimiter,
which allows tasks blocked on hot stores to still consume request-local slots
and throttle unrelated stores. Reorder the acquisition logic so that
acquireQueryCopStoreLimiter is called first, then only acquire
worker.requestRateLimit after successfully acquiring the store limiter. Update
the error handling paths to release the request-local token only when it has
been acquired, and ensure any pre-resolution errors follow the same
derr.ToTiDBErr path as SendReqCtx errors.
---
Nitpick comments:
In `@pkg/store/copr/coprocessor.go`:
- Around line 1896-1904: The comment block discussing the pre-resolution caveat
in SendReqCtx needs to document the intentional best-effort nature of this
store-selection approach. Either add a reference to an existing tracking issue
that addresses moving the per-store limiter acquisition into the client-go send
path, or add a targeted test case that explicitly documents and validates this
best-effort behavior when the actual store differs from the pre-resolved rpcCtx
during retries, stale cache lookups, leader changes, or store unavailability.
This ensures the caveat is tracked and not accidentally "fixed" without proper
coordination.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 165647d7-b814-4cdd-b686-74ac9358880f
📒 Files selected for processing (15)
pkg/distsql/context/context.gopkg/distsql/context/context_test.gopkg/distsql/distsql.gopkg/distsql/distsql_test.gopkg/executor/set_test.gopkg/kv/BUILD.bazelpkg/kv/kv.gopkg/kv/kv_test.gopkg/session/session.gopkg/sessionctx/vardef/tidb_vars.gopkg/sessionctx/variable/session.gopkg/sessionctx/variable/setvar_affect.gopkg/sessionctx/variable/sysvar.gopkg/store/copr/coprocessor.gopkg/util/mock/context.go
✅ Files skipped from review due to trivial changes (1)
- pkg/sessionctx/variable/setvar_affect.go
🚧 Files skipped from review as they are similar to previous changes (2)
- pkg/kv/BUILD.bazel
- pkg/distsql/context/context_test.go
c0482d1 to
a065cb6
Compare
…rrency - Added QueryCopStoreLimiter to limit the number of in-flight cop requests per TiKV store within a single query. - Updated DistSQLContext to include QueryCopStoreLimiter. - Modified Select function to apply the QueryCopStoreLimiter when sending requests to TiKV. - Enhanced tests to validate the behavior of the new limiter, ensuring it correctly manages concurrent requests. - Adjusted related variables and configurations to support the new functionality. Signed-off-by: Zhigao TONG <tongzhigao@pingcap.com>
…CoprRequestRateLimit directly Signed-off-by: Zhigao TONG <tongzhigao@pingcap.com>
36db0ab to
a8e8f3c
Compare
…stency Signed-off-by: Zhigao TONG <tongzhigao@pingcap.com>
890ae70 to
b1f3a99
Compare
|
/cc @windtalker |
Signed-off-by: Zhigao TONG <tongzhigao@pingcap.com>
|
/cc @gengliqi |
…acquisition logic Signed-off-by: Zhigao TONG <tongzhigao@pingcap.com>
Signed-off-by: Zhigao TONG <tongzhigao@pingcap.com>
…used test code Signed-off-by: Zhigao TONG <tongzhigao@pingcap.com>
Signed-off-by: Zhigao TONG <tongzhigao@pingcap.com>
…r consistency Signed-off-by: Zhigao TONG <tongzhigao@pingcap.com>
# Conflicts: # pkg/sessionctx/vardef/tidb_vars.go
| // the region/store selection work done inside SendReqCtx, and the selected | ||
| // store can still change there because of retries, stale cache, leader changes, | ||
| // or unavailable stores. | ||
| // TODO: Move query per-store limiter acquisition into the client-go send path |
There was a problem hiding this comment.
How about move it to client-go at this time
There was a problem hiding this comment.
OK. This PR shoudl be merged after tikv/client-go#2026
Signed-off-by: Zhigao TONG <tongzhigao@pingcap.com>
…elect test Signed-off-by: Zhigao TONG <tongzhigao@pingcap.com>
…structure Signed-off-by: Zhigao TONG <tongzhigao@pingcap.com>
…imits Signed-off-by: Zhigao TONG <tongzhigao@pingcap.com>
Signed-off-by: Zhigao TONG <tongzhigao@pingcap.com>
…s handling Signed-off-by: Zhigao TONG <tongzhigao@pingcap.com>
…g in tests Signed-off-by: Zhigao TONG <tongzhigao@pingcap.com>
Signed-off-by: Zhigao TONG <tongzhigao@pingcap.com>
# Conflicts: # DEPS.bzl # go.sum # pkg/distsql/select_result.go
…cument Signed-off-by: Zhigao TONG <tongzhigao@pingcap.com>
Signed-off-by: Zhigao TONG <tongzhigao@pingcap.com>
|
@solotzg: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
What problem does this PR solve?
Issue Number: close #69359
Problem Summary:
A large statement can issue many concurrent coprocessor RPC attempts to the same TiKV store.
tidb_distsql_scan_concurrencylimits worker concurrency for each DistSQL request, but it does not provide a statement-wide, per-store bound across multiple DistSQL requests or client-go retries. As a result, one statement can create uneven pressure on individual TiKV stores.What changed and how does it work?
tidb_query_cop_store_limit. It limits the number of in-flight TiKV coprocessor request attempts to each store within one statement. The default value is15; setting it to0disables the limit.kv.QueryCopStoreLimiterfor each statement when the variable is enabled. The limiter lazily creates one concrete*kv.CoprRequestLimiterfor each TiKV store ID.tikvrpc.RequestAttemptAdmissionhook introduced by tikv/client-go#2026. client-go invokes the hook after resolving the actual target store for every synchronous or asynchronous RPC attempt, including retries. Each admitted attempt releases its token exactly once when that attempt finishes, before another retry selects its store.QueryCopStoreLimiteris enabled, the request-local limiter is not acquired. When it is disabled, the request-local limiter remains the fallback for existing paths such as merge-sort index lookup.*util.RateLimitplumbing with the concrete*kv.CoprRequestLimiter. Admission supports a non-blocking fast path and cancellation by the request context or iterator shutdown without leaking tokens.LimiterWaitStatsasTotalTimeandMaxTime, and expose it inEXPLAIN ANALYZEcop task runtime statistics aslimiter_wait:{total:..., max:...}. No admission count is collected or displayed.This PR depends on tikv/client-go#2026. The TiDB
go.moddependency must be updated after that PR is merged.Check List
Tests
Unit tests and checks:
The related client-go PR includes synchronous and asynchronous retry tests that verify:
Manual verification used a TiDB and TiKV playground with a table split into multiple regions. The same
EXPLAIN ANALYZEquery was run with differenttidb_query_cop_store_limitvalues:Side effects
Documentation
Release note
Please refer to Release Notes Language Style Guide to write a quality release note.