RUBY-3706 Exponential backoff and jitter in retry by comandeo-mongo · Pull Request #2998 · mongodb/mongo-ruby-driver

comandeo-mongo · 2026-03-13T07:31:29Z

No description provided.

Copilot

Pull request overview

Adds client backpressure support by retrying overload errors (SystemOverloadedError + RetryableError) using exponential backoff with jitter, with an optional adaptive retry limiter (token bucket). This integrates the new policy into read/write retry paths, with_transaction, selected operations, and adds spec coverage (unified + unit/prose).

Changes:

Introduces Mongo::Retryable::Backpressure, TokenBucket, and RetryPolicy (including adaptiveRetries URI/client option).
Implements overload retry loops in read/write workers, cursor getMore, selected operations (Database#command, index create/drop, write aggregations), and transaction retries.
Adds unified spec test fixtures + runners and new unit/prose specs; app metadata now advertises backpressure: true.

Reviewed changes

Copilot reviewed 30 out of 32 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
spec/spec_tests/data/client_backpressure/getMore-retried.yml	Unified spec fixture for getMore overload retry behavior
spec/spec_tests/data/client_backpressure/getMore-retried.json	JSON form of the getMore overload unified spec
spec/spec_tests/data/client_backpressure/backpressure-retry-max-attempts.yml	Generated unified specs asserting maxAttempts=5 behavior
spec/spec_tests/data/client_backpressure/backpressure-retry-loop.yml	Generated unified specs asserting overload retry loop behavior
spec/spec_tests/client_backpressure_unified_spec.rb	Runs the new client backpressure unified specs
spec/mongo/session/with_transaction_overload_spec.rb	Unit tests for overload backoff integration in `with_transaction`
spec/mongo/server/app_metadata_backpressure_spec.rb	Ensures app metadata includes `backpressure: true`
spec/mongo/retryable/write_worker_overload_spec.rb	Unit tests for write overload retry loop + token bucket interactions
spec/mongo/retryable/read_worker_overload_spec.rb	Unit tests for read overload retry loop + token bucket interactions
spec/mongo/retryable/retry_policy_spec.rb	Unit tests for backoff + should-retry logic
spec/mongo/retryable/token_bucket_spec.rb	Unit tests for token bucket behavior + thread-safety
spec/mongo/retryable/backpressure_spec.rb	Unit tests for backpressure constants and delay calculation
spec/mongo/retryable/overload_error_helpers_spec.rb	Tests new overload label helper predicates
spec/mongo/retryable/client_backpressure_prose_spec.rb	Prose tests derived from the client-backpressure spec
spec/mongo/retryable/adaptive_retries_option_spec.rb	Verifies `adaptiveRetries` option parsing/defaults
lib/mongo/uri/options_mapper.rb	Adds `adaptiveRetries` URI mapping to `:adaptive_retries`
lib/mongo/client.rb	Initializes per-client `retry_policy`; adds `:adaptive_retries` to valid options
lib/mongo/retryable/backpressure.rb	Implements exponential backoff + jitter constants/helpers
lib/mongo/retryable/token_bucket.rb	Adds thread-safe token bucket implementation
lib/mongo/retryable/retry_policy.rb	Centralizes backoff + adaptive retry decision logic
lib/mongo/retryable/base_worker.rb	Adds retry policy accessor + overload label helpers
lib/mongo/retryable/read_worker.rb	Hooks overload retry loop + records success for token returns
lib/mongo/retryable/write_worker.rb	Hooks overload retry loop + records success for token returns
lib/mongo/retryable.rb	Requires new modules; adds generic `with_overload_retry` wrapper
lib/mongo/session.rb	Adds overload-aware backoff path to `with_transaction` retries
lib/mongo/server/app_metadata.rb	Advertises `backpressure: true` in handshake metadata
lib/mongo/database.rb	Wraps `Database#command` with overload retry wrapper
lib/mongo/index/view.rb	Wraps index create/drop execution with overload retry wrapper
lib/mongo/collection/view/iterable.rb	Wraps write-aggregation initial query with overload retry wrapper
lib/mongo/cursor.rb	Retries `getMore` on overload errors using overload retry wrapper

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

spec/mongo/retryable/token_bucket_spec.rb

lib/mongo/retryable/write_worker.rb

+            result = server.with_connection(connection_global_id: context.connection_global_id) do |connection|
              yield connection, nil, context
            end
+            retry_policy.record_success(is_retry: error_count > 0) if error_count > 0


spec/spec_tests/data/client_backpressure/backpressure-retry-loop.yml

+  documents: []
+
+_yamlAnchors:
+  bulWriteInsertNamespace: &client_bulk_write_ns retryable-writes-tests.coll


spec/spec_tests/data/client_backpressure/backpressure-retry-max-attempts.yml

+          # we expect 6 pairs of command started and succeeded events:
+          #   1 initial attempt and 5 retries.
+          - commandStartedEvent:
+              commandName: listDatabases
+          - commandFailedEvent:


Copilot

Pull request overview

This PR implements MongoDB client backpressure behavior in the Ruby driver by introducing an overload retry loop with exponential backoff + jitter (and optional adaptive retry limiting via a token bucket), and adds extensive automated coverage (unified spec tests + unit/prose specs) to validate the behavior.

Changes:

Add Mongo::Retryable::Backpressure, TokenBucket, and RetryPolicy to support exponential backoff/jitter and adaptive retries.
Integrate overload retry loops into read/write retry workers, cursor getMore, Database#command, index operations, and Session#with_transaction.
Add unified spec YAML/JSON fixtures and multiple RSpec suites validating backpressure retries, max attempts, token bucket behavior, and metadata reporting.

Reviewed changes

Copilot reviewed 31 out of 33 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
spec/spec_tests/data/client_backpressure/getMore-retried.yml	Unified spec fixture validating overload retries for `getMore` (including max attempts).
spec/spec_tests/data/client_backpressure/getMore-retried.json	JSON version of the `getMore` unified spec fixture.
spec/spec_tests/data/client_backpressure/backpressure-retry-max-attempts.yml	Generated unified specs asserting operations do not exceed max retry attempts under overload.
spec/spec_tests/data/client_backpressure/backpressure-retry-loop.yml	Generated unified specs asserting operations follow overload retry loop semantics and respect retryReads/retryWrites.
spec/spec_tests/client_backpressure_unified_spec.rb	Runner for the new client backpressure unified spec suite (with skip for clientBulkWrite).
spec/mongo/session/with_transaction_overload_spec.rb	Unit specs for overload backoff behavior within `with_transaction`.
spec/mongo/server/app_metadata_backpressure_spec.rb	Verifies handshake metadata includes `backpressure: true`.
spec/mongo/retryable_spec.rb	Updates existing retryable specs to provide a `retry_policy` on the mocked client.
spec/mongo/retryable/write_worker_overload_spec.rb	Unit specs for overload retry loop in write worker, including adaptive retries and notes behavior.
spec/mongo/retryable/token_bucket_spec.rb	Unit specs for token bucket semantics and concurrency behavior.
spec/mongo/retryable/retry_policy_spec.rb	Unit specs for retry policy (delay computation, deadline checks, token consumption/deposit).
spec/mongo/retryable/read_worker_overload_spec.rb	Unit specs for overload retry loop in read worker (including adaptive retries and record_success behavior).
spec/mongo/retryable/overload_error_helpers_spec.rb	Unit specs for new BaseWorker overload error helper predicates.
spec/mongo/retryable/client_backpressure_prose_spec.rb	Prose tests mirroring the client-backpressure spec’s narrative requirements.
spec/mongo/retryable/backpressure_spec.rb	Unit specs for backoff constants and `Backpressure.backoff_delay` behavior.
spec/mongo/retryable/adaptive_retries_option_spec.rb	Coverage for `adaptiveRetries` URI/client option parsing and defaults.
lib/mongo/uri/options_mapper.rb	Adds URI option mapping for `adaptiveRetries` -> `:adaptive_retries`.
lib/mongo/session.rb	Adds overload-aware backoff and retry limiting to transaction retry flow (including commit retry paths).
lib/mongo/server/app_metadata.rb	Advertises `backpressure: true` in the client handshake metadata.
lib/mongo/retryable/write_worker.rb	Implements overload retry loop for writes and integrates retry policy accounting (`record_success`, token handling).
lib/mongo/retryable/token_bucket.rb	Introduces a mutex-protected token bucket to rate-limit overload retries when adaptive retries are enabled.
lib/mongo/retryable/retry_policy.rb	Encapsulates backoff calculation, retry admission (max retries, deadline, tokens), and token accounting.
lib/mongo/retryable/read_worker.rb	Implements overload retry loop for reads and integrates retry policy accounting.
lib/mongo/retryable/base_worker.rb	Adds `retry_policy` accessor plus overload error helper predicates used by workers.
lib/mongo/retryable/backpressure.rb	Defines constants and exponential backoff/jitter helper for client backpressure.
lib/mongo/retryable.rb	Requires new backpressure modules and adds a generic `with_overload_retry` wrapper for operations.
lib/mongo/index/view.rb	Wraps create/drop index operations in `with_overload_retry` when retryWrites enabled.
lib/mongo/database.rb	Wraps `Database#command` execution in `with_overload_retry` with an explicit `Operation::Context`.
lib/mongo/cursor.rb	Adds overload retry wrapper around `getMore` and closes cursor when overload retries are exhausted.
lib/mongo/collection/view/iterable.rb	Wraps initial write aggregation cursor creation in `with_overload_retry` when retryWrites enabled.
lib/mongo/client.rb	Adds `:adaptive_retries` option and constructs a per-client `RetryPolicy`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

spec/mongo/retryable/token_bucket_spec.rb

+
+    it 'handles concurrent consume and deposit' do
+      run_concurrent_operations(bucket)
+      expect(bucket.tokens).to eq(1000)


lib/mongo/cursor.rb

+      # However, overload errors (SystemOverloadedError + RetryableError) are
+      # retried with exponential backoff since the server never processed
+      # the request.
+      with_overload_retry(context: possibly_refreshed_context) do


spec/spec_tests/data/client_backpressure/backpressure-retry-max-attempts.yml

+          # we expect 6 pairs of command started and succeeded events:
+          #   1 initial attempt and 5 retries.
+          - commandStartedEvent:
+              commandName: listDatabases
+          - commandFailedEvent:
+              commandName: listDatabases


lib/mongo/retryable/retry_policy.rb

+      # @param [ Mongo::CsotTimeoutHolder | nil ] context The operation
+      #   context (for CSOT deadline checking).
+      #
+      # @return [ true | false ] Whether the retry should proceed.
+      def should_retry_overload?(attempt, delay, context: nil)
+        return false if attempt > Backpressure::MAX_RETRIES
+        return false if exceeds_deadline?(delay, context)


spec/spec_tests/data/client_backpressure/backpressure-retry-loop.yml

+  documents: []
+
+_yamlAnchors:
+  bulWriteInsertNamespace: &client_bulk_write_ns retryable-writes-tests.coll


jamis · 2026-03-13T14:28:11Z

lib/mongo/retryable.rb

+      loop do
+        begin


You can use implicit begin/rescue/end in a do block:

loop do do_something rescue => e rescue_something else else_something ensure ensure_something end

RUBY-3706 Exponential backoff and jitter in retry

46cde18

comandeo-mongo requested a review from a team as a code owner March 13, 2026 07:31

comandeo-mongo requested review from Copilot and jamis March 13, 2026 07:31

Copilot started reviewing on behalf of comandeo-mongo March 13, 2026 07:31 View session

Copilot AI reviewed Mar 13, 2026

View reviewed changes

comandeo-mongo added 5 commits March 13, 2026 09:06

Skip bulkWrite spec tests

566e27f

Fix test failures

571637c

Remove non-ascii characters

c9a0258

Fix some rubocop offences

eeda8a4

Fix more rubocop insanity

765f208

comandeo-mongo requested a review from Copilot March 13, 2026 14:14

Copilot started reviewing on behalf of comandeo-mongo March 13, 2026 14:14 View session

Copilot AI reviewed Mar 13, 2026

View reviewed changes

jamis previously approved these changes Mar 13, 2026

View reviewed changes

Remove unnecessary spec test files

858dea2

comandeo-mongo dismissed jamis’s stale review via 858dea2 March 13, 2026 14:56

jamis approved these changes Mar 13, 2026

View reviewed changes

comandeo-mongo merged commit 6391c2a into mongodb:master Mar 16, 2026
169 of 171 checks passed

comandeo-mongo added the feature A PR for a new feature label Mar 16, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RUBY-3706 Exponential backoff and jitter in retry#2998

RUBY-3706 Exponential backoff and jitter in retry#2998
comandeo-mongo merged 7 commits intomongodb:masterfrom
comandeo-mongo:3706-bacpressure-retries

comandeo-mongo commented Mar 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

jamis Mar 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

comandeo-mongo commented Mar 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

jamis Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants