Skip to content

future: Fix double callback execution#468

Merged
wprzytula merged 1 commit into
scylladb:masterfrom
Lorak-mmk:fix-future-race-condition
Jun 2, 2026
Merged

future: Fix double callback execution#468
wprzytula merged 1 commit into
scylladb:masterfrom
Lorak-mmk:fix-future-race-condition

Conversation

@Lorak-mmk

@Lorak-mmk Lorak-mmk commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

ResolvableFuture::set_callback first set a callback under lock, released the lock, and then checked if result is set. If it is, then it invoked the callback.
The other path for callback calling is in future execution: when setting the result (under lock) new_from_future also reads a callback. If it exsits, then it is called.
the following race is possible:

  • set_callback sets callback, releases lock
  • new_from_future produces result, sees the callback, calls it
  • set_callback checks for the result, sees it exists, calls the callback.

This results in double callback execution, potentially causing UB and other issues.
This commit fixes that: in set_callback, while still under lock, we check if callback should be called. Then outside of lock we call it if needed. It is important to call the callback outside the lock, to avoid various deadlocks.

Fixes: https://scylladb.atlassian.net/browse/CUSTOMER-409

Pre-review checklist

  • I have split my patch into logically separate commits.
  • All commit messages clearly explain what they change and why.
  • PR description sums up the changes and reasons why they should be introduced.
  • I have implemented Rust unit tests for the features/changes introduced. - no clue how to test that
  • I have enabled appropriate tests in Makefile in {SCYLLA,CASSANDRA}_(NO_VALGRIND_)TEST_FILTER.
  • I added appropriate Fixes: annotations to PR description.

`ResolvableFuture::set_callback` first set a callback under lock,
released the lock, and then checked if result is set. If it is, then it
invoked the callback.
The other path for callback calling is in future execution: when setting
the result (under lock) new_from_future also reads a callback. If it
exsits, then it is called.
the following race is possible:
- `set_callback` sets callback, releases lock
- `new_from_future` produces result, sees the callback, calls it
- `set_callback` checks for the result, sees it exists, calls the
callback.

This results in double callback execution, potentially causing UB and
other issues.
This commit fixes that: in `set_callback`, while still under lock, we
check if callback should be called. Then outside of lock we call it if
needed. It is important to call the callback outside the lock, to avoid
various deadlocks.
@Lorak-mmk
Lorak-mmk requested a review from wprzytula June 2, 2026 08:05
@Lorak-mmk Lorak-mmk self-assigned this Jun 2, 2026
@Lorak-mmk Lorak-mmk added the P0 P0 item - absolute must have label Jun 2, 2026
@Lorak-mmk Lorak-mmk added this to the 1.1.0 milestone Jun 2, 2026
@coderabbitai

coderabbitai Bot commented Jun 2, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The PR refactors ResolvableFuture::set_callback to move the result-availability check into the state-lock critical section. Previously, the method checked whether self.result.get().is_some() after releasing the lock to decide on immediate callback invocation. Now, the availability check is captured as should_invoke_callback while the lock is held, and that flag is used to make the invocation decision after the lock is released. This ensures the callback invocation decision is atomic with any concurrent state updates.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely summarizes the main change: fixing a double callback execution bug in the future implementation.
Description check ✅ Passed The description thoroughly explains the race condition, the fix, and includes appropriate Fixes annotation, with pre-review checklist mostly completed.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions github-actions Bot added P1 P1 priority item - very important area/Driver_-_rust-driver and removed P0 P0 item - absolute must have labels Jun 2, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
scylla-rust-wrapper/src/future.rs (1)

427-443: ⚡ Quick win

Add a regression test for exactly-once callback delivery.

This closes the race in the implementation, but the current callback tests only prove “called at least once” with a bool flag. A future regression to double execution would still pass. Please add a test that races cass_future_set_callback() against resolution and asserts the callback count stays at 1.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scylla-rust-wrapper/src/future.rs` around lines 427 - 443, Add a regression
test that races cass_future_set_callback() against resolving the future and
asserts the callback is invoked exactly once: create a Future (the type
exercised by cass_future_set_callback and the code paths around
self.state.lock() and bound_cb.invoke), spawn one thread that repeatedly calls
cass_future_set_callback() (or equivalent wrapper) while another thread
resolves/completes the future, use an atomic counter inside the callback rather
than a bool, run the race loop many times (or use joins/barriers to increase
collision likelihood), and assert the atomic counter equals 1 after each
iteration to catch any double-invocation regressions.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@scylla-rust-wrapper/src/future.rs`:
- Around line 427-443: Add a regression test that races
cass_future_set_callback() against resolving the future and asserts the callback
is invoked exactly once: create a Future (the type exercised by
cass_future_set_callback and the code paths around self.state.lock() and
bound_cb.invoke), spawn one thread that repeatedly calls
cass_future_set_callback() (or equivalent wrapper) while another thread
resolves/completes the future, use an atomic counter inside the callback rather
than a bool, run the race loop many times (or use joins/barriers to increase
collision likelihood), and assert the atomic counter equals 1 after each
iteration to catch any double-invocation regressions.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f500f532-9134-4749-a476-da187fc54254

📥 Commits

Reviewing files that changed from the base of the PR and between 530d334 and f50548a.

📒 Files selected for processing (1)
  • scylla-rust-wrapper/src/future.rs

@wprzytula

Copy link
Copy Markdown
Contributor

Can you add a test that with high probability causes a race with the unfixed code?

@Lorak-mmk

Lorak-mmk commented Jun 2, 2026

Copy link
Copy Markdown
Contributor Author

I would love to, but I don't really have any idea how to write such test.

@wprzytula
wprzytula merged commit 18bd75c into scylladb:master Jun 2, 2026
13 of 15 checks passed
wprzytula added a commit that referenced this pull request Jun 2, 2026
Refs: #468

Add a probabilistic test (authored by Claude Opus 4.6) that verifies the
callback is invoked exactly once when set_callback races with future
resolution. The test creates a future whose resolution is synchronized
via a barrier, then calls set_callback at the same time, checking that
the atomic callback counter never exceeds 1.

The natural race window is unfortunately too narrow to hit reliably
without instrumentation. Manual testing on my machine showed the bug in
~20-40% runs, so this is enough to be hit in the CI from time to time.
@wprzytula wprzytula mentioned this pull request Jun 2, 2026
wprzytula added a commit that referenced this pull request Jun 2, 2026
`ResolvableFuture::set_callback` first set a callback under lock,
released the lock, and then checked if result is set. If it is, then it
invoked the callback.
The other path for callback calling is in future execution: when setting
the result (under lock) new_from_future also reads a callback. If it
exsits, then it is called.
the following race is possible:
- `set_callback` sets callback, releases lock
- `new_from_future` produces result, sees the callback, calls it
- `set_callback` checks for the result, sees it exists, calls the
callback.

This results in double callback execution, potentially causing UB and
other issues.
This commit fixes that: in `set_callback`, while still under lock, we
check if callback should be called. Then outside of lock we call it if
needed. It is important to call the callback outside the lock, to avoid
various deadlocks.

Fixes: https://scylladb.atlassian.net/browse/CUSTOMER-409
(cherry picked from commit 18bd75c)
wprzytula added a commit that referenced this pull request Jun 2, 2026
Refs: #468

Add a probabilistic test (authored by Claude Opus 4.6) that verifies the
callback is invoked exactly once when set_callback races with future
resolution. The test creates a future whose resolution is synchronized
via a barrier, then calls set_callback at the same time, checking that
the atomic callback counter never exceeds 1.

The natural race window is unfortunately too narrow to hit reliably
without instrumentation. Manual testing on my machine showed the bug in
~20-40% runs, so this is enough to be hit in the CI from time to time.

(cherry picked from commit ab2ce69)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/Driver_-_cpp-rs-driver P1 P1 priority item - very important

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants