future: Fix double callback execution#468
Conversation
`ResolvableFuture::set_callback` first set a callback under lock, released the lock, and then checked if result is set. If it is, then it invoked the callback. The other path for callback calling is in future execution: when setting the result (under lock) new_from_future also reads a callback. If it exsits, then it is called. the following race is possible: - `set_callback` sets callback, releases lock - `new_from_future` produces result, sees the callback, calls it - `set_callback` checks for the result, sees it exists, calls the callback. This results in double callback execution, potentially causing UB and other issues. This commit fixes that: in `set_callback`, while still under lock, we check if callback should be called. Then outside of lock we call it if needed. It is important to call the callback outside the lock, to avoid various deadlocks.
📝 WalkthroughWalkthroughThe PR refactors 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Comment |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
scylla-rust-wrapper/src/future.rs (1)
427-443: ⚡ Quick winAdd a regression test for exactly-once callback delivery.
This closes the race in the implementation, but the current callback tests only prove “called at least once” with a
boolflag. A future regression to double execution would still pass. Please add a test that racescass_future_set_callback()against resolution and asserts the callback count stays at 1.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scylla-rust-wrapper/src/future.rs` around lines 427 - 443, Add a regression test that races cass_future_set_callback() against resolving the future and asserts the callback is invoked exactly once: create a Future (the type exercised by cass_future_set_callback and the code paths around self.state.lock() and bound_cb.invoke), spawn one thread that repeatedly calls cass_future_set_callback() (or equivalent wrapper) while another thread resolves/completes the future, use an atomic counter inside the callback rather than a bool, run the race loop many times (or use joins/barriers to increase collision likelihood), and assert the atomic counter equals 1 after each iteration to catch any double-invocation regressions.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@scylla-rust-wrapper/src/future.rs`:
- Around line 427-443: Add a regression test that races
cass_future_set_callback() against resolving the future and asserts the callback
is invoked exactly once: create a Future (the type exercised by
cass_future_set_callback and the code paths around self.state.lock() and
bound_cb.invoke), spawn one thread that repeatedly calls
cass_future_set_callback() (or equivalent wrapper) while another thread
resolves/completes the future, use an atomic counter inside the callback rather
than a bool, run the race loop many times (or use joins/barriers to increase
collision likelihood), and assert the atomic counter equals 1 after each
iteration to catch any double-invocation regressions.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: f500f532-9134-4749-a476-da187fc54254
📒 Files selected for processing (1)
scylla-rust-wrapper/src/future.rs
|
Can you add a test that with high probability causes a race with the unfixed code? |
|
I would love to, but I don't really have any idea how to write such test. |
Refs: #468 Add a probabilistic test (authored by Claude Opus 4.6) that verifies the callback is invoked exactly once when set_callback races with future resolution. The test creates a future whose resolution is synchronized via a barrier, then calls set_callback at the same time, checking that the atomic callback counter never exceeds 1. The natural race window is unfortunately too narrow to hit reliably without instrumentation. Manual testing on my machine showed the bug in ~20-40% runs, so this is enough to be hit in the CI from time to time.
`ResolvableFuture::set_callback` first set a callback under lock, released the lock, and then checked if result is set. If it is, then it invoked the callback. The other path for callback calling is in future execution: when setting the result (under lock) new_from_future also reads a callback. If it exsits, then it is called. the following race is possible: - `set_callback` sets callback, releases lock - `new_from_future` produces result, sees the callback, calls it - `set_callback` checks for the result, sees it exists, calls the callback. This results in double callback execution, potentially causing UB and other issues. This commit fixes that: in `set_callback`, while still under lock, we check if callback should be called. Then outside of lock we call it if needed. It is important to call the callback outside the lock, to avoid various deadlocks. Fixes: https://scylladb.atlassian.net/browse/CUSTOMER-409 (cherry picked from commit 18bd75c)
Refs: #468 Add a probabilistic test (authored by Claude Opus 4.6) that verifies the callback is invoked exactly once when set_callback races with future resolution. The test creates a future whose resolution is synchronized via a barrier, then calls set_callback at the same time, checking that the atomic callback counter never exceeds 1. The natural race window is unfortunately too narrow to hit reliably without instrumentation. Manual testing on my machine showed the bug in ~20-40% runs, so this is enough to be hit in the CI from time to time. (cherry picked from commit ab2ce69)
ResolvableFuture::set_callbackfirst set a callback under lock, released the lock, and then checked if result is set. If it is, then it invoked the callback.The other path for callback calling is in future execution: when setting the result (under lock) new_from_future also reads a callback. If it exsits, then it is called.
the following race is possible:
set_callbacksets callback, releases locknew_from_futureproduces result, sees the callback, calls itset_callbackchecks for the result, sees it exists, calls the callback.This results in double callback execution, potentially causing UB and other issues.
This commit fixes that: in
set_callback, while still under lock, we check if callback should be called. Then outside of lock we call it if needed. It is important to call the callback outside the lock, to avoid various deadlocks.Fixes: https://scylladb.atlassian.net/browse/CUSTOMER-409
Pre-review checklist
I have implemented Rust unit tests for the features/changes introduced.- no clue how to test thatI have enabled appropriate tests inMakefilein{SCYLLA,CASSANDRA}_(NO_VALGRIND_)TEST_FILTER.Fixes:annotations to PR description.