Skip to content

Commit ecb9e39

Browse files
sbc100sbc100
authored
Add docs/design directory along with one initial design doc. (#26621)
1 parent 49f63ca commit ecb9e39

File tree

2 files changed

+150
-0
lines changed

2 files changed

+150
-0
lines changed

‎docs/design/01-precise-futex-wakeups.md‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
# Design Doc: Precise Futex Wakeups in Emscripten
2+
3+
- **Status**: Draft
4+
- **Bug**: https://github.com/emscripten-core/emscripten/issues/26633
5+
6+
## Context
7+
Currently, `emscripten_futex_wait` (in
8+
`system/lib/pthread/emscripten_futex_wait.c`) relies on a periodic wakeup loop
9+
for pthreads and the main runtime thread. This is done for two primary reasons:
10+
11+
1. **Thread Cancellation**: To check if the calling thread has been cancelled while it is blocked.
12+
2. **Main Runtime Thread Events**: To allow the main runtime thread (even when not the main browser thread) to process its mailbox/event queue.
13+
14+
The current implementation uses a 1ms wakeup interval for the main runtime
15+
thread and a 100ms interval for cancellable pthreads. This leads to unnecessary
16+
CPU wakeups and increased latency for events.
17+
18+
## Goals
19+
- Remove the periodic wakeup loop from `emscripten_futex_wait`.
20+
- Implement precise, event-driven wakeups for cancellation and mailbox events.
21+
- Maintain the existing `emscripten_futex_wait` API signature.
22+
- Focus implementation on threads that support `atomic.wait` (pthreads and workers).
23+
24+
## Non-Goals
25+
- **Main Browser Thread**: Changes to the busy-wait loop in `futex_wait_main_browser_thread` are out of scope.
26+
- **Direct Atomics Usage**: Threads that call `atomic.wait` directly (bypassing `emscripten_futex_wait`) will remain un-interruptible.
27+
- **Wasm Workers**: Wasm Worker do not have a `pthread` structure, they are not covered by this design.
28+
29+
## Proposed Design
30+
31+
The core idea is to allow "side-channel" wakeups (cancellation, mailbox events)
32+
to interrupt the `atomic.wait` call by having the waker call `atomic.wake` on the
33+
same address the waiter is currently blocked on.
34+
35+
As part of this design we will need to explicitly state that
36+
`emscripten_futex_wait` now supports spurious wakeups. i.e. it may return `0`
37+
(success) even if the underlying futex was not explicitly woken by the
38+
application.
39+
40+
### 1. `struct pthread` Extensions
41+
We will add the following fields to `struct pthread` (in
42+
`system/lib/libc/musl/src/internal/pthread_impl.h`). All operations on these
43+
fields must use `memory_order_seq_cst` to ensure the handshake is robust.
44+
45+
```c
46+
// The address the thread is currently waiting on in emscripten_futex_wait.
47+
// NULL if the thread is not currently in a futex wait.
48+
_Atomic(void*) waiting_on_address;
49+
50+
// A counter that is incremented every time the thread wakes up from a futex wait.
51+
// Used by wakers to ensure the target thread has actually acknowledged the wake.
52+
_Atomic(uint32_t) wait_counter;
53+
54+
// A bitmask of reasons why the thread was woken for a side-channel event.
55+
_Atomic(uint32_t) wait_reasons;
56+
57+
#define WAIT_REASON_CANCEL (1 << 0)
58+
#define WAIT_REASON_MAILBOX (1 << 1)
59+
```
60+
61+
### 2. Waiter Logic (`emscripten_futex_wait`)
62+
The waiter will follow this logic (using `SEQ_CST` for all atomic accesses):
63+
64+
1. **Pre-check**: Check `wait_reasons`. If non-zero, handle the reasons (e.g., process mailbox or handle cancellation).
65+
2. **Publish**: Set `waiting_on_address = addr`.
66+
3. **Counter Snapshot**: Read `current_counter = wait_counter`.
67+
4. **Double-check**: This is critical to avoid the race where a reason was added just before `waiting_on_address` was set. If `wait_reasons` is now non-zero, clear `waiting_on_address` and go to step 1.
68+
5. **Wait**: Call `ret = __builtin_wasm_memory_atomic_wait32(addr, val, timeout)`.
69+
6. **Unpublish**:
70+
- Set `waiting_on_address = NULL`.
71+
- Atomically increment `wait_counter`.
72+
7. **Post-check**: Check `wait_reasons`. If non-zero, handle the reasons.
73+
8. **Return**: Return the result of the wait to the caller.
74+
- If `ret == ATOMICS_WAIT_OK`, return `0`.
75+
- If `ret == ATOMICS_WAIT_TIMED_OUT`, return `-ETIMEDOUT`.
76+
- If `ret == ATOMICS_WAIT_NOT_EQUAL`, return `-EWOULDBLOCK`.
77+
78+
Note: We do **not** loop internally if `ret == ATOMICS_WAIT_OK`. Even if we suspect the wake was caused by a side-channel event, we must return to the user to avoid "swallowing" a simultaneous real application wake that might not have changed the memory value.
79+
80+
### 3. Waker Logic
81+
When a thread needs to wake another thread for a side-channel event (e.g., in `pthread_cancel` or `em_task_queue_enqueue`):
82+
83+
1. Atomically OR the appropriate bit into the target thread's `wait_reasons` (`SEQ_CST`).
84+
2. Read `target_addr = target->waiting_on_address` (`SEQ_CST`).
85+
3. If `target_addr` is not NULL:
86+
- Read `start_c = target->wait_counter` (`SEQ_CST`).
87+
- Enter a loop:
88+
- Call `emscripten_futex_wake(target_addr, 1)`.
89+
- Exit loop if `target->wait_counter != start_c` OR `target->waiting_on_address != target_addr`.
90+
- **Yield**: Call `sched_yield()` (or a small sleep) to allow the target thread to proceed if it is currently being scheduled.
91+
92+
### 4. Handling the Race Condition
93+
The "Lost Wakeup" race is handled by the combination of:
94+
- The waiter double-checking `wait_reasons` after publishing its `waiting_on_address`.
95+
- The waker looping `atomic.wake` until the waiter increments its `wait_counter`.
96+
97+
Even if the waker's first `atomic.wake` occurs after the waiter's double-check
98+
but *before* the waiter actually enters the `atomic.wait` instruction, the waker
99+
will continue to loop and call `atomic.wake` again. The subsequent call(s) will
100+
successfully wake the waiter once it is actually sleeping.
101+
102+
Multiple wakers can safely call this logic simultaneously; they will all exit
103+
the loop as soon as the waiter acknowledges the wake by incrementing the
104+
counter.
105+
106+
### 5. Overlapping and Spurious Wakeups
107+
The design must handle cases where "real" wakeups (triggered by the application) and "side-channel" wakeups (cancellation/mailbox) occur simultaneously.
108+
109+
1. **Spurious Wakeups for Other Threads**: If multiple threads are waiting on the same address (e.g., a shared mutex), a side-channel `atomic_wake(addr, 1)` targeted at Thread A might be delivered by the kernel to Thread B.
110+
- **Thread B's response**: It will wake up, increment its `wait_counter`, see that its `wait_reasons` are empty, and return `0` to its caller.
111+
- **Thread C (the waker)**: It will see that Thread A's `wait_counter` has *not* changed and `waiting_on_address` is still `addr`. It will therefore continue its loop and call `atomic_wake` again until Thread A is finally woken.
112+
- **Result**: Thread B experiences a "spurious" wakeup. This is acceptable and expected behavior for futex-based synchronization.
113+
2. **Handling Side-Channel Success**: If Thread A is woken by the side-channel, it handles the event and returns `0`. The user's code will typically see that its own synchronization condition is not yet met and immediately call `emscripten_futex_wait` again. This effectively "resumes" the wait from the user's perspective while having allowed the side-channel event to be processed.
114+
3. **No Lost "Real" Wakeups**: By returning to the caller whenever `atomic.wait` returns `OK`, we ensure that we never miss or swallow a real application-level `atomic.wake`.
115+
116+
### 6. Counter Wrap-around
117+
The `wait_counter` is a `uint32_t` and will wrap around to zero after $2^{32}$ wakeups. This is safe because:
118+
1. **Impossibility of Racing**: For the waker to "miss" a wake-up due to wrap-around, the waiter would have to wake up and re-enter a sleep state exactly $2^{32}$ times in the very brief window between the waker's `atomic_wake` and its subsequent check of `wait_counter`. Even at extreme wakeup frequencies (e.g., 1 million per second), this would take over an hour.
119+
2. **Address Change Check**: The waker loop also checks `target->waiting_on_address != target_addr`. If the waiter wakes up and either stops waiting or starts waiting on a *different* address, the waker will exit the loop regardless of the counter value.
120+
121+
### 6. Benefits
122+
- **Lower Power Consumption**: Threads can sleep indefinitely (or for the full duration of a user-requested timeout) without periodic wakeups.
123+
- **Lower Latency**: Mailbox events and cancellation requests are processed immediately rather than waiting for the next 1ms or 100ms tick.
124+
- **Simpler Loop**: The complex logic for calculating remaining timeout slices in `emscripten_futex_wait` is removed.
125+
126+
## Alternatives Considered
127+
- **Signal-based wakeups**: Not currently feasible in Wasm as signals are not
128+
implemented in a way that can interrupt `atomic.wait`.
129+
- **A single global "wake-up" address per thread**: This would require the
130+
waiter to wait on *two* addresses simultaneously (the user's futex and its
131+
own wakeup address), which `atomic.wait` does not support. The proposed
132+
design works around this by having the waker use the *user's* futex address.
133+
134+
## Security/Safety Considerations
135+
- The `waiting_on_address` must be managed carefully to ensure wakers don't
136+
call `atomic.wake` on stale addresses. The `wait_counter` and clearing the
137+
address upon wake mitigate this.
138+
- The waker loop should have a reasonable fallback (like a yield) to prevent a
139+
busy-wait deadlock if the waiter is somehow prevented from waking up (though
140+
`atomic.wait` is generally guaranteed to wake if `atomic.wake` is called).

‎docs/design/README.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# Emscripten Design Documents
2+
3+
This directory contains design documents for emscripten features and major
4+
changes/refactors.
5+
6+
We are experimenting with keeping these document here under source control with
7+
the hope that this will increase understandability of the codebase. This has
8+
some advantages over doing all planning in Google Docs or GitHub issues. For
9+
example, it allows us to track the history of designs and it allows them to be
10+
searchable using standard tools like `git grep`.

0 commit comments

Comments
 (0)