Add docs/design directory along with one initial design doc.

[sbc100]

No description provided.

[sbc100]

I add some more context to the README. I didn't completely remove the reference to AI here, since that is one of the main motivations here, but I did re-phrase to make it clear its only part of the rationale.

[kripken]

lgtm in general, and also the first design doc seems generally good to me, though I'm not an expert on the locking stuff.

[tlively]

This is going to be out of date once the new design lands. Should this be phrased as "the previous implementation" or edited out entirely to future-proof the doc?

[sbc100]

I'm not really sure how to address this problem. The perhaps by updating these documents if/when the bugs get closed. i.e. perhaps when Draft is changed to Done or Complete?

[tlively]

Maybe this should be named wake_counter.

[sbc100]

Its the waiter that increments it though. i.e. it counts how much times a given threads has called futex_wait.

[tlively]

Yes, but it increments upon waking. The current name suggests to me that it should be incremented when going to sleep, but in fact it is incremented when waking.

[tlively]

As a meta-comment, I'm not sure it's worth checking in a design doc if it is going to contain precise pseudocode listing like this. It might as well be real code.

[tlively]

Why wake just one waiter instead of all waiters at once? If all were woken at once, we could get rid of the wait_counter entirely, right? And maybe get rid of waiting_on_address as well?

[sbc100]

I agree, I don't see any reason not to wake all waiters.

Regarding getting rid waiting_on_address .. I don't see how that is possible? If you need to wake a given thread you need to know what address its currently waiting on don't you?

Regarding getting rid of wait_counter, did you grok the reason why its actually needed? i.e. to fix the possible race between setting waiting_on_address and the actual atomic.wait instruction. Maybe you have some other idea about how to solve that race?

[tlively]

did you grok

Probably not :)

[tlively]

Ok, I got it. We need the waiting_on_address so we know what address to notify when we want to send a waiting thread a message. It's possible we could send the notification before the waiting thread actually goes to sleep, so to protect against that we notify it repeatedly until we observe it changing its wake count, at which point we know it will proceed to handle the notification.

But instead of a wake count, perhaps we could just have the waiting thread clear its wait_reasons when it reads them after waking up. Then the notifying thread can just observe the wait_reasons get cleared to know the target thread has woken up.

[tlively]

There is no kernel in this case. Is this describing how native futexes work?

[sbc100]

I guess "kernel" should be replaced by "host" here. Would that makes sense here. This sentence does make sense to me with that replacment.

Its referring to the fact that normal wakups can overlap with the side channel wakeups.

[tlively]

This doesn't appear to be used for anything. (But it would be easier to be sure if looking at real code.)

[tlively]

This could be slightly simplified by using the low bits of waiting_on_address as the wait_reason, so both could atomically checked and updated in a CAS loop.

[sbc100]

Actually I think we can just remove wait_reasons completely.

That was something that Gemini added rather than me, and I don't think its actually safe since we could have multiple wakers trying to wake a sleeping thread for different reasons.

This is also why the counter is needed since it removes the multiple-wakers problem.

[tlively]

Ok, I got it. We need the waiting_on_address so we know what address to notify when we want to send a waiting thread a message. It's possible we could send the notification before the waiting thread actually goes to sleep, so to protect against that we notify it repeatedly until we observe it changing its wake count, at which point we know it will proceed to handle the notification.

But instead of a wake count, perhaps we could just have the waiting thread clear its wait_reasons when it reads them after waking up. Then the notifying thread can just observe the wait_reasons get cleared to know the target thread has woken up.

[tlively]

Proposed simpler protocol using a bit instead of a counter:

// Side-channel notify a target thread after enqueuing work.
addr = fetch_or(&target->wait_addr, NOTIFY_BIT);
if (addr == NULL) {
  // The thread is not waiting. We can leave the notify bit set.
  return;
}
if ((addr & NOTIFY_BIT) != 0) {
  // Some other thread is already notifying this target.
  return;
}
// We set the bit and are responsible for notifying the target. 
while (atomic_load(&target->wait_addr) == (addr | NOTIFY_BIT)) {
  // The target is not awake yet.
  notify_all(addr)
  sched_yield()
}

// Futex
while (!cas_weak(&self->wait_addr, NULL, addr)) {
  // Handle notifications
  atomic_store(&target->wait_addr, NULL)
  read_mailbox()
}
// From here, anyone trying to side-channel notify us will keep trying until they observe us wake up.
ret = atomic_wait(addr, expected, timeout)
if (atomic_exchange(&self->wait_addr, NULL) != addr) {
  // Handle notifications received while sleeping.
  read_mailbox();
}
return ret;

[sbc100]

I like it!