Add more sections

Author: luke-gruber

doc/concurrency_guide.md

@@ -9,11 +9,10 @@ language. It will go over:
 * The difference between the VM lock and the GVL.
 * How to write code that is ractor safe.
 * What a VM barrier is and when to use it.
-* The lock hierarchy of some important locks.
+* The lock ordering of some important locks.
 * How ruby interrupt handling works.
-* What happens when IO is performed through ruby.
 * The timer thread and what it's responsible for.
-
+* What happens when IO is performed through ruby.
 
 ## The VM Lock
 
@@ -80,29 +79,74 @@ is waiting (blocked) on this same native lock, it can't join the barrier and a d
 
 The VM Lock is a particular lock in the source code. There is only one VM Lock. The GVL, on the other hand, is more of a combination of locks.
 It is "acquired" when a ruby thread is about to run or is running. Since many ruby threads can run at the same time if they're in different ractors,
-there are many GVLs (1 per `SNT` + 1 for the main ractor). It can no longer be thought of as a "Global VM Lock".
+there are many GVLs (1 per `SNT` + 1 for the main ractor). It can no longer be thought of as a "Global VM Lock" like it once was.
 
 ## How To Write Ractor-Safe Code
 
 Before ractors, only one ruby thread could run at once. That didn't mean you could forget about concurrency issues, though. Context switches happen
 often and need to be taken into account when writing code. Also, threads without the GVL run too, like the timer thread. Sometimes these threads need
 to coordinate with ruby threads, and this coordination often needs locks or atomics.
 
-When you add ractors to the mix, it gets more complicated. Take the `fstring` table, for example. It's a global set of strings that each ractor can update
-concurrently, and it's used heavily. A lockless solution is preferred to using the VM lock in this case, as taking the VM Lock would cause too many OS context
-switches. A lockless solution is also preferable for dealing with call cache tables on classes. These are also updated often and can run from multiple ractors
-concurrently. Here, an RCU (Read-Copy-Update) solution is used. What was previously an `st_table` is now a ruby object, and the old and new tables are switched
-atomically.
+When you add ractors to the mix, it gets more complicated. However, ractors allow you to forget about synchronization for non-shareable objects because
+they aren't used across ractors. Only one ruby thread can touch the object at once. For shareable objects, they are deeply frozen so there isn't any
+mutation on the objects themselves, but things like reading/writing constants across ractors do need to be synchronized. Most synchronization though, is due
+to the VM internals that need to be protected. These internals include structures for the thread scheduler on each ractor, the global ractor scheduler, the
+coordination between ruby threads and ractors, global tables (for `fstrings`, encodings, symbols and global vars), etc.
 
 ## VM Barriers
 
-Sometimes, taking the VM Lock isn't enough and you need a guarantee that all ractors have stopped. This happens when running GC, for instance.
-A VM barrier is designed for this use case. It's not used often as taking the barrier slows ractor performance down considerably, but it's useful to
+Sometimes, taking the VM Lock isn't enough and you need a guarantee that all ractors have stopped. This happens when running `GC`, for instance.
+A `VM barrier` is designed for this use case. It's not used often as taking a barrier slows ractor performance down considerably, but it's useful to
 know about and is sometimes the only solution.
 
-## Lock Hierarchy
+## Lock Orderings
+
+It's a good idea to not hold more than 2 locks at once on the same thread. Locking multiple locks can introduce deadlocks, so do it with care. When locking
+multiple locks at once, follow an ordering that is consistent across the program. Here are the orderings of some important locks:
+
+* VM lock before ractor_sched_lock()
+* thread_sched_lock() before ractor_sched_lock()
+* interrupt_lock() before timer_th.waiting_lock()
+* timer_th.waiting_lock() before ractor_sched_lock()
+
+These orderings are subject to change, so check the source if you're not sure. On top of this:
+
+* During each `ubf` (unblock) function, the VM lock can be taken around it in some circumstances. See the "Interrupt Handling" section
+for more details.
 
 ## Ruby Interrupt Handling
 
 When the VM runs ruby code, ruby's threads intermittently check ruby-level interrupts. These software interrupts
-are for various things, like
+are for various things in ruby:
+
+* Ruby threads check when they should give up their timeslice and switch to another thread when their time is up.
+* The timer thread sends a "trap" interrupt to the main thread if any ruby-level signal handlers are pending.
+* Ruby threads can have other ruby threads run tasks for them by sending them an interrupt. For instance, ractors send
+the main thread an interrupt when they need to `require` a file so that it's done on the main thread. They wait for the
+main thread's result.
+* During VM shutdown, a "terminate" interrupt is sent to all ractor main threads top stop them right away.
+* When calling `Thread#raise`, the caller sends an interrupt to that thread telling it which exception to raise.
+* Unlocking a mutex sends the next waiter (if any) an interrupt telling it to grab the lock.
+* Signalling or broadcasting on a condition variable tells the waiter(s) to wake up.
+
+This isn't a complete list.
+
+When sending an interrupt to a ruby thread, the ruby thread can be blocked. For example, it could be in the middle of a `TCPSocket#read` call. If so,
+the receiving thread's `ubf` (unblock function) gets called from the thread (ruby thread or timer thread) that sent the interrupt.
+Each ruby thread has a `ubf` that is set when it enters a blocking operation. By default, this `ubf` function sends a
+`SIGVTALRM` to the receiving thread to try to unblock it from the kernel so it can check its interrupts. There are other `ubfs` that
+aren't associated with a syscall, such as when calling `Ractor#join` or `sleep`. All `ubfs` are called with the `interrupt_lock` held,
+so take that into account when using locks inside `ubfs`.
+
+Remember, `ubfs` can be called from the timer thread so you cannot assume an `ec` inside them.
+
+## The Timer Thread
+
+The timer thread has a few functions. They are:
+
+* Send interrupts to ruby threads that have run for their whole timeslice.
+* Wake up M:N ruby threads (threads in non-main ractors) blocked on IO or after a specified timeout. This
+uses `kqueue` or `epoll`, depending on the OS, to receive IO events on behalf of the threads.
+* Continue calling  the `SIGVTARLM` signal if a thread is still blocked on a syscall after the first `ubf` call.
+* Signal native threads (`SNT`) waiting on a ractor if there are ractors waiting in the queue.
+* Create more `SNT`s if some are blocked waiting, like on IO or calling `Ractor#join`.