Only check malloc_increase atomically if necessary

Previously we did a non-atomic read of malloc_increase to determine
whether or not we needed to gc on malloc. We also should only need to
check the value when we have actually flushed/committed a new increase.

Use relaxed load in atomic_sub_nounderflow

The previous code did a non-atomic load of val, which would work fine
(since the value would only be used for an atomic CAS) but resulted in
TSAN errors.

This alos adjusts the cas to use relaxed memory model, though I'm not
sure that actually makes a difference anywhere.

Use relaxed atomics for malloc increase as well

Use atomics for loading deferred

Use atomics for final slot count

Use flag for is_lazy_sweeping

WIP: simpler background thread (no-op for now)

Adjustments

Allow one more thread

WIP: getting closer to checking end of sweep condition correctly

dequeue

usleep

Attempt pre-sweep

Fix accounting of free slots

Sweep anything that !needs_cleanup

Add TODO

Free some T_OBJECTs in background thread

get id2ref working

Finish background sweeping before compaction

gc_abort() waits for background sweeping to finish

Add page->page_lock and lock it when changing freelist

Right now only the mutator and the background thread need to lock it.
We should re-init all these locks on fork due to Ractors (TODO).

Make sure not background sweeping during Process.warmup

Less locking/unlocking in gc_sweep_step_worker

Better sweep_lock management

reinit sweep_lock,sweep_cond after fork

Allow sweep thread to acquire VM lock.

It never joins a VM barrier.

Add simply T_DATA freeing

Also finish background freeing in ruby_vm_destruct.

Allow taking VM lock in sweep thread

Add fiber_pool_lock for cont.c

Don't take VM lock in background sweep thread

Make id2ref_tbl_lock re-entrant

We can get the following:

1) RB_VM_LOCK() // need this to allow GC when inserting into tbl
2) id2ref_tbl_lock() //
3) insert into id2ref_tbl, causes GC
4) free object id, which acquires id2ref_tbl_lock again

Therefore, the lock needs to be re-entrant.

freeing of id2ref object_id in background thread

Get gen fields freeing done in background sweep thread

First pass at making zombies in background sweep thread

add mutexes_lock lock for thread->keeping_mutexes

We need to lock this when manipulating this linked list, because freeing
a mutex, which can now be done in a background thread, manipulates it. I
made this lock global for now, but it should really be either per-ractor
or per-thread.

Add autoload_free_lock

We can't free autoload_data by 1 thread while also freeing an autoload_const
that's associated with it concurrently. This can happen currently if
they're on separate pages.

Add assertions after major GC that background thread is inactive.

I'm going to work on allowing background sweeping during a major (unless
explicitly requested via GC.start). This is probably more important even
than during minors.

Add more assertions and comments

whitelist making zombies in sweep thread

sweep some imemos in background sweep thread

tmp commit

stuck

tmp

Get 1 pass over pages mostly working

GC compaction tests are still broken. Not sure why.

TODO: when in background thread, never modify the page's freelist
directly in case user code is being run. Instead, each page should have
a deferred_freelist that the user thread will link in when the page is swept.

Merge freelist and deferred freelist when we process a page

some cleanup

Get GC compaction working, doesn't use background thread

Fix running GC in cleanup finalizers

stuck with GC compact

Fix GC.compact

Remove usage of page_lock mutex as we no longer need it.

Keep actual lock around, but I'll remove it in a separate commit.

GC: Remove unused page->page_lock mutex

cleanup

Remove unused code, add comments

Background thread only sweeps until ruby thread is done with that heap

There are some problems with the current approach:

1) The background thread can get ahead of the ruby thread on the current
heap and sweep more than is necessary instead of moving on to the next
heap. We should track `incremental_step_freed_objects` for each heap so
ruby thread and background thread are in sync, and background thread can
sweep next heap when necessary.

2) We need to restart the background sweeping when we exit from GC.
There should be a `objspace->background sweep_mode` after GC exits and
background sweeping begins.

checkin

Fix issues with parallel sweep

Issues were:

* post-fork issues
* gc_sweep_dequeue_page/heap_is_sweep_done/has_sweeping_pages trio
is tricky
* rb_ec_cleanup issues (aborting bg sweeping, stopping thread)

Fix issue with gc_sweep_rest() that could loop forever

It could happen when background sweeping got ahead of the ruby thread.

Fix more bugs

Attempt to make has_sweeping_pages() faster

We can make it even faster if we always let the ruby thread take the
last page. This is what it used to do, and I think it was the right
strategy in hindsight, just because of `has_sweeping_pages` and
`gc_sweep_finish`. Otherwise, the ruby thread could need to wait on the
background thread somtimes when it's called.

Simplify end conditions by ruby thread taking last sweeping_page

This is how it used to work, and I think it's a good idea to simplify
checks for when sweeping is finished.

Tracking down allocation bug

Fixed allocation bug

It had to do with adjacent bitfields being same memory object and used
concurrently. Changed them to bools and it fixed the issue.

Improve efficiency of requesting background sweep help

Keep track of heap->latest_swept_page

cleanup

unlink pages in sweep thread

Add to free_pages and pooled_pages in sweep thread

remove redundant work

Use atomic for heap->foreground_sweep_steps and separate swept_pages lock

Add heap->skip_sweep_continue

Add parallel sweep lock stats

Output sweep time at end of process

Add counts of sweep events

less conditionals

Change PSWEEP_LOCK_STATS to use per-callsite stats

Add wall clock psweep timings

first pass at rb_garbage_object_p with sweep thread

Fix WB issues with sweep thread

Use atomic operations for bitmaps that can be read/modified by both the
mutator and the sweep thread.

Avoid the tricky case of `gc_setup_mark_bits` by deferring it to sweep
finish. This way, it doesn't conflict with write barriers.

Make page->needs_setup_mark_bits its own memory object

tmp commit

before pairing

Bug fix for mark T_NONE

John found the fix.

Author: jhawthorn

class.c

@@ -625,6 +625,7 @@ class_get_subclasses_for_ns(struct st_table *tbl, VALUE box_id)
 static int
 remove_class_from_subclasses_replace_first_entry(st_data_t *key, st_data_t *value, st_data_t arg, int existing)
 {
+    RUBY_ASSERT(existing);
     *value = arg;
     return ST_CONTINUE;
 }
@@ -647,6 +648,7 @@ remove_class_from_subclasses(struct st_table *tbl, VALUE box_id, VALUE klass)
 
             if (first_entry) {
                 if (next) {
+                    // NOTE: doesn't allocate, just replaces existing value
                     st_update(tbl, box_id, remove_class_from_subclasses_replace_first_entry, (st_data_t)next);
                 }
                 else {

cont.c

@@ -287,6 +287,51 @@ rb_free_shared_fiber_pool(void)
 
 static ID fiber_initialize_keywords[3] = {0};
 
+rb_nativethread_lock_t fiber_lock;
+#ifdef RUBY_THREAD_PTHREAD_H
+pthread_t fiber_pool_lock_owner;
+#endif
+
+static inline void
+ASSERT_fiber_pool_locked(void)
+{
+#ifdef RUBY_THREAD_PTHREAD_H
+    VM_ASSERT(pthread_self() == fiber_pool_lock_owner);
+#endif
+}
+
+static inline void
+ASSERT_fiber_pool_unlocked(void)
+{
+#ifdef RUBY_THREAD_PTHREAD_H
+    VM_ASSERT(pthread_self() != fiber_pool_lock_owner);
+#endif
+}
+
+static inline void
+fiber_pool_lock(void) {
+    ASSERT_fiber_pool_unlocked();
+    rb_native_mutex_lock(&fiber_lock);
+#ifdef RUBY_THREAD_PTHREAD_H
+    fiber_pool_lock_owner = pthread_self();
+#endif
+}
+
+static inline void
+fiber_pool_unlock(void) {
+    ASSERT_fiber_pool_locked();
+#ifdef RUBY_THREAD_PTHREAD_H
+    fiber_pool_lock_owner = 0;
+#endif
+    rb_native_mutex_unlock(&fiber_lock);
+}
+
+void
+fiber_pool_lock_reset(void)
+{
+    rb_native_mutex_initialize(&fiber_lock);
+}
+
 /*
  * FreeBSD require a first (i.e. addr) argument of mmap(2) is not NULL
  * if MAP_STACK is passed.
@@ -508,10 +553,12 @@ fiber_pool_allocate_memory(size_t * count, size_t stride)
 // @return the allocated fiber pool.
 // @sa fiber_pool_allocation_free
 static struct fiber_pool_allocation *
-fiber_pool_expand(struct fiber_pool * fiber_pool, size_t count)
+fiber_pool_expand(struct fiber_pool * fiber_pool, size_t count, bool needs_lock, bool unlock_before_raise)
 {
     struct fiber_pool_allocation * allocation;
-    RB_VM_LOCK_ENTER();
+    allocation = RB_ALLOC(struct fiber_pool_allocation); // needs to be outside of fiber lock
+
+    if (needs_lock) fiber_pool_lock();
     {
         STACK_GROW_DIR_DETECTION;
 
@@ -522,11 +569,12 @@ fiber_pool_expand(struct fiber_pool * fiber_pool, size_t count)
         void * base = fiber_pool_allocate_memory(&count, stride);
 
         if (base == NULL) {
+            ruby_sized_xfree(allocation, sizeof(struct fiber_pool_allocation));
+            if (unlock_before_raise) fiber_pool_unlock();
             rb_raise(rb_eFiberError, "can't alloc machine stack to fiber (%"PRIuSIZE" x %"PRIuSIZE" bytes): %s", count, size, ERRNOMSG);
         }
 
         struct fiber_pool_vacancy * vacancies = fiber_pool->vacancies;
-        allocation = RB_ALLOC(struct fiber_pool_allocation);
 
         // Initialize fiber pool allocation:
         allocation->base = base;
@@ -552,6 +600,8 @@ fiber_pool_expand(struct fiber_pool * fiber_pool, size_t count)
 
             if (!VirtualProtect(page, RB_PAGE_SIZE, PAGE_READWRITE | PAGE_GUARD, &old_protect)) {
                 VirtualFree(allocation->base, 0, MEM_RELEASE);
+                ruby_sized_xfree(allocation, sizeof(struct fiber_pool_allocation));
+                if (unlock_before_raise) fiber_pool_unlock();
                 rb_raise(rb_eFiberError, "can't set a guard page: %s", ERRNOMSG);
             }
 #elif defined(__wasi__)
@@ -560,6 +610,8 @@ fiber_pool_expand(struct fiber_pool * fiber_pool, size_t count)
 #else
             if (mprotect(page, RB_PAGE_SIZE, PROT_NONE) < 0) {
                 munmap(allocation->base, count*stride);
+                ruby_sized_xfree(allocation, sizeof(struct fiber_pool_allocation));
+                if (unlock_before_raise) fiber_pool_unlock();
                 rb_raise(rb_eFiberError, "can't set a guard page: %s", ERRNOMSG);
             }
 #endif
@@ -590,7 +642,7 @@ fiber_pool_expand(struct fiber_pool * fiber_pool, size_t count)
         fiber_pool->vacancies = vacancies;
         fiber_pool->count += count;
     }
-    RB_VM_LOCK_LEAVE();
+    if (needs_lock) fiber_pool_unlock();
 
     return allocation;
 }
@@ -612,7 +664,7 @@ fiber_pool_initialize(struct fiber_pool * fiber_pool, size_t size, size_t count,
 
     fiber_pool->vm_stack_size = vm_stack_size;
 
-    fiber_pool_expand(fiber_pool, count);
+    fiber_pool_expand(fiber_pool, count, false, false);
 }
 
 #ifdef FIBER_POOL_ALLOCATION_FREE
@@ -665,7 +717,7 @@ static struct fiber_pool_stack
 fiber_pool_stack_acquire(struct fiber_pool * fiber_pool)
 {
     struct fiber_pool_vacancy * vacancy ;
-    RB_VM_LOCK_ENTER();
+    fiber_pool_lock();
     {
         vacancy = fiber_pool_vacancy_pop(fiber_pool);
 
@@ -679,7 +731,7 @@ fiber_pool_stack_acquire(struct fiber_pool * fiber_pool)
             if (count > maximum) count = maximum;
             if (count < minimum) count = minimum;
 
-            fiber_pool_expand(fiber_pool, count);
+            fiber_pool_expand(fiber_pool, count, false, true);
 
             // The free list should now contain some stacks:
             VM_ASSERT(fiber_pool->vacancies);
@@ -700,10 +752,9 @@ fiber_pool_stack_acquire(struct fiber_pool * fiber_pool)
 #ifdef FIBER_POOL_ALLOCATION_FREE
         vacancy->stack.allocation->used += 1;
 #endif
-
         fiber_pool_stack_reset(&vacancy->stack);
     }
-    RB_VM_LOCK_LEAVE();
+    fiber_pool_unlock();
 
     return vacancy->stack;
 }
@@ -769,7 +820,7 @@ fiber_pool_stack_free(struct fiber_pool_stack * stack)
 #endif
 }
 
-// Release and return a stack to the vacancy list.
+// Release and return a stack to the vacancy list. fiber_lock is acquired upon entry.
 static void
 fiber_pool_stack_release(struct fiber_pool_stack * stack)
 {
@@ -920,17 +971,6 @@ fiber_stack_release(rb_fiber_t * fiber)
     rb_ec_clear_vm_stack(ec);
 }
 
-static void
-fiber_stack_release_locked(rb_fiber_t *fiber)
-{
-    if (!ruby_vm_during_cleanup) {
-        // We can't try to acquire the VM lock here because MMTK calls free in its own native thread which has no ec.
-        // This assertion will fail on MMTK but we currently don't have CI for debug releases of MMTK, so we can assert for now.
-        ASSERT_vm_locking_with_barrier();
-    }
-    fiber_stack_release(fiber);
-}
-
 static const char *
 fiber_status_name(enum fiber_status s)
 {
@@ -1091,7 +1131,11 @@ cont_free(void *ptr)
     else {
         rb_fiber_t *fiber = (rb_fiber_t*)cont;
         coroutine_destroy(&fiber->context);
-        fiber_stack_release_locked(fiber);
+        fiber_pool_lock();
+        {
+            fiber_stack_release(fiber);
+        }
+        fiber_pool_unlock();
     }
 
     RUBY_FREE_UNLESS_NULL(cont->saved_vm_stack.ptr);
@@ -2756,9 +2800,11 @@ fiber_switch(rb_fiber_t *fiber, int argc, const VALUE *argv, int kw_splat, rb_fi
     // We cannot free the stack until the pthread is joined:
 #ifndef COROUTINE_PTHREAD_CONTEXT
     if (resuming_fiber && FIBER_TERMINATED_P(fiber)) {
-        RB_VM_LOCKING() {
+        fiber_pool_lock();
+        {
             fiber_stack_release(fiber);
         }
+        fiber_pool_unlock();
     }
 #endif
 
@@ -3475,6 +3521,7 @@ Init_Cont(void)
 #endif
     SET_MACHINE_STACK_END(&th->ec->machine.stack_end);
 
+    rb_native_mutex_initialize(&fiber_lock);
     fiber_pool_initialize(&shared_fiber_pool, stack_size, FIBER_POOL_INITIAL_SIZE, vm_stack_size);
 
     fiber_initialize_keywords[0] = rb_intern_const("blocking");

darray.h

@@ -47,6 +47,10 @@
 #define rb_darray_append_without_gc(ptr_to_ary, element) \
     rb_darray_append_impl(ptr_to_ary, element, rb_darray_realloc_mul_add_without_gc)
 
+//#define rb_darray_clear_and_free_without_gc(ptr_to_ary) \
+    //rb_darray_size(ptr_to_ary) ? (rb_darray_free_without_gc(ptr_to_ary)) : (void)0
+
+
 #define rb_darray_append_impl(ptr_to_ary, element, realloc_func) do {  \
     rb_darray_ensure_space((ptr_to_ary), \
                            sizeof(**(ptr_to_ary)), \
@@ -134,6 +138,21 @@ rb_darray_size(const void *ary)
 }
 
 
+/* Remove n items from the beginning of the array */
+#define rb_darray_shift_n(ary, n) rb_darray_shift_n_impl(ary, ary->data, n, sizeof((ary)->data[0]))
+
+static inline void
+rb_darray_shift_n_impl(void *ary, void *data, size_t n, size_t type_sz)
+{
+    rb_darray_meta_t *meta = ary;
+    RUBY_ASSERT(meta->size >= n);
+    char *dst = (char*)data;
+    if (n > 0) {
+        memmove(dst, dst + n * type_sz, (meta->size - n) * type_sz);
+        meta->size -= n;
+    }
+}
+
 static inline void
 rb_darray_pop(void *ary, size_t count)
 {
@@ -199,14 +218,23 @@ rb_darray_realloc_mul_add(void *orig_ptr, size_t x, size_t y, size_t z)
     return ptr;
 }
 
+bool is_sweep_thread_p(void);
+
 /* Internal function. Like rb_xrealloc_mul_add but does not trigger GC. */
 static inline void *
 rb_darray_realloc_mul_add_without_gc(void *orig_ptr, size_t x, size_t y, size_t z)
 {
     size_t size = rbimpl_size_add_or_raise(rbimpl_size_mul_or_raise(x, y), z);
 
     void *ptr = realloc(orig_ptr, size);
-    if (ptr == NULL) rb_bug("rb_darray_realloc_mul_add_without_gc: failed");
+    if (ptr == NULL) {
+        if (!is_sweep_thread_p()) {
+            rb_bug("rb_darray_realloc_mul_add_without_gc: failed");
+        }
+        else {
+            fprintf(stderr, "darray: realloc failed (from sweep thread)\n");
+        }
+    }
 
     return ptr;
 }

error.c

@@ -1108,11 +1108,16 @@ rb_bug_without_die(const char *fmt, ...)
     va_end(args);
 }
 
+bool is_sweep_thread_p(void);
+
 void
 rb_bug(const char *fmt, ...)
 {
     va_list args;
     va_start(args, fmt);
+    if (is_sweep_thread_p()) {
+        fprintf(stderr, "rb_bug() called from sweep_thread!\n");
+    }
     rb_bug_without_die_internal(fmt, args);
     va_end(args);
     die();