[Bug] Test for GC safety of blocking_operation in rb_fiber_scheduler_blocking_operation_wait

rb_fiber_scheduler_blocking_operation_wait creates `blocking_operation` as a
C-local VALUE, then calls rb_funcall(scheduler, :blocking_operation_wait, 1,
blocking_operation). If the scheduler causes a fiber switch during that call
(e.g. via rb_fiber_scheduler_block in a worker-pool implementation), the
calling fiber's C stack is suspended. The conservative GC may not scan
`blocking_operation` if it is held only in a machine register on the
suspended fiber rather than on the stack. The object then gets collected,
and get_blocking_operation() at scheduler.c:1104 reads stale/freed memory,
resulting in an rb_unexpected_object_type crash.

Confirmed by GC.disable workaround: disabling GC for the duration of the
rb_funcall prevents the crash (see socketry/io-event#170).

This commit adds a failing test to document the bug. Two trigger modes:
  :compact      - calls GC.compact inside the hook (may move blocking_operation
                  if it isn't pinned as a C-stack root)
  :fiber_switch - causes a fiber switch so the calling frame is suspended
                  during GC

The fix would be to RB_GC_GUARD(blocking_operation) after rb_funcall in
rb_fiber_scheduler_blocking_operation_wait, or to register it as a GC root
for the duration of the call.

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: samuel-williams-shopify

test/fiber/test_blocking_operation_gc.rb

@@ -0,0 +1,122 @@
+# frozen_string_literal: true
+require 'test/unit'
+require_relative 'scheduler'
+
+# Regression test for a GC bug in rb_fiber_scheduler_blocking_operation_wait.
+#
+# rb_fiber_scheduler_blocking_operation_wait creates `blocking_operation` as a
+# C-local VALUE, then calls rb_funcall(scheduler, :blocking_operation_wait, 1,
+# blocking_operation). If the scheduler's implementation causes a fiber switch
+# (via rb_fiber_scheduler_block), the calling fiber's C stack is suspended.
+# The conservative GC may not scan it correctly, allowing `blocking_operation`
+# to be collected. When Ruby reads it back via get_blocking_operation() at
+# scheduler.c:1104 after rb_funcall returns, it finds freed/reused memory ->
+# crash (rb_unexpected_object_type -> segfault).
+#
+# We can reliably trigger the same crash without a fiber switch by calling
+# GC.compact inside blocking_operation_wait: the compacting GC may move the
+# blocking_operation object without updating the C-local VALUE in the (now
+# inactive) scheduler.c frame, leaving a stale pointer.
+#
+# See: https://github.com/socketry/io-event/pull/166
+
+class BlockingOperationGCScheduler < Scheduler
+  # Two variants for triggering the bug:
+  #   :fiber_switch - cause a fiber switch (realistic: worker-pool-style)
+  #   :compact      - call GC.compact (does not require a fiber switch)
+  attr_accessor :trigger
+  attr_reader :blocking_operation_wait_call_count
+
+  def initialize(fiber = Fiber.current, trigger: :compact)
+    super(fiber)
+    @trigger = trigger
+    @gc_fiber = nil
+    @blocking_operation_wait_call_count = 0
+  end
+
+  def blocking_operation_wait(work)
+    @blocking_operation_wait_call_count += 1
+
+    # Execute the blocking operation so Ruby marks it done.
+    thread = Thread.new { work.call }
+    thread.join
+
+    case @trigger
+    when :compact
+      # GC.compact moves objects. If the C-local `blocking_operation` VALUE in
+      # rb_fiber_scheduler_blocking_operation_wait is not updated, the pointer
+      # becomes stale. Ruby then calls get_blocking_operation() on a moved/freed
+      # object -> wrong-type crash.
+      # Allocate garbage first to increase pressure on the GC.
+      1000.times { Object.new }
+      GC.compact
+    when :fiber_switch
+      # Transfer to another fiber so the calling fiber's C stack is suspended.
+      # The GC runs during the switch. If `blocking_operation` is in a register
+      # on the suspended fiber (not scanned by conservative GC), it gets collected.
+      @gc_fiber ||= Fiber.new do
+        loop do
+          1000.times { Object.new }
+          GC.start(full_mark: true, immediate_sweep: true)
+          Fiber.yield
+        end
+      end
+      @gc_fiber.resume
+    end
+  end
+end
+
+class TestBlockingOperationGC < Test::Unit::TestCase
+  def omit_unless_supported
+    omit "GC.compact not supported" unless GC.respond_to?(:compact)
+    omit "TCPServer not available" unless defined?(TCPServer)
+  end
+
+  def run_with_scheduler(trigger:)
+    scheduler = nil
+    Thread.new do
+      scheduler = BlockingOperationGCScheduler.new(trigger: trigger)
+      Fiber.set_scheduler(scheduler)
+
+      server = TCPServer.new("localhost", 0)
+      port = server.addr[1]
+      client = TCPSocket.new("localhost", port)
+      peer = server.accept
+
+      Fiber.schedule do
+        # IO#close calls maygvl_close -> IO_WITHOUT_GVL -> rb_nogvl(RB_NOGVL_OFFLOAD_SAFE)
+        # -> rb_fiber_scheduler_blocking_operation_wait when a fiber scheduler is active.
+        server.close
+      end
+    ensure
+      client&.close rescue nil
+      peer&.close rescue nil
+    end.join
+    scheduler
+  end
+
+  # Trigger via GC.compact inside blocking_operation_wait.
+  # GC.compact moves objects; if the C-local `blocking_operation` VALUE in
+  # rb_fiber_scheduler_blocking_operation_wait is not updated (e.g. it was only
+  # in a register on the suspended fiber), get_blocking_operation() reads a
+  # stale pointer -> wrong-type crash.
+  def test_blocking_operation_gc_compact
+    omit_unless_supported
+    assert_nothing_raised do
+      scheduler = run_with_scheduler(trigger: :compact)
+      omit "blocking_operation_wait was not called (Ruby version does not route IO#close through it)" if scheduler.blocking_operation_wait_call_count == 0
+    end
+  end
+
+  # Trigger via fiber switch inside blocking_operation_wait.
+  # When the calling fiber is suspended, the conservative GC may not find
+  # `blocking_operation` if it is only in a register (not on the stack).
+  # It gets collected, and get_blocking_operation() crashes on return from rb_funcall.
+  def test_blocking_operation_gc_fiber_switch
+    omit_unless_supported
+    assert_nothing_raised do
+      scheduler = run_with_scheduler(trigger: :fiber_switch)
+      omit "blocking_operation_wait was not called (Ruby version does not route IO#close through it)" if scheduler.blocking_operation_wait_call_count == 0
+    end
+  end
+end