fix crashes using multithreaded julia from python (#758)

* fix: rare crash due to race condition

The vectors PYJLVALUES and PYJLFREEVALUES are modified without any protection against GC-triggered re-entrancy.

  _pyjl_dealloc(o1) or PyJuliaValue_SetValue
    └→ push!(PYJLFREEVALUES, idx)          # begins resize (sets internal flag)
         └→ vector must grow → allocates → triggers Julia GC
              └→ GC collects a Py object → runs py_finalizer
                   └→ enqueue(ptr) → PyGILState_Check()==1 (same thread holds GIL)
                        └→ Py_DecRef(ptr) → refcount hits 0
                             └→ _pyjl_dealloc(o2)
                                  └→ push!(PYJLFREEVALUES, idx2)   ← BOOM: flag already set
                                     ConcurrencyViolationError!

1. push! needs to grow the vector (exceeds capacity), so it allocates
2. The allocation triggers Julia GC, which runs finalizers
3. A finalizer calls Py_DecRef (because enqueue sees the GIL is held on this same thread), dropping refcount to 0, which triggers _pyjl_dealloc re-entrantly on the same vector

This is especially likely during shutdown (at_jl_exit → jl_atexit_hook), because jl_gc_run_all_finalizers runs finalizers on the calling thread (the main thread, which holds the GIL).
Many Py objects are finalized at once, repeatedly pushing to PYJLFREEVALUES, and each push that triggers a reallocation can cause the re-entrant chain.

Disabled Julia's GC during the critical vector modifications, so that push!/pop! cannot trigger allocation-based GC,
which prevents the finalizer chain from re-entering.
Added a SpinLock as defense-in-depth against true multi-thread races.

* fix: no GC disable. avoid resize PYJLFREEVALUES in dealloc

---------

Co-authored-by: Dani Pinyol <dani@avatarcognition.com>

Author: dpinol

src/JlWrap/C.jl

@@ -23,8 +23,14 @@ const PyJuliaBase_Type = Ref(C.PyNULL)
 # we store the actual julia values here
 # the `value` field of `PyJuliaValueObject` indexes into here
 const PYJLVALUES = []
-# unused indices in PYJLVALUES
+# unused indices in PYJLVALUES; kept the same length as PYJLVALUES (extra slots
+# are pushed alongside PYJLVALUES so the dealloc/finalizer path never has to
+# resize this vector). Only the first PYJLFREEVALUECOUNT[] entries are valid
+# free indices; the rest are placeholders.
 const PYJLFREEVALUES = Int[]
+const PYJLFREEVALUECOUNT = Ref(0)
+# lock protecting PYJLVALUES, PYJLFREEVALUES and PYJLFREEVALUECOUNT
+const PYJL_LOCK = Threads.SpinLock()
 
 function _pyjl_new(t::C.PyPtr, ::C.PyPtr, ::C.PyPtr)
     alloc = C.PyType_GetSlot(t, C.Py_tp_alloc)
@@ -39,8 +45,14 @@ end
 function _pyjl_dealloc(o::C.PyPtr)
     idx = C.@ft UnsafePtr{PyJuliaValueObject}(o).value[]
     if idx != 0
+        # No allocation in this critical section: PYJLFREEVALUES has been pre-sized
+        # to match PYJLVALUES by PyJuliaValue_SetValue, so recording a free index is
+        # just a write to an existing slot plus a counter bump. This avoids the GC-triggered
+        lock(PYJL_LOCK)
         PYJLVALUES[idx] = nothing
-        push!(PYJLFREEVALUES, idx)
+        PYJLFREEVALUECOUNT[] += 1
+        PYJLFREEVALUES[PYJLFREEVALUECOUNT[]] = idx
+        unlock(PYJL_LOCK)
     end
     (C.@ft UnsafePtr{PyJuliaValueObject}(o).weaklist[!]) == C.PyNULL || C.PyObject_ClearWeakRefs(o)
     freeptr = C.PyType_GetSlot(C.Py_Type(o), C.Py_tp_free)
@@ -375,13 +387,19 @@ PyJuliaValue_SetValue(_o, @nospecialize(v)) = Base.GC.@preserve _o begin
     o = C.asptr(_o)
     idx = C.@ft UnsafePtr{PyJuliaValueObject}(o).value[]
     if idx == 0
-        if isempty(PYJLFREEVALUES)
+        lock(PYJL_LOCK)
+        if PYJLFREEVALUECOUNT[] == 0
+            # Grow both vectors together so length(PYJLFREEVALUES) == length(PYJLVALUES)
+            # is preserved. The 0 in PYJLFREEVALUES is just a placeholder
             push!(PYJLVALUES, v)
+            push!(PYJLFREEVALUES, 0)
             idx = length(PYJLVALUES)
         else
-            idx = pop!(PYJLFREEVALUES)
+            idx = PYJLFREEVALUES[PYJLFREEVALUECOUNT[]]
+            PYJLFREEVALUECOUNT[] -= 1
             PYJLVALUES[idx] = v
         end
+        unlock(PYJL_LOCK)
         C.@ft UnsafePtr{PyJuliaValueObject}(o).value[] = idx
     else
         PYJLVALUES[idx] = v