Skip to content

v1.2.0

Choose a tag to compare

@benoitc benoitc released this 15 Feb 17:27

Added

  • Context Affinity - Bind Erlang processes to dedicated Python workers for state persistence

    • py:bind() / py:unbind() - Bind current process to a worker, preserving Python state
    • py:bind(new) - Create explicit context handles for multiple contexts per process
    • py:with_context(Fun) - Scoped helper with automatic bind/unbind
    • Context-aware functions: py:ctx_call/4-6, py:ctx_eval/2-4, py:ctx_exec/2
    • Automatic cleanup via process monitors when bound processes die
    • O(1) ETS-based binding lookup for minimal overhead
    • New test suite: test/py_context_SUITE.erl
  • Python Thread Support - Any spawned Python thread can now call erlang.call() without blocking

    • Supports threading.Thread, concurrent.futures.ThreadPoolExecutor, and any other Python threads
    • Each spawned thread lazily acquires a dedicated "thread worker" channel
    • One lightweight Erlang process per Python thread handles callbacks
    • Automatic cleanup when Python thread exits via pthread_key_t destructor
    • New module: py_thread_handler.erl - Coordinator and per-thread handlers
    • New C file: py_thread_worker.c - Thread worker pool management
    • New test suite: test/py_thread_callback_SUITE.erl
    • New documentation: docs/threading.md - Threading support guide
  • Reentrant Callbacks - Python→Erlang→Python callback chains without deadlocks

    • Exception-based suspension mechanism interrupts Python execution cleanly
    • Callbacks execute in separate processes to prevent worker pool exhaustion
    • Supports arbitrarily deep nesting (tested up to 10+ levels)
    • Transparent to users - erlang.call() works the same, just without deadlocks
    • New test suite: test/py_reentrant_SUITE.erl
    • New examples: examples/reentrant_demo.erl and examples/reentrant_demo.py

Changed

  • Callback handlers now spawn separate processes for execution, allowing workers
    to remain available for nested py:eval/py:call operations
  • Modular C code structure - Split monolithic py_nif.c (4,335 lines) into
    logical modules for better maintainability:
    • py_nif.h - Shared header with types, macros, and declarations
    • py_convert.c - Bidirectional type conversion (Python ↔ Erlang)
    • py_exec.c - Python execution engine and GIL management
    • py_callback.c - Erlang callback support and asyncio integration
    • Uses #include approach for single compilation unit (no build changes needed)

Fixed

  • Multiple sequential erlang.call() - Fixed infinite loop when Python code makes
    multiple sequential erlang.call() invocations in the same function. The replay
    mechanism now falls back to blocking pipe behavior for subsequent calls after the
    first suspension, preventing the infinite replay loop.
  • Memory safety in C NIF - Fixed memory leaks and added NULL checks
    • nif_async_worker_new: msg_env now freed on pipe/thread creation failure
    • multi_executor_stop: shutdown requests now properly freed after join
    • create_suspended_state: binary allocations cleaned up on failure paths
    • Added NULL checks on all enif_alloc_resource and enif_alloc_env calls
  • Dialyzer warnings - Added {suspended, ...} return type to NIF specs for
    worker_call, worker_eval, and resume_callback functions
  • Dead code removal - Cleaned up unused code discovered during code review:
    • Removed execute_direct() function in py_exec.c (duplicated inline logic)
    • Removed unused ref field from async_pending_t struct in py_nif.h
    • Removed worker_recv/2 from py_nif.erl (declared but never implemented in C)

Documentation

  • Doxygen-style C documentation - Added documentation to all C source files:
    • Architecture overview with execution mode diagrams
    • Type mapping tables for conversions
    • GIL management patterns and best practices
    • Suspension/resume flow diagrams for callbacks
    • Function-level @param, @return, @pre, @warning, @see annotations