Skip to content

Releases: benoitc/erlang-python

v1.7.1

Choose a tag to compare

@benoitc benoitc released this 22 Feb 23:51

Fixed

  • Hex package missing priv directory - Added explicit files configuration to include priv/erlang_loop.py and other necessary files in the hex.pm package

v1.7.0

Choose a tag to compare

@benoitc benoitc released this 22 Feb 23:38

Added

  • Shared Router Architecture for Event Loops

    • Single py_event_router process handles all event loops (both shared and isolated)
    • Timer and FD messages include loop identity for correct dispatch
    • Eliminates need for per-loop router processes
    • Handle-based Python C API using PyCapsule for loop references
  • Isolated Event Loops - Create isolated event loops with ErlangEventLoop(isolated=True)

    • Default (isolated=False): uses the shared global loop managed by Erlang
    • Isolated (isolated=True): creates a dedicated loop with its own pending queue
    • Full asyncio support (timers, FD operations) for both modes
    • Useful for multi-threaded Python applications where each thread needs its own loop
    • See docs/asyncio.md for usage and architecture details

v1.6.1

Choose a tag to compare

@benoitc benoitc released this 22 Feb 22:01
7326786

Fixed

  • ASGI headers now correctly use bytes instead of str - Fixed ASGI spec compliance issue where headers were being converted to Python str objects instead of bytes. The ASGI specification requires headers to be list[tuple[bytes, bytes]]. This was causing authentication failures and form parsing issues with frameworks like Starlette and FastAPI, which search for headers using bytes keys (e.g., b"content-type").
    • Added explicit header handling in asgi_scope_from_map() to bypass generic conversion
    • Headers are now correctly converted using PyBytes_FromStringAndSize()
    • Supports both list [name, value] and tuple {name, value} header formats from Erlang
    • Fixes GitHub issue #1

v1.6.0

Choose a tag to compare

@benoitc benoitc released this 22 Feb 20:57

Added

  • Python Logging Integration - Forward Python's logging module to Erlang's logger

    • py:configure_logging/0,1 - Setup Python logging to forward to Erlang
    • erlang.ErlangHandler - Python logging handler that sends to Erlang
    • erlang.setup_logging(level, format) - Configure logging from Python
    • Fire-and-forget architecture using enif_send() for non-blocking messaging
    • Level filtering at NIF level for performance
    • Thread-safe - works from any Python thread
  • Distributed Tracing - Collect trace spans from Python code

    • py:enable_tracing/0, py:disable_tracing/0 - Enable/disable span collection
    • py:get_traces/0 - Retrieve collected spans
    • erlang.Span(name, **attrs) - Context manager for creating spans
    • erlang.trace(name) - Decorator for tracing functions
    • Automatic parent/child span linking via thread-local storage
  • New Erlang modules: py_logger, py_tracer

Performance

  • Type conversion optimizations - Faster Python ↔ Erlang marshalling

    • Use enif_is_identical for atom comparison instead of strcmp
    • Stack allocate small tuples/maps (≤16 elements) to avoid heap allocation
    • Use enif_make_map_from_arrays for O(n) map building vs O(n²) puts
  • Fire-and-forget NIF architecture - Log and trace calls never block Python execution

Fixed

  • Python 3.12+ event loop thread isolation - Fixed asyncio timeouts on Python 3.12+

    • ErlangEventLoop now only used for main thread; worker threads get SelectorEventLoop
    • Per-call ErlNifEnv for thread-safe timer scheduling in free-threaded mode
    • Fail-fast error handling in erlang_loop.py instead of silent hangs
    • Added gil_acquire()/gil_release() helpers to avoid GIL double-acquisition
  • Intermittent test failures on free-threaded Python - Added startup synchronization

v1.5.0

Choose a tag to compare

@benoitc benoitc released this 18 Feb 09:45

Added

  • py_asgi module - Optimized ASGI request handling with:

    • Pre-interned Python string keys (15+ ASGI scope keys)
    • Cached constant values (http type, HTTP versions, methods, schemes)
    • Thread-local response pooling (16 slots per thread, 4KB initial buffer)
    • Direct NIF path bypassing generic py:call()
    • ~60-80% throughput improvement over py:call()
    • Configurable runner module via runner option
    • Sub-interpreter and free-threading (Python 3.13+) support
  • py_wsgi module - Optimized WSGI request handling with:

    • Pre-interned WSGI environ keys
    • Direct NIF path for marshalling
    • ~60-80% throughput improvement over py:call()
    • Sub-interpreter and free-threading support
  • Web frameworks documentation - New documentation at docs/web-frameworks.md

v1.4.0 - Erlang-native asyncio event loop

Choose a tag to compare

@benoitc benoitc released this 18 Feb 03:16

Added

  • Erlang-native asyncio event loop - Custom asyncio event loop backed by Erlang's scheduler

    • ErlangEventLoop class in priv/erlang_loop.py
    • Sub-millisecond latency via Erlang's enif_select (vs 10ms polling)
    • Zero CPU usage when idle - no busy-waiting or polling overhead
    • Full GIL release during waits for better concurrency
    • Native Erlang scheduler integration for I/O events
    • Event loop policy via get_event_loop_policy()
  • TCP support for asyncio event loop

    • create_connection() - TCP client connections
    • create_server() - TCP server with accept loop
    • _ErlangSocketTransport - Non-blocking socket transport with write buffering
    • _ErlangServer - TCP server with serve_forever() support
  • UDP/datagram support for asyncio event loop

    • create_datagram_endpoint() - Create UDP endpoints with full parameter support
    • _ErlangDatagramTransport - Datagram transport implementation
    • Parameters: local_addr, remote_addr, reuse_address, reuse_port, allow_broadcast
    • DatagramProtocol callbacks: datagram_received(), error_received()
    • Support for both connected and unconnected UDP
  • Asyncio event loop documentation

    • New documentation: docs/asyncio.md

Performance

  • Event loop optimizations
    • Fixed run_until_complete callback removal bug
    • Cached ast.literal_eval lookup at module initialization
    • O(1) timer cancellation via handle-to-callback_id reverse map
    • Detach pending queue under mutex, build Erlang terms outside lock
    • O(1) duplicate event detection using hash set
    • Added PERF_BUILD cmake option for aggressive optimizations (-O3, LTO, -march=native)

Full Changelog: https://github.com/benoitc/erlang-python/blob/main/CHANGELOG.md

1.3.2

Choose a tag to compare

@benoitc benoitc released this 17 Feb 00:39

Fixed

  • torch/PyTorch introspection compatibility - Fixed AttributeError: 'erlang.Function' object has no attribute 'endswith' when importing torch or sentence_transformers in contexts where erlang_python callbacks are registered.
    • Root cause: torch does dynamic introspection during import, iterating through Python's namespace and calling .endswith() on objects. The erlang module's __getattr__ was returning ErlangFunction wrappers for any attribute access.
    • Solution: Added C-side callback name registry. Now __getattr__ only returns ErlangFunction wrappers for actually registered callbacks. Unregistered attributes raise AttributeError (normal Python behavior).
    • New test: test_callback_name_registry in py_reentrant_SUITE.erl

1.3.1

Choose a tag to compare

@benoitc benoitc released this 16 Feb 21:35

Fixed

  • Hex.pm packaging - Added files section to app.src to include build scripts (do_cmake.sh, do_build.sh) and other necessary files in the hex.pm package

v1.3.0

Choose a tag to compare

@benoitc benoitc released this 16 Feb 17:50

Added

  • Asyncio Support - New erlang.async_call() for asyncio-compatible callbacks
    • await erlang.async_call('func', arg1, arg2) - Call Erlang from async Python code
    • Integrates with asyncio event loop via add_reader()
    • No exceptions raised for control flow (unlike erlang.call())
    • Releases dirty NIF thread while waiting (non-blocking)
    • Works with FastAPI, Starlette, aiohttp, and other ASGI frameworks
    • Supports concurrent calls via asyncio.gather()

Fixed

  • Flag-based callback detection in replay path - Fixed SuspensionRequired exceptions leaking when ASGI middleware catches and re-raises exceptions

Changed

  • C code optimizations and refactoring
    • Thread safety fixes: Used pthread_once for async callback initialization
    • Timeout handling: Added read_with_timeout() and read_length_prefixed_data() helpers
    • Code deduplication: Merged suspended state creation functions, extracted helpers
    • Performance: Optimized list conversion using enif_make_list_cell()

v1.2.0

Choose a tag to compare

@benoitc benoitc released this 15 Feb 17:27

Added

  • Context Affinity - Bind Erlang processes to dedicated Python workers for state persistence

    • py:bind() / py:unbind() - Bind current process to a worker, preserving Python state
    • py:bind(new) - Create explicit context handles for multiple contexts per process
    • py:with_context(Fun) - Scoped helper with automatic bind/unbind
    • Context-aware functions: py:ctx_call/4-6, py:ctx_eval/2-4, py:ctx_exec/2
    • Automatic cleanup via process monitors when bound processes die
    • O(1) ETS-based binding lookup for minimal overhead
    • New test suite: test/py_context_SUITE.erl
  • Python Thread Support - Any spawned Python thread can now call erlang.call() without blocking

    • Supports threading.Thread, concurrent.futures.ThreadPoolExecutor, and any other Python threads
    • Each spawned thread lazily acquires a dedicated "thread worker" channel
    • One lightweight Erlang process per Python thread handles callbacks
    • Automatic cleanup when Python thread exits via pthread_key_t destructor
    • New module: py_thread_handler.erl - Coordinator and per-thread handlers
    • New C file: py_thread_worker.c - Thread worker pool management
    • New test suite: test/py_thread_callback_SUITE.erl
    • New documentation: docs/threading.md - Threading support guide
  • Reentrant Callbacks - Python→Erlang→Python callback chains without deadlocks

    • Exception-based suspension mechanism interrupts Python execution cleanly
    • Callbacks execute in separate processes to prevent worker pool exhaustion
    • Supports arbitrarily deep nesting (tested up to 10+ levels)
    • Transparent to users - erlang.call() works the same, just without deadlocks
    • New test suite: test/py_reentrant_SUITE.erl
    • New examples: examples/reentrant_demo.erl and examples/reentrant_demo.py

Changed

  • Callback handlers now spawn separate processes for execution, allowing workers
    to remain available for nested py:eval/py:call operations
  • Modular C code structure - Split monolithic py_nif.c (4,335 lines) into
    logical modules for better maintainability:
    • py_nif.h - Shared header with types, macros, and declarations
    • py_convert.c - Bidirectional type conversion (Python ↔ Erlang)
    • py_exec.c - Python execution engine and GIL management
    • py_callback.c - Erlang callback support and asyncio integration
    • Uses #include approach for single compilation unit (no build changes needed)

Fixed

  • Multiple sequential erlang.call() - Fixed infinite loop when Python code makes
    multiple sequential erlang.call() invocations in the same function. The replay
    mechanism now falls back to blocking pipe behavior for subsequent calls after the
    first suspension, preventing the infinite replay loop.
  • Memory safety in C NIF - Fixed memory leaks and added NULL checks
    • nif_async_worker_new: msg_env now freed on pipe/thread creation failure
    • multi_executor_stop: shutdown requests now properly freed after join
    • create_suspended_state: binary allocations cleaned up on failure paths
    • Added NULL checks on all enif_alloc_resource and enif_alloc_env calls
  • Dialyzer warnings - Added {suspended, ...} return type to NIF specs for
    worker_call, worker_eval, and resume_callback functions
  • Dead code removal - Cleaned up unused code discovered during code review:
    • Removed execute_direct() function in py_exec.c (duplicated inline logic)
    • Removed unused ref field from async_pending_t struct in py_nif.h
    • Removed worker_recv/2 from py_nif.erl (declared but never implemented in C)

Documentation

  • Doxygen-style C documentation - Added documentation to all C source files:
    • Architecture overview with execution mode diagrams
    • Type mapping tables for conversions
    • GIL management patterns and best practices
    • Suspension/resume flow diagrams for callbacks
    • Function-level @param, @return, @pre, @warning, @see annotations