Switch WAMP ubjson serializer from py-ubjson to bjdata (#1849)

Fixes #1849: py-ubjson is unmaintained and ships no wheels, so
`pip install --only-binary :all:` for autobahn failed. The WAMP "ubjson"
serializer is now backed by bjdata (Binary JData), a maintained,
wheel-shipping successor.

- serializer.py / message.py: import bjdata as the "ubjson" backend
  (serializer id unchanged for transport negotiation).
- pyproject.toml: drop the unconditional py-ubjson dependency; bjdata is an
  OPTIONAL dependency in the `serialization` extra (it pulls in numpy, which
  we keep out of a minimal install). A minimal `pip install autobahn` now
  installs cleanly from wheels only.
- PyPy: set PYBJDATA_NO_EXTENSION=1 in the test recipes and CI (serdes) to use
  bjdata's JIT-friendly pure-Python path (also avoids its numpy-ABI-fragile
  C extension).

WIRE-LEVEL CHANGE: bjdata's octet encoding is NOT identical to the prior
py-ubjson/UBJSON bytes (unsigned-int markers, little-endian). The wamp-proto
UBJSON test vectors will be regenerated in a follow-up wamp-proto PR after the
next autobahn-python release; until then the serdes byte-vector conformance
suite excludes "ubjson" (round-trip + cross-serializer coverage is retained
via test_wamp_serializer.py). See the changelog.

Note: This work was completed with AI assistance (Claude Code).

Author: oberstet

.github/workflows/main.yml

@@ -190,6 +190,10 @@ jobs:
           # Create output directory for test results
           mkdir -p test-results/serdes-${{ matrix.python-env }}
 
+          # On PyPy, use bjdata's pure-Python path (JIT-friendly; avoids the
+          # numpy-ABI-fragile C extension). See autobahn #1849.
+          if [[ "${{ matrix.python-env }}" == pypy* ]]; then export PYBJDATA_NO_EXTENSION=1; fi
+
           # Run tests and generate reports
           VENV_PYTHON=$(just --quiet _get-venv-python ${{ matrix.python-env }})
           ${VENV_PYTHON} -m pytest -v \

docs/changelog.rst

@@ -8,6 +8,12 @@ Changelog
 26.6.1
 ------
 
+**WAMP Serialization**
+
+* The WAMP ``ubjson`` serializer is now backed by ``bjdata`` (Binary JData) instead of the unmaintained, wheel-less ``py-ubjson``. ``py-ubjson`` is removed as a dependency, which fixes installation via wheels only (``pip install --only-binary :all:``) (#1849)
+* ⚠️ **Wire-level change to watch out for:** bjdata's octet-level encoding is NOT identical to the previous py-ubjson/UBJSON bytes (different integer markers, little-endian). The WAMP serializer id remains ``ubjson`` for transport negotiation. The ``wamp-proto`` UBJSON test vectors will be regenerated in a follow-up PR after this release; until then the ``ubjson`` serializer is excluded from the byte-vector conformance suite (round-trip and cross-serializer coverage retained) (#1849)
+* ``bjdata`` is an OPTIONAL dependency (it pulls in numpy): the ``ubjson`` serializer now requires the ``autobahn[serialization]`` extra, keeping numpy out of a minimal install. On PyPy, set ``PYBJDATA_NO_EXTENSION=1`` to use the JIT-friendly pure-Python path (#1849)
+
 **FlatBuffers**
 
 * Bump vendored FlatBuffers from v25.9.23 to v25.12.19, restoring the version-sync with zlmdb 26.6.1 (#1853)

examples/serdes/tests/conftest.py

@@ -12,6 +12,21 @@
 from .utils import load_test_vector, get_serializer_ids
 
 
+# The WAMP "ubjson" serializer is now backed by bjdata (autobahn #1849). Every
+# test in this conformance suite is built on the wamp-proto canonical byte vectors
+# (serialize-to / deserialize-from / cross-serializer all reference the stored
+# ``bytes_hex``). bjdata's octet-level encoding intentionally differs from the
+# legacy UBJSON bytes in those vectors, so "ubjson" is excluded from the byte-vector
+# conformance suite until the wamp-proto UBJSON vectors are regenerated (a follow-up
+# wamp-proto PR after the next autobahn-python release). bjdata round-trip
+# correctness is covered by src/autobahn/wamp/test/test_wamp_serializer.py.
+_VECTOR_EXCLUDED_SERIALIZERS = ("ubjson",)
+
+
+def _conformance_serializer_ids():
+    return [s for s in get_serializer_ids() if s not in _VECTOR_EXCLUDED_SERIALIZERS]
+
+
 @pytest.fixture(scope="session")
 def wamp_test_vector_publish():
     """Load PUBLISH test vector"""
@@ -48,12 +63,12 @@ def pytest_generate_tests(metafunc):
     This generates test parameters for serializer_id based on available serializers.
     """
     if "serializer_id" in metafunc.fixturenames:
-        serializer_ids = get_serializer_ids()
+        serializer_ids = _conformance_serializer_ids()
         metafunc.parametrize("serializer_id", serializer_ids)
 
     if "serializer_pair" in metafunc.fixturenames:
         # Generate all unique pairs of serializers for cross-serializer tests
-        serializer_ids = get_serializer_ids()
+        serializer_ids = _conformance_serializer_ids()
         pairs = []
         for i, ser1 in enumerate(serializer_ids):
             for ser2 in serializer_ids[i + 1 :]:

justfile

@@ -746,6 +746,10 @@ check-coverage-twisted venv="" use_nvx="": (install-tools venv) (install-dev ven
     VENV_PATH="{{ VENV_DIR }}/${VENV_NAME}"
     VENV_PYTHON=$(just --quiet _get-venv-python "${VENV_NAME}")
 
+    # On PyPy, use bjdata's pure-Python path (JIT-friendly; avoids the
+    # numpy-ABI-fragile C extension). See autobahn #1849.
+    if [[ "${VENV_NAME}" == pypy* ]]; then export PYBJDATA_NO_EXTENSION=1; fi
+
     # Handle NVX configuration
     USE_NVX="{{ use_nvx }}"
     if [ "${USE_NVX}" = "1" ]; then
@@ -1093,6 +1097,10 @@ test-twisted venv="" use_nvx="": (install-tools venv) (install-dev venv)
     VENV_PATH="{{ VENV_DIR }}/${VENV_NAME}"
     VENV_PYTHON=$(just --quiet _get-venv-python "${VENV_NAME}")
 
+    # On PyPy, use bjdata's pure-Python path (JIT-friendly; avoids the
+    # numpy-ABI-fragile C extension). See autobahn #1849.
+    if [[ "${VENV_NAME}" == pypy* ]]; then export PYBJDATA_NO_EXTENSION=1; fi
+
     # Handle NVX configuration
     USE_NVX="{{ use_nvx }}"
     if [ "${USE_NVX}" = "1" ]; then
@@ -1134,6 +1142,10 @@ test-asyncio venv="" use_nvx="": (install-tools venv) (install-dev venv)
     VENV_PATH="{{ VENV_DIR }}/${VENV_NAME}"
     VENV_PYTHON=$(just --quiet _get-venv-python "${VENV_NAME}")
 
+    # On PyPy, use bjdata's pure-Python path (JIT-friendly; avoids the
+    # numpy-ABI-fragile C extension). See autobahn #1849.
+    if [[ "${VENV_NAME}" == pypy* ]]; then export PYBJDATA_NO_EXTENSION=1; fi
+
     # Handle NVX configuration
     USE_NVX="{{ use_nvx }}"
     if [ "${USE_NVX}" = "1" ]; then
@@ -1162,6 +1174,10 @@ test-serdes venv="": (install-tools venv) (install-dev venv)
     fi
     VENV_PYTHON=$(just --quiet _get-venv-python "${VENV_NAME}")
 
+    # On PyPy, use bjdata's pure-Python path (JIT-friendly; avoids the
+    # numpy-ABI-fragile C extension). See autobahn #1849.
+    if [[ "${VENV_NAME}" == pypy* ]]; then export PYBJDATA_NO_EXTENSION=1; fi
+
     echo "==> Running WAMP message serdes conformance tests in ${VENV_NAME}..."
     echo "==> Test vectors loaded from: wamp-proto/testsuite/"
     ${VENV_PYTHON} -m pytest -v \

pyproject.toml

@@ -52,7 +52,10 @@ dependencies = [
     "u-msgpack-python>=2.1; platform_python_implementation != 'CPython'",
     "ujson>=4.0.2",  # Binary wheels for both CPython and PyPy
     "cbor2>=5.2.0",  # Binary wheels + pure Python fallback
-    "py-ubjson>=0.16.1",  # Pure Python implementation (set PYUBJSON_NO_EXTENSION=1 to skip C extension build)
+    # NOTE: the WAMP "ubjson" serializer (bjdata) is an OPTIONAL extra, NOT a base
+    # dependency: bjdata pulls in numpy, which we don't want in a minimal install.
+    # See the [serialization] extra below. This also fixes #1849 (the old, wheel-less
+    # py-ubjson made `pip install --only-binary :all:` fail).
     # flatbuffers is vendored - no external dependency needed
 ]
 
@@ -78,9 +81,14 @@ compress = [
     # Users who need snappy: pip install python-snappy
 ]
 
-# Backwards compatibility - serialization now included by default in base install
-# All WAMP serializers (JSON, MessagePack, CBOR, UBJSON, Flatbuffers) are always available
-serialization = []
+# Optional binary-JSON ("ubjson") WAMP serializer, backed by bjdata.
+# JSON, MessagePack, CBOR and (vendored) FlatBuffers are always available in the
+# base install; the "ubjson" serializer requires this extra because bjdata pulls
+# in numpy, which we keep out of a minimal autobahn install.
+# On PyPy, set PYBJDATA_NO_EXTENSION=1 to use the JIT-friendly pure-Python path.
+serialization = [
+    "bjdata>=0.6.0",
+]
 
 # TLS transport encryption, WAMP-cryptosign end-to-end encryption and authentication
 encryption = [

src/autobahn/wamp/message.py

@@ -774,9 +774,9 @@ def _deserialize_payload(self, data_bytes, ser_id):
             # msgpack supports memoryview (zero-copy)
             return msgpack.unpackb(data_bytes)
         elif ser_id == "ubjson":
-            import ubjson
+            # The WAMP "ubjson" serializer is backed by bjdata (see serializer.py)
+            import bjdata as ubjson
 
-            # ubjson supports memoryview (zero-copy)
             return ubjson.loadb(data_bytes)
         else:
             # Fallback to CBOR for unknown serializers

src/autobahn/wamp/serializer.py

@@ -909,11 +909,20 @@ def __init__(self, batched=False):
     __all__.append("CBORSerializer")
 
 
-# UBJSON serialization depends on the `py-ubjson` package being available
-# https://pypi.python.org/pypi/py-ubjson
-# https://github.com/Iotic-Labs/py-ubjson
+# The WAMP "ubjson" serializer is backed by the `bjdata` package (Binary JData),
+# a maintained, wheel-shipping successor of the (unmaintained, wheel-less)
+# `py-ubjson` package.
+# https://pypi.org/project/bjdata/  https://github.com/NeuroJSON/pybj
+#
+# NOTE: bjdata's on-the-wire (octet-level) encoding is NOT identical to the older
+# py-ubjson/UBJSON bytes (e.g. unsigned-integer markers, little-endian). The WAMP
+# serializer id remains "ubjson" for transport negotiation. See the changelog.
+#
+# `bjdata` is an OPTIONAL dependency (it pulls in numpy): install it via the
+# `autobahn[serialization]` extra. On PyPy, set PYBJDATA_NO_EXTENSION=1 to use the
+# JIT-friendly pure-Python path instead of the C extension.
 try:
-    import ubjson
+    import bjdata as ubjson
 except ImportError:
     pass
 else:

src/autobahn/wamp/test/test_wamp_serializer.py

@@ -229,9 +229,11 @@ def create_serializers(decimal_support=False):
         _serializers.append(serializer.MsgPackSerializer())
         _serializers.append(serializer.MsgPackSerializer(batched=True))
 
-        # roundtrip error
-        _serializers.append(serializer.UBJSONSerializer())
-        _serializers.append(serializer.UBJSONSerializer(batched=True))
+        # UBJSON (bjdata-backed) is optional: only present when the
+        # `autobahn[serialization]` extra (bjdata) is installed.
+        if hasattr(serializer, "UBJSONSerializer"):
+            _serializers.append(serializer.UBJSONSerializer())
+            _serializers.append(serializer.UBJSONSerializer(batched=True))
 
     # FIXME: implement full FlatBuffers serializer for WAMP
     # WAMP-FlatBuffers currently only supports Python 3