Fix 1849 (#1855)

* add new dev branch with audit file

* Switch WAMP ubjson serializer from py-ubjson to bjdata (#1849)

Fixes #1849: py-ubjson is unmaintained and ships no wheels, so
`pip install --only-binary :all:` for autobahn failed. The WAMP "ubjson"
serializer is now backed by bjdata (Binary JData), a maintained,
wheel-shipping successor.

- serializer.py / message.py: import bjdata as the "ubjson" backend
  (serializer id unchanged for transport negotiation).
- pyproject.toml: drop the unconditional py-ubjson dependency; bjdata is an
  OPTIONAL dependency in the `serialization` extra (it pulls in numpy, which
  we keep out of a minimal install). A minimal `pip install autobahn` now
  installs cleanly from wheels only.
- PyPy: set PYBJDATA_NO_EXTENSION=1 in the test recipes and CI (serdes) to use
  bjdata's JIT-friendly pure-Python path (also avoids its numpy-ABI-fragile
  C extension).

WIRE-LEVEL CHANGE: bjdata's octet encoding is NOT identical to the prior
py-ubjson/UBJSON bytes (unsigned-int markers, little-endian). The wamp-proto
UBJSON test vectors will be regenerated in a follow-up wamp-proto PR after the
next autobahn-python release; until then the serdes byte-vector conformance
suite excludes "ubjson" (round-trip + cross-serializer coverage is retained
via test_wamp_serializer.py). See the changelog.

Note: This work was completed with AI assistance (Claude Code).

* Docs: correct ubjson/bjdata packaging notes (sdist-only); drop py-ubjson refs (#1849)

Follow-up wording/doc fixes for the py-ubjson -> bjdata switch:

- README.md / docs/installation.rst: bjdata (like py-ubjson) is published
  sdist-only with no PyPI wheels and builds an optional C extension; document
  the optional `autobahn[serialization]` extra, PYBJDATA_NO_EXTENSION=1, and
  steer wheels-only/cross-arch users to cbor/msgpack. UBJSON is no longer
  described as "included by default".
- docs/changelog.rst: reframe the #1849 fix as "the binary-JSON dependency is
  now optional, so a base autobahn install is wheel-clean" (bjdata itself
  ships no wheels).
- serializer.py: drop the inaccurate "wheel-shipping" wording and state the
  sdist/C-extension reality.

Note: This work was completed with AI assistance (Claude Code).

* Make bjdata (ubjson serializer) CPython-only; PyPy can't build it (#1849)

bjdata's sdist build pulls numpy as an unconditional build dependency
(`oldest-supported-numpy` in build-system.requires, plus a top-level
`from numpy import get_include` in setup.py) even though it skips its C
extension on PyPy. On PyPy that numpy pin (1.23.2) has no wheel and fails to
compile, so `pip install autobahn[serialization]` cannot be installed on PyPy
at all -- this broke the wstest CI jobs. Reported upstream as NeuroJSON/pybj#6.

- pyproject.toml: restrict bjdata to CPython
  (`bjdata>=0.6.0; platform_python_implementation == 'CPython'`). On PyPy the
  UBJSON serializer is unavailable (graceful try/except ImportError); use
  cbor/msgpack instead.
- Remove the now-moot PyPy PYBJDATA_NO_EXTENSION lines from justfile and CI:
  the flag can't help, because the numpy build dependency is installed under
  PEP 517 build isolation before bjdata's setup.py is ever run.
- Update README, docs/installation.rst, docs/changelog.rst and the
  serializer.py comment to document bjdata as CPython-only, linking
  NeuroJSON/pybj#6.

Note: This work was completed with AI assistance (Claude Code).

* test: bump .proto submodule to wamp-proto master (#557 merged)

wamp-proto#557 (the canonical WAMP conformance test-vector suite) was
squash-merged to wamp-proto master as d8389b8. Bump the .proto submodule to
pick it up. This brings in the BJData (bjdata) UBJSON byte vectors, so the
serdes conformance suite can be re-enabled for the "ubjson" serializer
(follow-up commit).

The serdes suite still passes (446) with "ubjson" currently excluded.

Refs: #1849.

Note: This work was completed with AI assistance (Claude Code).

* test: re-enable ubjson in serdes conformance suite against bjdata vectors

With the wamp-proto vectors now carrying both the legacy py-ubjson and the new
bjdata/BJData byte representations for "ubjson" (.proto bumped in the previous
commit), re-enable the serializer in the byte-vector conformance suite:

- conftest: _VECTOR_EXCLUDED_SERIALIZERS is now empty (no serializer excluded).
- utils.require_decodable(): new helper implementing the "at least one must
  match" rule for the deserialize direction. A serializer's variant list may
  hold encodings produced by other implementations/versions that this backend
  cannot decode, or decodes to a valid-but-wrong message (py-ubjson bytes read
  as little-endian bjdata). It keeps variants whose decode re-serializes to a
  canonical variant (cosmetic variants + this backend's own encoding), falling
  back to decode-without-error for serializers whose re-serialization is not
  byte-canonical (flatbuffers). At least one variant must qualify.
- The 25 per-message test files use require_decodable() at their decode sites
  (deserialize-from-bytes, roundtrip, cross-serializer source) instead of
  iterating/indexing the raw variant list.

Serdes suite: 529 passed (up from 446 with ubjson excluded). On PyPy, bjdata
is unavailable so the ubjson serializer is simply not registered and these
tests do not run.

Refs: #1849.

Note: This work was completed with AI assistance (Claude Code).

Author: oberstet

.audit/oberstet_fix_1849.md

@@ -0,0 +1,8 @@
+- [ ] I did **not** use any AI-assistance tools to help create this pull request.
+- [x] I **did** use AI-assistance tools to *help* create this pull request.
+- [x] I have read, understood and followed the projects' [AI Policy](https://github.com/crossbario/autobahn-python/blob/main/AI_POLICY.md) when creating code, documentation etc. for this pull request.
+
+Submitted by: @oberstet
+Date: 2025-06-13
+Related issue(s): #1849
+Branch: oberstet:fix_1849

.proto

@@ -343,14 +343,14 @@ masking) and UTF-8 validation.
 
 ### WAMP Serializers
 
-**As of v25.11.1, all WAMP serializers are included by default** - batteries included!
+**The JSON, MessagePack, CBOR and FlatBuffers serializers are included by default** - batteries included! UBJSON is available via the optional `autobahn[serialization]` extra.
 
-Autobahn|Python now ships with full support for all WAMP serializers out-of-the-box:
+Autobahn|Python ships with the following WAMP serializers:
 
 - **JSON** (standard library) - always available
 - **MessagePack** - high-performance binary serialization
 - **CBOR** - IETF standard binary serialization (RFC 8949)
-- **UBJSON** - Universal Binary JSON
+- **UBJSON** - Universal Binary JSON *(optional: `pip install autobahn[serialization]`)*
 - **Flatbuffers** - Google's zero-copy serialization (vendored)
 
 #### Architecture & Performance
@@ -363,12 +363,12 @@ The serializer dependencies are optimized for both **CPython** and **PyPy**:
 | **msgpack** | Binary wheel (C extension) | u-msgpack-python (pure Python) | Native + Universal | PyPy JIT makes pure Python faster than C |
 | **ujson** | Binary wheel | Binary wheel | Native | Available for both implementations |
 | **cbor2** | Binary wheel | Pure Python fallback | Native + Universal | Binary wheels + py3-none-any |
-| **ubjson** | Pure Python | Pure Python | Source | Set `PYUBJSON_NO_EXTENSION=1` to skip C build |
+| **ubjson** *(optional)* | C ext (from sdist) | ❌ n/a (CPython only) | Source only — no wheels | Optional `autobahn[serialization]` extra (bjdata, pulls numpy). CPython-only: bjdata can't install on PyPy ([NeuroJSON/pybj#6](https://github.com/NeuroJSON/pybj/issues/6)); on PyPy use cbor/msgpack. On CPython without a compiler set `PYBJDATA_NO_EXTENSION=1` |
 | **flatbuffers** | Vendored | Vendored | Included | Always available, no external dependency |
 
 **Key Design Principles:**
 
-1. **Batteries Included**: All serializers available without extra install steps
+1. **Batteries Included**: Core serializers (JSON, MessagePack, CBOR, FlatBuffers) available without extra install steps; UBJSON via the optional `autobahn[serialization]` extra
 2. **PyPy Optimization**: Pure Python implementations leverage PyPy's JIT for superior performance
 3. **Binary Wheels**: Native wheels for all major platforms (Linux x86_64/ARM64, macOS x86_64/ARM64, Windows x86_64)
 4. **Zero System Pollution**: All dependencies install cleanly via wheels or pure Python
@@ -429,15 +429,15 @@ All dependencies follow these design principles:
 
 ### WAMP Serializers (Batteries Included)
 
-All serializers are now **included by default** in the base installation:
+All serializers **except UBJSON** are included by default in the base installation; UBJSON is an optional extra (`pip install autobahn[serialization]`):
 
 | Serializer | Purpose | CPython | PyPy | Wheel Coverage | Notes |
 |------------|---------|---------|------|----------------|-------|
 | **json** | JSON serialization | stdlib | stdlib | ✅ Always available | Python standard library |
 | **msgpack** | MessagePack serialization | msgpack (binary wheel) | u-msgpack-python (pure Python) | ✅ Excellent | 50+ wheels for CPython; PyPy JIT optimized |
 | **ujson** | Fast JSON (optional) | Binary wheel | Binary wheel | ✅ Excellent | 30+ wheels; both implementations |
 | **cbor2** | CBOR serialization (RFC 8949) | Binary wheel | Pure Python fallback | ✅ Excellent | 30+ binary wheels + universal fallback |
-| **py-ubjson** | UBJSON serialization | Pure Python | Pure Python | ✅ Good | Optional C extension (can skip with `PYUBJSON_NO_EXTENSION=1`) |
+| **bjdata** | UBJSON serialization *(optional)* | C ext (from sdist) | ❌ n/a (CPython only) | ⚠️ sdist only — no wheels | `autobahn[serialization]` extra; pulls numpy. CPython-only: can't install on PyPy ([NeuroJSON/pybj#6](https://github.com/NeuroJSON/pybj/issues/6)). On CPython without a compiler set `PYBJDATA_NO_EXTENSION=1`. For wheels-only/cross-arch/PyPy installs prefer cbor/msgpack |
 | **flatbuffers** | Google Flatbuffers | **Vendored** | **Vendored** | ✅ Perfect | Included in our wheel, zero external dependency |
 
 ### Optional: Twisted Framework

README.md

@@ -8,6 +8,13 @@ Changelog
 26.6.1
 ------
 
+**WAMP Serialization**
+
+* ``py-ubjson`` (unmaintained, sdist-only) is no longer an unconditional dependency. A base ``pip install autobahn`` — and the wheels-only / cross-arch case from #1849 (``pip download --only-binary :all: --platform ...``) — now resolves entirely from binary wheels (#1849)
+* The WAMP ``ubjson`` serializer is now backed by the maintained ``bjdata`` (Binary JData) package, provided as the OPTIONAL ``autobahn[serialization]`` extra (it also pulls in numpy), keeping both out of a minimal install (#1849)
+* ``bjdata`` is published sdist-only (no PyPI wheels) and is currently **CPython-only**: on PyPy its sdist build pulls an unbuildable numpy (upstream ``NeuroJSON/pybj#6``), so the ``ubjson`` serializer is unavailable on PyPy - use ``cbor``/``msgpack`` there. On CPython without a compiler, set ``PYBJDATA_NO_EXTENSION=1`` for a pure-Python build. For wheels-only or cross-arch deployments, also prefer ``cbor``/``msgpack`` (#1849)
+* ⚠️ **Wire-level change to watch out for:** bjdata's octet-level encoding is NOT identical to the previous py-ubjson/UBJSON bytes (different integer markers, little-endian). The WAMP serializer id remains ``ubjson`` for transport negotiation. The ``wamp-proto`` UBJSON test vectors will be regenerated in a follow-up PR after this release; until then the ``ubjson`` serializer is excluded from the byte-vector conformance suite (round-trip and cross-serializer coverage retained) (#1849)
+
 **FlatBuffers**
 
 * Bump vendored FlatBuffers from v25.9.23 to v25.12.19, restoring the version-sync with zlmdb 26.6.1 (#1853)

docs/changelog.rst

@@ -152,7 +152,7 @@ Install Variants
    * - ``scram``
      - Install WAMP-SCRAM authentication packages.
    * - ``serialization``
-     - Backwards-compatible no-op; WAMP serializers are included by default.
+     - Install ``bjdata`` to enable the optional UBJSON WAMP serializer. CPython-only - it cannot install on PyPy (upstream ``NeuroJSON/pybj#6``); on CPython without a compiler set ``PYBJDATA_NO_EXTENSION=1`` for a pure-Python build. JSON, MessagePack, CBOR and FlatBuffers are always available without this extra (use them on PyPy).
    * - ``nvx``
      - Backwards-compatible no-op; NVX acceleration is included in binary wheels where supported.
    * - ``all``

docs/installation.rst

@@ -12,6 +12,20 @@
 from .utils import load_test_vector, get_serializer_ids
 
 
+# The WAMP "ubjson" serializer is backed by bjdata (autobahn #1849). The
+# wamp-proto canonical vectors (via the .proto submodule) now carry BOTH the
+# legacy py-ubjson bytes and the new bjdata/BJData bytes for "ubjson" (each
+# tagged with a "note"), matched with "at least one must match" semantics. The
+# deserialize-direction tests use utils.require_decodable() to skip byte
+# variants this backend cannot decode (the two encodings are not mutually
+# decodable), so no serializer needs to be excluded from the byte-vector suite.
+_VECTOR_EXCLUDED_SERIALIZERS = ()
+
+
+def _conformance_serializer_ids():
+    return [s for s in get_serializer_ids() if s not in _VECTOR_EXCLUDED_SERIALIZERS]
+
+
 @pytest.fixture(scope="session")
 def wamp_test_vector_publish():
     """Load PUBLISH test vector"""
@@ -48,12 +62,12 @@ def pytest_generate_tests(metafunc):
     This generates test parameters for serializer_id based on available serializers.
     """
     if "serializer_id" in metafunc.fixturenames:
-        serializer_ids = get_serializer_ids()
+        serializer_ids = _conformance_serializer_ids()
         metafunc.parametrize("serializer_id", serializer_ids)
 
     if "serializer_pair" in metafunc.fixturenames:
         # Generate all unique pairs of serializers for cross-serializer tests
-        serializer_ids = get_serializer_ids()
+        serializer_ids = _conformance_serializer_ids()
         pairs = []
         for i, ser1 in enumerate(serializer_ids):
             for ser2 in serializer_ids[i + 1 :]:

examples/serdes/tests/conftest.py

@@ -14,6 +14,7 @@
 from autobahn.wamp.serializer import create_transport_serializer
 
 from .utils import (
+    require_decodable,
     load_test_vector,
     bytes_from_hex,
     matches_any_byte_representation,
@@ -62,7 +63,7 @@ def test_abort_deserialize_from_bytes(serializer_id, abort_samples, create_seria
         byte_variants = sample["serializers"][serializer_id]
 
         # Try deserializing each byte variant
-        for variant in byte_variants:
+        for variant in require_decodable(serializer, byte_variants):
             # Get bytes
             if "bytes_hex" in variant:
                 test_bytes = bytes_from_hex(variant["bytes_hex"])
@@ -129,7 +130,7 @@ def test_abort_roundtrip(serializer_id, abort_samples, create_serializer):
         if not byte_variants:
             continue
 
-        variant = byte_variants[0]
+        variant = require_decodable(serializer, byte_variants)[0]
         if "bytes_hex" in variant:
             original_bytes = bytes_from_hex(variant["bytes_hex"])
         elif "bytes" in variant:

examples/serdes/tests/test_abort.py

@@ -14,6 +14,7 @@
 from autobahn.wamp.serializer import create_transport_serializer
 
 from .utils import (
+    require_decodable,
     load_test_vector,
     bytes_from_hex,
     matches_any_byte_representation,
@@ -64,7 +65,7 @@ def test_authenticate_deserialize_from_bytes(
         byte_variants = sample["serializers"][serializer_id]
 
         # Try deserializing each byte variant
-        for variant in byte_variants:
+        for variant in require_decodable(serializer, byte_variants):
             # Get bytes
             if "bytes_hex" in variant:
                 test_bytes = bytes_from_hex(variant["bytes_hex"])
@@ -133,7 +134,7 @@ def test_authenticate_roundtrip(serializer_id, authenticate_samples, create_seri
         if not byte_variants:
             continue
 
-        variant = byte_variants[0]
+        variant = require_decodable(serializer, byte_variants)[0]
         if "bytes_hex" in variant:
             original_bytes = bytes_from_hex(variant["bytes_hex"])
         elif "bytes" in variant:

examples/serdes/tests/test_authenticate.py

@@ -14,6 +14,7 @@
 from autobahn.wamp.serializer import create_transport_serializer
 
 from .utils import (
+    require_decodable,
     load_test_vector,
     bytes_from_hex,
     matches_any_byte_representation,
@@ -62,7 +63,7 @@ def test_call_deserialize_from_bytes(serializer_id, call_samples, create_seriali
         byte_variants = sample["serializers"][serializer_id]
 
         # Try deserializing each byte variant
-        for variant in byte_variants:
+        for variant in require_decodable(serializer, byte_variants):
             # Get bytes
             if "bytes_hex" in variant:
                 test_bytes = bytes_from_hex(variant["bytes_hex"])
@@ -149,7 +150,7 @@ def test_call_roundtrip(serializer_id, call_samples, create_serializer):
         if not byte_variants:
             continue
 
-        variant = byte_variants[0]
+        variant = require_decodable(serializer, byte_variants)[0]
         if "bytes_hex" in variant:
             original_bytes = bytes_from_hex(variant["bytes_hex"])
         elif "bytes" in variant:

examples/serdes/tests/test_call.py

@@ -14,6 +14,7 @@
 from autobahn.wamp.serializer import create_transport_serializer
 
 from .utils import (
+    require_decodable,
     load_test_vector,
     bytes_from_hex,
     matches_any_byte_representation,
@@ -64,7 +65,7 @@ def test_cancel_deserialize_from_bytes(
         byte_variants = sample["serializers"][serializer_id]
 
         # Try deserializing each byte variant
-        for variant in byte_variants:
+        for variant in require_decodable(serializer, byte_variants):
             # Get bytes
             if "bytes_hex" in variant:
                 test_bytes = bytes_from_hex(variant["bytes_hex"])
@@ -131,7 +132,7 @@ def test_cancel_roundtrip(serializer_id, cancel_samples, create_serializer):
         if not byte_variants:
             continue
 
-        variant = byte_variants[0]
+        variant = require_decodable(serializer, byte_variants)[0]
         if "bytes_hex" in variant:
             original_bytes = bytes_from_hex(variant["bytes_hex"])
         elif "bytes" in variant: