Synchronize CI/CD, Flatbuffers vendoring and wamp-ai and wamp-cicd Submodules between autobahn-python and zlmdb (#1822)

* add feature branch audit file

* Add FlatBuffers/flatc bundling support and smoke tests

Add shared flatc infrastructure from wamp-cicd:
- Bump .cicd submodule to include new scripts/flatc/ files
- Add src/autobahn/_flatc/ module with get_flatc_path() and run_flatc()
- Add scripts/smoke_test.py and scripts/smoke_test_flatc.py
- Add justfile recipes: test-smoke, test-wheel-install, test-sdist-install

This syncs autobahn-python with zlmdb's FlatBuffers bundling approach,
using shared code from wamp-cicd/scripts/flatc/ for DRY.

Note: This work was completed with AI assistance (Claude Code).

* Add complete FlatBuffers vendoring with bundled flatc compiler

- hatch_build.py: Add _build_flatc(), _generate_reflection_bfbs(),
  and _update_flatbuffers_git_version() methods for wheel builds
- pyproject.toml: Add flatc console script entry point
- flatbuffers/__init__.py: Add version() function and _git_version import
- flatbuffers/_git_version.py: Git version tracking file
- flatbuffers/reflection.fbs: Copy reflection schema for runtime access
- scripts/update_flatbuffers.sh: External script for updating vendored flatbuffers
- justfile: Simplify update-flatbuffers recipe, add test-bundled-flatc
- .gitignore: Exclude flatc binary and reflection.bfbs build artifacts

The wheel now includes:
- Bundled flatc compiler built from deps/flatbuffers
- reflection.bfbs generated during build
- Git version tracking for vendored runtime

Tested: wheel install, sdist install, smoke tests all pass

Note: This work was completed with AI assistance (Claude Code).

* Fix manylinux ISA compatibility: use manylinux_2_28 instead of manylinux_2_34

The manylinux_2_34 container uses a toolchain that defaults to x86_64_v2
ISA level (SSE4.2, POPCNT), which causes auditwheel to fail with:

  ValueError: Cannot repair the wheel, because required library "flatc"
  could not be located. x86_64_v2 (ISA level) is not compatible with
  manylinux platform tag's (x86_64) requirement.

The manylinux_2_28 container uses baseline x86_64 ISA which is compatible
with all manylinux platform tags.

This matches zlmdb which already uses manylinux_2_28 successfully.

Note: This work was completed with AI assistance (Claude Code).

* Increase ARM64 build timeout to 30 minutes

CFFI compilation under QEMU ARM64 emulation is very slow.
CPython builds can take 20+ minutes due to emulation overhead.

This matches zlmdb which already uses 30 minute timeouts.

Note: This work was completed with AI assistance (Claude Code).

* Increase ARM64 build timeout to 60 minutes

CFFI and flatc compilation under QEMU ARM64 emulation is very slow.
Builds can take 45+ minutes due to emulation overhead.

Note: This work was completed with AI assistance (Claude Code).

* Update release.yml to use manylinux_2_28 artifact names

The wheels-docker workflow was changed to use manylinux_2_28 instead of
manylinux_2_34 for ISA compatibility. The release workflow must use the
matching artifact names.

Note: This work was completed with AI assistance (Claude Code).

* Add workflows README, WAMP .bfbs generation, and version sync check

1. Add .github/workflows/README.md documenting the CI/CD architecture,
   artifact flow, and platform coverage (adapted from zlmdb)

2. Add _generate_wamp_bfbs() to hatch_build.py to compile wamp.bfbs
   from the WAMP FlatBuffers schemas during wheel build. This binary
   schema file enables runtime schema introspection for WAMP messages.

3. Add check_zlmdb_flatbuffers_version_in_sync() to autobahn/__init__.py
   This is the reciprocal of zlmdb.check_autobahn_flatbuffers_version_in_sync()
   and verifies both libraries use the same vendored FlatBuffers runtime
   for reliable data-in-transit/data-at-rest interoperability.

4. Add wamp.bfbs to .gitignore as it's a build artifact

Note: This work was completed with AI assistance (Claude Code).

* Rename install-flatc to install-flatc-system with warning, add version check

1. Rename `install-flatc` to `install-flatc-system` with prominent warning
   explaining that users normally DON'T need this since autobahn bundles
   flatc in wheels. Adds interactive confirmation prompt.

2. Enhance smoke_test_flatc.py to verify flatc binary version matches
   the vendored FlatBuffers runtime version. This ensures consistency
   between the compiler and runtime library.

Note: This work was completed with AI assistance (Claude Code).

Author: oberstet

.audit/oberstet_sync-autobahn-zlmdb.md

@@ -0,0 +1,8 @@
+- [ ] I did **not** use any AI-assistance tools to help create this pull request.
+- [x] I **did** use AI-assistance tools to *help* create this pull request.
+- [x] I have read, understood and followed the project's AI_POLICY.md when creating code, documentation etc. for this pull request.
+
+Submitted by: @oberstet
+Date: 2025-12-13
+Related issue(s): #1821
+Branch: oberstet:sync-autobahn-zlmdb

.cicd

@@ -137,7 +137,7 @@ jobs:
 
               // Query artifacts from wheels-docker workflow
               const wheelsDockerRunId = latestRuns['wheels-docker']?.id;
-              core.setOutput('artifact_manylinux_x86_64', await findArtifact(wheelsDockerRunId, 'artifacts-manylinux_2_34_x86_64'));
+              core.setOutput('artifact_manylinux_x86_64', await findArtifact(wheelsDockerRunId, 'artifacts-manylinux_2_28_x86_64'));
 
               // Query artifacts from wheels-arm64 workflow
               const wheelsArm64RunId = latestRuns['wheels-arm64']?.id;
@@ -813,20 +813,20 @@ jobs:
           mode: strict
           keep-metadata: true  # Keep CHECKSUMS for user verification
           targets: |
-            cpy311-linux-x86_64-manylinux_2_34
+            cpy311-linux-x86_64-manylinux_2_28
             cpy311-linux-aarch64-manylinux_2_28
             cpy311-win-amd64
-            cpy312-linux-x86_64-manylinux_2_34
+            cpy312-linux-x86_64-manylinux_2_28
             cpy312-win-amd64
             cpy313-macos-arm64
-            cpy313-linux-x86_64-manylinux_2_34
+            cpy313-linux-x86_64-manylinux_2_28
             cpy313-linux-aarch64-manylinux_2_28
             cpy313-win-amd64
             cpy314-macos-arm64
-            cpy314-linux-x86_64-manylinux_2_34
+            cpy314-linux-x86_64-manylinux_2_28
             cpy314-win-amd64
             pypy311-macos-arm64
-            pypy311-linux-x86_64-manylinux_2_34
+            pypy311-linux-x86_64-manylinux_2_28
             pypy311-linux-aarch64-manylinux_2_17
             pypy311-win-amd64
             source
@@ -1642,20 +1642,20 @@ jobs:
           mode: strict
           keep-metadata: true  # Keep CHECKSUMS for user verification
           targets: |
-            cpy311-linux-x86_64-manylinux_2_34
+            cpy311-linux-x86_64-manylinux_2_28
             cpy311-linux-aarch64-manylinux_2_28
             cpy311-win-amd64
-            cpy312-linux-x86_64-manylinux_2_34
+            cpy312-linux-x86_64-manylinux_2_28
             cpy312-win-amd64
             cpy313-macos-arm64
-            cpy313-linux-x86_64-manylinux_2_34
+            cpy313-linux-x86_64-manylinux_2_28
             cpy313-linux-aarch64-manylinux_2_28
             cpy313-win-amd64
             cpy314-macos-arm64
-            cpy314-linux-x86_64-manylinux_2_34
+            cpy314-linux-x86_64-manylinux_2_28
             cpy314-win-amd64
             pypy311-macos-arm64
-            pypy311-linux-x86_64-manylinux_2_34
+            pypy311-linux-x86_64-manylinux_2_28
             pypy311-linux-aarch64-manylinux_2_17
             pypy311-win-amd64
             source
@@ -2262,20 +2262,20 @@ jobs:
           mode: strict
           # keep-metadata: false (default - removes CHECKSUMS for PyPI)
           targets: |
-            cpy311-linux-x86_64-manylinux_2_34
+            cpy311-linux-x86_64-manylinux_2_28
             cpy311-linux-aarch64-manylinux_2_28
             cpy311-win-amd64
-            cpy312-linux-x86_64-manylinux_2_34
+            cpy312-linux-x86_64-manylinux_2_28
             cpy312-win-amd64
             cpy313-macos-arm64
-            cpy313-linux-x86_64-manylinux_2_34
+            cpy313-linux-x86_64-manylinux_2_28
             cpy313-linux-aarch64-manylinux_2_28
             cpy313-win-amd64
             cpy314-macos-arm64
-            cpy314-linux-x86_64-manylinux_2_34
+            cpy314-linux-x86_64-manylinux_2_28
             cpy314-win-amd64
             pypy311-macos-arm64
-            pypy311-linux-x86_64-manylinux_2_34
+            pypy311-linux-x86_64-manylinux_2_28
             pypy311-linux-aarch64-manylinux_2_17
             pypy311-win-amd64
             source

.github/workflows/README.md

@@ -130,7 +130,9 @@ jobs:
         if: matrix.target.build_type == 'custom'
         uses: nick-fields/retry@v3
         with:
-          timeout_minutes: 10
+          # CFFI and flatc compilation under QEMU ARM64 emulation is very slow
+          # Builds can take 45+ minutes due to emulation overhead
+          timeout_minutes: 60
           max_attempts: 3
           retry_on: error
           warning_on_retry: true
@@ -153,7 +155,9 @@ jobs:
       - name: Build ARM64 wheels with NVX extension (with retry for QEMU flakiness)
         uses: nick-fields/retry@v3
         with:
-          timeout_minutes: 15
+          # CFFI and flatc compilation under QEMU ARM64 emulation is very slow
+          # Builds can take 45+ minutes due to emulation overhead
+          timeout_minutes: 60
           max_attempts: 3
           retry_on: error
           warning_on_retry: true

.github/workflows/release.yml

@@ -45,22 +45,14 @@ jobs:
       fail-fast: false
       matrix:
         target:
-          # manylinux_2_34 (glibc 2.34+) - PEP 600 compliant for modern Linux
+          # manylinux_2_28 (glibc 2.28+) - PEP 600 compliant, wide compatibility
+          # IMPORTANT: manylinux_2_28 uses baseline x86_64 ISA level (no SSE4.2/POPCNT)
+          # which is required for auditwheel to repair binaries like flatc.
+          # manylinux_2_34 uses x86_64_v2 which causes auditwheel failures.
+          # Matches zlmdb which also uses manylinux_2_28.
           # see: https://github.com/pypa/manylinux
-          - name: "manylinux_2_34_x86_64"
-            base_image: "quay.io/pypa/manylinux_2_34_x86_64"
-
-          # Future manylinux images can be added here:
-          # - name: "manylinux_2_34_aarch64"
-          #   base_image: "quay.io/pypa/manylinux_2_34_aarch64"
-
-          # Deactivated for now - focusing on standard manylinux wheels:
-          # - name: "debian12-amd64"
-          #   base_image: "debian:12"
-          # - name: "rocky9-amd64"
-          #   base_image: "rockylinux:9"
-          # - name: "ubuntu2404-amd64"
-          #   base_image: "ubuntu:24.04"
+          - name: "manylinux_2_28_x86_64"
+            base_image: "quay.io/pypa/manylinux_2_28_x86_64"
 
     steps:
       - name: Checkout code
@@ -197,22 +189,13 @@ jobs:
           command -v auditwheel >/dev/null || { echo "auditwheel missing, aborting"; exit 1; }
           mkdir -p wheelhouse
 
-          # Convert linux_x86_64 wheels to multi-platform tag (incl. manylinux_2_34_x86_64) using auditwheel
+          # Convert linux_x86_64 wheels to manylinux_2_28_x86_64 using auditwheel
           echo ""
-          echo "==> Converting wheels to multi-platform tag (incl. manylinux_2_34_x86_64) format..."
+          echo "==> Converting wheels to manylinux_2_28_x86_64 format..."
           for wheel in dist/*.whl; do
             if [[ "$wheel" == *"linux_x86_64"* ]]; then
               echo "Converting: $(basename $wheel)"
-
-              # show autodetected set of platform tags
               auditwheel show "$wheel"
-
-              # fix/rename wheel based on autodetected platform tags and store in wheelhouse/
-              # -> auditwheel will encode all (!) supported tags into the wheel filename
-              # -> Python packaging ecosystem allows wheels to carry multiple platform tags
-              # -> pip will then pick the most specific tag that works for the current system
-              # -> so we do not need to manually set: --plat manylinux_2_34_x86_64 as in:
-              #     auditwheel repair "$wheel" --plat manylinux_2_34_x86_64 -w wheelhouse/
               auditwheel repair "$wheel" -w wheelhouse/
             else
               echo "Copying non-linux wheel: $(basename $wheel)"

.github/workflows/wheels-arm64.yml

@@ -45,6 +45,15 @@ node.key
 src/autobahn/nvx/_nvx_*.c
 src/_nvx_*.c
 
+# Build artifact: flatc binary (compiled from deps/flatbuffers during wheel build)
+src/autobahn/_flatc/bin/
+
+# Build artifact: reflection.bfbs (generated from reflection.fbs during wheel build)
+src/autobahn/flatbuffers/reflection.bfbs
+
+# Build artifact: wamp.bfbs (generated from wamp.fbs during wheel build)
+src/autobahn/wamp/flatbuffers/wamp.bfbs
+
 .wheels
 get-pip.py
 docs/autoapi/