Skip to content

Commit 70b4b42

Browse files
authored
Fix 1834 (#1858)
* start new dev branch; add audit file * Default NVX builds to a portable arch baseline; -march=native opt-in (#1834) Fixes cross-compilation of the NVX CFFI extensions (e.g. Buildroot/Yocto targeting aarch64 on an x86-64 host), which failed because the cross toolchain was handed the host-only -march=native flag: cc1: error: unknown value 'native' for '-march' Changes (implementing the design agreed in PR #1835): - get_compile_args(): the portable, safe architecture baseline is now the default for ALL build contexts (wheels, local source installs, and cross-compilation). -march=native is no longer the implicit default for local source builds; it is available opt-in via AUTOBAHN_ARCH_TARGET=native (build host == run host only). An unknown/cross target arch emits no -march flag, letting the toolchain defaults / distro CFLAGS decide. - Detect the TARGET architecture via sysconfig.get_platform() instead of platform.machine() (which reports the build host under cross-compilation). Approach contributed by @jameshilliard in PR #1835. - is_building_wheel() is retained for backward compatibility but no longer gates the default (documented as such). - Add src/autobahn/nvx/test/test_compile_args.py covering the default-safe / opt-in-native behavior, the arch->flag mapping, target detection, and the unknown-target (no -march) cross-compile case. Thanks to @jameshilliard for the original report (#1834) and PR #1835. Note: This work was completed with AI assistance (Claude Code).
1 parent c7e7f26 commit 70b4b42

4 files changed

Lines changed: 200 additions & 39 deletions

File tree

.audit/oberstet_fix_1834.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
- [ ] I did **not** use any AI-assistance tools to help create this pull request.
2+
- [x] I **did** use AI-assistance tools to *help* create this pull request.
3+
- [x] I have read, understood and followed the projects' [AI Policy](https://github.com/crossbario/autobahn-python/blob/main/AI_POLICY.md) when creating code, documentation etc. for this pull request.
4+
5+
Submitted by: @oberstet
6+
Date: 2026-06-16
7+
Related issue(s): #1834
8+
Branch: oberstet:fix_1834

docs/changelog.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ Changelog
2828
* Fix ``scripts/update_flatbuffers.sh`` git-version capture for submodule checkouts (``.git`` is a file, not a directory) (#1853)
2929
* Bump the ``.cicd`` (wamp-cicd) submodule to pick up the script/shell-injection fix in the shared ``identifiers.yml`` reusable workflow (untrusted GitHub event fields are now passed via ``env:`` as quoted data with a fail-closed branch-name allowlist) (#1856)
3030
* Fail wheel builds hard when NVX was requested (``AUTOBAHN_USE_NVX``) but the CFFI extension did not compile, instead of silently degrading to a pure-Python (``py3-none-any``) wheel. A transient native-compile crash (e.g. a ``gcc`` SIGSEGV under QEMU ARM64 emulation) now aborts the build with a non-zero exit so CI can retry it, rather than uploading a structurally valid but unintended artifact. Building with ``AUTOBAHN_USE_NVX=0`` still produces a pure-Python wheel as before (#1856)
31+
* Fix NVX native-extension builds breaking under cross-compilation (e.g. Buildroot/Yocto for aarch64), where the cross toolchain rejected the host-only ``-march=native`` flag (``unknown value 'native' for '-march'``). The default architecture target is now the portable baseline for *all* build contexts (wheels, local source installs, and cross-compilation), with ``-march=native`` available opt-in via ``AUTOBAHN_ARCH_TARGET=native``. The target architecture is detected via ``sysconfig.get_platform()`` so the correct baseline is chosen when cross-compiling. Thanks to @jameshilliard for the original report and approach (#1834, #1835)
3132

3233
25.12.2
3334
-------

src/autobahn/nvx/_compile_args.py

Lines changed: 77 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -34,13 +34,15 @@
3434
Strategy
3535
--------
3636
37-
For **WHEEL BUILDS** (distribution via PyPI):
38-
Use safe, portable baseline architectures to ensure wheels work on a wide
39-
range of CPUs without causing SIGILL (Illegal Instruction) crashes.
37+
**By default** - for *every* build context (PyPI wheels, local source installs,
38+
and cross-compilation) - a safe, portable baseline architecture is used. This
39+
keeps the resulting binaries runnable on a wide range of CPUs without SIGILL
40+
(Illegal Instruction) crashes, and never hands a cross-compilation toolchain the
41+
host-only ``-march=native`` flag (which it rejects).
4042
41-
For **SOURCE BUILDS** (local installation):
42-
Use -march=native to generate optimal code for the specific CPU where
43-
the build is happening, maximizing performance.
43+
Maximum-performance ``-march=native`` code generation is **opt-in** via
44+
``AUTOBAHN_ARCH_TARGET=native``; use it only when the build host is also the run
45+
host (e.g. Gentoo/Arch packages, dedicated single-machine deployments).
4446
4547
Architecture Baselines
4648
----------------------
@@ -61,9 +63,10 @@
6163
6264
AUTOBAHN_ARCH_TARGET : str, optional
6365
User/distro override for architecture target:
64-
- "native" : Force -march=native (maximum performance, may break portability)
65-
- "safe" : Force portable baseline (ensures compatibility)
66-
- Not set : Auto-detect based on build context (recommended)
66+
- "native" : Force -march=native (maximum performance, build host only;
67+
unsafe for distributed wheels and cross-compilation)
68+
- "safe" : Force portable baseline (explicit; same as the default)
69+
- Not set : Portable baseline (the safe default; works for cross-compilation)
6770
6871
AUTOBAHN_WHEEL_BUILD : str, optional
6972
Explicit marker for wheel builds ("true" or "1")
@@ -81,13 +84,19 @@
8184
Examples
8285
--------
8386
84-
**GitHub Actions building wheels:**
85-
>>> # Automatically detects CI=true, uses -march=x86-64-v2
87+
**GitHub Actions building wheels (default):**
88+
>>> # Default is the portable baseline
8689
>>> get_compile_args()
8790
['-std=c99', '-Wall', '-Wno-strict-prototypes', '-O3', '-march=x86-64-v2']
8891
89-
**User installing from source:**
90-
>>> # Detects local build, uses -march=native
92+
**User installing from source (default):**
93+
>>> # Default is the portable baseline (safe for cross-compilation too)
94+
>>> get_compile_args()
95+
['-std=c99', '-Wall', '-Wno-strict-prototypes', '-O3', '-march=x86-64-v2']
96+
97+
**Opting in to -march=native (build host == run host):**
98+
>>> import os
99+
>>> os.environ['AUTOBAHN_ARCH_TARGET'] = 'native'
91100
>>> get_compile_args()
92101
['-std=c99', '-Wall', '-Wno-strict-prototypes', '-O3', '-march=native']
93102
@@ -115,6 +124,8 @@
115124
Related Issues
116125
--------------
117126
- #1717: SIGILL crashes from -march=native in distributed wheels
127+
- #1834: -march=native breaks cross-compilation; the default is now the portable
128+
baseline for all build contexts, with -march=native available opt-in.
118129
119130
See Also
120131
--------
@@ -124,13 +135,21 @@
124135

125136
import os
126137
import sys
138+
import sysconfig
127139
import platform
128140

129141

130142
def is_building_wheel():
131143
"""
132144
Detect if we're building a wheel for distribution vs. a local source install.
133145
146+
.. note::
147+
148+
As of #1834 the default architecture target is the portable baseline for
149+
*every* build context (wheels, local source installs, cross-compilation),
150+
so this helper no longer influences :func:`get_compile_args`. It is
151+
retained for backward compatibility and for external callers/tooling.
152+
134153
Returns
135154
-------
136155
bool
@@ -194,7 +213,7 @@ def get_compile_args():
194213
return ["/O2", "/W3"]
195214

196215
# GCC/Clang on POSIX (Linux, macOS, *BSD)
197-
machine = platform.machine().lower()
216+
machine = _get_target_machine()
198217

199218
# Base flags for all POSIX platforms
200219
base_args = [
@@ -208,34 +227,53 @@ def get_compile_args():
208227
arch_override = os.environ.get("AUTOBAHN_ARCH_TARGET", "").lower()
209228

210229
if arch_override == "native":
211-
# User explicitly wants -march=native (maximum performance)
212-
# Use case: Gentoo, Arch Linux, performance-critical deployments
230+
# Explicit opt-in only: -march=native generates code for the exact CPU of
231+
# the *build* host (maximum performance). Use this when the build host is
232+
# also the run host (e.g. Gentoo, Arch Linux, performance-critical
233+
# deployments). It is NOT safe for distributed wheels or cross-compilation.
213234
return base_args + ["-march=native"]
214235

215-
elif arch_override == "safe":
216-
# User explicitly wants portable baseline
217-
# Use case: Debian, Ubuntu, RHEL package builds
218-
arch_flag = _get_safe_march_flag(machine)
219-
if arch_flag:
220-
return base_args + [arch_flag]
221-
else:
222-
# Unknown arch: use compiler defaults
223-
return base_args
224-
225-
elif is_building_wheel():
226-
# Building wheel for distribution: use portable baseline
227-
# These wheels may run on CPUs different from build machine
228-
arch_flag = _get_safe_march_flag(machine)
229-
if arch_flag:
230-
return base_args + [arch_flag]
231-
else:
232-
# Unknown arch: use compiler defaults
233-
return base_args
236+
# Default for everyone (AUTOBAHN_ARCH_TARGET unset or "safe"): a portable
237+
# baseline architecture. Defaulting to "safe" rather than -march=native is
238+
# what makes cross-compilation work out of the box - a cross toolchain
239+
# rejects -march=native ("unknown value 'native' for '-march'", #1834) - and
240+
# also prevents SIGILL crashes from over-optimized distributed wheels (#1717).
241+
arch_flag = _get_safe_march_flag(machine)
242+
if arch_flag:
243+
return base_args + [arch_flag]
234244

235-
else:
236-
# Building from source locally: use -march=native for maximum performance
237-
# Build machine = runtime machine, so native optimizations are safe and optimal
238-
return base_args + ["-march=native"]
245+
# Unknown / cross target architecture: emit no -march flag and let the
246+
# toolchain defaults (or distro-supplied CFLAGS, e.g. Buildroot/Yocto)
247+
# govern code generation.
248+
return base_args
249+
250+
251+
def _get_target_machine():
252+
"""
253+
Return the *target* machine architecture for the build.
254+
255+
Uses ``sysconfig.get_platform()`` rather than ``platform.machine()`` so that
256+
the correct architecture is detected when cross-compiling (e.g.
257+
Buildroot/Yocto building aarch64 on an x86-64 host): ``platform.machine()``
258+
reports the build host (``uname``), whereas ``sysconfig`` reflects the
259+
interpreter's configured target platform. Falls back to
260+
``platform.machine()`` when no architecture token can be derived.
261+
262+
(Target-detection approach contributed in PR #1835 by @jameshilliard.)
263+
264+
Returns
265+
-------
266+
str
267+
Lower-cased target architecture token, e.g. "x86_64", "aarch64",
268+
"arm64".
269+
"""
270+
plat = sysconfig.get_platform().lower()
271+
# Examples: 'linux-x86_64', 'linux-aarch64', 'macosx-11.0-arm64',
272+
# 'win-amd64', 'win32'.
273+
if plat == "win32":
274+
return "x86"
275+
machine = plat.rsplit("-", 1)[-1]
276+
return machine or platform.machine().lower()
239277

240278

241279
def _get_safe_march_flag(machine):
Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
###############################################################################
2+
#
3+
# The MIT License (MIT)
4+
#
5+
# Copyright (c) typedef int GmbH
6+
#
7+
# Permission is hereby granted, free of charge, to any person obtaining a copy
8+
# of this software and associated documentation files (the "Software"), to deal
9+
# in the Software without restriction, including without limitation the rights
10+
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
11+
# copies of the Software, and to permit persons to whom the Software is
12+
# furnished to do so, subject to the following conditions:
13+
#
14+
# The above copyright notice and this permission notice shall be included in
15+
# all copies or substantial portions of the Software.
16+
#
17+
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
18+
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
19+
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
20+
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
21+
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
22+
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
23+
# THE SOFTWARE.
24+
#
25+
###############################################################################
26+
27+
import os
28+
import sys
29+
import unittest
30+
from unittest import mock
31+
32+
from autobahn.nvx import _compile_args
33+
from autobahn.nvx._compile_args import (
34+
get_compile_args,
35+
_get_safe_march_flag,
36+
_get_target_machine,
37+
)
38+
39+
40+
class TestSafeMarchFlag(unittest.TestCase):
41+
"""Architecture -> safe -march flag mapping (#1834, #1717)."""
42+
43+
def test_x86_64_variants(self):
44+
for machine in ("x86_64", "amd64", "x64"):
45+
self.assertEqual(_get_safe_march_flag(machine), "-march=x86-64-v2")
46+
47+
def test_arm64_variants(self):
48+
for machine in ("aarch64", "arm64"):
49+
self.assertEqual(_get_safe_march_flag(machine), "-march=armv8-a")
50+
51+
def test_unknown_arch_returns_none(self):
52+
# Unknown architectures must not get a -march flag (let the toolchain
53+
# defaults / CFLAGS decide), so cross-compilation is never broken.
54+
self.assertIsNone(_get_safe_march_flag("riscv64"))
55+
56+
57+
class TestTargetMachine(unittest.TestCase):
58+
def test_returns_nonempty_lowercase(self):
59+
machine = _get_target_machine()
60+
self.assertIsInstance(machine, str)
61+
self.assertTrue(machine)
62+
self.assertEqual(machine, machine.lower())
63+
64+
def test_uses_sysconfig_target_arch(self):
65+
# When cross-compiling, sysconfig.get_platform() reflects the *target*
66+
# (e.g. aarch64) even on an x86-64 build host - this is the whole point
67+
# of preferring it over platform.machine() (#1834 / PR #1835).
68+
with mock.patch(
69+
"autobahn.nvx._compile_args.sysconfig.get_platform",
70+
return_value="linux-aarch64",
71+
):
72+
self.assertEqual(_get_target_machine(), "aarch64")
73+
74+
75+
class TestGetCompileArgs(unittest.TestCase):
76+
"""Default-safe / opt-in-native behaviour (#1834)."""
77+
78+
def test_default_is_never_native(self):
79+
# The core #1834 guard: with no override, -march=native must NEVER be
80+
# emitted (it breaks cross-compilation and can SIGILL distributed wheels).
81+
with mock.patch.dict("os.environ", {}, clear=False):
82+
os.environ.pop("AUTOBAHN_ARCH_TARGET", None)
83+
self.assertNotIn("-march=native", get_compile_args())
84+
85+
def test_safe_matches_default(self):
86+
with mock.patch.dict("os.environ", {}, clear=False):
87+
os.environ.pop("AUTOBAHN_ARCH_TARGET", None)
88+
default_args = get_compile_args()
89+
with mock.patch.dict("os.environ", {"AUTOBAHN_ARCH_TARGET": "safe"}):
90+
self.assertEqual(get_compile_args(), default_args)
91+
92+
@unittest.skipIf(sys.platform == "win32", "MSVC path uses /arch, not -march")
93+
def test_native_is_opt_in(self):
94+
with mock.patch.dict("os.environ", {"AUTOBAHN_ARCH_TARGET": "native"}):
95+
self.assertIn("-march=native", get_compile_args())
96+
97+
@unittest.skipIf(sys.platform == "win32", "MSVC path uses /arch, not -march")
98+
def test_unknown_target_emits_no_march(self):
99+
# An unrecognized cross-compile target must yield no -march flag at all.
100+
with mock.patch.dict("os.environ", {}, clear=False):
101+
os.environ.pop("AUTOBAHN_ARCH_TARGET", None)
102+
with mock.patch.object(
103+
_compile_args, "_get_target_machine", return_value="riscv64"
104+
):
105+
args = get_compile_args()
106+
self.assertFalse(any(a.startswith("-march") for a in args))
107+
108+
@unittest.skipIf(sys.platform != "win32", "MSVC-specific flags")
109+
def test_windows_uses_msvc_flags(self):
110+
self.assertEqual(get_compile_args(), ["/O2", "/W3"])
111+
112+
113+
if __name__ == "__main__":
114+
unittest.main()

0 commit comments

Comments
 (0)