feat: Add cross-compilation support for sdist packages with native extensions#1363
feat: Add cross-compilation support for sdist packages with native extensions#1363xangcastle wants to merge 14 commits into
Conversation
✨ Aspect Workflows Tasks📅 Thu Jul 23 18:45:26 UTC 2026 ✅ 40 successful tasks
⏱ Last updated Thu Jul 23 18:50:10 UTC 2026 · 📊 GitHub API quota 542/15,000 (4% used, resets in 49m) |
py_binary startup benchmark
sys.path quality
Bazel analysis benchmark
Auxiliary metrics
|
b74997f to
0888721
Compare
5e9c4f1 to
cd6dfa7
Compare
xangcastle
changed the title
tamird
left a comment
tamird
left a comment
There was a problem hiding this comment.
Reviewing this draft at the author's explicit request.
rules_pycross already provides the native CC layer, separate execution and target Python interpreters, PEP 517 cross-build environment, and target-aware sysconfig that this change reimplements. Its v2 alpha also exposes those primitives through a public backend integration API. [0] [1] [2] [3]
The proposed implementation instead infers cross-compilation from an absent optional native toolchain, omits target native dependency and runtime closures, hard-codes macOS deployment tags, and rejects valid Windows wheels. The Linux-only fixture checks the ELF architecture and filename without proving that a cross-built native extension actually loads.
Please first evaluate integration with rules_pycross, including the v2 alpha's transitive-dependency compatibility. If that integration is not viable, document the actual blocker and propose the smallest reusable upstream interface before introducing a parallel cross-compilation implementation. [4]
[0] https://github.com/jvolkman/rules_pycross/blob/v2.0.0-alpha.2/pycross/backend.bzl#L63-L83
[1] https://github.com/jvolkman/rules_pycross/blob/74ee87c7d8eab76a673c07df3ec9e55d1e629e49/pycross/toolchain.bzl#L33-L55
[2] https://github.com/jvolkman/rules_pycross/blob/74ee87c7d8eab76a673c07df3ec9e55d1e629e49/pycross/private/build/actions/cc_layer.bzl#L126-L182
[3] https://github.com/jvolkman/rules_pycross/blob/74ee87c7d8eab76a673c07df3ec9e55d1e629e49/pycross/private/build/tools/utils/sysconfig_utils.py#L97-L169
[4] https://github.com/bazelbuild/bazel-central-registry/blob/main/modules/rules_pycross/2.0.0-alpha.2/MODULE.bazel
— tamirdex
|
|
||
| target_os, target_cpu = get_target_platform(ctx) | ||
|
|
||
| return struct( |
There was a problem hiding this comment.
This returns compiler tools and flags, but not the CcInfo headers, dependent static/shared libraries, or target C++ runtime that a real source-built extension requires. The wheel action cannot materialize that dependency closure; inspecting the toy geohash ELF cannot establish that the resulting extension links or loads. The existing pycross layer explicitly gathers all three. Please compose with that layer or explain and test the real alternative. [0]
— tamirdex
| sysconfig_file = _find_sysconfigdata(runtime) | ||
| if sysconfig_file: | ||
| extra_inputs.append(depset([sysconfig_file])) | ||
| env["RULES_PY_TARGET_SYSCONFIGDATA"] = sysconfig_file.path |
There was a problem hiding this comment.
Copying one target _sysconfigdata file into a build still executed by the host interpreter does not create a target Python environment. Build backends continue to observe host interpreter identity, packaging tags, and platform behavior; the later validation checks only the final platform segment, so a host cp313 ABI can pass for a cp312 target with the same OS and CPU. pycross carries both interpreters and builds the appropriate cross environment instead. [0]
— tamirdex
| return "linux-" + _PYTHON_CPU_MAP.get(target_cpu, target_cpu) | ||
| if target_os == "darwin": | ||
| cpu = "arm64" if target_cpu == "aarch64" else target_cpu | ||
| return "macosx-11.0-" + cpu |
There was a problem hiding this comment.
The macOS deployment version is a property of the target interpreter/SDK, not always 11.0. Hard-coding it produces an incorrect compatibility tag for targets requiring another deployment version. Derive the platform and deployment target from target sysconfig, as pycross already does, and add an actual Darwin cross-build regression. [0]
— tamirdex
| ) | ||
| exit(1) | ||
|
|
||
| if expected_cpu not in platform_tag: |
There was a problem hiding this comment.
A valid Windows x86-64 wheel has platform tag win_amd64, but _expected_cpu_in_tag returns x86_64 here. Consequently every such wheel fails this validation despite being correct; Windows x86 has the analogous win32/i686 mismatch. Conversely unrecognized target OS/CPU values silently skip validation above. Derive supported wheel tags from actual target metadata and fail closed for unsupported targets.
— tamirdex
| tools = [":check_so_arch.sh"], | ||
| ) | ||
|
|
||
| build_test( |
There was a problem hiding this comment.
These new checks prove the packaged Linux ELF architecture and filename, but never import or execute the cross-built extension. They therefore pass if the Python ABI is wrong or a target C/C++ dependency is absent at runtime. Add a target-side load/runtime regression with a real native dependency, and cover native-toolchain absence and a non-Linux target, before treating these checks as proof of general cross-compilation.
— tamirdex
a70f8ff to
c1e9648
Compare
c1e9648 to
4a95d04
Compare
tamird
left a comment
tamird
left a comment
There was a problem hiding this comment.
Re-reviewing because Jason explicitly requested review of 9f3b372.
The new execution transition still selects host_platform and repository-time host libc, rather than the wheel action’s actual execution platform. The associated frontend test removes its execution-platform assertion. This cannot build correctly under heterogeneous remote execution; the new documentation itself confirms that the design assumes exec == host.
The five existing review threads remain current and unanswered. The replacement CC layer does not stage transitive CcInfo headers, native libraries or runtime dependencies; the PEP 517 frontend still runs the host interpreter with only a target sysconfig file; macOS wheel tags still hard-code deployment target 11.0; and wheel validation still rejects valid win_amd64 tags. The Linux fixture checks one patched extension’s ELF and filename but never executes the cross-built artifact or verifies a native dependency, target Python ABI, Darwin or remote worker.
rules_pycross already publicly exports target-aware CC extraction, PEP 517 actions, wheel repair and execution-platform transitions. Please evaluate composing those existing primitives before maintaining a second incomplete cross builder. If a dependency or compatibility issue prevents reuse, identify it and add real target-runtime and remote-execution coverage before calling this general cross-compilation.
https://github.com/jvolkman/rules_pycross/blob/v2.0.0-alpha.2/pycross/backend.bzl#L28-L39
https://github.com/jvolkman/rules_pycross/blob/v2.0.0-alpha.2/pycross/backend.bzl#L63-L83
— tamirdex
2 participants
Enable pep517_native_whl to cross-compile Python sdists containing C or C++ extensions when the target platform differs from the exec host.
The native_build_toolchain sentinel is now optional, so a missing resolution signals cross-compilation mode and falls through to a user-registered cross CC toolchain (such as
toolchains_llvm) instead of hard-failing.A new
cc_layer.bzlmodule extracts compiler paths,CFLAGS, andLDFLAGSfrom the resolved cross CC toolchain at analysis time, and build_helper.py generates compiler wrappers that re-inject -target/--sysroot identity flags, filter incompatible linker flags from the exec host, and override LDSHARED to use the cross CC toolchain's link flags. The target interpreter's_sysconfigdatais loaded via_PYTHON_SYSCONFIGDATA_NAMEso setuptools produces correctEXT_SUFFIXandSOABItags, and a post-build check validates the wheel's platform tag against the target before accepting the output.Changes are visible to end-users: yes/no
Test plan