aws-lc-sys fails to link on Windows CI: __builtin_bswap unresolved externals

[jamiepine]

Problem

aws-lc-sys v0.37.0 produces unresolved external symbol errors for __builtin_bswap32, __builtin_bswap64, and __builtin_bswap16 when building on Windows CI runners (Blacksmith blacksmith-4vcpu-windows-2025). This blocks all Windows desktop release builds.

Branch: release-build-fixes

Error (identical across 20+ CI runs):

libaws_lc_sys-f3cfa6bd59120afa.rlib(5e7b0e08b07b6019-bcm.o) : error LNK2019: unresolved external symbol __builtin_bswap32 referenced in function CRYPTO_bswap4
libaws_lc_sys-f3cfa6bd59120afa.rlib(5e7b0e08b07b6019-bcm.o) : error LNK2019: unresolved external symbol __builtin_bswap64 referenced in function CRYPTO_bswap8
libaws_lc_sys-f3cfa6bd59120afa.rlib(aa3883b9c76e19d7-cbb.o) : error LNK2019: unresolved external symbol __builtin_bswap16 referenced in function CRYPTO_bswap2
C:\runner\_work\spacedrive\spacedrive\target\release\deps\iroh_relay-f3f13ac3990b19b7.dll : fatal error LNK1120: 3 unresolved externals

The linker is MSVC's link.exe. The affected crates are iroh-relay and rustls-platform-verifier, both linking against aws_lc_sys.

Root Cause Analysis

aws-lc-sys has a compiler feature detection step in its build script (cc_builder.rs line ~732):

if self.compiler_check("builtin_swap_check", Vec::<&'static str>::new()) {
    cc_build.define("AWS_LC_BUILTIN_SWAP_SUPPORTED", Some("1"));
}

This test compiles builtin_swap_check.c which uses __builtin_bswap16/32/64. If the detected compiler supports these GCC builtins, it defines AWS_LC_BUILTIN_SWAP_SUPPORTED, causing the actual C source (crypto/internal.h) to use __builtin_bswap* instead of MSVC's _byteswap_ulong.

The problem is a compiler mismatch: aws-lc-sys uses the cc crate to detect the compiler (which finds clang-cl from the LLVM installation we install for bindgen), so the builtin check passes. But the actual CMake build uses the Visual Studio generator which compiles with MSVC cl.exe. MSVC treats __builtin_bswap* as regular extern function calls → unresolved at link time.

Critically, the cc crate does not respect CMAKE_C_COMPILER — it only checks CC, CC_<target>, and TARGET_CC environment variables. We spent many iterations setting CMAKE_C_COMPILER before discovering this.

Why LLVM is Installed

We install LLVM 15 via KyleMayes/install-llvm-action@v2 specifically for bindgen, which needs libclang.dll. The install-llvm-action adds C:\Program Files\LLVM\bin to PATH via core.addPath(), which makes clang-cl.exe discoverable to the cc crate.

Everything We Tried

All attempts on the release-build-fixes branch, each taking ~20 min to validate:

Attempt 1: Remove lld-link from xtask linker detection

• Removed lld-link from xtask/src/system.rs get_best_linker()
• Result: Same error, just with link.exe instead of lld-link (this was a linker issue, not the root cause)

Attempt 2: Install NASM

• Added ilammy/setup-nasm@v1 for Windows in release.yml
• Theory: NASM would let aws-lc-sys use assembly instead of C fallbacks
• Result: Same error — NASM doesn't eliminate C compilation for bswap code paths

Attempt 3: Set CC=cl and CXX=cl on tauri-action build step

• Added CC: cl and CXX: cl env vars to the build step
• Result: Same error — suspected cargo cache restoring old artifact

Attempt 4: Bust cargo cache (stable-cache-v2)

• Changed Swatinem/rust-cache shared-key from stable-cache to stable-cache-v2
• Result: Same error

Attempt 5: Add CMAKE_C_COMPILER and CMAKE_CXX_COMPILER

• Added these env vars to the build step
• Result: Same error — aws-lc-sys ignores these (uses cc crate, not cmake env vars)

Attempt 6: Set CC=cl via GITHUB_ENV in setup-system

• Set CC=cl, CXX=cl, CMAKE_C_COMPILER=cl, CMAKE_CXX_COMPILER=cl via $GITHUB_ENV
• Result: NEW error: ToolNotFound: failed to find tool "cl" — the cc crate can't find cl via simple PATH lookup; its built-in MSVC detection requires CC to NOT be set

Attempt 7: Remove LLVM from PATH with sed

• Tried removing LLVM bin from PATH using sed in a bash step, set LIBCLANG_PATH for bindgen
• Result: Same original error — GITHUB_PATH entries (from core.addPath()) are always prepended to PATH at every step, overriding GITHUB_ENV changes

Attempt 8: Rename clang executables

• Physically renamed clang.exe, clang++.exe, clang-cl.exe to .bak in LLVM bin
• Set LIBCLANG_PATH for bindgen, didn't set CC/CXX
• Result: Same error — cache from v2 may have been poisoned

Attempt 9: Cache bust to v3 + verification step

• Bumped shared-key to stable-cache-v3
• Added verification step that fails if Get-Command clang still finds clang
• Result: Same error

Attempt 10: Delete ALL .exe files from LLVM\bin

• Instead of renaming 3 specific files, deleted every .exe in C:\Program Files\LLVM\bin
• Also removed LLVM CMake config directory
• Also checked for VS-bundled Clang at multiple VS installation paths
• Set CMAKE_C_COMPILER to cl.exe full path via vswhere
• Bumped cache to v4
• Result: Same error — CMAKE_C_COMPILER is irrelevant because aws-lc-sys uses the cc crate

Attempt 11: Set CC to cl.exe full path via vswhere

• Used vswhere to find cl.exe absolute path
• Set CC, CXX, CMAKE_C_COMPILER, CMAKE_CXX_COMPILER via GITHUB_ENV
• Bumped cache to v5
• Result: Same error — still happening, the CC env var via GITHUB_ENV may not be reaching the build step, or the cc crate's behavior with full paths needs investigation

What We Know For Sure

1. The .rlib hash f3cfa6bd59120afa is a cargo metadata hash (deterministic, not cache-dependent)
2. The error persists through fresh caches (v2 through v5), confirming it's freshly compiled each time
3. The cc crate ignores CMAKE_C_COMPILER — only reads CC/CC_<target>/TARGET_CC
4. aws-lc-sys's builtin_swap_check creates a mismatch between compiler detection (cc crate → finds clang-cl) and actual compilation (CMake Visual Studio generator → uses MSVC cl.exe)
5. iroh already uses ring (not aws-lc-rs), but aws-lc-rs is still pulled in by other transitive deps (tokio-rustls defaults, sqlx, etc.)

Potential Solutions (Not Yet Tried)

1. Remove aws-lc-rs from the dependency tree entirely — switch all rustls consumers to use ring instead. Complex due to transitive dependency feature unification (tokio-rustls, sqlx, etc. all enable aws-lc-rs by default).

2. Don't install full LLVM — download only libclang.dll for bindgen instead of the full LLVM toolchain. Eliminates clang-cl from the system entirely.

3. Investigate why CC env var via GITHUB_ENV isn't reaching the build — the tauri-action might have its own env handling that overrides GITHUB_ENV.

4. Set CC directly on the tauri-action build step — instead of via GITHUB_ENV in an earlier step, set it in the env: block of the build step itself using vswhere output.

5. File upstream issue on aws-lc-rs — the compiler detection mismatch between cc crate and cmake builder is arguably a bug in aws-lc-sys.

Related Upstream Issues

• aws/aws-lc-rs#780 — Building aws-lc-sys under MSVC is brittle
• aws/aws-lc-rs#771 — Other libraries that rely on AWS LC SYS do not compile properly in a Windows environment
• rustls/rustls#2133 — Unable to build aws-lc-sys on Windows

[jamiepine]

Turns out issue was never about clang-cl detection. All previous attempts were solving the wrong problem.

The bug

In release.yml, the tauri-action build step had:

CFLAGS: ${{ matrix.settings.arch == 'aarch64' && '-march=armv8.2-a+crypto' || '' }}

For Windows x86_64, this evaluates to CFLAGS="" — an empty string, not unset. The cc crate (used by aws-lc-sys for compiler feature detection) treats a set-but-empty CFLAGS as "the user is managing compiler flags" and skips adding default warning flags like /W3.

Without /W3, MSVC defaults to /W1 (Level 1 warnings only). Warning C4013 ("function undefined; assuming extern returning int") is a Level 2 warning — so it doesn't fire at /W1.

This means cl.exe silently compiles builtin_swap_check.c containing __builtin_bswap32() calls as implicit extern function declarations. The .obj is created successfully (no linking happens during try_compile_intermediates()), so the compiler_check returns true, AWS_LC_BUILTIN_SWAP_SUPPORTED gets defined, and all of aws-lc's crypto source uses __builtin_bswap* instead of MSVC's _byteswap_ulong. At link time → unresolved externals.

The fix

Moved CFLAGS/CXXFLAGS out of the static env block into a conditional step that only runs for aarch64 targets (where the flags are actually needed):

- name: Set ARM crypto flags
  if: ${{ matrix.settings.arch == 'aarch64' }}
  shell: bash
  run: |
    echo "CFLAGS=-march=armv8.2-a+crypto" >> $GITHUB_ENV
    echo "CXXFLAGS=-march=armv8.2-a+crypto" >> $GITHUB_ENV

For non-aarch64 targets (including Windows x86_64), CFLAGS is now simply not present in the environment. The cc crate adds its default /W3, C4013 fires for the __builtin_bswap* calls, /WX (from warnings_into_errors(true)) promotes it to an error, the check correctly fails, and aws-lc-sys falls back to MSVC's native _byteswap_* intrinsics.

Also cleaned up

• Replaced the full LLVM toolchain install + 60 lines of clang-cl removal workaround with a minimal libclang.dll-only download for bindgen
• Bumped cargo cache key to avoid stale artifacts

Windows desktop build now passes in ~28 minutes.