Stop committing linux-x86-64 .so; guard the GLIBC floor in CI

The committed linux-x86-64/libquestdb.so had regressed to a GLIBC_2.33
load floor (stat@GLIBC_2.33 / fstat@GLIBC_2.33), up from the intended
2.14, making it fail to load on RHEL/Rocky/Alma 8, Ubuntu 20.04, Amazon
Linux 2, Debian 10 and RHEL 7. It was also the last committed client
binary, contradicting the "no longer committed" CI comment.

- Delete the committed linux-x86-64 .so. The release and CI jobs build
  the native libs from source in low-glibc containers, and Os.java loads
  them via the prd/bin-local paths, so nothing depends on the committed
  file.
- Add .github/scripts/check-glibc-floor.sh: extracts every versioned
  GLIBC import via objdump and fails if the highest node exceeds the
  allowed floor, naming the offending symbols.
- ci.yml: new glibc-floor matrix guard that rebuilds the linux libs in
  the same manylinux_2_28 containers as release and asserts the floor
  (2.14 on x86-64, 2.17 -- the lowest glibc offers -- on aarch64). The
  build-jdk8 job builds on ubuntu-latest for functional tests only and
  cannot catch a floor regression.
- maven_central_release.yml: assert the floor in the linux smoke-tests
  so a regressed library can never be shipped.
- rebuild_native_libs.yml: assert the floor before committing so a
  regressed library can never be committed.

Author: bluestreak01

.github/scripts/check-glibc-floor.sh

@@ -0,0 +1,80 @@
+#!/usr/bin/env bash
+# Assert the glibc runtime floor of a Linux native library.
+#
+# Usage: check-glibc-floor.sh <path-to-libquestdb.so> <max-glibc-version>
+#   e.g. check-glibc-floor.sh core/.../linux-x86-64/libquestdb.so 2.14
+#        check-glibc-floor.sh core/.../linux-aarch64/libquestdb.so 2.17
+#
+# The dynamic linker resolves .gnu.version_r at load time, so the HIGHEST
+# GLIBC_x.y version node the library imports is its hard load floor: a host
+# whose glibc is older than that node fails System.loadLibrary/dlopen with
+# `version 'GLIBC_x.y' not found`. This script extracts every versioned import
+# and fails if the highest one exceeds the allowed floor.
+#
+# Why the floors are what they are:
+#   * linux-x86-64 -> 2.14. The oldest node we intentionally keep is
+#     memcpy@GLIBC_2.14; clock_gettime is pinned back to GLIBC_2.2.5 by
+#     src/main/c/share/glibc_compat.h, and stat/fstat resolve to the inline
+#     __xstat/__fxstat@GLIBC_2.2.5 wrappers when built in a low-glibc container.
+#     A build on a modern host (glibc >= 2.33) instead emits stat@GLIBC_2.33 /
+#     fstat@GLIBC_2.33 and trips this guard -- that is exactly the regression it
+#     exists to catch.
+#   * linux-aarch64 -> 2.17. glibc gained aarch64 support in 2.17, so 2.17 is
+#     the lowest floor physically achievable on that architecture.
+#
+# Portable to bash 3.2 (no mapfile / no negative array indices) so it can be run
+# locally on macOS as well as in the glibc build containers.
+set -euo pipefail
+
+lib="${1:?usage: check-glibc-floor.sh <lib.so> <max-glibc-version>}"
+floor="${2:?usage: check-glibc-floor.sh <lib.so> <max-glibc-version>}"
+
+if [ ! -f "$lib" ]; then
+  echo "::error::check-glibc-floor: library not found: $lib"
+  exit 1
+fi
+
+# All distinct versioned GLIBC nodes (e.g. 2.14, 2.2.5), sorted ascending.
+# objdump prints them as (GLIBC_x.y) or GLIBC_x.y depending on the toolchain;
+# the -o regex captures the token regardless of surrounding parentheses.
+# GLIBC_PRIVATE has no digit after the underscore, so it is naturally excluded.
+versions="$(
+  objdump -T "$lib" \
+    | grep -oE 'GLIBC_[0-9]+(\.[0-9]+)+' \
+    | sed 's/^GLIBC_//' \
+    | sort -Vu
+)"
+
+if [ -z "$versions" ]; then
+  echo "::error::check-glibc-floor: no versioned GLIBC symbols found in $lib (unexpected)."
+  exit 1
+fi
+
+highest="$(printf '%s\n' "$versions" | tail -n1)"
+
+echo "GLIBC version nodes required by $lib:"
+printf '%s\n' "$versions" | sed 's/^/  GLIBC_/'
+echo "Highest required: GLIBC_${highest} (allowed floor: GLIBC_${floor})"
+
+# leq A B -> succeeds when version A <= version B: sorting {A, B} with -V puts B
+# last, or they are equal.
+leq() {
+  [ "$1" = "$2" ] && return 0
+  [ "$(printf '%s\n%s\n' "$1" "$2" | sort -V | tail -n1)" = "$2" ]
+}
+
+if leq "$highest" "$floor"; then
+  echo "OK: $lib floor is GLIBC_${highest} (<= GLIBC_${floor})."
+  exit 0
+fi
+
+echo "::error::GLIBC floor regression in $lib: requires GLIBC_${highest}, above the GLIBC_${floor} floor."
+echo "::error::This library will fail to load on hosts with glibc < ${highest}."
+echo "Offending nodes above the floor and the symbols that pull them in:"
+printf '%s\n' "$versions" | while IFS= read -r v; do
+  if ! leq "$v" "$floor"; then
+    echo "  GLIBC_${v}:"
+    objdump -T "$lib" | grep -E "GLIBC_${v//./\\.}([^0-9]|\$)" | awk '{print "    " $NF}' | sort -u
+  fi
+done
+exit 1

.github/workflows/ci.yml

@@ -44,9 +44,12 @@ jobs:
 
       - name: Build native libquestdb.so
         # JAVA_HOME points at the JDK 8 above, so the lib is compiled against the
-        # Java 8 JNI headers -- the artifact's floor. Copy it into src resources
-        # (not target/) so it survives the `mvn clean` in the next step and gets
-        # packaged + loaded via the production bin/<platform> path.
+        # Java 8 JNI headers -- the artifact's Java floor. Copy it into src
+        # resources (not target/) so it survives the `mvn clean` in the next step
+        # and gets packaged + loaded via the production bin/<platform> path.
+        # NOTE: this builds on ubuntu-latest for FUNCTIONAL testing only; the
+        # library's glibc runtime floor is validated separately by the
+        # `glibc-floor` job, which rebuilds in the release low-glibc container.
         run: |
           cd core
           cmake -DCMAKE_BUILD_TYPE=Release -B cmake-build-release -S.
@@ -81,3 +84,80 @@ jobs:
 
       - name: Compile (main + test) and build javadoc (no tests run)
         run: mvn -B -ntp -P javadoc -DskipTests clean package
+
+  # GLIBC floor guard. The native libraries are built at release time in
+  # low-glibc manylinux containers (see maven_central_release.yml) and are NOT
+  # committed, so a floor regression is invisible to the functional test job
+  # above (it builds on ubuntu-latest, whose glibc is new enough to load almost
+  # anything). This job rebuilds the linux libraries in the SAME low-glibc
+  # environment as release and asserts the runtime floor with objdump, so a
+  # change that raises the floor (e.g. a new stat/fstat call pulling in
+  # stat@GLIBC_2.33 on a modern build host) fails the PR instead of silently
+  # shipping a library that cannot load on older distros.
+  #
+  #   * linux-x86-64  -> GLIBC_2.14 (the intended floor: memcpy@GLIBC_2.14).
+  #   * linux-aarch64 -> GLIBC_2.17 (the lowest floor glibc offers on aarch64).
+  #
+  # Uses manylinux_2_28 for both arches (stock Node 24, no glibc-2.17 shadow
+  # hack). The x86-64 floor is identical in manylinux2014 (2.17) and
+  # manylinux_2_28 (2.28) -- both resolve stat/fstat to the inline
+  # __xstat/__fxstat@GLIBC_2.2.5 wrappers -- so this validates the real shipped
+  # floor without the heavier manylinux2014 release toolchain.
+  glibc-floor:
+    name: GLIBC floor guard (${{ matrix.platform }})
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - platform: linux-x86-64
+            os: ubuntu-latest
+            image: quay.io/pypa/manylinux_2_28_x86_64
+            jdk_arch: x64
+            floor: "2.14"
+            cmake_args: ""
+            build_dir: cmake-build-release
+          - platform: linux-aarch64
+            os: ubuntu-22.04-arm
+            image: quay.io/pypa/manylinux_2_28_aarch64
+            jdk_arch: aarch64
+            floor: "2.17"
+            cmake_args: "-DCMAKE_TOOLCHAIN_FILE=./src/main/c/toolchains/linux-arm64.cmake"
+            build_dir: cmake-build-release-arm64
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 45
+    container:
+      image: ${{ matrix.image }}
+    steps:
+      - name: Check out
+        uses: actions/checkout@v4
+        with:
+          # zstd is required to compile the native library.
+          submodules: recursive
+
+      - name: Install tooling
+        # binutils provides objdump for the floor check; nasm/zstd are build deps.
+        run: |
+          yum update -y
+          yum install -y wget nasm zstd binutils
+
+      - name: Install Temurin JDK 8 (for jni.h)
+        # Build against the Java 8 JNI headers -- JDK 8 is the artifact's floor.
+        # The JDK version does not affect the glibc floor; it only supplies jni.h.
+        run: |
+          wget -v --timeout=180 -O jdk8.tar.gz \
+            "https://api.adoptium.net/v3/binary/latest/8/ga/linux/${{ matrix.jdk_arch }}/jdk/hotspot/normal/eclipse"
+          mkdir jdk8
+          tar xfz jdk8.tar.gz -C jdk8 --strip-components=1
+          echo "JAVA_HOME=$(pwd)/jdk8" >> "$GITHUB_ENV"
+
+      - name: Build native libquestdb.so
+        run: |
+          cd core
+          cmake ${{ matrix.cmake_args }} -DCMAKE_BUILD_TYPE=Release -B ${{ matrix.build_dir }} -S.
+          cmake --build ${{ matrix.build_dir }} --config Release
+
+      - name: Assert GLIBC floor
+        run: |
+          ./.github/scripts/check-glibc-floor.sh \
+            core/target/classes/io/questdb/client/bin-local/libquestdb.so \
+            "${{ matrix.floor }}"

.github/workflows/maven_central_release.yml

@@ -295,6 +295,8 @@ jobs:
             echo "::error::libquestdb.so has unresolved dependencies."
             exit 1
           fi
+          # Refuse to ship if a symbol raised the glibc floor above 2.14.
+          ./.github/scripts/check-glibc-floor.sh "$lib" 2.14
           cat > LoadCheck.java <<'EOF'
           public class LoadCheck {
               public static void main(String[] args) {
@@ -360,6 +362,8 @@ jobs:
             echo "::error::libquestdb.so has unresolved dependencies."
             exit 1
           fi
+          # 2.17 is the lowest floor glibc offers on aarch64.
+          ./.github/scripts/check-glibc-floor.sh "$lib" 2.17
           cat > LoadCheck.java <<'EOF'
           public class LoadCheck {
               public static void main(String[] args) {

.github/workflows/rebuild_native_libs.yml

@@ -108,6 +108,11 @@ jobs:
           mkdir -p src/main/resources/io/questdb/client/bin/linux-x86-64/
           mkdir -p src/main/bin/linux-x86-64/
           cp target/classes/io/questdb/client/bin-local/libquestdb.so src/main/resources/io/questdb/client/bin/linux-x86-64/
+      - name: Assert GLIBC floor (2.14)
+        # Never commit a library whose glibc floor regressed above 2.14.
+        run: |
+          bash ./.github/scripts/check-glibc-floor.sh \
+            core/src/main/resources/io/questdb/client/bin/linux-x86-64/libquestdb.so 2.14
       - name: Save linux-x86-64 Libraries to Cache
         uses: actions/cache/save@v3
         with:
@@ -143,6 +148,11 @@ jobs:
           mkdir -p src/main/resources/io/questdb/client/bin/linux-aarch64/
           mkdir -p src/main/bin/linux-aarch64/
           cp target/classes/io/questdb/client/bin-local/libquestdb.so src/main/resources/io/questdb/client/bin/linux-aarch64/
+      - name: Assert GLIBC floor (2.17)
+        # 2.17 is the lowest floor glibc offers on aarch64.
+        run: |
+          bash ./.github/scripts/check-glibc-floor.sh \
+            core/src/main/resources/io/questdb/client/bin/linux-aarch64/libquestdb.so 2.17
       - name: Save linux-aarch64 Libraries to Cache
         uses: actions/cache/save@v3
         with: