Skip to content

Commit 3f065bf

Browse files
committed
docs: Add comprehensive multi-platform build architecture documentation
Created BUILD.md documenting the entire JRE bundling system: - Complete architecture overview (hybrid Docker + native builds) - Build pipeline (2-job strategy with single upstream JAR filtering) - Platform-specific details (5 platforms with sizes and runner info) - JAR exclusion system (single source of truth via jar_exclusions.txt) - Test parsing strategy (JUnit XML for cross-platform reliability) - Docker multi-stage build explanation - QEMU emulation setup for linux/arm64 - All 7 issues resolved and their solutions - Size breakdown (before/after optimization: 195.9M → 157-160M) - Performance notes and future improvements Updated README.md: - Added reference to BUILD.md for developers - Documented build system achievement This represents significant engineering work: - 5 platform support with zero Java required for users - ~40MB savings per wheel (gRPC exclusion) - Reliable CI/CD on GitHub Actions free tier - Platform-specific JREs (not cross-compiled) - Production-ready as of CI run ArcadeData#87 (2025-01-29)
1 parent bb8e48f commit 3f065bf

10 files changed

Lines changed: 482 additions & 669 deletions

File tree

bindings/python/BUILD.md

Lines changed: 421 additions & 0 deletions
Large diffs are not rendered by default.

bindings/python/README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,8 @@ Supported platforms: `linux/amd64`, `linux/arm64`, `darwin/amd64`, `darwin/arm64
131131

132132
> **Note:** `linux/arm64` uses QEMU emulation for builds. Additional platforms like `windows/arm64` may be added in future releases if demand justifies it.
133133
134+
> **Developer Note:** See [BUILD.md](BUILD.md) for comprehensive documentation of the multi-platform build architecture, including how we achieve platform-specific JRE bundling across 5 platforms on GitHub Actions.
135+
134136
!!! note "Package Contents"
135137
The package includes optimized ArcadeDB JARs. Some components are excluded for size optimization - see `jar_exclusions.txt` for details.
136138

bindings/python/TODO.md

Lines changed: 59 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -37,54 +37,70 @@ This document tracks all pending work for the ArcadeDB Python bindings project,
3737

3838
---
3939

40-
## 🔧 URGENT: Fix Multi-Platform Build System
41-
42-
**Status:** CRITICAL BUG - All 5 platform builds create identical linux-x64 wheels!
43-
44-
### Problem Discovery (2025-10-28)
45-
46-
**Root Cause:** The current Docker-based build uses `jlink` which can ONLY create JREs for the platform it's running on.
47-
48-
**Current Broken Behavior:**
49-
- All 5 matrix builds run on `ubuntu-latest` (linux-x64)
50-
- Docker `--platform` flag is passed but ignored by `jlink`
51-
- `jlink` always creates linux-x64 JRE (from the host JDK)
52-
- Result: 5 identical wheels with linux-x64 JRE
53-
- SHA256 checksums differ slightly (timestamps) but binaries are the same
54-
- All tests pass because all wheels have linux-x64 JRE running on ubuntu-latest
55-
56-
**Why We Didn't Notice:**
57-
- Tests all passed (they're all testing the same linux-x64 wheel)
58-
- Wheel sizes were identical (expected for same content)
59-
- Platform matrix looked correct (but wasn't actually building different platforms)
60-
61-
### Solution: Native Runners (Option 2 - Simple and Effective!)
40+
## ✅ Multi-Platform Build System - COMPLETE!
41+
42+
**Status:** ✅ COMPLETE - All 5 platforms working with optimized builds!
43+
44+
### Final Solution: Native Runners + QEMU for ARM64
45+
46+
**Supported Platforms (5):**
47+
-**linux/amd64** - `ubuntu-latest` (Docker native build)
48+
-**linux/arm64** - `ubuntu-latest` (Docker + QEMU emulation)
49+
-**darwin/amd64** - `macos-13` (native build, Intel Mac)
50+
-**darwin/arm64** - `macos-latest` (native build, Apple Silicon)
51+
-**windows/amd64** - `windows-latest` (native build)
52+
53+
### Accomplishments (CI run #87 - 2025-01-29)
54+
55+
**Final Wheel Sizes:**
56+
- linux/amd64: 160.9M ✅
57+
- linux/arm64: 159.9M ✅
58+
- darwin/amd64: 157.8M ✅
59+
- darwin/arm64: 156.7M ✅
60+
- windows/amd64: 157.4M ✅
61+
62+
**All Platforms:**
63+
- 43 tests passing
64+
- 167.4M JARs (83 files, gRPC excluded)
65+
- ~40MB savings per wheel (was 195.9M, now ~157-160M)
66+
67+
**Architecture Improvements:**
68+
1. **Single JAR filtering point**: download-jars job filters once (Ubuntu bash), uploads artifact
69+
2. **Artifact strategy**: Pre-filtered JARs for native builds, skip artifact for Docker
70+
3. **Structured test data**: pytest --junitxml for reliable cross-platform parsing
71+
4. **Code reduction**: Removed ~90 lines of duplicate filtering logic
72+
5. **QEMU support**: docker/setup-qemu-action@v3.2.0 for linux/arm64
73+
74+
### Issues Resolved
75+
1. ✅ Docker python-builder copying wrong JARs → Fixed: copy from jre-builder not java-builder
76+
2. ✅ Linux artifact conflict → Fixed: skip artifact download for Docker builds
77+
3. ✅ linux/arm64 host testing → Fixed: skip host tests (architecture mismatch), Docker only
78+
4. ✅ Windows glob pattern failure → Fixed: moved JAR filtering upstream to Ubuntu
79+
5. ✅ JAR filtering duplication → Fixed: single upstream filter in download-jars job
80+
6. ✅ Test count parsing failure → Fixed: grep -P (GNU-only) → JUnit XML (POSIX)
81+
7. ✅ Bash counter increment → Fixed: set -e compatibility
82+
83+
### Why This Approach Works
84+
1. GitHub Actions provides native macOS and Windows runners for free
85+
2. Docker provides consistent Linux builds with multi-platform support
86+
3. QEMU enables ARM64 emulation on AMD64 runners (slower but functional)
87+
4. Each platform builds with platform-specific JRE via native `jlink`
88+
5. True platform-specific wheels served by PyPI based on user's platform
89+
6. Single upstream JAR filtering eliminates duplication and cross-platform issues
6290

63-
**Philosophy: Only support what we can test natively**
91+
### Implementation Tasks
6492

65-
**Currently Supported Platforms (4):**
66-
-**linux/amd64** - `ubuntu-latest` (free, native)
67-
-**darwin/amd64** - `macos-13` (free with limits, native Intel Mac)
68-
-**darwin/arm64** - `macos-latest` (free with limits, native Apple Silicon Mac)
69-
-**windows/amd64** - `windows-latest` (free with limits, native x64)
93+
#### Phase 3.6: Multi-Platform Optimization and Refinement - ✅ COMPLETE
7094

71-
**Future Optional Platforms (2):**
72-
- 🔮 **linux/arm64** - Requires self-hosted runner or paid service
73-
- 🔮 **windows/arm64** - Requires `windows-11-arm64` runner (currently in beta)
95+
**NOTE:** Phase 3.6 original plan (refactor to native runners for 4 platforms) was superseded by improved solution:
96+
- Added linux/arm64 support via QEMU emulation (5 platforms total)
97+
- Implemented single upstream JAR filtering (jar_exclusions.txt)
98+
- Switched to JUnit XML for reliable cross-platform test parsing
99+
- Achieved ~40MB savings per wheel via gRPC exclusion
74100

75-
**Why This Approach:**
76-
1. GitHub Actions provides native macOS and Windows runners for free
77-
2. Each platform builds and tests on its native hardware
78-
3. No Docker cross-compilation complexity
79-
4. No QEMU emulation slowness
80-
5. True platform-specific JREs via native `jlink` on each platform
81-
6. Can add linux/arm64 and windows/arm64 later if demand justifies:
82-
- linux/arm64: Self-hosted runner or paid service (e.g., Actuated)
83-
- windows/arm64: GitHub's `windows-11-arm64` runner (currently in beta)
84-
85-
### Implementation Tasks
101+
**All tasks obsolete - replaced by superior implementation documented above.**
86102

87-
#### Phase 3.6: Refactor to Native Runners
103+
**Original Plan (Obsolete):**
88104

89105
**Build System Changes:**
90106

bindings/python/prototype/.gitignore

Lines changed: 0 additions & 8 deletions
This file was deleted.

bindings/python/prototype/Dockerfile.jre-prototype

Lines changed: 0 additions & 52 deletions
This file was deleted.

bindings/python/prototype/README.md

Lines changed: 0 additions & 46 deletions
This file was deleted.

bindings/python/prototype/RESULTS.md

Lines changed: 0 additions & 142 deletions
This file was deleted.

0 commit comments

Comments
 (0)