Skip to content

DUAL_RUNNER_COMPARISON.md

Latest commit

 

History

History
193 lines (153 loc) · 6.18 KB

DUAL_RUNNER_COMPARISON.md

File metadata and controls

193 lines (153 loc) · 6.18 KB

Matrix Strategy vs Single Runner: Separate Architecture Binaries

This document compares the current single-runner approach with the new matrix strategy approach for building separate architecture binaries.

Current Approach: Single Runner with CMAKE_OSX_ARCHITECTURES

How it works:

runs-on: macos-latest  # GitHub ARM64 runner
cmake -DCMAKE_OSX_ARCHITECTURES="arm64;x86_64" ..
make

Architecture:

[GitHub ARM64 Runner] -> [Universal Binary]
                     |
                     +-> Build both arm64 + x86_64
                     +-> Output: Universal binary

New Approach: Matrix Strategy with Separate Binaries

How it works:

strategy:
  matrix:
    arch: [arm64, x86_64]
    include:
      - arch: arm64
        runs-on: macos-latest  # GitHub ARM64 runner
      - arch: x86_64  
        runs-on: [self-hosted, macos-intel]  # Your Intel Mac

# Each job builds for its specific architecture
cmake -DCMAKE_OSX_ARCHITECTURES=${{ matrix.arch }} ..

Architecture:

[GitHub ARM64 Runner] -> [ARM64 Binary] -> [ARM64 DMG]
                                             
[Intel Mac Self-hosted] -> [x86_64 Binary] -> [x86_64 DMG]

Detailed Comparison

Aspect Current (Single Runner) New (Matrix Strategy)
Build Time ⚠️ Slower (sequential cross-compile) ✅ Faster (parallel native builds)
GitHub Actions Cost 💰 Higher (all minutes on GitHub) 💰 Lower (x86_64 on your hardware)
Debugging ❌ Hard to isolate arch issues ✅ Easy per-architecture debugging
Reliability ⚠️ Cross-compilation issues ✅ Native compilation reliability
Setup Complexity ✅ Simple (single runner) ❌ Complex (self-hosted setup)
Resource Usage ❌ GitHub runner does everything ✅ Distributed across machines
Architecture Separation ❌ Same config for both ✅ Different configs possible
Distribution ❌ Universal binary (larger) ✅ Separate binaries (smaller each)
User Choice ❌ One size fits all ✅ Users download their architecture
Testing ❌ Can't test individual archs ✅ Can test each arch separately
Maintenance ✅ Zero maintenance ❌ Runner maintenance required

Performance Analysis

Current Approach Timing:

GitHub Runner (ARM64):
├── Configure CMake (universal): ~2 min
├── Build ARM64 + x86_64: ~15 min
├── Package: ~3 min
└── Total: ~20 min

New Approach Timing:

Parallel Jobs:
├── GitHub Runner (ARM64):
│   ├── Configure: ~1 min
│   ├── Build ARM64: ~7 min
│   ├── Package ARM64 DMG: ~2 min
│   └── Subtotal: ~10 min
│
├── Intel Mac (x86_64):
│   ├── Configure: ~1 min  
│   ├── Build x86_64: ~7 min
│   ├── Package x86_64 DMG: ~2 min
│   └── Subtotal: ~10 min

Total: ~10 min (fully parallel)

Result: ~50% faster builds 🚀

Reliability Comparison

Current Issues We've Encountered:

  • "bad CPU type in executable" errors
  • Cross-compilation Qt framework issues
  • Architecture-specific dependency problems
  • Hard to debug which architecture causes issues

New Approach Benefits:

  • Each architecture builds natively (more reliable)
  • Easy to isolate problems to specific architecture
  • Independent dependency resolution
  • Better error messages and debugging

Cost Analysis

GitHub Actions Minutes Usage:

Current Approach:

20 minutes × macOS multiplier (10x) = 200 GitHub minutes per build

New Approach:

GitHub Runner: 10 minutes × 10x = 100 minutes
Intel Mac: 10 minutes × 0x = 0 minutes (your hardware)
Total: 100 GitHub minutes per build

Savings: 50% reduction in GitHub Actions costs 💰

Migration Strategy

Phase 1: Setup (This Week)

  1. Set up Intel Mac as self-hosted runner
  2. Test the new workflow on a branch
  3. Verify universal binaries work correctly

Phase 2: Parallel Running (Next Week)

  1. Run both workflows in parallel
  2. Compare build times and reliability
  3. Gather feedback and metrics

Phase 3: Switch Over (Week 3)

  1. Make dual-runner the default for releases
  2. Keep single-runner as backup
  3. Monitor for any issues

Phase 4: Optimization (Ongoing)

  1. Fine-tune runner performance
  2. Add architecture-specific optimizations
  3. Implement dependency caching

Risk Assessment

Risks of New Approach:

  • Self-hosted runner maintenance: Need to keep Intel Mac online and updated
  • Network dependency: Intel Mac needs stable internet
  • Single point of failure: If Intel Mac is down, no x86_64 builds
  • Security: Self-hosted runner has repository access

Mitigation Strategies:

  • Backup runner: Set up secondary Intel Mac
  • Monitoring: Add runner health checks
  • Fallback: Keep single-runner workflow as backup
  • Security: Use dedicated runner user account

Recommendation

Recommended approach: Matrix Strategy with Separate Binaries

Why:

  1. Better reliability: Native builds are more reliable than cross-compilation
  2. Cost savings: 50% reduction in GitHub Actions minutes
  3. Performance: 50% faster build times
  4. Debugging: Much easier to troubleshoot architecture-specific issues
  5. User choice: Smaller downloads, users pick their architecture
  6. Future-proofing: Better foundation for additional architectures

Implementation Plan:

  1. Follow SELF_HOSTED_RUNNER_SETUP.md to set up your Intel Mac
  2. Test using the new workflow .github/workflows/macos-matrix-build.yml
  3. Compare results with current approach
  4. Switch over once confident in the new system

Monitoring Success

Key Metrics to Track:

  • Build success rate: Should improve with native builds
  • Build time: Target 50% improvement
  • GitHub Actions cost: Target 50% reduction
  • Issue resolution time: Should be faster with better debugging

Success Criteria:

  • ✅ Architecture-specific binaries pass all tests
  • ✅ Build times under 12 minutes
  • ✅ 95%+ build success rate
  • ✅ Zero architecture-related issues for 2 weeks

This matrix strategy approach gives you clean, separate architecture binaries with much better performance! 🎯