Skip to content

Epic: Fortran engine - cross-platform binary distribution + auto-detected backend #147

Description

@neuromechanist

Goal

Promote the reference Fortran amica binary from a benchmark-only fixture to a first-class,
selectable pyAMICA engine (alongside AMICA_NumPy, AMICATorchNG, AMICAMLXNG): given real
EEG/EMG data, pyAMICA auto-detects the host platform, fetches/locates a compatible prebuilt
amica binary, runs it, and returns results through the same interface (mixing/unmixing
matrices, sources, log-likelihood) as the other backends.

Why

Two things already exist but were never connected:

  • benchmarks/fortran/build_amica.sh -- an open-source, portable build recipe (gfortran +
    OpenMPI + LAPACK) for amica15.f90/funmod2.f90, upstreamed to sccn/amica#53.
  • _run_fortran in benchmarks/benchmark_dimsweep.py -- a working adapter that writes .fdt
    (Fortran-order) + a param file, invokes the binary, and parses its output, but it is
    benchmark-only plumbing (no public class, no result objects, not platform-aware).

Meanwhile pyAMICA/sample_data/amica15mac is a single Mac x86_64 binary that isn't packaged for
distribution at all (pyproject.toml package-data ships only pyAMICA/data/* and
numpy_impl/params.json). There is no way today for a user to say "run the actual reference
Fortran implementation" through pyAMICA -- they'd have to hand-build and drive the binary
themselves.

Running the literal reference implementation is also the strongest possible parity oracle:
instead of comparing pyAMICA against Fortran, users could run Fortran as a pyAMICA backend when
they want ground truth, with no separate toolchain.

Distribution approach (decided)

GitHub Release assets, fetched on demand and cached locally -- not bundled in the PyPI wheel.
CI builds one binary per supported platform per release and attaches it as a release asset; the
new engine downloads (with checksum verification) and caches the matching binary on first use
(similar model to Playwright fetching browser binaries). This keeps the wheel small and avoids
per-platform wheel splitting, at the cost of needing network access (or a pre-populated cache) the
first time the Fortran engine is used on a given machine.

Phases (proposed)

  1. Platform-detection + binary-fetch utility. A small module that identifies OS/arch (and
    relevant ABI details -- glibc vs musl on Linux, macOS arm64 vs x86_64), maintains a manifest of
    known-good release-asset binaries per platform, downloads + checksums + caches to e.g.
    ~/.cache/pyamica/bin/, and raises a clear, actionable error (pointing at
    benchmarks/fortran/build_amica.sh) when no prebuilt binary matches the host.
  2. AMICAFortran engine class. Promote _run_fortran's write-.fdt/invoke/parse logic out of
    benchmarks/benchmark_dimsweep.py into a proper backend (own module, e.g. fortran_impl/)
    that exposes the same post-fit surface as the other backends (mixing/unmixing matrices,
    sources via transform, final_ll_, ll_history_) so it drops into validate_implementations.py
    and existing test patterns. Uses the Phase 1 utility to resolve its binary; no bundled
    binary shipped in the wheel itself.
  3. CI: build + publish binaries -- Linux + macOS. Extend CI to build amica for each
    platform on tag/release and attach the artifacts as GitHub Release assets, so the Phase 1
    manifest has real URLs to point at. Matrix: Linux x86_64 (glibc), Linux arm64 (aarch64) if the
    existing gfortran+OpenMPI+LAPACK recipe cross-builds cleanly, macOS arm64 (native, replacing
    the Rosetta-only amica15mac), macOS x86_64.
  4. Windows build spike + CI. build_amica.sh targets a POSIX toolchain (gfortran + OpenMPI +
    LAPACK via apt/brew); Windows has no equivalent out of the box, so this needs its own spike
    before it's a CI job: MSYS2/MinGW-w64 gfortran + MS-MPI (or MPICH) + a LAPACK source, or
    cross-compiling from Linux with MinGW. Target both Windows x86_64 and Windows arm64 if the
    toolchain supports arm64 (MSYS2/MinGW-w64 arm64 support is newer and may not); if arm64 proves
    infeasible, ship x86_64 only (it runs under emulation on Windows-on-ARM) and document the gap.
    Once a build path works, wire it into the same CI release-asset job as phase 3.
  5. Docs + validation wiring. Document the new engine (usage, cache location, offline/manual
    build fallback per platform) and add it as an option in validate_implementations.py / parity
    docs so "run the real Fortran binary" becomes a one-line comparison instead of a manual
    benchmark invocation.

Platform matrix (target)

OS Arch Status
Linux x86_64 (glibc) existing build recipe, should just work
Linux arm64 existing recipe, needs a cross-build/native-arm64 CI runner to confirm
macOS arm64 existing recipe (native, replaces Rosetta amica15mac)
macOS x86_64 existing recipe
Windows x86_64 new toolchain spike (phase 4)
Windows arm64 new toolchain spike (phase 4); may be dropped if MinGW-w64 arm64 Fortran+MPI is not viable, falling back to x86_64-under-emulation

Open questions (for phase 1/2 design, not blocking epic creation)

  • API shape: a new standalone class (AMICAFortran, consistent with how AMICA_NumPy /
    AMICATorchNG / AMICAMLXNG are separate classes today -- the AMICA wrapper has no
    backend= switch at present) vs. adding a unified backend= selector to AMICA itself. Leaning
    towards the standalone-class pattern to match existing architecture; revisit if this changes.
  • Multi-model / Newton / adaptive-PDF / share_comps support: the binary supports all of these
    natively (it is the reference), so this is mostly result-parsing scope, not a numerical
    question -- but the parser needs to handle every param file variant already covered in
    numpy_impl/load.py.
  • Whether release-asset binaries are signed/notarized (macOS Gatekeeper may block an unsigned
    downloaded binary from running without user approval) -- needs a spike in phase 1.

Constraints

  • Real EEG only (NO MOCKS) for validating engine output against pyAMICA's own backends.
  • No secrets/credentials in the fetch path; GitHub Release assets are public, no auth needed for
    read.
  • Follows the existing epic/phase workflow (gh sub-issue add, worktree-per-phase).

Metadata

Metadata

Assignees

No one assigned

    Labels

    epicEpic tracking issuefeatureNew feature or enhancement

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions