Goal
Promote the reference Fortran amica binary from a benchmark-only fixture to a first-class,
selectable pyAMICA engine (alongside AMICA_NumPy, AMICATorchNG, AMICAMLXNG): given real
EEG/EMG data, pyAMICA auto-detects the host platform, fetches/locates a compatible prebuilt
amica binary, runs it, and returns results through the same interface (mixing/unmixing
matrices, sources, log-likelihood) as the other backends.
Why
Two things already exist but were never connected:
benchmarks/fortran/build_amica.sh -- an open-source, portable build recipe (gfortran +
OpenMPI + LAPACK) for amica15.f90/funmod2.f90, upstreamed to sccn/amica#53.
_run_fortran in benchmarks/benchmark_dimsweep.py -- a working adapter that writes .fdt
(Fortran-order) + a param file, invokes the binary, and parses its output, but it is
benchmark-only plumbing (no public class, no result objects, not platform-aware).
Meanwhile pyAMICA/sample_data/amica15mac is a single Mac x86_64 binary that isn't packaged for
distribution at all (pyproject.toml package-data ships only pyAMICA/data/* and
numpy_impl/params.json). There is no way today for a user to say "run the actual reference
Fortran implementation" through pyAMICA -- they'd have to hand-build and drive the binary
themselves.
Running the literal reference implementation is also the strongest possible parity oracle:
instead of comparing pyAMICA against Fortran, users could run Fortran as a pyAMICA backend when
they want ground truth, with no separate toolchain.
Distribution approach (decided)
GitHub Release assets, fetched on demand and cached locally -- not bundled in the PyPI wheel.
CI builds one binary per supported platform per release and attaches it as a release asset; the
new engine downloads (with checksum verification) and caches the matching binary on first use
(similar model to Playwright fetching browser binaries). This keeps the wheel small and avoids
per-platform wheel splitting, at the cost of needing network access (or a pre-populated cache) the
first time the Fortran engine is used on a given machine.
Phases (proposed)
- Platform-detection + binary-fetch utility. A small module that identifies OS/arch (and
relevant ABI details -- glibc vs musl on Linux, macOS arm64 vs x86_64), maintains a manifest of
known-good release-asset binaries per platform, downloads + checksums + caches to e.g.
~/.cache/pyamica/bin/, and raises a clear, actionable error (pointing at
benchmarks/fortran/build_amica.sh) when no prebuilt binary matches the host.
AMICAFortran engine class. Promote _run_fortran's write-.fdt/invoke/parse logic out of
benchmarks/benchmark_dimsweep.py into a proper backend (own module, e.g. fortran_impl/)
that exposes the same post-fit surface as the other backends (mixing/unmixing matrices,
sources via transform, final_ll_, ll_history_) so it drops into validate_implementations.py
and existing test patterns. Uses the Phase 1 utility to resolve its binary; no bundled
binary shipped in the wheel itself.
- CI: build + publish binaries -- Linux + macOS. Extend CI to build
amica for each
platform on tag/release and attach the artifacts as GitHub Release assets, so the Phase 1
manifest has real URLs to point at. Matrix: Linux x86_64 (glibc), Linux arm64 (aarch64) if the
existing gfortran+OpenMPI+LAPACK recipe cross-builds cleanly, macOS arm64 (native, replacing
the Rosetta-only amica15mac), macOS x86_64.
- Windows build spike + CI.
build_amica.sh targets a POSIX toolchain (gfortran + OpenMPI +
LAPACK via apt/brew); Windows has no equivalent out of the box, so this needs its own spike
before it's a CI job: MSYS2/MinGW-w64 gfortran + MS-MPI (or MPICH) + a LAPACK source, or
cross-compiling from Linux with MinGW. Target both Windows x86_64 and Windows arm64 if the
toolchain supports arm64 (MSYS2/MinGW-w64 arm64 support is newer and may not); if arm64 proves
infeasible, ship x86_64 only (it runs under emulation on Windows-on-ARM) and document the gap.
Once a build path works, wire it into the same CI release-asset job as phase 3.
- Docs + validation wiring. Document the new engine (usage, cache location, offline/manual
build fallback per platform) and add it as an option in validate_implementations.py / parity
docs so "run the real Fortran binary" becomes a one-line comparison instead of a manual
benchmark invocation.
Platform matrix (target)
| OS |
Arch |
Status |
| Linux |
x86_64 (glibc) |
existing build recipe, should just work |
| Linux |
arm64 |
existing recipe, needs a cross-build/native-arm64 CI runner to confirm |
| macOS |
arm64 |
existing recipe (native, replaces Rosetta amica15mac) |
| macOS |
x86_64 |
existing recipe |
| Windows |
x86_64 |
new toolchain spike (phase 4) |
| Windows |
arm64 |
new toolchain spike (phase 4); may be dropped if MinGW-w64 arm64 Fortran+MPI is not viable, falling back to x86_64-under-emulation |
Open questions (for phase 1/2 design, not blocking epic creation)
- API shape: a new standalone class (
AMICAFortran, consistent with how AMICA_NumPy /
AMICATorchNG / AMICAMLXNG are separate classes today -- the AMICA wrapper has no
backend= switch at present) vs. adding a unified backend= selector to AMICA itself. Leaning
towards the standalone-class pattern to match existing architecture; revisit if this changes.
- Multi-model / Newton / adaptive-PDF / share_comps support: the binary supports all of these
natively (it is the reference), so this is mostly result-parsing scope, not a numerical
question -- but the parser needs to handle every param file variant already covered in
numpy_impl/load.py.
- Whether release-asset binaries are signed/notarized (macOS Gatekeeper may block an unsigned
downloaded binary from running without user approval) -- needs a spike in phase 1.
Constraints
- Real EEG only (NO MOCKS) for validating engine output against pyAMICA's own backends.
- No secrets/credentials in the fetch path; GitHub Release assets are public, no auth needed for
read.
- Follows the existing epic/phase workflow (
gh sub-issue add, worktree-per-phase).
Goal
Promote the reference Fortran
amicabinary from a benchmark-only fixture to a first-class,selectable pyAMICA engine (alongside
AMICA_NumPy,AMICATorchNG,AMICAMLXNG): given realEEG/EMG data, pyAMICA auto-detects the host platform, fetches/locates a compatible prebuilt
amicabinary, runs it, and returns results through the same interface (mixing/unmixingmatrices, sources, log-likelihood) as the other backends.
Why
Two things already exist but were never connected:
benchmarks/fortran/build_amica.sh-- an open-source, portable build recipe (gfortran +OpenMPI + LAPACK) for
amica15.f90/funmod2.f90, upstreamed tosccn/amica#53._run_fortraninbenchmarks/benchmark_dimsweep.py-- a working adapter that writes.fdt(Fortran-order) + a param file, invokes the binary, and parses its output, but it is
benchmark-only plumbing (no public class, no result objects, not platform-aware).
Meanwhile
pyAMICA/sample_data/amica15macis a single Mac x86_64 binary that isn't packaged fordistribution at all (
pyproject.tomlpackage-data ships onlypyAMICA/data/*andnumpy_impl/params.json). There is no way today for a user to say "run the actual referenceFortran implementation" through pyAMICA -- they'd have to hand-build and drive the binary
themselves.
Running the literal reference implementation is also the strongest possible parity oracle:
instead of comparing pyAMICA against Fortran, users could run Fortran as a pyAMICA backend when
they want ground truth, with no separate toolchain.
Distribution approach (decided)
GitHub Release assets, fetched on demand and cached locally -- not bundled in the PyPI wheel.
CI builds one binary per supported platform per release and attaches it as a release asset; the
new engine downloads (with checksum verification) and caches the matching binary on first use
(similar model to Playwright fetching browser binaries). This keeps the wheel small and avoids
per-platform wheel splitting, at the cost of needing network access (or a pre-populated cache) the
first time the Fortran engine is used on a given machine.
Phases (proposed)
relevant ABI details -- glibc vs musl on Linux, macOS arm64 vs x86_64), maintains a manifest of
known-good release-asset binaries per platform, downloads + checksums + caches to e.g.
~/.cache/pyamica/bin/, and raises a clear, actionable error (pointing atbenchmarks/fortran/build_amica.sh) when no prebuilt binary matches the host.AMICAFortranengine class. Promote_run_fortran's write-.fdt/invoke/parse logic out ofbenchmarks/benchmark_dimsweep.pyinto a proper backend (own module, e.g.fortran_impl/)that exposes the same post-fit surface as the other backends (mixing/unmixing matrices,
sources via
transform,final_ll_,ll_history_) so it drops intovalidate_implementations.pyand existing test patterns. Uses the Phase 1 utility to resolve its binary; no bundled
binary shipped in the wheel itself.
amicafor eachplatform on tag/release and attach the artifacts as GitHub Release assets, so the Phase 1
manifest has real URLs to point at. Matrix: Linux x86_64 (glibc), Linux arm64 (aarch64) if the
existing gfortran+OpenMPI+LAPACK recipe cross-builds cleanly, macOS arm64 (native, replacing
the Rosetta-only
amica15mac), macOS x86_64.build_amica.shtargets a POSIX toolchain (gfortran + OpenMPI +LAPACK via apt/brew); Windows has no equivalent out of the box, so this needs its own spike
before it's a CI job: MSYS2/MinGW-w64 gfortran + MS-MPI (or MPICH) + a LAPACK source, or
cross-compiling from Linux with MinGW. Target both Windows x86_64 and Windows arm64 if the
toolchain supports arm64 (MSYS2/MinGW-w64 arm64 support is newer and may not); if arm64 proves
infeasible, ship x86_64 only (it runs under emulation on Windows-on-ARM) and document the gap.
Once a build path works, wire it into the same CI release-asset job as phase 3.
build fallback per platform) and add it as an option in
validate_implementations.py/ paritydocs so "run the real Fortran binary" becomes a one-line comparison instead of a manual
benchmark invocation.
Platform matrix (target)
amica15mac)Open questions (for phase 1/2 design, not blocking epic creation)
AMICAFortran, consistent with howAMICA_NumPy/AMICATorchNG/AMICAMLXNGare separate classes today -- theAMICAwrapper has nobackend=switch at present) vs. adding a unifiedbackend=selector toAMICAitself. Leaningtowards the standalone-class pattern to match existing architecture; revisit if this changes.
natively (it is the reference), so this is mostly result-parsing scope, not a numerical
question -- but the parser needs to handle every param file variant already covered in
numpy_impl/load.py.downloaded binary from running without user approval) -- needs a spike in phase 1.
Constraints
read.
gh sub-issue add, worktree-per-phase).