You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/guides/validation.md
+11Lines changed: 11 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -55,6 +55,17 @@ distance:
55
55
The fixed source-density families are bit-exact against the literal Fortran score/derivative expressions (~1e-15),
56
56
and the backend converges to the binary's solution within ~0.005 log-likelihood on either dataset.
57
57
58
+
### Newton-enabled runs and the initialization basin
59
+
60
+
The comparison above disables Newton (`do_newton=0`) to isolate the algorithm from its starting point.
61
+
With Newton enabled (`do_newton=1`, the default), agreement at the full 2000-iteration budget depends on the initialization, not on any dynamics difference between the backends.
62
+
From an *identical* initialization (the same starting mixing matrix and densities fed to both), `pamica` and Fortran converge to the same solution:
63
+
mean Hungarian-matched correlation ~0.997 with no collapsed components on the full 70-channel recording, the residual being floating-point summation-order noise between two implementations rather than an algorithmic gap.
64
+
From *independent* random initializations the picture differs, because the two backends' random number generators do not share a state, so a fixed seed does not map to a matched start.
65
+
At the long Newton budget this occasionally settles a few of the weakest, under-determined components into a different but equally likely (equal- or higher-likelihood) optimum.
66
+
Fortran is more robust to its own random inits (run-to-run self-consistency ~0.9997) than `pamica` is, so the effect appears as a `pamica`-specific spread on those components, not a divergence from the reference.
67
+
It is therefore an initialization-basin property (like the non-identifiable multi-model case below), not a parity defect; see issue #145 and the optional init-robustness follow-up #198.
68
+
58
69
### Source-density families are bit-exact
59
70
60
71
AMICA models each source with one of the reference's five `pdftype` density families.
Copy file name to clipboardExpand all lines: paper.md
+12-12Lines changed: 12 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -75,15 +75,16 @@ logistic, sub-Gaussian, and the extended-Infomax kurtosis switcher), a mixture o
75
75
and the Amari distance [@amari1996new], a standard unmixing-matrix comparison metric that needs no assignment step since it is permutation- and scale-invariant by construction.
76
76
Both implementations were run for AMICA's usual 2000 iterations with Newton off (`do_newton=0`) and otherwise-default parameters
77
77
(two separate configuration files drive them, JSON for `pamica` and Fortran's own text format, with a transcribed subset of settings; a shared-format reader is planned).
78
-
The single-model headline (Table 1) uses an external recording (OpenNeuro ds002718, 70 channels, $k\approx153$), well past the ~60 threshold where cross-backend agreement plateaus (documentation);
79
-
the bundled 32-channel sample used below sits at that threshold's boundary ($k\approx30$) and gives a consistent Amari distance.
78
+
Newton is disabled here to isolate the algorithm from initialization: with Newton on, independent random seeds (the backends share no random state) can settle a few under-determined components in different but equally likely optima, whereas from a matched initialization the two agree (documentation).
79
+
The single-model headline (Table 1) uses an external recording (OpenNeuro ds002718, 70 channels, $k\approx153$), well past the ~60 threshold where cross-backend agreement plateaus;
80
+
the bundled 32-channel sample below ($k\approx30$) gives a consistent Amari distance.
80
81
Score functions and per-block sufficient statistics are exact to floating-point resolution against the literal Fortran expressions on the bundled sample.
81
82
A mixture of ICA models is not partition-identifiable, so exact partition parity is the wrong bar for the multi-model case;
82
83
it is instead assessed by whether the two implementations sample a similar distribution of solutions, across ensembles of 20 bundled-sample runs each (\autoref{fig:ensemble}).
83
-
A run-level permutation test, which permutes the 40 runs as intact units to respect the dependence among pairwise values, finds no evidence that cross-implementation
84
+
A run-level permutation test (the 40 runs permuted as intact units) finds no evidence that cross-implementation
84
85
agreement is worse than Fortran's own run-to-run agreement, so single-run values reflect intrinsic estimator spread rather than a shortfall.
85
86
The multi-model log-likelihood distributions still differ slightly at a matched 100-iteration budget
86
-
(`pamica` reaches Fortran's mean with about twice as many iterations), so full-likelihood similarity is not yet claimed. Per-run detail for both metrics is in the documentation.
87
+
(`pamica` reaches Fortran's mean with about twice as many iterations), so full-likelihood similarity is not yet claimed.
@@ -108,7 +109,7 @@ On Apple Silicon the MLX backend is the fastest option and is flat with channel
108
109
about eight times faster than double-precision multithreaded CPU; PyTorch-MPS is not a win (at or worse than the CPU).
109
110
On NVIDIA hardware double-precision CUDA is the reproducible path and is overhead-bound at EEG scale,
110
111
so single precision gives it little additional speedup (Table 2). Native Fortran itself scales with CPU cores, unlike the CPU backends above:
111
-
with enough cores pinned it beats the CUDA GPU on the workstation (Table 2), though only by dedicating most cores of a larger, hotter host than a laptop GPU;
112
+
with enough cores pinned it beats the CUDA GPU on the workstation (Table 2), though only on a larger, hotter host;
112
113
it does not catch Apple's MLX on laptop hardware.
113
114
A data-size sweep further shows cross-backend component agreement rising with frames per channel and plateauing near 0.98 once the decomposition is well-determined,
114
115
where two independent double-precision implementations (native Fortran and PyTorch-CUDA) agree at a mean of 0.995;
@@ -128,21 +129,20 @@ single-precision runs agree with double precision to four to five significant di
128
129
`pdftype`=0, `block_size`=512; warm, minimum of repeated runs).
129
130
CPU, MPS, and MLX on Apple Silicon; CUDA on a separate NVIDIA RTX 4090;
130
131
CUDA float32 is comparable (~36 ms).
131
-
The two native-Fortran rows are from a separate core-count sweep (documentation) on the same two machines, at each backend's throughput plateau;
132
-
the other CPU rows above use the platformdefault thread count, so they are not core-matched to Fortran.
132
+
The two native-Fortran rows are from a separate core-count sweep (documentation) at each backend's plateau;
133
+
the other CPU rows use platform-default threads, so they are not core-matched to Fortran.
133
134
Unlike the correctness comparison, this benchmark uses external data (OpenNeuro ds002718, one subject so far) and specific GPU hardware.
134
135
135
136
The correctness harness compares `pamica` against Fortran with two metrics, Hungarian-matched component correlation and Amari distance, and never uses synthetic data;
136
137
the multi-model and score-function checks need no external download (bundled sample only).
137
-
The full per-channel and multi-model performance tables, the per-run Amari-distance detail,
138
-
and the data-size sweep, along with the step-by-step commands to reproduce every number here,
139
-
are in the documentation (<https://eeglab.org/pAMICA/guides/validation/>).
138
+
The full performance tables, per-run Amari-distance detail, data-size sweep,
139
+
and step-by-step reproduction commands are in the documentation (<https://eeglab.org/pAMICA/guides/validation/>).
140
140
141
141
# State of the field
142
142
143
-
`pamica` complements rather than replaces EEGLAB [@delorme2004eeglab] and its Fortran AMICA plugin: it reads and writes the same output format, so results move between the two, while adding GPU support and a Python API.
143
+
`pamica` complements rather than replaces EEGLAB [@delorme2004eeglab] and its Fortran AMICA plugin: it reads and writes the same output format, so results move between the two, adds GPU support and a Python API, and can run the reference Fortran itself from Python through a bundled dependency-free native build (no MKL or MPI runtime).
144
144
Two other Python AMICA reimplementations have appeared [@esmaeili2025amica; @herforth2026pyamica],
145
-
both of which provide MNE-Python-compatible objects; `pamica`instead offers a scikit-learn-style array API and byte-identical EEGLAB I/O, and does not yet ship an MNE-Python wrapper.
145
+
both of which provide MNE-Python-compatible objects; `pamica`leads with a scikit-learn-style array API and byte-identical EEGLAB I/O, and also ships an MNE-Python wrapper (an optional extra).
146
146
What distinguishes `pamica` is the rigor and scope of its Fortran-parity validation,
147
147
beyond what either alternative publishes: source-density score functions exact to floating-point resolution ($\sim\!10^{-15}$),
148
148
single-model component correlation and Amari distance against the reference, and a distributional-similarity framework for the non-identifiable multi-model case,
0 commit comments