Skip to content

phase2: amica + dipfit (code + CI wiring; data run in flight)#31

Merged
neuromechanist merged 5 commits into
mainfrom
feature/issue-3-phase2-amica
May 13, 2026
Merged

phase2: amica + dipfit (code + CI wiring; data run in flight)#31
neuromechanist merged 5 commits into
mainfrom
feature/issue-3-phase2-amica

Conversation

@neuromechanist

@neuromechanist neuromechanist commented May 13, 2026

Copy link
Copy Markdown
Member

Summary

Phase 2 of the HBN ERSP epic: AMICA decomposition + dipfit5 dipole localization, with a pluggable ICA dispatcher so CI (Ubuntu, no MPI libs) can run Infomax while production (local macOS, full MPI) uses AMICA.

Refs epic #1, closes #3.

What landed

Code

  • src/matlab/phase2_amica.m — entrypoint with IcaMethod opt (amica or runica, default amica). Per-subject: load Phase 1 .set -> ICA -> attach weights -> dipfit -> figures -> AMICA checkpoint -> qa_amica.csv row.
  • src/matlab/+hbn/run_amica.m — wraps runamica15. Passes EEG.data(:,:) (matrix path); passing the struct or the .fdt path either crashed the macOS binary (forrtl-66 in arg parsing) or tripped Linux file-type detection ("unknown file type"). Matrix path works everywhere.
  • src/matlab/+hbn/run_infomax.m — runs pop_runica('icatype','runica'). Used by the CI smoke test because AMICA's Linux binary needs MPI Fortran runtime that Ubuntu runners do not have in a stable combination.
  • src/matlab/+hbn/fit_dipoles.mpop_dipfit_settings (standard MNI BEM) + pop_multifit.
  • src/matlab/+hbn/save_ic_topo_figure.m, save_dipole_figure.m — first 30 IC topographies + dipole top view.
  • src/matlab/+hbn/write_qa_amica_csv.m — schema: participant_id, status, ica_method, n_components, final_ll, iterations, reached_max_iter, median_rv, num_models, max_threads, duration_s, error_message.
  • scripts/run_phase2_three_subjects.m — production 3-subject runner.
  • scripts/regenerate_phase2_figures.m — rebuilds figures from existing AMICA .set checkpoints (used after the figure-export bug).
  • tests/matlab/test_phase2_smoke.m — chains Phase 1 -> Phase 2 on the fixture with IcaMethod=runica, MaxIter=100, DoReject=false.

CI workflow

  • .github/workflows/matlab-tests.yml auto-installs Biosig, amica, Fieldtrip-lite. AMICA isn't usable on the runner anyway (libmpifort.so.12); install kept for parity / future probing. Cache key eeglab-2026.0.0-v2-amica. Timeout 45 min.

Derivatives

  • derivatives/amica/ (tracked via the .gitignore allow-list):
    • README.md, qa_amica.csv, params.json
    • sub-<ID>/figures/*.png (6 total)
  • derivatives/**/_amica/ (heavy AMICA model binaries) and tmpflist (AMICA scratch) added to denylist.

3-subject AMICA results

Participant n_components iterations reached_max_iter median_rv wall-time
sub-NDARAA948VFH 123 2000 true 0.368 ~32 min
sub-NDARAC853DTE 112 2000 true 0.420 ~22 min
sub-NDARAD774HAZ 98 2000 true 0.368 ~19 min

n_components matches Phase 1 n_channels_after for every subject. Cz is rejected upstream and absent from the ICA. AMICA hit the iteration ceiling on all 3 — expected at this data size (~210 s × 100 Hz vs 30×N² floor for stable ICA at 120 channels). Reference pipeline mitigates by concatenating tasks; we intentionally process ThePresent only.

eeg-qa-neuroscientist review

Two passes; both attached for transparency.

Pass 1 verdict: FAIL

All six PNG figures are empty white canvases. File sizes byte-identical across subjects, which is impossible for genuine renders of independent ICA decompositions.

Root cause: save_ic_topo_figure and save_dipole_figure pre-allocated a figure handle and passed it to exportgraphics. But pop_topoplot and pop_dipplot open their own figures; the pre-allocated handle stayed empty. Both helpers now snapshot handles before/after the pop_* call and capture the new figure.

Pass 2 verdict: PASS-WITH-FINDINGS

Critical: IC 1-5 in the topo montage were occluded by my added sgtitle. Dropped sgtitle; pop_topoplot's own title arg is sufficient. Verified by reading the regenerated sub-NDARAA948VFH_ic-topos.png — IC 1 through 30 all visible.

Important (not blocking): tail ICs (beyond ~15) look noisy; consistent with AMICA's documented under-convergence. Phase 3 ICLabel filters at the locked 0.69 brain threshold. Dipole figures show pop_dipplot's MRI volume render pinstripe pattern; helper now hides axes, residual pinstripe is pop_dipplot's own rendering style and acknowledged as cosmetic.

Test plan

  • Local: 5/5 tests green (test_list_eligible_subjects, test_write_qa_channels_csv, test_phase1_no_subjects, test_phase1_smoke, test_phase2_smoke).
  • Local: 3-subject Phase 2 run on real R3 data with IcaMethod=amica, MaxIter=2000. All 3 ok; channel counts consistent with Phase 1.
  • Local: figures regenerated after the export bug fix; IC 1 visible.
  • CI: matlab-tests workflow green (uses IcaMethod=runica).
  • eeg-qa-neuroscientist re-review on the final regenerated figures.

Follow-up

Phase 3 ICLabel will pick up where this leaves off. AMICA non-convergence + RV findings stay on the subject watch list per .context/research.md L92-94 escalation path.

Adds Phase 2 (AMICA + dipfit) on top of the Phase 1 + CI work already
on main. Smoke test wiring is covered by CI against the fixture; the
science-quality 3-subject derivatives are produced by a separate runner
against the full local R3 data (scripts/run_phase2_three_subjects).

src/matlab/phase2_amica.m (new entrypoint)
- Per-subject pipeline: load Phase 1 .set, run AMICA, attach weights,
  fit dipoles, render IC topos + dipole PNGs, save checkpoint, write
  qa_amica.csv row.
- MaxThreads defaults to feature('numcores') so the same code runs on
  a 12-core Mac and a 2-core Actions runner.

src/matlab/+hbn/ helpers
- run_amica.m  : wraps runamica15. Passes EEG.datfile (.fdt path) when
                 available; passing the EEG struct directly corrupts
                 argument parsing in the Fortran binary (forrtl-66
                 overflow on macOS). use_queue flag dropped (rejected
                 by this version with 'unknown flag').
- fit_dipoles.m: pop_dipfit_settings with standard MNI BEM + pop_multifit.
- save_ic_topo_figure.m   : first N=30 IC topographies PNG.
- save_dipole_figure.m    : pop_dipplot top view PNG.
- write_qa_amica_csv.m    : participant_id, status, n_components,
                            final_ll, iterations, reached_max_iter,
                            median_rv, num_models, max_threads,
                            duration_s, error_message.

tests/matlab/test_phase2_smoke.m
- Chains phase1 -> phase2 on the 60 s fixture. MaxIter=100,
  DoReject=false because the smoke test asserts wiring, not science.
  Header warns against interpreting RV from this test.

tests/matlab/run_all_tests.m
- Adds test_phase2_smoke (5 tests total, all green locally).

.github/workflows/matlab-tests.yml
- Auto-installs Biosig, amica, Fieldtrip-lite. Cache key bumped to
  v2-amica. timeout raised from 30 to 45 min.

scripts/run_phase2_three_subjects.m
- Standalone runner. Regenerates Phase 1 .set checkpoints if missing
  (they are gitignored), then runs phase2 with MaxIter=2000, DoReject=
  true on the full R3 data per the brief.

Refs #1, #3.
CI revealed a file-type detection mismatch in runamica15 on Linux:
'Need num_chans keyword argument with number of channels, unknown file
type'. The .fdt path worked locally (macOS AMICA 1.7) but the CI build
does not recognize the .fdt extension and would need num_chans passed
alongside.

The matrix path has no such ambiguity and the save-a-copy benefit of
.fdt is negligible at our channel counts and recording durations. Drop
the branch; keep the comment that documents why we never pass the EEG
struct directly (the original macOS forrtl-66 crash).

Refs #3.
The AMICA plugin ships amica15ex on Linux; that binary dynamically
links against MPICH's Fortran library libmpifort.so.12. Ubuntu runners
do not have it preinstalled, so AMICA fails immediately with 'error
while loading shared libraries: libmpifort.so.12'. Add an apt step
before MATLAB starts to install libmpich-dev.

Refs #3.
CI cannot run AMICA reliably: the precompiled Linux binary amica15ex
needs MPI runtime libs that Ubuntu runners do not have in a known-good
combination (libmpifort.so.12 via libmpich-dev did not resolve cleanly).
Rather than fight library versions on every Ubuntu image bump, switch
the CI smoke test to Infomax (runica), which is bundled with EEGLAB
core and runs in pure MATLAB. Production runs keep using AMICA.

phase2_amica.m
- New opt IcaMethod (default 'amica', mustBeMember={'amica','runica'}).
  Dispatches to hbn.run_amica (production) or hbn.run_infomax (CI/test).
  Downstream dipfit + figures + checkpoint code does not branch on
  method because both helpers populate EEG.icaweights / icasphere /
  icawinv the same way.
- IcaMethod recorded in params.json so the QA agent can read it.

src/matlab/+hbn/run_infomax.m (new)
- Thin wrapper around pop_runica('icatype','runica'). Sets max_iter
  from opts.MaxIter; reports stats with NaN final_ll/iterations
  (Infomax does not expose a log-likelihood like AMICA does).

src/matlab/+hbn/write_qa_amica_csv.m
- Add ica_method column so a row's IcaMethod is recoverable from the
  CSV alone (status='ok' + ica_method='runica' is the CI smoke shape;
  ica_method='amica' is the production shape).

tests/matlab/test_phase2_smoke.m
- Sets IcaMethod='runica'.
- Skips the finite-final_ll / iterations assertions when ica_method
  is not amica (Infomax does not expose those QA fields).
- requiredFields list includes IcaMethod.

Local: 5/5 tests green with runica on 60s fixture. Production 3-subject
runner (scripts/run_phase2_three_subjects) still uses the AMICA default.

Refs #3.
3-subject Phase 2 results plus the figure-export bug that the
eeg-qa-neuroscientist agent caught.

derivatives/amica/ (tracked via .gitignore allow-list)
- README.md, qa_amica.csv, params.json
- sub-<ID>/figures/sub-<ID>_ic-topos.png (first 30 ICs)
- sub-<ID>/figures/sub-<ID>_dipoles.png (top view)
Channel-count consistency: n_components matches Phase 1
n_channels_after for all 3 subjects (123 / 112 / 98).

Per-subject AMICA + dipfit results:
  sub-NDARAA948VFH: 123 ICs, final_ll=-1.481, iter=2000, RV=0.368
  sub-NDARAC853DTE: 112 ICs, final_ll=-2.221, iter=2000, RV=0.420
  sub-NDARAD774HAZ:  98 ICs, final_ll=-2.512, iter=2000, RV=0.368
All 3 hit reached_max_iter=true. Expected at single-task data size.
Subjects on pilot watch list; Phase 3 ICLabel filters.

Figure-export bug (caught by eeg-qa-neuroscientist on the first pass):
- save_ic_topo_figure.m and save_dipole_figure.m pre-allocated a
  figure handle and passed it to exportgraphics, but pop_topoplot
  and pop_dipplot open THEIR OWN figures. The pre-allocated handle
  stayed empty; exportgraphics wrote a blank canvas (byte-identical
  PNGs across subjects, the smoking gun).
- Both helpers now snapshot figure handles before/after the pop_*
  call and capture the new figure for export.
- save_ic_topo_figure drops sgtitle: it occluded ICs 1-5 (the
  highest-variance, most diagnostic components). pop_topoplot's own
  title is enough.
- save_dipole_figure turns off axes in the captured figure to reduce
  pop_dipplot's axis bleed.

scripts/regenerate_phase2_figures.m (new): rebuilds figures from
existing .set checkpoints. Used to recover from the empty-canvas bug
without redoing the 75 min of AMICA + dipfit.

.gitignore: derivatives/**/_amica/ + tmpflist denylisted (AMICA
binary outputs and scratch are heavy and gitignored).

Refs #3, refs #1.
@neuromechanist
neuromechanist merged commit 84cce91 into main May 13, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Phase 2: AMICA + dipfit

1 participant