Status: PROPOSAL (draft)
Scope: All data-extraction code in CanlabCore — object methods (@fmri_data, @image_vector, @region, @atlas) and the standalone Data_extraction/ (and peak_coordinates/) functions.
Audience: CanlabCore maintainers.
This document is grounded in a full read of the actual .m sources and a repo-wide call-graph/usage analysis (grep -rn --include='*.m'). Function signatures and call sites cited below were extracted from source, not guessed.
The toolbox currently exposes at least 15 distinct functions that all do some variant of "pull voxel values out of images, restrict to a mask/atlas/region, and (optionally) reduce to per-region means or pattern expression." They have accreted over ~20 years across three eras: (a) the legacy SPM clusters-struct era, (b) the standalone procedural era, and (c) the current object-oriented era. The result is redundancy, silent behavioral drift, and dead code.
Concrete symptoms found in the source:
-
Two near-identical standalone twins.
Data_extraction/extract_image_data.mandData_extraction/extract_from_rois.mshare essentially the same body and H1 text. They differ only in default averaging mode ('contiguous_regions'vs'unique_mask_values') and output argument order. Neither has any live caller — the onlyextract_image_data(...)call in the repo resolves to a local subfunction insideParcellation_tools/parcel_images.m, not the standalone. -
Two overlapping OO
extract_roi_averagesimplementations that have drifted.@fmri_data/extract_roi_averages.m([cl, cl_roisum, cl_demeanedpattern] = extract_roi_averages(obj, mask_image, varargin)) is the rich version with pattern expression,statistic_image/fmri_mask_imagespecial-casing, and atlas-aware resampling.@image_vector/extract_roi_averages.m(cl = extract_roi_averages(obj, mask, varargin)) is a thinner copy that explicitlyerrors on'pattern_expression'and whose actual default ('unique_mask_values') disagrees with its own help text (which says contiguous). Its own header admits: "Better to have only one function of record in the future." -
Inconsistent resampling direction across methods doing the same job.
extract_roi_averages(both),apply_mask, andapply_parcellationresample the mask/atlas → data space.@atlas/extract_dataresamples the data → atlas space (thenapply_parcellationre-matches internally).@region/extract_datadoes no resampling at all (mm-coordinate intersection). Same conceptual operation, three different space contracts — a correctness and reproducibility hazard. -
A documented-but-unimplemented method.
apply_atlasis named in the help headers of@image_vector/image_vector.m(lines 23, 143) and@fmri_data/fmri_data.m(lines 23, 141) as "Computes the mean value or pattern expression for each reference region specified in an atlas object," but noapply_atlas.mfile exists anywhere. Users following the docs hit a missing method. The real implementations are@image_vector/apply_parcellation.mand@atlas/extract_data.m. -
An unresolved merge-conflict file shipped in the repo.
Data_extraction/sphere_roi_tool_2008_conflict.mliterally contains<<<<<<< .mine/=======/>>>>>>> .r1197markers (lines 1-5, 43-52, 59-65) and a duplicatefunction sphere_roi_tool_2008definition. It cannot parse and is a name-collision hazard. -
A fully dead
Grandfathered/folder.Data_extraction/Grandfathered/timeseries4.mandcheck_timeseries_vals.mhave no external callers; every live call resolves to local subfunctions insidecluster_orthviews.m. -
A legacy clusters-struct chain with an orphaned head.
extract_raw_data→tor_extract_rois/extract_indiv_peak_data.extract_raw_datahas no live external caller, andextract_indiv_peak_datais reachable only through it. -
Doc-quality debt.
poorly_documented_functions.txtflagsapply_mask(line 27, 0/3),check_timeseries_vals(200),timeseries4(201),sphere_roi_tool_2008+ its conflict twin (204-205), andextract_raw_data(521).
The cost: contributors cannot tell which function is canonical; bug fixes land in one copy and not its twin (e.g. the @fmri_data version got the 'nearest'-interp fix for integer atlas codes and the atlas2region fix; the @image_vector version did not); and users are pointed (by help text) at dead standalones and a non-existent apply_atlas.
There should be exactly one computational core for "restrict-to-mask + reduce." Every public method becomes a thin, well-documented wrapper that normalizes its inputs to that core and shapes the outputs to its return type. No two functions should independently re-implement voxel intersection, weight normalization, or pattern expression.
The strongest existing candidate for the core is @image_vector/apply_parcellation.m plus canlab_pattern_similarity (the pattern-expression primitive that almost everything already funnels through). Reasons it should be the engine of record:
- It already produces the full superset of useful outputs:
[parcel_means, parcel_pattern_expression, parcel_valence, rmsv_pos, rmsv_neg, voxel_count, parcel_ste]. - It handles the integer-label-preserving resample correctly (
resample_space(parcels, dat, 'nearest')), full-width output viacondf2indic(parcels.dat, 'integers', n_orig_parcels), and theprobability_maps = []speedup for atlases. - It computes true weighted means via a scaled matrix product and routes pattern expression through
canlab_pattern_similarity, which is the same primitiveapply_mask,@region/extract_data, and@fmri_data/extract_roi_averagesalready call. @atlas/extract_datais already a thin wrapper over it.@fmri_data/extract_measures_batchalready calls it directly.
Recommended concrete structure:
canlab_extract_core(data_obj, mask_or_atlas, 'reduce', <mean|pattern|both>, ...) % NEW or = refactored apply_parcellation
|
|-- handles: input normalization (char/struct/stat_image/atlas -> canonical),
| ONE documented space contract (resample mask/atlas -> data, 'nearest' for integer labels),
| voxel intersection, weight L1/L2 normalization, canlab_pattern_similarity dispatch
v
returns the full matrix superset (means, pattern, valence, rmsv, counts, ste)
Wrappers (public, stable names, shape outputs only):
@fmri_data/extract_roi_averages -> region-object output (cl, cl_roisum, cl_demeanedpattern)
@image_vector/extract_roi_averages-> region-object output (mean only) — OR deprecate-merge (see table)
@atlas/extract_data -> matrix/table output (already thin)
@region/extract_data -> region-object output, KEEPS its mm-coordinate path (distinct contract — see below)
apply_parcellation -> public alias / direct core entry (keep name; it IS the core)
apply_mask -> KEEP as the global (non-region-aware) masking primitive
@region/extract_data must not be naively folded into the resampling core. Its mm-coordinate intersection (mm2voxel + intersect(...,'rows'), no interpolation) is a different and intentional contract — its header explicitly notes results differ from a resample-then-extract approach. It should keep its coordinate path but share the reduction step (mean + canlab_pattern_similarity weighting) with the core via a small shared helper, so the "what does pattern expression mean" logic lives in one place.
Pick one documented default: resample the mask/atlas/parcels to the data space, using 'nearest' for integer-labeled parcellations and the atlas-aware path for atlases. @atlas/extract_data's data→atlas direction should be retained only as an explicit, documented performance option ('resample','data_to_atlas') for large-N, not as a silent divergence. Every wrapper documents which contract it uses and why.
- Either implement
apply_atlasas a one-line alias toapply_parcellation/extract_data, or remove it from the class help headers. (Recommend: add a thinapply_atlasalias so the documented name resolves — lowest-surprise for users.) - Correct the
@image_vector/extract_roi_averageshelp/default mismatch. - Repoint the "non-object-oriented alternative" help in
@fmri_data/extract_roi_averages.m:96and@atlas/extract_data.m:114away from the deadextract_image_datastandalone.
Verdicts: KEEP-CORE (canonical, becomes/anchors the engine) · KEEP (distinct, still needed) · CONSOLIDATE-INTO-X (merge behavior into X, leave thin wrapper) · DEPRECATE (keep callable, emit warning, schedule removal) · REMOVE (delete now or end of plan).
| Function | Path (relative to CanlabCore/) | Live callers | Disposition | Rationale & risk |
|---|---|---|---|---|
@image_vector/apply_parcellation |
@image_vector/apply_parcellation.m |
5 (atlas/extract_data, fmri_data/extract_measures_batch, image_vector/wedge_plot_by_atlas) | KEEP-CORE | Becomes the consolidation target / engine. Produces the full output superset; already the computational backbone. Risk: low; central, so any refactor must be test-guarded hard. |
@image_vector/apply_mask |
@image_vector/apply_mask.m |
57 (13 distinct @class methods incl. @statistic_image/threshold) |
KEEP-CORE | Most-used masking primitive in the toolbox; global (one-mask) masking, not region-aware. Keep as the masking layer the core uses. Doc debt: flagged 0/3 (poorly_documented_functions.txt:27) — improve header. Risk: very high blast radius; do NOT change semantics, only document. |
@fmri_data/extract_roi_averages |
@fmri_data/extract_roi_averages.m |
~5 (region/region.m, fmri_data/canlab_connectivity_preproc) | KEEP-CORE (wrapper) | The primary OO ROI extractor and the version with the bug fixes (nearest-interp for unique values, atlas2region). Re-home its compute onto the core; keep its region-object output contract and the cl_roisum/cl_demeanedpattern outputs. Risk: medium — output struct/region shape must be byte-stable for callers. |
@image_vector/extract_roi_averages |
@image_vector/extract_roi_averages.m |
tests + inheritance | CONSOLIDATE-INTO @fmri_data version / core | Thinner drifted copy; errors on pattern expression; help/default mismatch; its own header asks for a single function of record. Make it delegate to the shared core (mean-only path) so the two cannot drift again. Risk: medium — it is the inherited superclass method; subclasses other than fmri_data rely on it. Must verify class dispatch unchanged. |
@atlas/extract_data |
@atlas/extract_data.m |
1 (region/ttest_table_by_condition) + public | KEEP (thin wrapper) | Already delegates to apply_parcellation. Keep; align its space contract documentation (data→atlas) as an explicit option. Risk: low. |
@region/extract_data |
@region/extract_data.m |
1 (region/ttest_table_by_condition) + public | KEEP (distinct contract) | mm-coordinate, no-resample path is intentional and unique. Share only the reduction helper with the core. Risk: low if coordinate path untouched. |
@image_vector/extract_gray_white_csf |
@image_vector/extract_gray_white_csf.m |
callers via batch | KEEP | Purpose-built tissue-compartment summaries; thin over apply_mask. No redundancy. Risk: none. |
@fmri_data/extract_measures_batch |
@fmri_data/extract_measures_batch.m |
orchestrator | KEEP | Orchestration only; already calls the core (apply_parcellation) directly. Risk: none. |
@region/check_extracted_data |
@region/check_extracted_data.m |
QC | KEEP | Standalone QC sanity check. Minor latent bug (isok overwritten each loop) — fix opportunistically. Risk: none. |
extract_image_data |
Data_extraction/extract_image_data.m |
0 live | DEPRECATE → REMOVE | Orphaned standalone twin of extract_from_rois; superseded by OO methods. The OO help text falsely advertises it. Deprecate with a pointer to extract_roi_averages, remove in Phase 3. Risk: low — but external/downstream scripts may call it; hence deprecate-first, not delete-now. |
extract_from_rois |
Data_extraction/extract_from_rois.m |
0 live | DEPRECATE → REMOVE | Older duplicate of the above (different default + arg order). Same path. Risk: low; deprecate-first for downstream safety. |
extract_raw_data |
Data_extraction/extract_raw_data.m |
0 live external | DEPRECATE (KEEP-LEGACY-WRAPPER) | Head of a legacy EXPT/clusters chain; no live external caller but it drives tor_extract_rois + extract_indiv_peak_data. Deprecate; do not remove until the chain is retired. Doc-flagged (521). Risk: medium — old user scripts/EXPT pipelines. |
extract_indiv_peak_data |
Data_extraction/extract_indiv_peak_data.m |
1 (only extract_raw_data) |
DEPRECATE → REMOVE (with its parent) | Dead once extract_raw_data goes. Remove together. Risk: low. |
tor_extract_rois |
Data_extraction/tor_extract_rois.m |
34 across ~14 legacy files | KEEP-LEGACY-WRAPPER | High usage but only from legacy procedural/cluster tooling (mask2clusters, cluster_orthviews, mask_*, parcel_*, hewma_*, classify_naive_bayes*); never from an @class method. Long pole — keep until those tools migrate. Note local-subfunction shadow in cluster_orthviews.m:503. Risk: high to remove; safe to keep. |
extract_contrast_data |
Data_extraction/extract_contrast_data.m |
2 (cluster_tool_getbetas) |
KEEP-LEGACY-WRAPPER | Used only by the GUI cluster_tool; chains to tor_extract_rois + extract_ind_peak. Keep until GUI migrates. Risk: medium. |
extract_ind_peak |
peak_coordinates/extract_ind_peak.m |
2 (cluster_barplot, extract_contrast_data) |
KEEP-LEGACY-WRAPPER | Legacy individual-peak helper. Name-collision hazard with extract_indiv_peak_data. Keep until its two callers retire. Risk: medium. |
canlab_maskstats |
Data_extraction/canlab_maskstats.m |
2 (canlab_glm_maskstats) |
KEEP-LEGACY-WRAPPER | Overlaps apply_mask 'pattern_expression' + extract_roi_averages but still wired into the GLM_Batch_tools pipeline. Keep until that pipeline is refactored, then DEPRECATE. Risk: medium. |
canlab_load_ROI |
Data_extraction/canlab_load_ROI.m |
active | KEEP | Current, object-aware (returns region/atlas). Not an extractor of data — a region loader. No change. Risk: none. |
canlab_extract_ventricle_wm_timeseries |
Data_processing_tools/canlab_extract_ventricle_wm_timeseries.m |
active | KEEP | Current; aCompCor-style nuisance extraction with PCA comps; not replaced by extract_gray_white_csf. Risk: none. |
timeseries_extract_slice |
Data_extraction/timeseries_extract_slice.m |
active | KEEP | Low-level single-slice I/O; no object equivalent. Risk: none. |
sphere_roi_tool_2008 |
Data_extraction/sphere_roi_tool_2008.m |
4 (ROI-building scripts) | KEEP | Active, clean, has 2023/2025 examples bridging to region via cluster2region. Risk: none. |
sphere_roi_tool_2008_conflict.m |
Data_extraction/sphere_roi_tool_2008_conflict.m |
0 | REMOVE NOW | Unresolved merge-conflict file; will not parse; duplicate function name. No callers. Zero risk. Doc-flagged (205). |
Grandfathered/timeseries4.m |
Data_extraction/Grandfathered/timeseries4.m |
0 (live calls hit local subfn in cluster_orthviews.m) |
REMOVE (end of Phase 1) | Fully dead in Grandfathered/. Doc-flagged (201). Risk: very low — verify the cluster_orthviews.m:765 local subfunction shadow is the actual resolver before deleting. |
Grandfathered/check_timeseries_vals.m |
Data_extraction/Grandfathered/check_timeseries_vals.m |
0 (used only by sibling Grandfathered file) | REMOVE (with timeseries4) | Dead validation helper. Doc-flagged (200). Risk: very low. |
apply_atlas (documented, no file) |
(none) | — | CREATE alias OR REMOVE from docs | No file exists; named only in class help headers. Add a thin alias to apply_parcellation, or strike from help. Risk: low; user-facing doc correctness. |
Each phase is gated by canlab_run_all_tests passing. Tests are function-based (functiontests(localfunctions)), discovered by the custom glob in Unit_tests/canlab_run_all_tests.m (files must be named canlab_test_*.m). Fixtures come from Unit_tests/helpers/canlab_get_sample_fmri_data.m (emotionreg, 30 images) and canlab_get_sample_thresholded_t.m.
git rm Data_extraction/sphere_roi_tool_2008_conflict.m(broken, 0 callers).- Confirm via
grep -rn 'timeseries4\|check_timeseries_vals'that the only resolvers for the live calls are the local subfunctions incluster_orthviews.m; thengit rmthe twoGrandfathered/files (or, if any doubt remains, defer their deletion to end of Phase 1 after the characterization tests below exist). - Commit. Run
canlab_run_all_tests; expect green (these files have no callers, so nothing should change).
Goal: lock in current behavior of the KEEP-CORE functions before refactoring, and start warning on the doomed ones.
Steps:
- Add value-correctness tests (extend, do not duplicate, the existing
Unit_tests/image_vector/canlab_test_extract_roi.m). The existing file already covers(data, mask_filename)return type and.dat-row-count==n-images. Add new local test functions covering the currently uncovered ground that the refactor will touch:- Synthetic known-value test: build an
fmri_datawhose.datis a known constant (e.g. all 7s) overbrainmask_canlab.nii, runapply_parcellationagainstload_atlas('canlab2024'), andverifyEqualthat every parcel mean == 7 (within tolerance). Pin the analytic mean so the core refactor is guarded. - Atlas-input /
'unique_mask_values'multi-region path (the case the oldold_to_integrate/check_roi_extraction.mexercised) — verify number of regions and per-region means. - On-the-fly resampling: extract with a mask in a different space; verify it runs warning-free and returns sane shapes.
- Pattern-expression equivalence: verify
@fmri_data/extract_roi_averages(..., 'pattern_expression')and the correspondingapply_parcellation(..., 'pattern_expression', w)agree within tolerance on a shared ROI — this is the cross-check that lets us merge the two later. - Cross-method space-contract test: document/verify current resampling direction of each method so changes are visible.
- Assertion style per house convention:
tc.verifyEqual,tc.verifyClass,tc.assumeNotEmpty(which('brainmask_canlab.nii'), ...)to skip cleanly when data is off-path.
- Synthetic known-value test: build an
- Add deprecation warnings (callable, no behavior change) to:
extract_image_data,extract_from_rois,extract_raw_data,extract_indiv_peak_data. Use a standard one-liner at the top of each:Warnings must NOT change return values (downstream scripts keep working).warning('CanlabCore:deprecated', ... '%s is deprecated and will be removed. Use @fmri_data/extract_roi_averages or @atlas/extract_data instead.', mfilename);
- Fix docs-vs-reality now (cheap, high value): correct the
@image_vector/extract_roi_averageshelp/default mismatch; repoint the "non-object-oriented alternative" pointers (@fmri_data/extract_roi_averages.m:96,@atlas/extract_data.m:114); add theapply_atlasalias (or strike it from the class headers). - Run
canlab_run_all_tests. Green gate. The new characterization tests now define "correct."
How tests guard: the synthetic known-value and pattern-expression-equivalence tests pin the exact numerical contract of the core before any code moves, so Phase 2 cannot silently alter results.
Goal: make every wrapper delegate to one engine; eliminate drift.
Steps:
- Factor the shared reduction logic (voxel intersection + weight L1/L2 normalization +
canlab_pattern_similaritydispatch + the full output superset) into the core (apply_parcellation, possibly fronted by acanlab_extract_corehelper). Keepapply_parcellation's public signature unchanged. - Re-point
@fmri_data/extract_roi_averagesto compute via the core, then shape intoregionobjects (cl,cl_roisum,cl_demeanedpattern). Preserve the exact output struct/region fields. - Make
@image_vector/extract_roi_averagesdelegate to the same core (mean-only path). Remove the divergent inline resampling. This kills the fmri_data-vs-image_vector drift. - Unify the space contract: single documented default (mask/atlas → data,
'nearest'for integer labels), with@atlas/extract_data's data→atlas as an explicit documented'resample'option. - Share the reduction helper into
@region/extract_datawhile preserving its mm-coordinate intersection path. - After each step, run
canlab_run_all_tests. The Phase 1 tests (value-correctness, pattern-expression equivalence, shape, cross-method agreement) are the guardrail: any numeric or shape change fails the gate.
How tests guard: because Phase 1 pinned analytic means and cross-method equivalence, a refactor that changes resampling, normalization, or output shape fails immediately. The pattern-expression-equivalence test specifically guards the riskiest merge (fmri_data ↔ image_vector ↔ apply_parcellation).
Goal: delete what Phase 1 warned about, once a deprecation window (recommend: one tagged release / ~6 months) has elapsed.
Steps:
- Remove
extract_image_data.mandextract_from_rois.m(0 callers, deprecated since Phase 1). - Remove
extract_raw_data.mtogether withextract_indiv_peak_data.m(its only caller) — only if the EXPT/clusters legacy pipeline is confirmed retired. Otherwise keep them as KEEP-LEGACY-WRAPPER and re-evaluate next cycle. - Remove the
Grandfathered/directory if not already done in Phase 0. - Re-run the full suite + a spot-check of
CANlab_help_exampleswalkthroughs (see Risks). - For functions that must keep their public name for backward compat but whose body is gone, leave a thin wrapper that calls the core and emits the deprecation warning (see §5).
How tests guard: removal of zero-caller functions cannot break the suite by definition; the walkthrough spot-check (Phase 3 step 4) catches example scripts that the unit suite does not exercise.
- Deprecate, never silently delete, anything with a non-zero (or uncertain) external surface. Functions named in user-facing help, in
CANlab_help_examples, or plausibly called by downstream repos get a deprecation warning first and a removal only after a full release cycle. - Keep function names as thin wrappers. When
extract_image_data/extract_from_roisbehavior is fully subsumed by the OO methods, the name can survive as a one-screen wrapper that (a) emitswarning('CanlabCore:deprecated', ...)once, and (b) forwards toextract_roi_averages/apply_parcellation, mapping args and return order. This keeps old scripts running while steering new code to the core. - Stable public signatures.
apply_parcellation,apply_mask,@fmri_data/extract_roi_averages,@atlas/extract_data,@region/extract_datakeep their exact input/output signatures throughout. Internal re-homing onto the core must not change argument names, option strings, or output order/fields. - Warning hygiene. Use a single warning identifier namespace (
CanlabCore:deprecated) so users canwarning('off', 'CanlabCore:deprecated')if needed, and so the unit tests can assert deprecation warnings deliberately (verifyWarning) withoutverifyWarningFreechecks elsewhere tripping. apply_atlasresolution. Add the documented-but-missingapply_atlasas a real (thin) alias rather than removing the promise from the docs — least surprising for users who copied the help.- Changelog + deprecation table. Maintain a short "deprecated / removed" list in the docs and release notes so downstream maintainers can grep their code ahead of removal.
-
CANlab_help_exampleswalkthroughs and tutorials are not exercised by the unit suite (the runner skipswalkthroughs/by default). They are the most likely place to callextract_roi_averages/extract_data/apply_parcellationwith subtle expectations about output shape. Mitigation: before any Phase 3 removal and after Phase 2 consolidation, run a grep ofCANlab_help_examplesfor every touched function name and execute the affected walkthroughs headless. -
Downstream CANlab repos (e.g.
canlab_single_trials,CanlabPrivate, project analysis scripts) may call the standaloneextract_image_data/extract_from_rois/extract_raw_dataeven though there are zero callers inside CanlabCore. The repo-internal call-graph cannot see them. Mitigation: this is exactly why these are DEPRECATE-then-REMOVE, not REMOVE-now; the warning gives downstream a release cycle to migrate. -
Output-shape sensitivity.
@fmri_data/extract_roi_averagesreturnsregionobjects with specific fields (.dat,.all_data,.val,.val_descrip) and the secondarycl_roisum/cl_demeanedpatternoutputs populated only under'pattern_expression'+nargout>1. Callers like@region/region.m:286-288andcanlab_connectivity_preprocdepend on this. Mitigation: Phase 1 characterization tests pin these fields; refactor must keep them byte-stable. -
Resampling-direction change is a correctness risk. Unifying the space contract could shift values for code that implicitly relied on
@atlas/extract_data's data→atlas direction (different interpolation than mask→data). Mitigation: keep data→atlas as an explicit option; add the cross-method agreement test in Phase 1; document the chosen default loudly. -
Local-subfunction shadowing.
tor_extract_rois,timeseries4,check_timeseries_vals, andextract_image_dataeach have like-named local subfunctions insidecluster_orthviews.m/parcel_images.mthat currently satisfy the live calls. Deleting the standalone is safe only if the local subfunction is the true resolver. Mitigation: confirm withwhich -all/ a path-order check in MATLAB before each deletion (Phase 0 step 2, Phase 3 step 3). -
Name-collision hazards between
extract_ind_peak(peak_coordinates) andextract_indiv_peak_data(Data_extraction) — similar purpose, different signatures. Removing one must not change which the other's callers resolve to. Mitigation: verify callers explicitly; these are KEEP/DEPRECATE, not REMOVE-now. -
The custom test runner only discovers
canlab_test_*.m. New guard tests must follow that exact naming or they will silently not run. Mitigation: add new local functions into the existingcanlab_test_extract_roi.m(and a newcanlab_test_apply_parcellation.mif warranted), and confirm they appear in thecanlab_run_all_testssuite count.
Delete the obviously dead/broken items now (conflict file, Grandfathered/), pin current behavior with synthetic value-correctness and cross-method-equivalence tests, converge the OO extract_*/apply_parcellation methods onto a single space-correct core (apply_parcellation + canlab_pattern_similarity) while keeping @region/extract_data's mm-coordinate path distinct, and deprecate-then-remove the orphaned standalones over one release cycle behind CanlabCore:deprecated warnings.