Ensure equal sampling of age across cross validation folds#108
Ensure equal sampling of age across cross validation folds#108HippocampusGirl wants to merge 3 commits into
Conversation
We split age into five quantiles and then ensure that these are equally distributed across train and test using StratifiedShuffleSplit
There was a problem hiding this comment.
Pull request overview
Adjusts cross-validation splitting for the age regression task to better balance the age distribution between train/test folds by stratifying on age quantile bins.
Changes:
- Replaced
ShuffleSplitwithStratifiedShuffleSplitfor the regression CV strategy. - Added quantile binning (
pd.qcut) of the regression target to enable stratification. - Precomputed CV splits using the binned target labels.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| bins = pd.qcut(y_train, q=5, labels=False, duplicates="drop") | ||
| cv_strategy = StratifiedShuffleSplit(n_splits=n_splits, test_size=0.2, random_state=random_state) | ||
| splits = list(cv_strategy.split(np.zeros_like(bins), bins)) | ||
| cv_strategy = splits |
There was a problem hiding this comment.
StratifiedShuffleSplit can raise at runtime if the binned target has too few samples in one or more bins (e.g., a bin count < 2, or only a single unique bin after duplicates="drop"). Consider adding a guard/fallback (e.g., dynamically reduce q based on bin counts/unique values, or fall back to non-stratified splitting when stratification isn’t feasible) so CV doesn’t fail on small or low-variance datasets.
| cv_strategy = StratifiedShuffleSplit(n_splits=n_splits, test_size=0.2, random_state=random_state) | ||
| splits = list(cv_strategy.split(np.zeros_like(bins), bins)) | ||
| cv_strategy = splits |
There was a problem hiding this comment.
cv_strategy is used for two different concepts (first a splitter instance, then a concrete list of index splits). This makes the code harder to read/debug. Consider using distinct names like cv_splitter and cv_splits (and pass cv_splits into cross_validate) to keep intent clear.
|
|
||
| bins = pd.qcut(y_train, q=5, labels=False, duplicates="drop") | ||
| cv_strategy = StratifiedShuffleSplit(n_splits=n_splits, test_size=0.2, random_state=random_state) | ||
| splits = list(cv_strategy.split(np.zeros_like(bins), bins)) |
There was a problem hiding this comment.
Using np.zeros_like(bins) as a dummy X is non-obvious, and it depends on bins shape/dtype semantics. Prefer a clearer placeholder that only encodes length (e.g., np.empty((len(bins), 1))), and/or add a short comment explaining that X is unused and bins supplies the stratification labels.
| splits = list(cv_strategy.split(np.zeros_like(bins), bins)) | |
| # X is unused; we only need the correct number of samples for the splitter. | |
| splits = list(cv_strategy.split(np.empty((len(bins), 1)), bins)) |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #108 +/- ##
==========================================
- Coverage 54.03% 53.91% -0.12%
==========================================
Files 26 26
Lines 1414 1417 +3
==========================================
Hits 764 764
- Misses 650 653 +3
🚀 New features to boost your workflow:
|
We split age into five quantiles and then ensure that these are equally distributed across train and test using StratifiedShuffleSplit. This may fix unusually high variability of model scores in the OASIS dataset.