Add files via upload

README_ADD_TO_GITHUB.md

@@ -0,0 +1,11 @@
+# How to add these files to the PHUSE GitHub repository
+
+1. Clone the repository or open your local copy.
+2. Checkout the add-missing-data-imputation-structure branch.
+3. Create a new working branch.
+4. Copy these folders into the repository root.
+5. Add the YAML snippet from datasets/datasets_yaml_snippet.yaml into datasets/datasets.yaml.
+6. Commit and push the branch.
+7. Open a Pull Request into main.
+
+Important: do not commit raw ADNI CSV data, patient IDs, subject-level predictions, imputed patient-level data, or any row-level outputs.

analysis_methods/simulation_designs/adni_artificial_missingness_benchmark.md

@@ -0,0 +1,142 @@
+# ADNI Alzheimer Longitudinal Missing-Data Imputation Benchmark
+
+## Objective
+Compare standard, machine-learning, and newer imputation methods for non-IID longitudinal Alzheimer clinical data.
+
+The benchmark evaluates:
+
+1. imputation quality under realistic missingness mechanisms;
+2. downstream clinical model performance after imputation.
+
+## Dataset structure
+The dataset is subject-visit level. Multiple rows may belong to the same subject. Observations are non-IID and all train/test splitting must be performed by subject ID.
+
+## Primary data type
+D2: longitudinal / repeated-measures clinical data.
+
+## Secondary data type
+D1: structured/tabular clinical data.
+
+## Missingness types
+The benchmark distinguishes:
+
+1. ordinary missingness;
+2. structural missingness;
+3. planned/protocol-driven missingness;
+4. longitudinal dropout;
+5. site/phase-related missingness.
+
+## Primary benchmark outcome
+Next-visit clinical diagnosis:
+
+- Dementia vs non-Dementia.
+
+Alternative multiclass outcome:
+
+- CN vs MCI vs Dementia.
+
+## Secondary benchmark outcomes
+- next-visit MMSE;
+- next-visit ADAS13;
+- change from baseline in MMSE;
+- change from baseline in ADAS13;
+- NPI-Q total symptom burden.
+
+## Data split
+All splitting must be subject-level:
+
+- 70% subjects for training;
+- 15% subjects for validation;
+- 15% subjects for testing.
+
+Rows from the same subject must never appear in both training and testing sets.
+
+## Imputation methods
+The benchmark may compare:
+
+1. complete-case analysis;
+2. median/mode imputation;
+3. median/mode imputation with missingness indicators;
+4. LOCF/BOCF sensitivity method;
+5. MICE / fully conditional specification;
+6. longitudinal or multilevel MICE;
+7. mixed-model-based imputation;
+8. KNN imputation;
+9. random-forest imputation / missForest-style methods;
+10. gradient-boosting-based imputation;
+11. deep-learning or sequence-aware imputation methods, such as BRITS/SAITS-style methods, if implementation is available.
+
+## Artificial missingness scenarios
+Observed values will be masked to create benchmark scenarios:
+
+1. MCAR: randomly mask observed cells.
+2. MAR: mask values depending on diagnosis, age, visit month, site, or ADNI phase.
+3. Block missingness: remove whole clinical domains such as imaging or biomarkers.
+4. Longitudinal dropout: remove future visits for selected subjects.
+5. Site/phase missingness: remove values depending on site or protocol phase.
+
+Recommended masking rates:
+
+- 10%;
+- 20%;
+- 30%;
+- 40%.
+
+## Imputation-quality metrics
+Continuous variables:
+
+- RMSE;
+- MAE;
+- normalized RMSE;
+- distributional similarity.
+
+Categorical variables:
+
+- accuracy;
+- balanced accuracy;
+- macro-F1.
+
+Longitudinal consistency:
+
+- within-subject trajectory plausibility;
+- clinically appropriate smoothness checks.
+
+## Downstream model metrics
+Classification:
+
+- AUROC;
+- AUPRC;
+- balanced accuracy;
+- F1 score;
+- Brier score;
+- calibration slope/intercept.
+
+Regression:
+
+- RMSE;
+- MAE;
+- R-squared;
+- calibration of predicted versus observed values.
+
+Robustness:
+
+- grouped cross-validation by subject;
+- site/phase sensitivity analysis;
+- feature-importance stability.
+
+## Data leakage prevention
+The imputer must be fit only on the training subjects. Validation and test data must be transformed using the fitted imputation process. Full-dataset imputation before train/test split is not allowed.
+
+## Reporting
+The final report should include:
+
+1. dataset metadata, not raw data;
+2. missingness summary by variable;
+3. missingness summary by visit;
+4. missingness summary by diagnosis group;
+5. missingness summary by site and ADNI phase;
+6. description of structural NPI-Q missingness handling;
+7. imputation benchmark results;
+8. downstream model results;
+9. sensitivity analyses;
+10. limitations and data-use restrictions.

datasets/34_adni_adnimerge_npiq/dataset_card.md

@@ -0,0 +1,50 @@
+# ADNI ADNIMERGE + NPI-Q Longitudinal Dataset Card
+
+## Dataset name
+ADNI ADNIMERGE + NPI-Q Combined Longitudinal Dataset
+
+## Data source
+Alzheimer's Disease Neuroimaging Initiative (ADNI).
+
+## Access status
+Controlled access. Data are available through ADNI/LONI only after approval and acceptance of the applicable ADNI Data Use Agreement.
+
+## GitHub data policy
+No raw participant-level ADNI data are included in this repository. This folder contains metadata only.
+
+## Dataset type
+D2: longitudinal / repeated-measures clinical data.
+
+Secondary type: D1 structured/tabular clinical data, because each subject-visit row contains structured clinical, cognitive, imaging, biomarker, and neuropsychiatric variables.
+
+## Clinical area
+Alzheimer's disease, mild cognitive impairment, cognitive decline, and neuropsychiatric symptoms.
+
+## Unit of observation
+Subject-visit level.
+
+## Non-IID structure
+Rows are not independent. A single subject may contribute multiple visits. Analyses must therefore split data by subject identifier rather than by row.
+
+## Key identifiers
+- RID: subject identifier
+- ADNI_EXAMDATE: ADNI visit date
+- NPIQ_VISDATE: NPI-Q visit date
+- ADNI_VISCODE: visit code
+- ADNI_SITE: clinical site
+- ADNI_COLPROT / ADNI_ORIGPROT: ADNI phase/protocol
+
+## Main clinical domains
+- Demographics
+- Diagnosis
+- Cognitive scores
+- Neuropsychiatric symptoms
+- MRI measures
+- PET biomarkers
+- CSF biomarkers
+
+## Missing-data relevance
+This dataset is suitable for benchmarking missing-data imputation because it contains repeated measures, subject-level correlation, irregular visit patterns, site and protocol effects, structural missingness in NPI-Q severity variables, and high missingness in selected imaging and biomarker variables.
+
+## Recommended benchmark use
+Approved ADNI users may run the benchmark in a secure local environment. GitHub materials should include only metadata, code templates, benchmark definitions, and aggregate results.

datasets/datasets.yaml

@@ -0,0 +1,8 @@
+# Add this block to datasets/datasets.yaml under the most appropriate section.
+# If the file does not already have a neurodegeneration section, this can be added as a new section.
+
+neurodegeneration_longitudinal:
+  - name: ADNI ADNIMERGE + NPI-Q
+    folder: 34_adni_adnimerge_npiq
+    access: Controlled
+    notes: "Longitudinal Alzheimer clinical dataset combining ADNI ADNIMERGE and NPI-Q variables. Suitable for non-IID missing-data imputation benchmarking. Raw participant-level data must not be committed to GitHub."

docs/d2_longitudinal_data/adni_adnimerge_npiq_missingness_profile.md

@@ -0,0 +1,37 @@
+# ADNI ADNIMERGE + NPI-Q Missingness Profile
+
+## Purpose
+This document defines how missingness should be interpreted and handled for the ADNI ADNIMERGE + NPI-Q longitudinal benchmark.
+
+## Missingness types
+
+### 1. Structural missingness
+NPI-Q symptom variables and severity variables require special handling. For each NPI-Q symptom domain, the symptom variable is binary and the severity variable is meaningful only when the symptom is present.
+
+Recommended rule:
+
+1. If symptom = 0, derive severity_for_model = 0.
+2. If symptom = 1, severity should be 1, 2, or 3.
+3. If symptom is missing, impute symptom first.
+4. If symptom is imputed as 0, set severity_for_model = 0.
+5. If symptom is imputed as 1, impute severity conditionally among valid positive severity levels.
+6. Do not apply ordinary continuous imputation directly to raw severity fields without respecting this structure.
+
+### 2. Planned or protocol-driven missingness
+Some biomarker and imaging variables may be missing because they were collected only in selected ADNI phases, visits, substudies, or modality-specific cohorts.
+
+Recommended handling:
+
+- keep missingness indicators;
+- perform primary models without extreme-missing variables;
+- perform domain-specific subcohort analyses for biomarkers or imaging;
+- avoid forcing full-dataset imputation for variables with extreme planned missingness.
+
+### 3. Ordinary clinical missingness
+Clinical and cognitive variables may be imputed using standard or ML methods, but the imputation model should include subject, visit, diagnosis, site, and phase information.
+
+### 4. Longitudinal dropout
+Later visits may be missing because of dropout, disease progression, loss to follow-up, death, or administrative reasons. Sensitivity analyses should evaluate dropout-related assumptions.
+
+## Non-IID rule
+All splitting and resampling must respect the subject-level structure. Rows from the same RID must not be split across training and testing datasets.

imputation_methods/d1_structured_data/machine_learning_imputation/adni_ml_imputation_plan.md

@@ -0,0 +1,31 @@
+# ADNI Machine-Learning Imputation Plan
+
+## Objective
+Compare ML-based imputation methods against standard statistical methods for ADNI subject-visit clinical data.
+
+## Candidate methods
+- KNN imputation;
+- random-forest imputation;
+- gradient-boosting-based imputation;
+- iterative imputation with ML estimators;
+- matrix factorization for high-dimensional clinical matrices.
+
+## Important restrictions
+All ML imputation models must be trained only on the training subjects. Test subjects must not be used to fit imputers.
+
+## Non-IID handling
+Because rows are clustered within subjects:
+
+- train/test split must be by RID;
+- cross-validation must be grouped by RID;
+- longitudinal features should include visit month and previous visit values;
+- performance uncertainty should use subject-level bootstrap or grouped cross-validation.
+
+## Benchmark comparison
+Each imputation method should be evaluated using:
+
+1. artificial masking accuracy;
+2. downstream clinical prediction;
+3. calibration;
+4. feature-importance stability;
+5. runtime and reproducibility.

imputation_methods/d2_longitudinal_data/longitudinal_multiple_imputation/adni_longitudinal_mice_plan.md

@@ -0,0 +1,49 @@
+# ADNI Longitudinal MICE Imputation Plan
+
+## Purpose
+Apply multiple imputation to ADNI longitudinal clinical data while respecting subject-level repeated measurements.
+
+## Required design features
+- Use subject ID as a clustering variable.
+- Include visit month/time as a predictor.
+- Include baseline diagnosis.
+- Include site and ADNI phase.
+- Include previous observed values where available.
+- Use variable-specific imputation models.
+
+## Variable-specific methods
+Continuous variables:
+
+- predictive mean matching;
+- linear regression;
+- mixed-effects imputation where available.
+
+Binary variables:
+
+- logistic imputation.
+
+Multicategory variables:
+
+- multinomial imputation.
+
+Ordinal variables:
+
+- proportional odds or ordered categorical imputation.
+
+NPI-Q severity variables:
+
+- conditional imputation based on symptom presence.
+
+## Multiple imputation
+Use multiple imputed datasets, for example:
+
+- m = 20 for the main analysis;
+- m = 50 for sensitivity analysis if missingness is high.
+
+## Pooling
+For statistical models, combine estimates using Rubin's rules.
+
+For ML prediction benchmarks, either:
+
+1. train and evaluate models separately in each imputed dataset and summarize performance; or
+2. stack imputations with appropriate subject-level grouping and sensitivity reporting.

imputation_methods/d2_longitudinal_data/mixed_model_based_methods/adni_mixed_model_imputation_plan.md

@@ -0,0 +1,41 @@
+# ADNI Mixed-Model-Based Imputation Plan
+
+## Purpose
+Use mixed-model-based imputation for longitudinal ADNI clinical variables where repeated measures within subjects are important.
+
+## Rationale
+Rows are correlated within subjects. Mixed models can represent subject-specific random effects and time trends, which are useful for longitudinal clinical scores such as MMSE, ADAS13, CDRSB, FAQ, and NPI-Q total score.
+
+## Candidate variables
+- MMSE
+- ADAS11
+- ADAS13
+- CDRSB
+- FAQ
+- MOCA
+- RAVLT measures
+- NPI-Q total score
+
+## Candidate predictors
+- baseline diagnosis;
+- visit month;
+- age;
+- sex;
+- education;
+- APOE4;
+- site;
+- ADNI phase;
+- previous observed clinical score;
+- other cognitive and clinical domains.
+
+## Model structure
+A basic mixed model may include:
+
+- fixed effects for time, diagnosis, demographics, site, and phase;
+- random intercept for subject;
+- random slope for time when supported by the data.
+
+## Sensitivity analyses
+- compare random-intercept and random-slope models;
+- compare models with and without site/phase effects;
+- evaluate robustness under dropout-related missingness assumptions.

phuse_adni_recommendations/README_ADD_TO_GITHUB.md

@@ -0,0 +1,11 @@
+# How to add these files to the PHUSE GitHub repository
+
+1. Clone the repository or open your local copy.
+2. Checkout the add-missing-data-imputation-structure branch.
+3. Create a new working branch.
+4. Copy these folders into the repository root.
+5. Add the YAML snippet from datasets/datasets_yaml_snippet.yaml into datasets/datasets.yaml.
+6. Commit and push the branch.
+7. Open a Pull Request into main.
+
+Important: do not commit raw ADNI CSV data, patient IDs, subject-level predictions, imputed patient-level data, or any row-level outputs.