Skip to content

DATA_PROVENANCE.md

Latest commit

 

History

History
215 lines (187 loc) · 12.1 KB

DATA_PROVENANCE.md

File metadata and controls

215 lines (187 loc) · 12.1 KB
Raw

Data Provenance — unified repo

Phase 1 complete: 2026-05-18. Authoritative record of the data corpus, its derivation chain, and the six integrity remediations. Verified against actual files (not agent summaries / stale state-logs).

1. Canonical corpus = PI data/labels/ (proven strict superset)

Adopted verbatim from ilae-skill-certification-test-multi-main-PI-code (MD5-verified 1:1 copy: 186 label files identical, labels.csv md5 match).

"Nothing lost" proof (direct, multiplicity-aware): every one of MINE's 1,303,201 labels.csv observations is present in PI's labels.csv with ≥ multiplicity (0 lost). Headers identical. The 3 shared source_datasets are exactly equal in both (sn1_combined_v2 964,206; sparcnet50K 290,268; pd_rda_profiler 48,727); PI adds 792,592 Centaur-2025 + Kong-2025 rows.

Post-gold-ingest canonical table sizes (data rows):

table PI base + gold ingest unified
labels.csv 2,095,793 +20,000 2,115,793
segments.csv 95,327 0 95,327
raters.csv 5,303 +3 5,306
datasets.csv 4 +1 5
segment_labels.csv 84,555 (STALE) regenerated 95,327

2. Derivation chain (ground truth from the reference repo)

raw annotations (SN1_combined_v2.h5 [EXTERNAL, PHI, never vendored - D9]
                 + sparcnet50K + pd-rda-profiler + Centaur2025 + Kong2025)
  -> build_unified_labels.py  -> labels/segments/raters/datasets.csv
        (base build is NOT re-runnable here: needs the external PHI h5,
         the absent pd-rda-profiler repo, and a hardcoded ROOT. PI's
         ingest_*.py appended Centaur2025/Kong rows to the 4 tables.)
  -> Phase-1 ingest_centaur_iiic_expert.py -> +20,000 gold rows
  -> Rasch main effects (reference fit_main_effects.py): per-case c_j +
     per-rater l_i on the LOGIT scale  ->  x(1/1.7) logit->probit bridge
  -> probit-lapse MLE (lambda=0.025 fixed): per-rater (sigma_hat, theta_hat)
  -> Youden on TRAIN pool (experts 70/30): sigma*/ell*  [Phase 3 recompute]
  -> engine_inputs/ (Phase 5 regen) + deployment_prior/ (Phase 4 re-freeze)

Invariants to enforce downstream (reference ground truth): LOGIT_TO_PROBIT = 1/1.7; lambda = 0.025; sigma*/ell* on TRAIN pool only; expert split 70/30 (reference CLAUDE.md's "50/50" is STALE); the EXPERTS set is the authoritative expert membership (not gold_standard_raters.yaml).

3. The six integrity remediations (as executed)

  • R1 - segment_labels.csv regenerated. Was byte-identical & STALE in both repos (84,555 rows, pre-Centaur/Kong). Regenerated by the verbatim pure function build_segment_labels(labels_df, segments_df) (zero logic drift) on the post-gold canonical tables -> 95,327 rows (one per segment), full original 33-column schema incl. iiic_vote_other (the K=7 task). Script: pipeline/regen_segment_labels.py.
  • R2 - consolidated/ NOT shipped. It is a losslessly reconstructible, zero-consumer convenience view (no engine/deployment code reads it). Canonical tier truth = raters.csv.expertise_level. The script is kept for ad-hoc use only (pipeline/consolidate_label_tiers.py).
  • R3 - Centaur double-count eliminated. The original consolidate script unioned labels.csv (disjoint) raw Centaur (conservation hardwired to the pre-ingest corpus). PI's labels.csv already ingested Centaur 2025. The carried script is R3-corrected to tier ONLY the canonical labels.csv; raw Centaur stays provenance-only and is never re-unioned. Validated: lossless round-trip on all 2,115,793 rows.
  • R4 - legacy Sigma_l_fitted.npy archived. The frozen 6x6 (15-rater era; max corr drift 0.715 vs current) moved to archive/. The engine migrates to PI's fitted 12x12->14x14 Sigma (deployment_prior / fits_hier); consistent with the methodology repo's own AUDIT.
  • R5 - rater_id namespace pre-flight (findings recorded). Gold ingest introduced 0 rater_id collisions (97, 99000001/2/3 each appear once). KNOWN, deferred to Phase 5: engine_inputs/sdt_fits.csv rater_id is a domain-local index (42 ids 0,1,10...), NOT a PI raters.csv rater_id - must join by rater_name, never numeric id. cross_domain_rater_matrix.csv is name-keyed; 4/29 names need crosswalk resolution against PI canonical_name (Hiba Haider->Hiba Arif [documented marriage name change], Osman Gamaleldin->Gamal Osman, Zubeda Karim->Zubeda Sheikh, Aaron Struck). Phase 5 regenerates engine_inputs from the PI corpus, resolving identities by construction; a Phase-1 test asserts the gold-ingest collision-freeness now.
  • R6 - Full-7 / K=7 (data side). The real 7th task is the SPARCNET other/IIC class, present in canonical labels.csv as pattern_class=='other' and (post-ingest) in the gold panel. PI's single-task fits/iic/ exists; the hierarchical fits are K=6 and are extended to K=7 in Phase 4. No data is missing for K=7.

4. Centaur-IIIC gold panel ingest (Phase-1 decision)

The 4-expert gold panel (centaur_iiic_expert_labels.xlsx, 5000 cases x {mbw,cal,matt,tianyu}) existed ONLY in MINE's raw files (absent from PI's labels.csv) and is the Paper-1 Centaur-IIIC validation reference.

  • Join (asserted, not assumed): case_id -> external/centaur_2025/ case_id_to_seg_id.csv -> seg_id; all 5000/5000 cases map; every seg_id exists in segments.csv AND already carries the aligned centaur_2025_iiic novice reads (panel is purely additive).
  • Taxonomy: gold is 7-class {...,bipd,birds}; canonical is 6-class. Per AUDIT_centaur_iiic_novice_expert.md section 6 ("the mapping is clean: collapse {bipd,birds}->other -> exact 6-class compatibility"), ingested with {bipd,birds}->other (1,852 of 20,000 collapsed). Native 7-class is losslessly recoverable from the raw xlsx kept verbatim under data/labels/external/centaur_iiic_goldpanel_raw/ (Paper-2 / native-7).
  • rater_ids: mbw -> existing canonical 97 (M. Brandon Westover); cal/matt/tianyu -> new collision-free block 99000001/2/3 (expertise_level=expert). source_dataset = centaur_iiic_expert.
  • Method: pipeline/ingest_centaur_iiic_expert.py - repo-relative paths, idempotent (aborts if already ingested), APPEND-ONLY (the 2,095,793 prior labels rows are a byte-identical prefix; verified), reversible (*.bak.*.csv). Provenance sidecar: external/centaur_iiic_goldpanel_raw/INGEST_PROVENANCE.json.

5. Carry-forwards & dispositions

item from unified location role
labels/segments/raters/datasets, fits*/, deployment_prior/, external/centaur_2025/ PI data/... canonical
SENSITIVE.md MINE data/SENSITIVE.md PHI governance
engine_inputs/ (4 files) MINE data/engine_inputs/ carried as-is; Phase 5 regenerates from PI corpus
raw Centaur (novice/expert/survey/AUDIT) MINE data/labels/external/centaur_iiic_goldpanel_raw/ provenance only; never re-ingested
Sigma_l_fitted.npy MINE root archive/ legacy/frozen (R4)
8 shared aux files (annotations.csv, channel_.json, discharge_times.json, rda_wave_labels.json, raters_aliases_, README.md) byte-identical MINE==PI data/labels/ (PI copy) unchanged
consolidated/ - NOT shipped (R2) reconstructible view

SN1_combined_v2.h5 (PHI EEG source) stays EXTERNAL and is never vendored (D9); datasets.csv records its S3 path only.

6. Phase 5 — engine_inputs provenance reconciliation (executed 2026-05-19)

Audit (referencing the actual code) found the original Phase-5 plan stale vs reality — §2/§3/§5 above said "Phase 5 regenerates engine_inputs from the PI corpus"; that framing is superseded by what was actually found + done:

Findings. (1) scripts/build_engine_inputs.py read the legacy out-of-repo prepared fits and produced the 219-row legacy artifact — re-running it would regress engine_inputs + violate D9. (2) The live data/engine_inputs/sdt_fits.csv (14,214 rows, 6 IIIC domains) is already the correct Phase-3 recompute on the unified data/labels, true producer = pipeline/ run_unified_calibration.py STEP d, already suite-gated. (3) README.md + (4) MANIFEST.json were both stale (claimed the legacy builder/source; output_sha256 did not match the live files; no data/labels D3 anchor). (5) sdt_fits.phase3.csv was a stray, NOT-engine-consumed snapshot differing materially from the authoritative file (σ Δ≈5.0 in ~1.9k rows). (6) the plan gate's data/consolidated/ does not exist (retired Phase 1).

Decisions (locked 2026-05-19) + outcome. No content regeneration (the Phase-3 artifact is authoritative + gated): sdt_fits.csv = authoritative (engine_paths.SDT_FITS); sdt_fits.legacy_oldcorpus.csv = the historical 219-row legacy (recorded, never rebuilt); sdt_fits.spike.csv = the Phase-3 spike-domain fits (kept). build_engine_inputs.py repurposed into a no-sibling MANIFEST verifier/regenerator that self-checks the gated contract (6 IIIC domains; 29 Q2-locked + 4 R5 name-variants join by canonical name) and cross-checks D3 — MANIFEST.json data_labels_provenance sha256 == deployment_prior/summary.json data_labels_provenance == the live data/labels/{labels,raters}.csv ⇒ engine_inputs and deployment_prior are provably the same unified corpus (D3, proven True). Stray sdt_fits.phase3.csv removed. MANIFEST.json + README.md rewritten truthfully. Gate = the already-green Phase-3 / Mode-A / Mode-B suite on sdt_fits.csv + tests/ test_phase5_engine_inputs.py; the nonexistent consolidated/ is dropped from the Phase-5 gate. (The §2/§5 "Phase 5 regenerates" wording above is retained for history but is superseded by this §6.)

7. Nature Medicine readiness audit (2026-05-28) — data-structure tour + gaps

A reviewer-grade audit was executed 2026-05-28 targeting a Nature Medicine submission. Findings live at docs/NATURE_MEDICINE_AUDIT.md with file:line references throughout. For an end-to-end tour of the data structures documented in this file (corpus tables, calibration intermediates, engine inputs, deployment prior, replay artefacts, curated banks) + the data-side gaps for Nature Medicine submission, read docs/NATURE_MEDICINE_AUDIT.md §3.

Data-side findings (summarised; full details in audit doc §6):

  • Single-institution clustering (E1). All 5 datasets trace to the MGH / BIDMC / Harvard / Yale circle under IRBs BIDMC 2016P000058 and MGH 2013P001024. No external US-academic, VA, community-hospital, or international site contributes signals.
  • Patient demographics nearly absent (E2). Sex 4.2 %, age 16.4 %, race/ethnicity 0 %, comorbidity 0 %, ICU/EMU/outpatient setting 0 %, etiology 0 %, encephalopathy grade 0 %. Forecloses fairness analyses without backfill from BDSP source records under IRB amendment.
  • Pediatric-heavy spike substrate (E3). Where age is recorded (n=15,670), median 16–21 y; 51 % pediatric vs adult-ICU deployment framing.
  • No site_id in segments.csv (E4). Forecloses site-disjoint CV.
  • Cross-dataset label-comparability never validated (E5). 5 datasets used different annotation protocols; no anchor-segment study; no bridge analysis.
  • Centaur IED panel (5,000 segs, 643 raters) not in K=7 banks (E6). Either implicit Paper-2 carry or undocumented gap.
  • Rater-pool fragmentation (E7). 2,872 spike-only + 2,047 IIIC-only
    • only 322 with both; median 54 segs/IIIC and 69 spike — bank exhaustion drives ≈50 % of REFER verdicts at floor=10.

Five Tier-1 actions on data integrity (docs/NATURE_MEDICINE_AUDIT.md §9) do not require new acquisition and close all four §4 (integrity / leakage) gaps and three of six §5 (methodology) gaps:

T1.# Title Effort
T1.1 Pin v13 ℓ* numerically in test suite 30 min
T1.2 Leakage-aware ℓ* sensitivity rerun (replay-disjoint calibration) 1–2 days
T1.3 Run scripts/run_lapse_sensitivity.py on unified corpus minutes
T1.4 Brier + CITL + reliability diagrams per task 4–6 hours
T1.5 Formal hypothesis test for replay-vs-Bernoulli 1–2 days

OPEN_DECISIONS items 6–15 carry the Nature Medicine deltas (docs/OPEN_DECISIONS.md).