Register the eICU dataset

eICU (multi-center US ICU) — `MEDS_extract-eICU` from the upstream
`eICU-MEDS` package handles the extraction. Unlike the sister per-dataset
PRs, eICU's upstream ships an end-to-end demo recipe (`do_demo=True`),
so this PR keeps `demo_available` at its default of True and exercises
the demo in the integration test matrix.

Bumped the requirements pin from the contributor's `eICU-MEDS==0.0.1`
to the latest `0.0.2` to pick up any demo / MEDS_transforms compatibility
fixes; the 0.0.1 demo failed end-to-end in CI on the bundled PR #299
(shard_events stage couldn't find its parquet lock under demo mode).

Also lands the shared `demo_available` registry mechanism (identical
across sister per-dataset PRs).

Replicated from the closed bundled PR #299 (which itself replicated
@rvandewater's #258).

Co-Authored-By: Robin P. van de Water <rvandewater@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: 3 people

src/MEDS_DEV/datasets/__init__.py

@@ -38,23 +38,28 @@ class DatasetMetadata(Metadata):
         access_policy: The level of accessibility of the dataset. Limited to the values in the AccessPolicy
             enum.
         access_details: A string describing the access policy in more detail. May be empty.
+        demo_available: Whether the dataset has a usable ``build_demo`` recipe. Datasets where the upstream
+            extractor offers no public demo set this to False; integration tests skip them and they are
+            excluded from the CI dataset matrix.
 
     Examples:
         >>> DatasetMetadata(description="foo", contacts=[{"name": "bar"}]) # doctest: +NORMALIZE_WHITESPACE
         DatasetMetadata(description='foo',
                         contacts=[Contact(name='bar', email=None, github_username=None)],
                         links=None,
                         access_policy=<AccessPolicy.PRIVATE_SINGLE_USE: 'private_single_use'>,
-                        access_details=None)
+                        access_details=None,
+                        demo_available=True)
         >>> DatasetMetadata(
         ...     description="foo", contacts=[{"name": "bar"}], access_policy="public_unrestricted",
-        ...     access_details="baz"
+        ...     access_details="baz", demo_available=False
         ... ) # doctest: +NORMALIZE_WHITESPACE
         DatasetMetadata(description='foo',
                         contacts=[Contact(name='bar', email=None, github_username=None)],
                         links=None,
                         access_policy=<AccessPolicy.PUBLIC_UNRESTRICTED: 'public_unrestricted'>,
-                        access_details='baz')
+                        access_details='baz',
+                        demo_available=False)
         >>> DatasetMetadata(
         ...     description="foo", contacts=[{"name": "bar"}], access_policy="foo"
         ... ) # doctest: +NORMALIZE_WHITESPACE
@@ -74,6 +79,7 @@ class DatasetMetadata(Metadata):
 
     access_policy: AccessPolicy = AccessPolicy.PRIVATE_SINGLE_USE
     access_details: str | None = None
+    demo_available: bool = True
 
     def __post_init__(self):
         super().__post_init__()

src/MEDS_DEV/datasets/eICU/README.md

@@ -0,0 +1,33 @@
+# eICU Collaborative Research Database
+
+## Description
+
+The eICU Collaborative Research Database is a multi-center database comprising de-identified health data associated with over 200,000 admissions to ICUs across the United States between 2014-2015. The database includes vital sign measurements, care plan documentation, severity of illness measures, diagnosis information, and treatment information. Data is collected through the Philips eICU program, a critical care telehealth program that delivers information to caregivers at the bedside.[1]
+
+## Access Requirements
+
+Taken from [PhysioNet](https://physionet.org/content/eicu-crd/2.0/):
+
+- **Access Policy**: Complete the credentialed data access requirements on PhysioNet[2]
+- **License (for files)**: PhysioNet Credentialed Health Data License Version 1.5.0[2]
+- **Data Use Agreement**: Agreement requires verified institutional affiliation and commitment to use data solely for lawful scientific research[2]
+- **Required training**: Valid CITI training certification in human research subject protection and HIPAA regulations[2]
+- **License Term**: 3 years from account creation date[2]
+- **Code Sharing**: Agreement to contribute code associated with publications to open research repository[2]
+
+## Supported Tasks
+
+- `tasks/mortality/in_icu/first_24h.yaml`
+
+## MEDS-transformation
+
+eICU is available in three formats: Original (compatible with benchmark repository), MEDS (Medical Event Data Standard), and OMOP (full OMOP table dumps). The MEDS version contains the same exact data as the Original dataset but in MEDS-compatible format.[1]
+
+## Sources
+
+1. [eICU Physionet Repository](https://physionet.org/content/eicu-crd/2.0/)
+2. [PhysioNet Credentialed Access](https://physionet.org/content/eicu-crd/view-license/2.0/)
+
+## Disclaimer
+
+Please refer to the data owners and the most up-to-date information when using this dataset in your research. The eICU dataset has not been reviewed or approved by the Food and Drug Administration and is for non-clinical, research and education use only.[2]

src/MEDS_DEV/datasets/eICU/dataset.yaml

@@ -0,0 +1,23 @@
+metadata:
+  description: >-
+    A MEDS version of eICU.
+  links:
+    - https://github.com/Medical-Event-Data-Standard/eICU_MEDS
+    - https://physionet.org/content/eicu-crd/
+  contacts:
+    - name: "Robin P. van de Water"
+      github_username: "rvandewater"
+commands:
+  build_full: >-
+    MEDS_extract-eICU
+    raw_input_dir="{temp_dir}/raw"
+    pre_MEDS_dir="{temp_dir}/pre_MEDS"
+    MEDS_cohort_dir="{output_dir}"
+    log_dir="{output_dir}/.pipeline_logs"
+  build_demo: >-
+    MEDS_extract-eICU
+    do_demo=True
+    raw_input_dir="{temp_dir}/raw"
+    pre_MEDS_dir="{temp_dir}/pre_MEDS"
+    MEDS_cohort_dir="{output_dir}"
+    log_dir="{output_dir}/.pipeline_logs"

src/MEDS_DEV/datasets/eICU/predicates.yaml

@@ -0,0 +1,162 @@
+predicates:
+  hospital_admission:
+    code: { regex: "^HOSPITAL_ADMISSION//.*" }
+  hospital_discharge:
+    code: { regex: "^HOSPITAL_DISCHARGE//.*" }
+
+  ED_registration:
+    code: { regex: "^ED_REGISTRATION//.*" }
+  ED_discharge:
+    code: { regex: "^ED_OUT//.*" }
+
+  icu_admission:
+    code: { regex: "^ICU_ADMISSION//.*" }
+  icu_discharge:
+    code: { regex: "^ICU_DISCHARGE//.*" }
+
+  creatinine_1:
+    code: LAB//50912//mg/dL
+  creatinine_2:
+    code: LAB//52546//mg/dL
+  abnormally_high_creatinine_1:
+    code: LAB//50912//mg/dL
+    value_min: 1.3 # mg/dL
+    value_min_inclusive: False
+    value_max: null
+  abnormally_high_creatinine_2:
+    code: LAB//52546//mg/dL
+    value_min: 1.3 # mg/dL
+    value_min_inclusive: False
+    value_max: null
+  creatinine:
+    expr: or(creatinine_1, creatinine_2)
+  abnormally_high_creatinine:
+    expr: or(abnormally_high_creatinine_1, abnormally_high_creatinine_2)
+
+  sodium_1:
+    code: LAB//220645//mEq/L
+  sodium_2:
+    code: LAB//50983//mEq/L
+  sodium_3:
+    code: LAB//52623//mEq/L
+  abnormally_low_sodium_1:
+    code: LAB//220645//mEq/L
+    value_min: null
+    value_max: 135 # mEq/L
+    value_max_inclusive: False
+  abnormally_low_sodium_2:
+    code: LAB//50983//mEq/L
+    value_min: null
+    value_max: 135 # mEq/L
+    value_max_inclusive: False
+  abnormally_low_sodium_3:
+    code: LAB//52623//mEq/L
+    value_min: null
+    value_max: 135 # mEq/L
+    value_max_inclusive: False
+  sodium:
+    expr: or(sodium_1, sodium_2, sodium_3)
+  abnormally_low_sodium:
+    expr: or(abnormally_low_sodium_1, abnormally_low_sodium_2, abnormally_low_sodium_3)
+
+  bicarbonate_1:
+    code: LAB//227443//mEq/L
+  bicarbonate_2:
+    code: LAB//50882//mEq/L
+  abnormally_low_bicarbonate_1:
+    code: LAB//227443//mEq/L
+    value_min: null
+    value_max: 22 # mEq/L
+    value_max_inclusive: False
+  abnormally_low_bicarbonate_2:
+    code: LAB//50882//mEq/L
+    value_min: null
+    value_max: 22 # mEq/L
+    value_max_inclusive: False
+  bicarbonate:
+    expr: or(bicarbonate_1, bicarbonate_2)
+  abnormally_low_bicarbonate:
+    expr: or(abnormally_low_bicarbonate_1, abnormally_low_bicarbonate_2)
+
+  hemoglobin_1:
+    code: LAB//220228//g/dl
+  hemoglobin_2:
+    code: LAB//50811//g/dL
+  abnormally_low_hemoglobin_1:
+    code: LAB//220228//g/dl
+    value_min: null
+    value_max: 13 # g/dL
+    value_max_inclusive: False
+  abnormally_low_hemoglobin_2:
+    code: LAB//50811//g/dL
+    value_min: null
+    value_max: 13 # g/dL
+    value_max_inclusive: False
+  hemoglobin:
+    expr: or(hemoglobin_1, hemoglobin_2)
+  abnormally_low_hemoglobin:
+    expr: or(abnormally_low_hemoglobin_1, abnormally_low_hemoglobin_2)
+
+  wbc_1:
+    code: LAB//220546//K/uL
+  wbc_2:
+    code: LAB//51300//K/uL
+  abnormally_high_wbc_1:
+    code: LAB//220546//K/uL
+    value_min: 11 # K/uL
+    value_min_inclusive: False
+    value_max: null
+  abnormally_high_wbc_2:
+    code: LAB//51300//K/uL
+    value_min: 11 # K/uL
+    value_min_inclusive: False
+    value_max: null
+  wbc:
+    expr: or(wbc_1, wbc_2)
+  abnormally_high_wbc:
+    expr: or(abnormally_high_wbc_1, abnormally_high_wbc_2)
+
+  platelets_1:
+    code: LAB//227457//K/uL
+  platelets_2:
+    code: LAB//51265//K/uL
+  abnormally_low_platelets_1:
+    code: LAB//227457//K/uL
+    value_min: null
+    value_max: 150 # K/uL
+    value_max_inclusive: False
+  abnormally_low_platelets_2:
+    code: LAB//51265//K/uL
+    value_min: null
+    value_max: 150 # K/uL
+    value_max_inclusive: False
+  platelets:
+    expr: or(platelets_1, platelets_2)
+  abnormally_low_platelets:
+    expr: or(abnormally_low_platelets_1, abnormally_low_platelets_2)
+
+  map_1:
+    code: LAB//220052//mmHg
+  map_2:
+    code: LAB//220181//mmHg
+  map_3:
+    code: LAB//225312//mmHg
+  abnormally_low_map_1:
+    code: LAB//220052//mmHg
+    value_min: null
+    value_max: 65 # mmHg
+    value_max_inclusive: False
+  abnormally_low_map_2:
+    code: LAB//220181//mmHg
+    value_min: null
+    value_max: 65 # mmHg
+    value_max_inclusive: False
+  abnormally_low_map_3:
+    code: LAB//225312//mmHg
+    value_min: null
+    value_max: 65 # mmHg
+    value_max_inclusive: False
+  map:
+    expr: or(map_1, map_2, map_3)
+  abnormally_low_map:
+    expr: or(abnormally_low_map_1, abnormally_low_map_2, abnormally_low_map_3)

src/MEDS_DEV/datasets/eICU/refs.bib

@@ -0,0 +1,15 @@
+@article{pollardEICUCollaborativeResearch2018,
+  title = {The {{eICU Collaborative Research Database}}, a Freely Available Multi-Center Database for Critical Care Research},
+  author = {Pollard, Tom J. and Johnson, Alistair E. W. and Raffa, Jesse D. and Celi, Leo A. and Mark, Roger G. and Badawi, Omar},
+  year = 2018,
+  month = dec,
+  journal = {Scientific Data},
+  volume = {5},
+  number = {1},
+  pages = {180178},
+  issn = {2052-4463},
+  doi = {10.1038/sdata.2018.178},
+  urldate = {2022-03-09},
+  langid = {english},
+  keywords = {_tablet},
+}

src/MEDS_DEV/datasets/eICU/requirements.txt

@@ -0,0 +1 @@
+eICU-MEDS==0.0.2

tests/conftest.py

@@ -325,6 +325,9 @@ def get_opts(config, opt: str) -> list[str]:
     if isinstance(out, dict):
         out = list(out.keys())
 
+    if opt == "dataset":
+        out = [name for name in out if DATASETS[name]["metadata"].demo_available]
+
     return out
 
 

tests/test_registry_validation.py

@@ -31,6 +31,10 @@ def test_all_datasets_have_commands():
     for name, dataset in DATASETS.items():
         commands = dataset.get("commands")
         assert commands is not None, f"Dataset {name} missing commands"
+        metadata = dataset.get("metadata")
+        if metadata is not None and not metadata.demo_available:
+            # Datasets that explicitly opt out of a demo recipe don't need build_demo.
+            continue
         assert "build_demo" in commands, f"Dataset {name} missing build_demo command"