chore: drop concept SQL changes for separate PRs

Per maintainer feedback on MIT-LCP#2124, keep load-path / transpile / CI fixes here and move concept logic to individual PRs.

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: Chessing234

mimic-iii/concepts/README.md

@@ -5,7 +5,7 @@ The scripts are intended to be run against the BigQuery instantiation of MIMIC-I
 Concepts are categorized into folders if possible, otherwise they remain in the top-level directory.
 A table of contents is provided below: [List of concepts](#list-of-concepts).
 
-You can read about cloud access to MIMIC-III, including via Google BigQuery, on the [cloud page](https://mimic.physionet.org/gettingstarted/cloud/).
+You can read about cloud access to MIMIC-III, including via Google BigQuery, on the [cloud page](https://mimic.mit.edu/docs/gettingstarted/cloud/).
 
 The rest of this README describes:
 
@@ -14,7 +14,7 @@ The rest of this README describes:
 
 ## Generating the concepts in BigQuery
 
-You do not need to generate the concepts if you are using BigQuery! They have already been generated for you. If you have access to MIMIC-III on BigQuery, look under `physionet-data.mimic_derived`. If you would like to generate the concepts again, for example on your own dataset, you must modify the `TARGET_DATASET` variable within the [make-concepts.sh](/concepts/make-concepts.sh) script. The script assumes you have installed and configured the [Google Cloud SDK](https://cloud.google.com/sdk/docs/install).
+You do not need to generate the concepts if you are using BigQuery! They have already been generated for you. If you have access to MIMIC-III on BigQuery, look under `physionet-data.mimiciii_derived`. If you would like to generate the concepts again, for example on your own dataset, you must modify the `TARGET_DATASET` variable within the [make-concepts.sh](/mimic-iii/concepts/make-concepts.sh) script. The script assumes you have installed and configured the [Google Cloud SDK](https://cloud.google.com/sdk/docs/install).
 
 ## Generating the concepts in PostgreSQL or DuckDB
 
@@ -42,6 +42,7 @@ Folder | Table | Description
 --- | --- | ---
 . | [echo_data](echo_data.sql) | Text extracted from echocardiography reports using regular expressions.
 . | [code_status](code_status.sql) | Whether the patient has restrictions on life saving resuscitation.
+. | [rrt](rrt.sql) | Whether the patient received any dialysis during their ICU stay.
 comorbidity | [elixhauser_ahrq_v37](comorbidity/elixhauser_ahrq_v37.sql)                    | Comorbidities in categories proposed by Elixhauser et al. AHRQ produced the mapping.
 comorbidity | [elixhauser_ahrq_v37_no_drg](comorbidity/elixhauser_ahrq_v37_no_drg.sql)      | As above, but DRG codes are not used to exclude primary conditions.
 comorbidity | [elixhauser_quan](comorbidity/elixhauser_quan.sql)                            | Comorbidities in categories proposed by Elixhauser et al. using an algorithm by Quan et al.
@@ -76,7 +77,7 @@ durations | [neuroblock_dose](durations/neuroblock_dose.sql)
 durations | [norepinephrine_dose](durations/norepinephrine_dose.sql)                        | Dose administered with start/stop times for norepinephrine.
 durations | [phenylephrine_dose](durations/phenylephrine_dose.sql)                          | Dose administered with start/stop times for phenylephrine.
 durations | [vasopressin_dose](durations/vasopressin_dose.sql)                              | Dose administered with start/stop times for vasopressin.
-durations | [ventilation_classification](durations/ventilation_classification.sql)          | Classifies patient settings as implying mechanical ventilation.
+durations | [ventilation_classification](durations/ventilation_classification.sql)          | Classifies ventilator settings (tidal volume, PEEP, etc.) as implying mechanical ventilation.
 durations | [ventilation_durations](durations/ventilation_durations.sql)                    | Start and stop times for mechanical ventilation.
 durations | [weight_durations](durations/weight_durations.sql)                              | Start and stop times for daily weight measurements.
 **firstday** | | The first day subfolder contains scripts to summarizes a patient's health on their first ICU day.
@@ -93,6 +94,8 @@ firstday | [weight_first_day](firstday/weight_first_day.sql)
 **fluid_balance** | | Tables which track fluid input and output for the patient.
 fluid_balance | [colloid_bolus](fluid_balance/colloid_bolus.sql)                            | Times at which a patient received a bolus of colloidal fluid.
 fluid_balance | [crystalloid_bolus](fluid_balance/crystalloid_bolus.sql)                    | Times at which a patient received a bolus of crystalloid fluid.
+fluid_balance | [ffp_transfusion](fluid_balance/ffp_transfusion.sql)                        | Times at which a patient received a transfusion of fresh frozen plasma.
+fluid_balance | [rbc_transfusion](fluid_balance/rbc_transfusion.sql)                        | Times at which a patient received a transfusion of red blood cells.
 fluid_balance | [urine_output](fluid_balance/urine_output.sql)                              | Urine output for a patient with the time of documentation.
 **organfailure** | | Summarizations of the degree of organ failure for single organ systems.
 organfailure | [kdigo_creatinine](organfailure/kdigo_creatinine.sql)                        | Creatinine values with baseline creatinine as defined by KDIGO.
@@ -103,6 +106,7 @@ organfailure | [kdigo_stages_7day](organfailure/kdigo_stages_7day.sql)
 organfailure | [meld](organfailure/meld.sql)                                                | The MELD score, often used to assess health of liver transplant candidates.
 **pivot** |                                                                                 | Pivoted views contain the patient `icustay_id`, the `charttime`, and a number of variables. They are useful to acquiring a time series of values for patient stays.
 pivot | [pivoted_bg](pivot/pivoted_bg.sql)                                                  | Blood gas measurements.
+pivot | [pivoted_bg_art](pivot/pivoted_bg_art.sql)                                          | Arterial blood gas measurements, explicitly labelled or inferred from SpO2.
 pivot | [pivoted_fio2](pivot/pivoted_fio2.sql)                                              | Fraction of inspired oxygen.
 pivot | [pivoted_gcs](pivot/pivoted_gcs.sql)                                                | Glasgow Coma Scale.
 pivot | [pivoted_height](pivot/pivoted_height.sql)                                          | Height.
@@ -113,7 +117,6 @@ pivot | [pivoted_oasis](pivot/pivoted_oasis.sql)
 pivot | [pivoted_rrt](pivot/pivoted_rrt.sql)                                                | Renal replacement therapy.
 pivot | [pivoted_sofa](pivot/pivoted_sofa.sql)                                              | The Sequential Organ Failure Assessment (SOFA) scale.
 pivot | [pivoted_uo](pivot/pivoted_uo.sql)                                                  | Urine output.
-pivot | [pivoted_vent_setting](pivot/pivoted_vent_setting.sql)                              | Ventilator settings (tidal volume, PEEP, etc).
 pivot | [pivoted_vital](pivot/pivoted_vital.sql)                                            | Vital signs.
 **sepsis** | | Definitions of sepsis, a common cause of mortality for intensive care unit patients.
 sepsis | [angus](sepsis/angus.sql)                                                          | Sepsis defined using billing codes validated by Angus et al.

mimic-iii/concepts/comorbidity/elixhauser_ahrq_v37.sql

@@ -384,10 +384,8 @@ select hadm_id, seq_num, icd9_code
   when icd9_code = '3091' then 1
   when icd9_code = '311'         then 1
 		end as depress  /* Depression */
-from (
-  select hadm_id, seq_num, RTRIM(icd9_code) as icd9_code
-  from `physionet-data.mimiciii_clinical.diagnoses_icd`
-) icd
+from `physionet-data.mimiciii_clinical.diagnoses_icd` icd
+WHERE seq_num = 1
 )
 -- collapse the icd9_code specific flags into hadm_id specific flags
 -- this groups comorbidities together for a single patient admission

mimic-iii/concepts/comorbidity/elixhauser_ahrq_v37_no_drg.sql

@@ -390,11 +390,8 @@ select hadm_id, seq_num, icd9_code
   when icd9_code = '3091' then 1
   when icd9_code = '311'         then 1
 		end as depress  /* Depression */
-from (
-  select hadm_id, seq_num, RTRIM(icd9_code) as icd9_code
-  from `physionet-data.mimiciii_clinical.diagnoses_icd`
-  where seq_num != 1
-) icd
+from `physionet-data.mimiciii_clinical.diagnoses_icd` icd
+WHERE seq_num = 1
 )
 -- collapse the icd9_code specific flags into hadm_id specific flags
 -- this groups comorbidities together for a single patient admission

mimic-iii/concepts/comorbidity/elixhauser_quan.sql

@@ -174,10 +174,7 @@ select hadm_id, seq_num, icd9_code
   when SUBSTR(icd9_code, 1, 4) in ('2962','2963','2965','3004') then 1
   when SUBSTR(icd9_code, 1, 3) in ('309','311') then 1
   else 0 end as depress  /* Depression */
-from (
-  select hadm_id, seq_num, RTRIM(icd9_code) as icd9_code
-  from `physionet-data.mimiciii_clinical.diagnoses_icd`
-) icd
+from `physionet-data.mimiciii_clinical.diagnoses_icd` icd
 where seq_num != 1 -- we do not include the primary icd-9 code
 )
 -- collapse the icd9_code specific flags into hadm_id specific flags

mimic-iii/concepts/diagnosis/README.md

@@ -4,7 +4,7 @@ The Clinical Classification Software (CCS) categorizes ICD-9 coded diagnoses int
 
 The `ccs_multi_dx.csv.gz` data file must be uploaded to `physionet-data.mimiciii_derived.ccs_multi_dx`.
 
-The BigQuery schema definition is available in this folder as [ccs_multi_dx.json](/ccs_multi_dx.json).
+The BigQuery schema definition is available in this folder as [ccs_multi_dx.json](ccs_multi_dx.json).
 
 ## Creation of the ccs_multi_dx.csv.gz file
 
@@ -24,7 +24,7 @@ Use Python to convert all apostrophes in `ccs_multi_dx_tool_2015.csv` into doubl
 
 ```python
 import pandas as pd
-df = pd.read_csv('ccs_multi_dx_tool_2015.csv.gz')
+df = pd.read_csv('ccs_multi_dx_tool_2015.csv')
 # remove apostrophes from header names and relabel
 df.rename(columns={"'ICD-9-CM CODE'": "icd9_code", "'CCS LVL 1'": "ccs_level1", "'CCS LVL 1 LABEL'": "ccs_group1", "'CCS LVL 2'": "ccs_level2", "'CCS LVL 2 LABEL'": "ccs_group2", "'CCS LVL 3'": "ccs_level3", "'CCS LVL 3 LABEL'": "ccs_group3", "'CCS LVL 4'": "ccs_level4", "'CCS LVL 4 LABEL'": "ccs_group4", }, inplace=True)
 

mimic-iii/concepts/diagnosis/ccs_dx.sql

@@ -1,5 +1,5 @@
 -- add in matched ID, name, and ccs_id
---  matched id (ccs_mid): the ccs ID with the hierachy, e.g. 7.1.2.1
+--  matched id (ccs_mid): the ccs ID with the hierarchy, e.g. 7.1.2.1
 --  name (ccs_name): the most granular CCS category the diagnosis is in
 --  ID (ccs_id): the CCS identifier for the ICD-9 code (integer)
 SELECT

mimic-iii/concepts/durations/adenosine_durations.sql

@@ -198,39 +198,11 @@ and
 (
   select
     icustay_id, linkorderid
-    -- Keep each MV row: min/max by linkorderid includes pause gaps (#1808).
-    , starttime, endtime
+    , min(starttime) as starttime, max(endtime) as endtime
   FROM `physionet-data.mimiciii_clinical.inputevents_mv`
   where itemid = 221282 -- adenosine
   and statusdescription != 'Rewritten' -- only valid orders
-)
-
-, vasomv_grp as
-(
-  -- Merge overlapping/abutting intervals; pause gaps stay separate (#1808).
-  SELECT
-    s1.icustay_id
-    , s1.starttime
-    , MIN(t1.endtime) AS endtime
-  FROM vasomv s1
-  INNER JOIN vasomv t1
-    ON s1.icustay_id = t1.icustay_id
-    AND s1.starttime <= t1.endtime
-    AND NOT EXISTS
-    (
-      SELECT 1 FROM vasomv t2
-      WHERE t1.icustay_id = t2.icustay_id
-        AND t1.endtime >= t2.starttime
-        AND t1.endtime < t2.endtime
-    )
-  WHERE NOT EXISTS
-  (
-    SELECT 1 FROM vasomv s2
-    WHERE s1.icustay_id = s2.icustay_id
-      AND s1.starttime > s2.starttime
-      AND s1.starttime <= s2.endtime
-  )
-  GROUP BY s1.icustay_id, s1.starttime
+  group by icustay_id, linkorderid
 )
 
 select
@@ -252,6 +224,6 @@ select
   , DATETIME_DIFF(endtime, starttime, HOUR) AS duration_hours
   -- add durations
 from
-  vasomv_grp
+  vasomv
 
 order by icustay_id, vasonum;

mimic-iii/concepts/durations/arterial_line_durations.sql

@@ -163,45 +163,16 @@ with mv as
   group by icustay_id, arterial_line_rownum
   having min(charttime) != max(charttime)
 )
--- collapse overlapping MetaVision arterial line intervals (#371)
-, mv_dur as
-(
-  SELECT
-    s1.icustay_id
-    , s1.starttime
-    , MIN(t1.endtime) AS endtime
-    , DATETIME_DIFF(MIN(t1.endtime), s1.starttime, HOUR) AS duration_hours
-  FROM mv s1
-  INNER JOIN mv t1
-    ON s1.icustay_id = t1.icustay_id
-    AND s1.arterial_line = 1
-    AND t1.arterial_line = 1
-    AND s1.starttime <= t1.endtime
-    AND NOT EXISTS
-    (
-      SELECT 1 FROM mv t2
-      WHERE t1.icustay_id = t2.icustay_id
-        AND t2.arterial_line = 1
-        AND t1.endtime >= t2.starttime
-        AND t1.endtime < t2.endtime
-    )
-  WHERE s1.arterial_line = 1
-    AND NOT EXISTS
-    (
-      SELECT 1 FROM mv s2
-      WHERE s1.icustay_id = s2.icustay_id
-        AND s2.arterial_line = 1
-        AND s1.starttime > s2.starttime
-        AND s1.starttime <= s2.endtime
-    )
-  GROUP BY s1.icustay_id, s1.starttime
-)
 select icustay_id
   -- , arterial_line_rownum
   , starttime, endtime, duration_hours
 from cv_dur
 UNION ALL
+--TODO: collapse metavision durations if they overlap
 select icustay_id
-  , starttime, endtime, duration_hours
-from mv_dur
+  -- , ROW_NUMBER() over (PARTITION BY icustay_id ORDER BY starttime) as arterial_line_rownum
+  , starttime, endtime
+  , DATETIME_DIFF(endtime, starttime, HOUR) AS duration_hours
+from mv
+where arterial_line = 1
 order by icustay_id, starttime;

mimic-iii/concepts/durations/central_line_durations.sql

@@ -158,45 +158,16 @@ with mv as
   group by icustay_id, central_line_rownum
   having min(charttime) != max(charttime)
 )
--- collapse overlapping MetaVision central line intervals (#371)
-, mv_dur as
-(
-  SELECT
-    s1.icustay_id
-    , s1.starttime
-    , MIN(t1.endtime) AS endtime
-    , DATETIME_DIFF(MIN(t1.endtime), s1.starttime, HOUR) AS duration_hours
-  FROM mv s1
-  INNER JOIN mv t1
-    ON s1.icustay_id = t1.icustay_id
-    AND s1.central_line = 1
-    AND t1.central_line = 1
-    AND s1.starttime <= t1.endtime
-    AND NOT EXISTS
-    (
-      SELECT 1 FROM mv t2
-      WHERE t1.icustay_id = t2.icustay_id
-        AND t2.central_line = 1
-        AND t1.endtime >= t2.starttime
-        AND t1.endtime < t2.endtime
-    )
-  WHERE s1.central_line = 1
-    AND NOT EXISTS
-    (
-      SELECT 1 FROM mv s2
-      WHERE s1.icustay_id = s2.icustay_id
-        AND s2.central_line = 1
-        AND s1.starttime > s2.starttime
-        AND s1.starttime <= s2.endtime
-    )
-  GROUP BY s1.icustay_id, s1.starttime
-)
 select icustay_id
   -- , central_line_rownum
   , starttime, endtime, duration_hours
 from cv_dur
 UNION ALL
+--TODO: collapse metavision durations if they overlap
 select icustay_id
-  , starttime, endtime, duration_hours
-from mv_dur
+  -- , ROW_NUMBER() over (PARTITION BY icustay_id ORDER BY starttime) as central_line_rownum
+  , starttime, endtime
+  , DATETIME_DIFF(endtime, starttime, HOUR) AS duration_hours
+from mv
+where central_line = 1
 order by icustay_id, starttime;

mimic-iii/concepts/durations/dobutamine_durations.sql

@@ -195,39 +195,11 @@ and
 (
   select
     icustay_id, linkorderid
-    -- Keep each MV row: min/max by linkorderid includes pause gaps (#1808).
-    , starttime, endtime
+    , min(starttime) as starttime, max(endtime) as endtime
   FROM `physionet-data.mimiciii_clinical.inputevents_mv`
   where itemid = 221653 -- dobutamine
   and statusdescription != 'Rewritten' -- only valid orders
-)
-
-, vasomv_grp as
-(
-  -- Merge overlapping/abutting intervals; pause gaps stay separate (#1808).
-  SELECT
-    s1.icustay_id
-    , s1.starttime
-    , MIN(t1.endtime) AS endtime
-  FROM vasomv s1
-  INNER JOIN vasomv t1
-    ON s1.icustay_id = t1.icustay_id
-    AND s1.starttime <= t1.endtime
-    AND NOT EXISTS
-    (
-      SELECT 1 FROM vasomv t2
-      WHERE t1.icustay_id = t2.icustay_id
-        AND t1.endtime >= t2.starttime
-        AND t1.endtime < t2.endtime
-    )
-  WHERE NOT EXISTS
-  (
-    SELECT 1 FROM vasomv s2
-    WHERE s1.icustay_id = s2.icustay_id
-      AND s1.starttime > s2.starttime
-      AND s1.starttime <= s2.endtime
-  )
-  GROUP BY s1.icustay_id, s1.starttime
+  group by icustay_id, linkorderid
 )
 
 select
@@ -249,6 +221,6 @@ select
   , DATETIME_DIFF(endtime, starttime, HOUR) AS duration_hours
   -- add durations
 from
-  vasomv_grp
+  vasomv
 
 order by icustay_id, vasonum;