Merge pull request #2018 from MIT-LCP/alistair/transpile

Transpile BigQuery to Postgres/duckdb; handful of concept fixes

Author: alistairewj

.github/workflows/lint_sqlfluff.yml

@@ -14,7 +14,7 @@ jobs:
         with:
           python-version: "3.12"
       - name: install sqlfluff
-        run: "pip install sqlfluff==3.5.0"
+        run: "pip install sqlfluff==4.1.0"
       - name: Get changed files
         id: get_file_changes
         uses: trilom/file-changes-action@v1.2.4

.gitignore

@@ -1,3 +1,15 @@
+## Local docker-postgres raw data (do not commit large CSVs)
+mimic-iv/docker-postgres/mimic-data/
+
+## Allow example env, ignore real env
+mimic-iv/docker-postgres/.env
+
+# duckdb / sqlite db files
+*.db
+
+# credentials
+key.json
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]
@@ -60,4 +72,4 @@ target/
 .DS_Store
 
 # PyCharm / JetBrains
-.idea
+.idea

.sqlfluff

@@ -18,4 +18,8 @@ capitalisation_policy = upper
 extended_capitalisation_policy = upper
 
 [sqlfluff:rules:capitalisation.identifiers]
-extended_capitalisation_policy = lower
+extended_capitalisation_policy = lower
+
+# Boolean/null literals (TRUE, FALSE, NULL) upper case
+[sqlfluff:rules:capitalisation.literals]
+capitalisation_policy = upper

README.md

@@ -10,18 +10,20 @@ The various MIMIC databases are available on Google Cloud Platform (GCP) and Ama
 
 ## Navigating this repository
 
-This repository contains code for five databases on PhysioNet:
+This repository contains code for the following databases on PhysioNet:
 
 - [MIMIC-III](https://physionet.org/content/mimiciii/) - critical care data for patients admitted to ICUs at the BIDMC between 2001 - 2012
 - [MIMIC-IV](https://physionet.org/content/mimiciv/) - hospital and critical care data for patients admitted to the ED or ICU between 2008 - 2019
+- [MIMIC-IV-Note](https://physionet.org/content/mimic-iv-note) - deidentified free-text clinical notes
 - [MIMIC-IV-ED](https://physionet.org/content/mimic-iv-ed/) - emergency department data for individuals attending the ED between 2011 - 2019
 - MIMIC-IV Waveforms (TBD) - this dataset has yet to be published.
 - [MIMIC-CXR](https://physionet.org/content/mimic-cxr/) - chest x-ray imaging and deidentified free-text radiology reports for patients admitted to the ED from 2012 - 2016
 
 The repository contains one top-level folder containing community developed code for each datasets:
 
 - [mimic-iii](/mimic-iii) - build scripts for MIMIC-III, derived concepts which are available on the `physionet-data.mimiciii_derived` dataset on BigQuery, and tutorials.
-- [mimic-iv](/mimic-iv) - build scripts for MIMIC-IV, derived concepts which are available on the `physionet-data.mimic_derived` dataset on BigQuery, and tutorials.
+- [mimic-iv](/mimic-iv) - build scripts for MIMIC-IV, derived concepts which are available on release-specific datasets such as `physionet-data.mimiciv_3_1_derived` on BigQuery, and tutorials.
+- [mimic-iv-note](/mimic-iv-note) - build scripts
 - [mimic-iv-cxr](/mimic-iv-cxr) - code for loading and analyzing both dicom (mimic-iv-cxr/dcm) and text (mimic-iv-cxr/txt) data. In order to clearly indicate that MIMIC-CXR can be linked with MIMIC-IV, we have named this folder mimic-iv-cxr, and any references to MIMIC-CXR / MIMIC-IV-CXR are interchangeable.
 - [mimic-iv-ed](/mimic-iv-ed) - build scripts for MIMIC-IV-ED.
 - mimic-iv-waveforms - TBD
@@ -32,10 +34,41 @@ Each subfolder has a README with further detail regarding its content.
 
 MIMIC-III is available on AWS (and MIMIC-IV will be available in the future). Use the below Launch Stack button to deploy access to the MIMIC-III dataset into your AWS account.  This will give you real-time access to the MIMIC-III data in your AWS account without having to download a copy of the MIMIC-III dataset.  It will also deploy a Jupyter Notebook with access to the content of this GitHub repository in your AWS account.    Prior to launching this, please login to the [MIMIC PhysioNet website](https://mimic.mit.edu/), [input your AWS account number](https://physionet.org/settings/cloud/), and [request access to the MIMIC-III Clinical Database on AWS](https://physionet.org/projects/mimiciii/1.4/request_access/2).  
 
-To start this deployment, click the Launch Stack button.  On the first screen, the template link has already been specified, so just click next.  On the second screen, provide a Stack name (letters and numbers) and click next, on the third screen, just click next.  On the forth screen, at the bottom, there is a box that says **I acknowledge that AWS CloudFormation might create IAM resources.**.  Check that box, and then click **Create**.  Once the Stack has complete deploying, look at the **Outputs** tab of the AWS CloudFormation console for links to your Juypter Notebooks instance.
+To start this deployment, click the Launch Stack button.  On the first screen, the template link has already been specified, so just click next.  On the second screen, provide a Stack name (letters and numbers) and click next, on the third screen, just click next.  On the forth screen, at the bottom, there is a box that says **I acknowledge that AWS CloudFormation might create IAM resources.**.  Check that box, and then click **Create**.  Once the Stack has complete deploying, look at the **Outputs** tab of the AWS CloudFormation console for links to your Jupyter Notebooks instance.
 
 [![cloudformation-launch-stack](/mimic-iii/buildmimic/aws-athena/cloudformation-launch-stack.png)](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=MIMIC&templateURL=https://aws-bigdata-blog.s3.amazonaws.com/artifacts/biomedical-informatics-studies/mimic-iii-athena.yaml)
 
+## mimic_utils package
+
+This package contains utilities for working with the MIMIC datasets; primarily transpiling SQL code.
+
+### Installation
+
+```bash
+pip install -e ".[test]"
+```
+
+### Development
+
+The repository includes a locked dependency file (`requirements-lock.txt`) generated by `pip-compile` for reproducible builds on Python 3.9. To use it:
+
+```bash
+pip install -r requirements-lock.txt
+pip install -e . --no-deps
+```
+
+To update the lock file after changing dependencies in `pyproject.toml`:
+
+```bash
+pip-compile pyproject.toml --output-file requirements-lock.txt --all-extras --strip-extras --upgrade
+```
+
+### Running tests
+
+```bash
+pytest tests/
+```
+
 ## Other useful tools
 
 * [Bloatectomy](https://github.com/MIT-LCP/bloatectomy) ([paper](https://github.com/MIT-LCP/bloatectomy/blob/master/paper/paper.md)) - A python based package for removing duplicate text in clinical notes
@@ -72,7 +105,7 @@ Our team has worked hard to create and share the MIMIC datasets. We encourage yo
 * Commit your changes to the forked repository.
 * Submit a pull request to the [MIMIC code repository](https://github.com/MIT-LCP/mimic-code), using the method described at: https://help.github.com/articles/using-pull-requests/
 
-We encourage users to share concepts they have extracted by writing code which generates a materialized view. These materialized views can then be used by researchers around the world to speed up data extraction. For example, ventilation durations can be acquired by creating the ventdurations view in [concepts/durations/ventilation_durations.sql](https://github.com/MIT-LCP/mimic-code/tree/new_consol/mimic-iii/concepts/durations/ventilation_durations.sql).
+We encourage users to share concepts they have extracted by writing code which generates a table. These derived tables can then be used by researchers around the world to speed up data extraction. See the [mimic-iv/concepts](mimic-iv/concepts/) folder for examples.
 
 ### License
 

README_mimic_utils.md

@@ -0,0 +1,31 @@
+# MIMIC-IV-Note
+
+* [buildmimic](/mimic-iv-note/buildmimic) - Scripts to build MIMIC-IV-Note in various database systems
+    * [postgres](/mimic-iv-note/buildmimic/postgres) - PostgreSQL (v10+)
+    * [duckdb](/mimic-iv-note/buildmimic/duckdb) - DuckDB (serverless, single-file)
+    * [bigquery](/mimic-iv-note/buildmimic/bigquery) - BigQuery schemas
+
+MIMIC-IV-Note contains deidentified free-text clinical notes for patients in MIMIC-IV, including discharge summaries and radiology reports. It is a separate download from MIMIC-IV and is available on [PhysioNet](https://physionet.org/content/mimic-iv-note/).
+
+There are currently no derived concept scripts for MIMIC-IV-Note.
+
+## Building the database
+
+### PostgreSQL
+
+```sh
+psql -d mimiciv -f mimic-iv-note/buildmimic/postgres/create.sql
+psql -d mimiciv -v ON_ERROR_STOP=1 -v mimic_data_dir=<path to mimic-iv-note>/note \
+  -f mimic-iv-note/buildmimic/postgres/load_gz.sql
+```
+
+See the [postgres README](/mimic-iv-note/buildmimic/postgres/README.md) for full details.
+
+### DuckDB
+
+```sh
+cd mimic-iv-note/buildmimic/duckdb
+./import_duckdb.sh <path to mimic-iv-note>
+```
+
+See the [duckdb README](/mimic-iv-note/buildmimic/duckdb/README.md) for full details.

mimic-iv-note/README.md

@@ -23,7 +23,7 @@ The BigQuery [physionet-data.mimic_derived](https://console.cloud.google.com/big
 These concepts assume the output schema is `mimiciv_derived`. If you would like a different schema, you will need to make a few edits to the scripts.
 
 All concepts are originally written in the **BigQuery Standard SQL Dialect**. A Python package is used to convert these BigQuery scripts into other dialects such as PostgreSQL.
-These scripts have been converted to PostgreSQL by a script. To generate the concepts in PostgreSQL, see the [MIMIC-IV postgresql concepts subfolder](/mimic-iv/concepts/postgres).
+These scripts have been converted to PostgreSQL and DuckDB by a transpilation script. To generate the concepts in PostgreSQL, see the [MIMIC-IV postgresql concepts subfolder](/mimic-iv/concepts_postgres).
 [See below for how scripts in non-bigquery dialects were generated](#transpile).
 
 ### BigQuery
@@ -39,29 +39,70 @@ bq query --use_legacy_sql=False --replace --destination_table=my_bigquery_datase
 
 ### PostgreSQL
 
-The [postgres](/mimic-iv/concepts_postgres) folder contains concepts in a PostgreSQL compatible dialect.
+The [concepts_postgres](/mimic-iv/concepts_postgres) folder contains concepts in a PostgreSQL compatible dialect.
+To run them against a local PostgreSQL database:
+
+```sh
+psql -d mimiciv -f mimic-iv/concepts_postgres/postgres-make-concepts.sql
+```
 
 ### DuckDB
 
-The [duckdb](/mimic-iv/concepts_duckdb) folder contains concepts in a DuckDB compatible dialect.
+The [concepts_duckdb](/mimic-iv/concepts_duckdb) folder contains concepts in a DuckDB compatible dialect.
+To load all derived concepts into an existing DuckDB database (e.g. one built from CSV files):
+
+```python
+import duckdb
+
+con = duckdb.connect("~/data/mimic-iv.duckdb")
+con.execute("SET search_path = mimiciv_derived")
+
+import os
+from pathlib import Path
+concepts_dir = Path("mimic-iv/concepts_duckdb")
+duckdb_sql = (concepts_dir / "duckdb.sql").read_text()
+for line in duckdb_sql.splitlines():
+    line = line.strip()
+    if line.startswith(".read "):
+        sql = (concepts_dir / line[6:]).read_text()
+        con.execute(sql)
+```
+
+Or from the duckdb CLI:
+
+```sh
+duckdb ~/data/mimic-iv.duckdb -c "SET search_path = mimiciv_derived" \
+  < mimic-iv/concepts_duckdb/duckdb.sql
+```
 
 ## Transpile
 
-The Python package [sqlglot](https://sqlglot.com/) is used to convert from concepts in BigQuery syntax to the other syntaxes ("transpile").
-This package parses the SQL into an abstract syntax tree (AST), after which it can be re-written into a specific dialect.
-Not all functions are supported by sqlglot, so a helper package was written which adds support for the missing functions.
-Most of this process is done the [transpile.py](/src/mimic_utils/transpile.py) file.
+The Python package [sqlglot](https://sqlglot.com/) is used to convert concepts from BigQuery SQL to other dialects ("transpile").
+It parses the SQL into an abstract syntax tree (AST) and re-writes it for the target dialect.
+The `mimic_utils` helper package handles functions not natively supported by sqlglot; see [transpile.py](/src/mimic_utils/transpile.py).
+
+**The transpiled files are already committed** to `concepts_postgres` and `concepts_duckdb`.
+You only need to re-run transpilation if you modify the source BigQuery SQL in `concepts/`.
+
+First install the package from the repo root:
 
-An entrypoint is provided for convenience. To transpile a single file, run:
+```sh
+pip install -e .
+```
+
+To transpile a single file:
 
 ```sh
-# convert_file <source_file> <destination_file> --destination_dialect <dialect>
+# mimic_utils convert_file <source_file> <destination_file> [--destination_dialect postgres|duckdb]
 mimic_utils convert_file mimic-iv/concepts/demographics/age.sql age.sql --destination_dialect duckdb
 ```
 
-To transpile all files in a folder, run:
+To re-transpile all concepts at once:
 
 ```sh
-# convert_folder <source_folder> <destination_folder> --destination_dialect <dialect>
+# To DuckDB:
 mimic_utils convert_folder mimic-iv/concepts mimic-iv/concepts_duckdb --destination_dialect duckdb
+
+# To PostgreSQL:
+mimic_utils convert_folder mimic-iv/concepts mimic-iv/concepts_postgres --destination_dialect postgres
 ```