Add biobank merge code#1304
Merged
mandawilson merged 8 commits intoJun 18, 2026
Merged
Conversation
4 tasks
Move biobank download/merge into fetch-dmp-data-for-import.sh with best-effort aws s3 cp so a missing biobank file does not abort the import, and set BIOBANK_SKIPPED flags when biobank data is omitted.
Extract return-based try_upload_to_s3 and try_download_from_s3 so biobank fetch can skip failed downloads without changing existing upload_to_s3 and download_from_s3 behavior.
Remove unused BIOBANK_SKIPPED flags and the download_biobank_file_from_s3 wrapper; document rm/touch before biobank downloads in merge_biobank_clinical_data.
Merge biobank clinical data when the file is already present from downloadFromS3AllStudies; keep timeline on clinical merge failure and clean up the merged file only when combine fails.
callachennault
commented
Jun 18, 2026
| # Files not in S3 will be removed from the local destination (for directories) | ||
| # PATH_TO_OVERWRITE must end in PATH_IN_S3 (if provided), and cannot be "/" | ||
| function download_from_s3() { | ||
| if ! try_download_from_s3 "$1" "$2" "$3"; then |
Contributor
Author
There was a problem hiding this comment.
same comment as above - would document args
Add Args blocks to upload_to_s3, download_from_s3, and the try_* wrappers so callers can see path, S3 prefix, and bucket parameters.
mandawilson
approved these changes
Jun 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For msk_solid_heme and msk_archer, the following steps have been added:
data_clinical_patient_biobankintodata_clinical_patient, removedata_clinical_patient_biobankdata_timeline_biobank_specimenandALIQUOT_STATUSnow included in study importquestions:
data_clinical_patient_biobankbe a standalone file or should it be merged intodata_clinical_patient?data_clinical_patient_biobankanddata_clinical_patientfails?TODO