Summary
Scott Olesen requesting an ETL workflow be updated to use cloudops, orchestrated with dagster, and versioned. Still waiting for more details, but the process of this change should be thoroughly documented and presented back to his team, so they can make these type of improvements going forward.
It looks like the info is stored here: https://github.com/CDCgov/nis-py-api/blob/main/nisapi/metadata.json
We would save each of these datasets as individual datasets in the catalog.
Notes
CDC publishes a number of vaccination datasets on data.cdc.gov. For some CFA analyses, we need to grab multiple datasets and clean them, to get them in one place. My team wrote this package to download, clean, and collate the data.
Request to run that process on some regular cadence (dagster) and put the results in a versioned place in Azure.
Then ideally we could query that to run, e.g., this codebase. Right now, that pipeline uses the package above to pull data, then analyzes; the ideal state is that it could just point to the Azure-stored data.
Summary
Scott Olesen requesting an ETL workflow be updated to use cloudops, orchestrated with dagster, and versioned. Still waiting for more details, but the process of this change should be thoroughly documented and presented back to his team, so they can make these type of improvements going forward.
It looks like the info is stored here: https://github.com/CDCgov/nis-py-api/blob/main/nisapi/metadata.json
We would save each of these datasets as individual datasets in the catalog.
Notes
CDC publishes a number of vaccination datasets on data.cdc.gov. For some CFA analyses, we need to grab multiple datasets and clean them, to get them in one place. My team wrote this package to download, clean, and collate the data.
Request to run that process on some regular cadence (dagster) and put the results in a versioned place in Azure.
Then ideally we could query that to run, e.g., this codebase. Right now, that pipeline uses the package above to pull data, then analyzes; the ideal state is that it could just point to the Azure-stored data.