| description | What are efficient ways to process your data? |
|---|
{% embed url="https://www.youtube.com/watch?v=foD8r33JM\_8" caption="Processing - Data Management" %}
- The simplest thing we can do is a Makefile to specify what action(s) depend on.
- You will probably need a workflow management system. Airflow is the current winner of this space.
- Try to keep things simple and don't over-engineer your processing pipeline.