We are currently on the back foot if errors in the data are discovered as users spot them. We need to support more robust observability and quality checks on the pipelines. Some examples:
- ingestion errors: What went wrong with a pipeline?
- freshness: Indicators of how when data tables/models were last updated
- source errors: Has a source stopped producing data unexpectedly?
- pipeline tests (unit and data): support developers to know pipelines are correct
There is overlap with #78 but this is also a separate issue understanding the state of the data itself.
Recent examples of where this has gone wrong:
We are currently on the back foot if errors in the data are discovered as users spot them. We need to support more robust observability and quality checks on the pipelines. Some examples:
There is overlap with #78 but this is also a separate issue understanding the state of the data itself.
Recent examples of where this has gone wrong: