Description / Context
A running triggerer process allows deferrable operators to be used in DAGs so that worker management can be improved e.g. on scheduler re-starts and for providing HA: see here for more details.
Problem
Currently the airflow-operator does not implement an airflow triggerer process/role.
Proposal
Introduce a new optional role to go alongside the existing ones:
- webservers
- schedulers
- celeryExecutors / kubernetesExecutors
- triggerers
This new role will encapsulate a command that starts the triggerer.
The CRD would look something like this:
---
apiVersion: airflow.stackable.tech/v1alpha1
kind: AirflowCluster
metadata:
name: airflow
spec:
...
webservers:
...
celeryExecutors:
...
schedulers:
...
triggerers:
roleGroups:
default:
replicas: 1
Acceptance criteria
- add an airflow role
triggerer
- create a test that uses a DAG/deferrable operator that keeps running over a re-deployment/scheduler re-start
Description / Context
A running triggerer process allows deferrable operators to be used in DAGs so that worker management can be improved e.g. on scheduler re-starts and for providing HA: see here for more details.
Problem
Currently the airflow-operator does not implement an airflow triggerer process/role.
Proposal
Introduce a new optional role to go alongside the existing ones:
This new role will encapsulate a command that starts the triggerer.
The CRD would look something like this:
Acceptance criteria
triggerer