Skip to content

database-connections.adoc

Latest commit

 

History

History
75 lines (65 loc) · 2.46 KB

database-connections.adoc

File metadata and controls

75 lines (65 loc) · 2.46 KB
Raw

Database connections

Airflow requires a metadata database for storing e.g. DAG, task and job data. The actual connection string is calculated by the operator so that the user does not need to remember the exact structure.

Typed connections

spec:
  clusterConfig:
    metadataDatabase:
      postgresql: # (1)
        host: airflow-postgresql
        database: airflow
        credentialsSecretName: airflow-postgresql-credentials # (2)

  1. A reference to one of the supported database backends (e.g. postgresql).

  2. A reference to a Secret which must contain the two fields username and password.

The queue/broker metadata and URL is only needed when running the celery executor. The celeryResultsBackend definition uses the same structure as metadataDatabase shown above. The celeryBroker definition requires Redis connection details.

spec:
  clusterConfig:
    celeryResultsBackend:
      postgresql: # (1)
        host: airflow-postgresql
        database: airflow
        credentialsSecretName: airflow-postgresql-credentials # (2)
    celeryBroker:
      redis: # (3)
        host: airflow-redis-master
        credentialsSecretName: airflow-redis-credentials # (2)
  celeryExecutors:
    roleGroups:
      default:
        replicas: 1

  1. A reference to one of the supported database backends (e.g. postgresql).

  2. A reference to a secret which must contain the two fields username and password.

  3. A reference to one of the supported queue brokers (e.g. redis).

Generic connections

Alternatively, these connections can also be defined in full in a referenced secret:

spec:
  clusterConfig:
    metadataDatabase:
      generic:
        connectionUrlSecretName: postgresql-metadata # (1)
spec:
  clusterConfig:
    celeryResultsBackend:
      generic:
        connectionUrlSecretName: postgresql-celery # (2)
    celeryBroker:
      generic:
        connectionUrlSecretName: redis-celery # (3)

  1. A reference to a secret which must contain the single fields connectionUrl e.g. postgresql+psycopg2://airflow:airflow@airflow-postgresql/airflow

  2. A reference to a secret which must contain the single fields connectionUrl e.g. db+postgresql://airflow:airflow@airflow-postgresql/airflow

  3. A reference to a secret which must contain the single fields connectionUrl e.g. redis://:redis@airflow-redis-master:6379/0