|
1 | 1 | = Mounting DAGs |
2 | | -:description: Mount DAGs in Airflow via ConfigMap for single DAGs or use git-sync for multiple DAGs. git-sync pulls from a Git repo and handles updates automatically. |
| 2 | +:description: Mount DAGs in Airflow via ConfigMap for single DAGs, git-sync for multiple DAGs from a single repo, or DAG bundles (Airflow 3.x) for multiple repositories. |
3 | 3 | :git-sync: https://github.com/kubernetes/git-sync/tree/v4.2.4 |
4 | 4 |
|
5 | | -DAGs can be mounted by using a ConfigMap or `git-sync`. |
| 5 | +DAGs can be mounted by using a ConfigMap, `git-sync`, or - on Airflow 3.x - DAG bundles. |
6 | 6 | This is best illustrated with an example of each, shown in the sections below. |
7 | 7 |
|
8 | 8 | == Via ConfigMap |
@@ -87,3 +87,168 @@ include::example$example-airflow-gitsync-ssh.yaml[] |
87 | 87 |
|
88 | 88 | NOTE: git-sync can be used with DAGs that make use of Python modules, as Python is configured to use the git-sync target folder as the "root" location when looking for referenced files. |
89 | 89 | See the xref:usage-guide/applying-custom-resources.adoc[] example for more details. |
| 90 | + |
| 91 | +== Via DAG bundles (Airflow 3.x) |
| 92 | + |
| 93 | +https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/dag-bundles.html[DAG bundles] are an Airflow 3.x feature that natively supports loading DAGs from multiple sources - including multiple Git repositories - without requiring a git-sync sidecar. |
| 94 | +This is particularly useful when DAGs are maintained in separate repositories by different teams. |
| 95 | + |
| 96 | +The Stackable Airflow operator does not have first-class CRD support for DAG bundles, but they can be configured using `envOverrides` to set the `AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST` environment variable. |
| 97 | +No changes to the Stackable Airflow image are required: the `apache-airflow-providers-git` package and the `git` binary are both included in the standard image. |
| 98 | + |
| 99 | +=== When to use DAG bundles instead of git-sync |
| 100 | + |
| 101 | +Use `git-sync` (via `dagsGitSync`) when: |
| 102 | + |
| 103 | +* DAGs come from a single repository. |
| 104 | +* You need per-repository TLS/CA certificate configuration. |
| 105 | + |
| 106 | +Use DAG bundles when: |
| 107 | + |
| 108 | +* DAGs come from **multiple repositories** and must all be visible to Airflow. |
| 109 | +* You want DAG versioning (each DAG run is pinned to the Git commit at the time it was created). |
| 110 | + |
| 111 | +=== Prerequisites |
| 112 | + |
| 113 | +* Airflow 3.x (the `dag_bundle_config_list` setting does not exist in Airflow 2.x). |
| 114 | +* Each `GitDagBundle` requires an https://airflow.apache.org/docs/apache-airflow-providers-git/stable/connections/git.html[Airflow Git connection] - even for public repositories. |
| 115 | + For public repos, the connection only needs a `host` (the repository URL) and no credentials. |
| 116 | + Connections can be created via the Airflow UI, CLI, a https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html[secrets backend], or - as shown in the example below - via `AIRFLOW_CONN_*` environment variables. |
| 117 | + The operator does not manage Airflow connections. |
| 118 | + |
| 119 | +=== Example |
| 120 | + |
| 121 | +The following example configures two DAG bundles, each pulling from a public Git repository. |
| 122 | +The Airflow connections are defined as `AIRFLOW_CONN_*` environment variables alongside the bundle configuration. |
| 123 | + |
| 124 | +WARNING: This example points both bundles at the same repository and subdirectory for illustrative purposes. |
| 125 | +In practice, each bundle should reference a different repository (or at least a different subdirectory) with distinct DAG files. |
| 126 | +Airflow requires that https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/multi-team.html[DAG IDs are unique across the entire deployment]: |
| 127 | +if two bundles define the same DAG ID, the last one parsed silently overwrites the other - with no error or warning - and the DAG may flip-flop between bundles on each parse cycle. |
| 128 | + |
| 129 | +NOTE: The `envOverrides` are set at the role level (not the role group level) in all cases, so that they apply to all role groups within that role. |
| 130 | + |
| 131 | +[source,yaml] |
| 132 | +---- |
| 133 | +include::example$example-airflow-dag-bundles.yaml[] |
| 134 | +---- |
| 135 | + |
| 136 | +<1> The credentials Secret for database and admin user access (same as any other Airflow cluster). |
| 137 | +<2> `dagsGitSync` is intentionally not configured. |
| 138 | + DAG bundles replace the git-sync sidecar entirely. |
| 139 | +<3> A YAML anchor is used to define the environment variables once at the role level and reuse them across all roles. |
| 140 | +<4> Each bundle requires an Airflow Git connection. |
| 141 | + Connections are defined as `AIRFLOW_CONN_<CONN_ID>` environment variables with a JSON value containing `conn_type` and `host` (the repository URL). |
| 142 | + For private repositories, add `login` (username) and `password` (access token) fields for HTTPS auth, or `key_file` / `private_key` in the `extra` dict for SSH auth. |
| 143 | + The connection ID in the env var name must be uppercase (e.g. `AIRFLOW_CONN_REPO1`), while the `git_conn_id` in the bundle config uses the lowercase form (`repo1`). |
| 144 | +<5> The `AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST` environment variable is a JSON list of bundle definitions. |
| 145 | + Each entry specifies a `name`, the `classpath` of the bundle backend, and `kwargs` passed to the bundle constructor. |
| 146 | + For `GitDagBundle`, the key kwargs are `git_conn_id` (referencing an Airflow connection), `tracking_ref` (branch or tag), and `subdir` (subdirectory within the repository containing DAGs). |
| 147 | +<6> The YAML anchor is referenced on all other roles so that every Airflow component sees the same bundle configuration. |
| 148 | + |
| 149 | +=== Airflow Git connection reference |
| 150 | + |
| 151 | +When using `GitDagBundle` with private repositories, credentials are configured via an https://airflow.apache.org/docs/apache-airflow-providers-git/stable/connections/git.html[Airflow Git connection]. |
| 152 | +The table below shows which capabilities of the operator's `dagsGitSync` fields have equivalents in a Git connection or `GitDagBundle`. |
| 153 | +The connection field names (`login`, `password`, `extra`) refer to the https://airflow.apache.org/docs/apache-airflow-providers-git/stable/connections/git.html[JSON field names] used in `AIRFLOW_CONN_*` environment variables. |
| 154 | +The `GitDagBundle` kwargs are documented in the https://airflow.apache.org/docs/apache-airflow-providers-git/stable/bundles/index.html[git provider bundles reference]. |
| 155 | + |
| 156 | +[cols="2,3,1"] |
| 157 | +|=== |
| 158 | +|`dagsGitSync` field |Git connection / `GitDagBundle` equivalent |Parity |
| 159 | + |
| 160 | +|`repo` |
| 161 | +|Connection `host` field, or `GitDagBundle` `repo_url` kwarg. |
| 162 | +|Full |
| 163 | + |
| 164 | +|`branch` |
| 165 | +|`GitDagBundle` `tracking_ref` kwarg. Accepts branches, tags, or commit hashes. |
| 166 | +|Full |
| 167 | + |
| 168 | +|`gitFolder` |
| 169 | +|`GitDagBundle` `subdir` kwarg. |
| 170 | +|Full |
| 171 | + |
| 172 | +|`wait` |
| 173 | +|`GitDagBundle` `refresh_interval` kwarg (integer, in seconds). |
| 174 | +|Full |
| 175 | + |
| 176 | +|`credentials.basicAuthSecretName` |
| 177 | +|Connection username and access token fields (JSON keys `login` and `password` in `AIRFLOW_CONN_*` env vars). |
| 178 | +|Full - but the user must create the Airflow connection rather than referencing a Kubernetes Secret directly. |
| 179 | + |
| 180 | +|`credentials.sshPrivateKeySecretName` (key) |
| 181 | +|Connection extra https://airflow.apache.org/docs/apache-airflow-providers-git/stable/connections/git.html[`key_file`] (path to a mounted key file) or `private_key` (inline key content). Mutually exclusive. |
| 182 | +|Full |
| 183 | + |
| 184 | +|`credentials.sshPrivateKeySecretName` (knownHosts) |
| 185 | +|Connection extra https://airflow.apache.org/docs/apache-airflow-providers-git/stable/connections/git.html[`known_hosts_file`] (path to a mounted file) and `strict_host_key_checking` (defaults to `"no"`). |
| 186 | +To replicate git-sync's known-hosts verification, set `strict_host_key_checking` to `"yes"` and provide a `known_hosts_file`. |
| 187 | +|Full |
| 188 | + |
| 189 | +|`depth` |
| 190 | +|No equivalent. `GitDagBundle` always performs a full clone. |
| 191 | +|None |
| 192 | + |
| 193 | +|`gitSyncConf` |
| 194 | +|No equivalent. There is no pass-through mechanism for arbitrary git options. |
| 195 | +|None |
| 196 | + |
| 197 | +|`tls.verification.none` |
| 198 | +|Not supported by the Git provider. Workaround: set the `GIT_SSL_NO_VERIFY=true` environment variable on the pod (applies globally to all repositories). |
| 199 | +|None |
| 200 | + |
| 201 | +|`tls.verification.server.caCert.webPki` |
| 202 | +|Implicit - Git uses the operating system's CA trust store by default. |
| 203 | +|Implicit |
| 204 | + |
| 205 | +|`tls.verification.server.caCert.secretClass` |
| 206 | +|Not supported by the Git provider. Workaround: mount the CA certificate and set the `GIT_SSL_CAINFO` environment variable on the pod, but this applies globally to *all* repositories, not per-repo. |
| 207 | +|None (global workaround only) |
| 208 | +|=== |
| 209 | + |
| 210 | +The https://airflow.apache.org/docs/apache-airflow-providers-git/stable/connections/git.html[Git connection] also supports several extra keys *not* available in `dagsGitSync`: |
| 211 | + |
| 212 | +[cols="1,3"] |
| 213 | +|=== |
| 214 | +|Connection extra key |Description |
| 215 | + |
| 216 | +|`private_key_passphrase` |
| 217 | +|Passphrase for encrypted SSH private keys. |
| 218 | + |
| 219 | +|`ssh_config_file` |
| 220 | +|Path to a custom SSH configuration file. |
| 221 | + |
| 222 | +|`host_proxy_cmd` |
| 223 | +|SSH `ProxyCommand` for connecting through bastion or jump hosts. |
| 224 | + |
| 225 | +|`ssh_port` |
| 226 | +|Non-default SSH port (set via `-p` on the SSH command). |
| 227 | +|=== |
| 228 | + |
| 229 | +`GitDagBundle` itself also accepts two additional kwargs (see the https://airflow.apache.org/docs/apache-airflow-providers-git/stable/bundles/index.html[bundles reference]): |
| 230 | + |
| 231 | +[cols="1,1,3"] |
| 232 | +|=== |
| 233 | +|Kwarg |Default |Description |
| 234 | + |
| 235 | +|`submodules` |
| 236 | +|`false` |
| 237 | +|Initialise and update Git submodules recursively. |
| 238 | + |
| 239 | +|`prune_dotgit_folder` |
| 240 | +|`true` |
| 241 | +|Remove the `.git` folder from version clones to save disk space. Forced to `false` when `submodules` is `true`. |
| 242 | +|=== |
| 243 | + |
| 244 | +=== Limitations |
| 245 | + |
| 246 | +* **No per-repository TLS/CA certificates.** The Airflow Git provider does not support custom CA certificates per connection. |
| 247 | + The only workaround is setting `GIT_SSL_CAINFO` as a pod-level environment variable, which applies to all repositories. |
| 248 | +* **No clone depth control.** `GitDagBundle` always performs a full clone. |
| 249 | + For large repositories this may increase pod startup time, particularly with the Kubernetes executor where each short-lived worker pod clones independently. |
| 250 | +* **No `gitSyncConf` equivalent.** There is no mechanism to pass arbitrary git or git-sync options through to the bundle. |
| 251 | +* **Triggerer limitation.** The Airflow triggerer does not initialise DAG bundles. |
| 252 | + Custom trigger classes cannot be loaded from a bundle and must be installed as Python packages in the image. |
| 253 | +* **Static configuration.** Bundle definitions are read from configuration at process startup. |
| 254 | + Adding or removing a bundle requires updating the `envOverrides` and restarting the affected pods. |
0 commit comments