You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -135,7 +135,7 @@ Click on the `run_every_minute` box in the centre of the page to select the logs
135
135
In this demo, the KubernetesExecutor is deployed which means that logs are only preserved (and available in the UI) if either remote logging or the SDP logging framework is configured.
136
136
In this demo we set up remote logging using S3/Minio.
137
137
Since Minio in this case is set up with TLS, the Airflow connection requires that the webserver has access to a relevant certificate and that every pod has environment variables containing the access and secret keys.
138
-
See the https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/kubernetes.html#managing-dags-and-logs[Airflow Documentation] for more details.
138
+
See the https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/kubernetes_executor.html#managing-dags-and-logs[Airflow Documentation{external-link-icon}^] for more details.
139
139
140
140
If you are interested in persisting the logs using the SDP logging framework, take a look at the xref:logging.adoc[] demo.
The DAG can always be triggered manually again via REST or from within the Webserver UI.
157
157
158
158
image::airflow-scheduled-job/airflow_10.png[]
@@ -270,15 +270,15 @@ If you switch to the `Code` tab you will see the following:
270
270
----
271
271
272
272
The task checks the configuration, runs a task that inserts some dummy data into a table, and then runs some tests to verify the result.
273
-
The details of the simple DBT project can be found https://github.com/stackabletech/demos/tree/main/demos/airflow-scheduled-job/dbt/dbt_test[in the demos repository].
273
+
The details of the simple DBT project can be found https://github.com/stackabletech/demos/tree/main/demos/airflow-scheduled-job/dbt/dbt_test[in the demos repository{external-link-icon}^].
274
274
275
275
== Patching Airflow to stress-test DAG parsing using relevant environment variables
276
276
277
277
Make sure you are still logged in as `admin`.
278
278
The demo also created a third DAG in the ConfigMap, called `dag_factory.py`, which was not mounted to the cluster and therefore does not appear in the UI.
279
279
This DAG can be used to create a number of individual DAGs on-the-fly, thus allowing a certain degree of stress-testing of the DAG scan/register steps (the generated DAGs themselves are trivial and so this approach will not really increase the burden of DAG _parsing_).
280
280
To show these individual DAGs in the overall list (and to remove the existing ones), adjust the volumeMounts as shown below.
281
-
The patch also sets some environment variables that can be used to change the frequency of certain operations. The descriptions can be found here: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html[window=_blank].
281
+
The patch also sets some environment variables that can be used to change the frequency of certain operations. The descriptions can be found in the https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html[Airflow configuration reference{external-link-icon}^].
0 commit comments