deploy_airflow.md

Composer Integration

StarThinker can be easily deployed to Apache Airflow stack and Google Cloud Composer.

All Scripts Available As Airflow Dags

Airflow can wrap any Python function in a PythonOperator. Because StarThinker tasks are just python functions with JSON parameters, they can be quickly deployed to Airflow using a simple DAG factory.

Instructions

1. First install airflow and the StarThinker package.

apache-airflow
starthinker

1. Copy a StarThinker DAG to your Airflow folder or bucket.
2. Modify the copied file.
3. Execute the DAG like any other Airflow DAG.

Sample DAG Recipe

The airflow.json recipe has three tasks in it. These three tasks represent the three possible ways to use ST with Airflow.

1. Using a native Airflow Operator in a recipe.
2. Using a custom Airflow Operator in a recipe.
3. Using any StarThinker task in a recipe.

cp dags/sample_dag.py ~/airflow/dags/sample_dag.py
python ~/airflow/dags/sample_dag.py

1. Run a native Airflow Operator as a task from the recipe:

airflow tasks test "sample" airflow_1 2020-09-14

1. Run a custom Airflow Operator as a task from the recipe:

airflow tasks test "sample" starthinker_airflow_1 2020-09-14

1. Run a StarThinker native python function as a task from the recipe:

airflow tasks test "sample" hello_1 2020-09-14

Using User Credentials In Airflow

Storing user credentials inside a connector variable will work for single jobs because the refresh token is refreshed when loaded. However, for multiple jobs the refresh token must be shared across jobs. StarThinker solves this by allowing a GCP Storage path for user credentials.

Cloud Resources

• Google Cloud Composer - where your instance will deploy.
• Google Cloud Storage - where your user credentials will be stored ( keep this secure ).
• Google Cloud Credentials - where you manage your credentials.
• Google Cloud Billing - examine costs in real time.

---

© 2019 Google LLC - Apache License, Version 2.0