Skip to content

Commit 70485e2

Browse files
authored
Migrate from PAT to Github App Authentication (GoogleCloudPlatform#773)
Update readme.md. Migrate from PAT to Github App Authentication.
1 parent cab3a78 commit 70485e2

3 files changed

Lines changed: 246 additions & 88 deletions

File tree

plugins/README.md

Lines changed: 70 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -3,17 +3,21 @@ The "on_failure_actions" plugin listens to the DAGs run results and takes action
33
## Pre-requisites:
44
To leverage the "on_failure_actions" plugin, ensure the following conditions are met:
55

6-
### 1. **DAG Opt-In:**
6+
### 1. **DAG Opt-In:**
77
Each DAG intended to utilize this feature **must include the `"on_failure_alert"` tag** within its DAG definition. DAGs without this specific tag will be ignored by the plugin's failure-handling logic, and no GitHub issue will be filed for their failures.
88

99
with DAG(
1010
dag_id='my_critical_dag',
1111
# ... other DAG parameters ...
12-
tags=['data_ingestion', 'critical', 'on_failure_alert'], # <--- Add this tag
12+
tags = [
13+
'data_ingestion',
14+
'critical',
15+
'on_failure_alert' # <--- Add this tag
16+
],
1317
) as dag:
1418
# ... tasks ...
1519

16-
### 2. **GitHub Owner Mapping:**
20+
### 2. **GitHub Owner Mapping:**
1721
For accurate issue assignment, ensure that the `owner` property defined for tests within your DAGs corresponds directly to valid **GitHub usernames**. The plugin will collect unique test owners from the failed DAG and attempt to assign the GitHub issue to these users.
1822

1923
#### Example task definition
@@ -29,20 +33,71 @@ For accurate issue assignment, ensure that the `owner` property defined for test
2933
def task_a():
3034
pass
3135

36+
### 2.5 **GitHub Owner Mapping for Test Config:**
37+
If you are using test_config.py in your DAG, you should fill test_owner attribute with your GitHub username
38+
39+
#### Example test definition
40+
maxtext_v4_configs_test = gke_config.get_gke_config(
41+
num_slices=slice_num,
42+
cluster=clusters[accelerator],
43+
time_out_in_min=60,
44+
test_name=f"maxtext-checkpointing-{mode.value}-{chkpt_mode}",
45+
run_model_cmds=command,
46+
docker_image=image.value,
47+
test_owner='github_username_here', # This should be a valid GitHub username
48+
).run()
49+
3250
## Configuration and Installation:
51+
### Create a GitHub APP
52+
Refer to this [procedure](https://docs.github.com/en/apps/creating-github-apps/registering-a-github-app/registering-a-github-app) to register a GitHub App.
53+
54+
- Give the app a unique name, for example "ml-auto-solutions-app", and record it as "APP_ID".
55+
- Choose "Issues" for the "Repository permissions".
56+
- Install it to the Target GitHub Repo (ml-auto-solutions). Record the "INSTALLATION_ID" from the web URL. For example, if the URL is https://github.com/apps/ml-auto-solutions-app/installations/78297659, then 78297659 is the INSTALLATION_ID.
57+
- Create a private key and record it as PRIVATE_KEY.
58+
59+
The APP_ID, INSTALLATION_ID, and PRIVATE_KEY will be used in the next steps.
60+
61+
### Set up via GCP console UI
3362
1. From GCP console UI, Your Composer Env -> Tab -> Pypi packages -> Edit -> Add 'apache-airflow-providers-github' -> Save
3463

35-
2. From GCP console UI, search for "Secret Manager", and add conn_id 'github_default' into Secret Manager.
36-
1. key: airflow-connections-<composer_environment_name>-github_default
37-
2. value:
38-
{
39-
"conn_type": "http",
40-
"host": "https://api.github.com",
41-
"password": "\<GitHub Personal Access Token\>"
42-
}
64+
2. From GCP console UI, search for "Secret Manager". Go to this page and click '+ CREATE SECRET' button.
65+
1. Name: airflow-connections-<composer_environment_name>-github_app
66+
Secret value:
67+
```
68+
{
69+
"app_id": "<APP_ID>",
70+
"installation_id": "<INSTALLATION_ID>",
71+
"private_key": "<PRIVATE_KEY>"
72+
}
73+
```
74+
75+
3. Upload 'on_failure_actions.py' to \<DAG Bucket\>/plugins/
76+
77+
### Set up via Google Cloud CLI
78+
1. Set the environment variables in your terminal:
79+
```
80+
COMPOSER_ENVIRONMENT_NAME=<Composer Environment Name>
81+
COMPOSER_LOCATION=<Composer Location>
82+
COMPOSER_DAG_BUCKET=gs://<DAG Bucket>
83+
APP_ID=<GitHub App Id>
84+
INSTALLATION_ID=<GitHub Installation Id>
85+
PRIVATE_KEY=<GitHub App Private Key>
86+
```
87+
88+
2. Add `apache-airflow-providers-github` Pypi package to your Composer environment.
89+
```
90+
gcloud composer environments update ${COMPOSER_ENVIRONMENT_NAME} --location=${COMPOSER_LOCATION} --update-pypi-package='apache-airflow-providers-github'
91+
```
92+
93+
3. Add a new Secret into Secret Manager for accessing GitHub App.
94+
```
95+
echo "{"app_id": "${APP_ID}", "installation_id": "${INSTALLATION_ID}", "private_key": "${PRIVATE_KEY}"}" | gcloud secrets create airflow-connections-${COMPOSER_ENVIRONMENT_NAME}-github_app --data-file=-
96+
```
97+
98+
4. Upload Composer plugins to the folder for dags in your Composer bucket
99+
```
100+
gcloud storage cp ./on_failure_actions.py ${COMPOSER_DAG_BUCKET}/plugins/
101+
```
43102
44-
3. Composer -> Airflow configuration overrides -> Edit, reference: https://cloud.google.com/composer/docs/composer-1/configure-secret-manager
45-
1. secrets | backend | airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
46-
2. secrets | backends_order | custom,environment_variable,metastore
47103
48-
4. Upload 'on_failure_actions.py' to \<DAG Bucket\>/plugins/

0 commit comments

Comments
 (0)