Skip to content

Add EC2 start / stop functions for use in public & genie dags#1309

Open
jamesqo wants to merge 2 commits into
masterfrom
ec2-functions
Open

Add EC2 start / stop functions for use in public & genie dags#1309
jamesqo wants to merge 2 commits into
masterfrom
ec2-functions

Conversation

@jamesqo
Copy link
Copy Markdown
Contributor

@jamesqo jamesqo commented Mar 18, 2026

/cc @sheridancbio @averyniceday

  • Add turn_ec2_on and turn_ec2_off tasks that use EC2Start/StopInstanceOperator to control the KS node directly from the Airflow server
    • Approach: use saml2aws (installed on the Airflow server) to dynamically generate a new Airflow AWS connection based on the SAML credentials. Then, pass to the EC2 operators via aws_conn_id
  • Only shut down the ec2 node if it is determined that there are no active imports running
    • Go into the importer mgmt database on Clickhouse and check the status for each of the relevant portals -- in this case, public and genie
    • Decided to implement this using Airflow metadata DB -- check if import_genie_dag is running from the public DAG, and vice versa

@jamesqo jamesqo changed the title [Draft] Add EC2 start / stop functions, use in public & genie dags Add EC2 start / stop functions for use in public & genie dags Apr 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant