How to deploy to AWS EKS and start up Airflow properly in Production?

[Rietty]

Hello folks! I'm trying to deploy Airflow to our AWS EKS Environments (Dev/UAT/Prod) and am having trouble setting it up properly, specifically the Airflow instance I am trying to install it from the Python Package and get it up and running.

Currently we have some EKS stuff that we deploy so I have followed the same template and got it to at least deploy. However my issue is that the system says not to run standalone in production so I'm not sure what to change.

Currently I have a pyproject.toml with a simple project, dependencies (apache-airflow==3.1.7) and a [[tool.uv.index]] for our internal PyPI mirror.

My Dockerfile simply copies these things over, installs uv resolves dependencies via sync and then just uses CMD['uv', 'run', 'airflow', 'standalone'] however this complains about the above not to use standalone in production.

So my question is how do I properly deploy app properly so:

1. Kubernetes will properly scale workers and such when workloads become high.
2. Handle things like postgresdb that are required by airflow to function? I assume there's some way to have a built-in one that can be the default. (This is how our old/existing one works.)
3. Is there some documentation about this stuff?

Please don't suggest stuff like Amazon's Managed Airflow (It's very expensive!) or to use data-on-eks which is very strongly opinionated and will not work for us as we are not looking to create a new AWS EKS cluster.

[potiuk]

You just need to manage your components in some way - systemd or the like, or even have containers or vms per container. Or you can use Docker Compose or Helm Chart / Kubernetes - but it's generally up to you to configure and deploy Airflow. All possible options are nicely described in https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html - you have all the known installation methods, describing their pros/cons - why you might want to use one, and what are your responsibilities as Deployment Manager.

Maybe you will not get detailed instructions step-by-step to do precisely what you want (deployment definition and building is Deployment Manager's responsibility) - you will find enough leads examples, quick-starts, and the Helm Chart of Airflow is used by people in production quite heavily - and it's highly customizable (too highly one might say).

[Rietty]

Is the helm chart customizable enough to use it with AWS EKS if I configure the variables for it properly?

[potiuk]

Hard to answer it yes/no. It depends mostly I guess on the many ways you choose to deploy it on EKS - each k8s installation has many options, so authoritative answer might not be possible.

But You should try. If you look at issues here-⁰ lots of people use it and you always can customize the chart by having your own and extend it Extending the Chart-with your own things https://share.google/7WCA1oQRqa4rOc17g

[akhoyannh-a11y]

• ---

[akhoyannh-a11y]