upgrade python to 3.12 (#1850)

## Overview
This commit upgrades the data.all framework from Python 3.9 to Python
3.12, modernizing the runtime environment and improving performance,
security, and compatibility.

## Key Changes

### 🐍 **Python Runtime Upgrade**
- **Core upgrade**: Python 3.9 → Python 3.12 across all environments
- **New runtime configuration**: Added
`deploy/stacks/runtime_options.py` to centrally manage Python version
settings
- **Lambda runtime**: Updated to use Python 3.12 runtime for AWS Lambda
functions

### 📦 **Dependency Updates** (either because they were incompatible with
3.12 or because they raised known vulnerabilities)
- **ariadne**: 0.17.0 → 0.26.2 (GraphQL library upgrade)
- **fastapi**: 0.115.5 → 0.116.1 (minor version bump)
- **PyYAML**: 6.0 → 6.0.2 (patch update)
- **aws-cdk-lib**: 2.177.0 → 2.208.0 (significant CDK upgrade)

### 🐳 **Docker Infrastructure Improvements**
- **Replace dev/Dockerfile**: Refactored prod Dockerfiles and compose to
use those in our local setup which reduces duplication but also the
local environment is close to the prod.
- **Simplified base images**: Removed custom Elasticsearch and
PostgreSQL Dockerfiles, now using official images
- **Health checks**: Added comprehensive health checks for all services
in docker-compose
- **Service dependencies**: Improved service dependency management with
condition-based startup
- **Removed wait-for-it.sh**: Eliminated custom wait script in favor of
Docker health checks
- **Use Configs, Secrets and Watch**: Replace volume mounts with the
more modern Configs (bind configs in containers), Secrets (bind secrets
in containers) and Watch (monitor for changes and sync them in the
container)

### 🏗️ **Container Orchestration Enhancements**
- **PostgreSQL**: Switched to official `postgres:16.4` image (match the
version used in Aurora)
- **Elasticsearch**: Switched to official
`docker.elastic.co/elasticsearch/elasticsearch:7.9.3` image
- **Service isolation**: Better container naming and networking
configuration
- **Startup reliability**: Enhanced service startup sequence with proper
health monitoring

### 🧹 **Code Cleanup**
- **Removed files**: 
  - `backend/docker/dev/wait-for-it.sh` (replaced by health checks)
  - `compose/elasticsearch/Dockerfile` and config (using official image)
  - `compose/postgres/Dockerfile` (using official image)

Resolve #1842

Author: petrkalos

.github/workflows/alembic-tests.yml

@@ -16,7 +16,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [ 3.9 ]
+        python-version: [ 3.12 ]
     services:
       postgres:
         image: postgres
@@ -41,4 +41,4 @@ jobs:
       - name: Drop tables
         run: make drop-tables
       - name: Upgrade tables
-        run: make upgrade-db
+        run: make upgrade-db

.github/workflows/ash.yml

@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [3.9]
+        python-version: [3.12]
     steps:
       - uses: actions/checkout@v4
       - name: Set up Python ${{ matrix.python-version }}

.github/workflows/bandit.yml

@@ -17,7 +17,7 @@ jobs:
   bandit:
     strategy:
       matrix:
-        python-version: [3.9]
+        python-version: [3.12]
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4

.github/workflows/cdk-nag.yml

@@ -21,7 +21,7 @@ jobs:
   cdk-nag:
     strategy:
       matrix:
-        python-version: [3.9]
+        python-version: [3.12]
     env:
       CDK_DEFAULT_REGION: eu-west-1
       CDK_DEFAULT_ACCOUNT: 111111111111

.github/workflows/checkov.yml

@@ -24,7 +24,7 @@ jobs:
       DATAALL_REPO_BRANCH: main
     strategy:
       matrix:
-        python-version: [3.9]
+        python-version: [3.12]
     services:
       postgres:
         image: postgres

.github/workflows/integration-tests.yml

@@ -14,7 +14,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [ 3.9 ]
+        python-version: [ 3.12 ]
     services:
       postgres:
         image: postgres
@@ -37,4 +37,4 @@ jobs:
           python-version: ${{ matrix.python-version }}
           cache: 'pip'
       - name: coverage
-        run: make coverage
+        run: make coverage

.github/workflows/snyk.yaml

@@ -14,7 +14,7 @@ jobs:
   security:
     strategy:
       matrix:
-        python-version: [3.9]
+        python-version: [3.12]
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4

Makefile

@@ -79,7 +79,7 @@ coverage: upgrade-pip install-backend install-cdkproxy install-tests
 		--color=yes
 
 deploy-image:
-	docker build -f backend/docker/prod/${type}/Dockerfile -t ${image-tag}:${image-tag} . && \
+	docker build ${build-args} -f backend/docker/prod/${type}/Dockerfile -t ${image-tag}:${image-tag} . && \
 	aws ecr get-login-password --region ${region} | docker login --username AWS --password-stdin ${account}.dkr.ecr.${region}.amazonaws.com && \
 	docker tag ${image-tag}:${image-tag} ${account}.dkr.ecr.${region}.amazonaws.com/${repo}:${image-tag} && \
 	docker push ${account}.dkr.ecr.${region}.amazonaws.com/${repo}:${image-tag}

backend/dataall/base/cdkproxy/cdk_cli_wrapper.py

@@ -216,7 +216,7 @@ def describe_stack(stack, engine: Engine = None, stackid: str = None):
 
 
 def cdk_installed():
-    cmd1 = ['.', '~/.nvm/nvm.sh']
+    cmd1 = ['bash', '~/.nvm/nvm.sh']
     logger.info(f'Running command {" ".join(cmd1)}')
     subprocess.run(
         cmd1,

backend/dataall/base/cdkproxy/requirements.txt

@@ -1,9 +1,9 @@
-aws-cdk-lib==2.177.0
+aws-cdk-lib==2.208.0
 boto3==1.35.26
 boto3-stubs==1.35.26
 cdk-nag==2.7.2
-fastapi == 0.115.5
-PyYAML==6.0
+fastapi == 0.116.1
+PyYAML==6.0.2
 requests==2.32.4
 tabulate==0.8.9
 uvicorn==0.15.0