Merge pull request #7 from dnks0/feature/dbx-proxy

* multi instance support
* haproxy maxconn override and automatic default calculation (conservative)
* introduced deployment-modes in terraform stack: bootstrap, proxy-only
* dbx-proxy now uses arm based compute

Author: dnks0

README.md

@@ -23,15 +23,15 @@ Include the module in your Terraform stack:
 ```hcl
 module "dbx_proxy" {
 
-  source = "github.com/dnks0/dbx-proxy//terraform/aws?ref=v0.1.0"
+  source = "github.com/dnks0/dbx-proxy//terraform/aws?ref=v<release>"
 
   # aws config
   region    = "eu-central-1"
   tags      = {}
   ...
 
   # dbx-proxy config
-  dbx_proxy_image_version = "0.1.0"
+  dbx_proxy_image_version = "<release>"
   dbx_proxy_health_port   = 8080
 
   # Example: forward TCP/443 to a private target in your VPC
@@ -58,6 +58,8 @@ More details about the Terraform module and configurations can be found [here](t
 
 You will still need to configure the Databricks-side objects like NCC, private endpoint rules and accept the connection on your endpoint-service.
 
+By default the module runs in `deployment_mode = "bootstrap"` and can create networking and the NLB/endpoint service. If you already have networking use `deployment_mode = "bootstrap"` and provide `vpc_id`, and `subnet_ids`. If you already have networking/NLB, set `deployment_mode = "proxy-only"` and provide `vpc_id`, `subnet_ids`, and `nlb_arn` (see Terraform docs for details).
+
 ### Troubleshooting
 
 To validate that the proxy is up and reachable,run the following from a serverless notebook:

terraform/README.md

@@ -13,14 +13,19 @@ This repository provides a **Terraform module** for deploying `dbx-proxy` across
 
 **Always deployed (AWS implementation):**
 - **Compute**: EC2 Launch Template + Auto Scaling Group (ASG) running `dbx-proxy`
-- **Load balancing**: internal **Network Load Balancer (NLB)**
-- **Private connectivity**: **VPC Endpoint Service (PrivateLink)** backed by the NLB
 - **Networking & security**: Security Group, IAM Role + Instance Profile
 
-**Conditionally deployed:**
-- VPC, private subnets
-- Optional IGW + public subnet + NAT gateway (internet connectivity needed to pull images, etc.!)
-- Route tables + associations
+**Deployment-mode dependent:**
+- **Load balancing**
+  - `bootstrap`: creates an internal **Network Load Balancer (NLB)**
+  - `proxy-only`: attaches listeners/target groups to an existing NLB
+- **Private connectivity**
+  - `bootstrap`: creates a **VPC Endpoint Service (PrivateLink)** backed by the NLB
+  - `proxy-only`: uses the existing endpoint service (if any) outside this module
+- **Networking (when bootstrapping)**
+  - VPC, private subnets
+  - Optional IGW + public subnet + NAT gateway (internet connectivity needed to pull images, etc.!)
+  - Route tables + associations
 
 ---
 
@@ -36,15 +41,15 @@ In your existing Terraform stack, add:
 
 ```hcl
 module "dbx_proxy" {
-  source = "github.com/dnks0/dbx-proxy//terraform/aws?ref=v0.1.0"
+  source = "github.com/dnks0/dbx-proxy//terraform/aws?ref=v<release>"
 
   # AWS config
   region = "eu-central-1"
   tags   = {}
   ...
 
   # dbx-proxy config
-  dbx_proxy_image_version = "0.1.0"
+  dbx_proxy_image_version = "<release>"
   dbx_proxy_health_port   = 8080
   dbx_proxy_listener      = []
 }
@@ -57,7 +62,7 @@ terraform init
 terraform apply
 ```
 
-After apply, use the output `vpc_endpoint_service_name` when creating Databricks private endpoint rules (see Databricks guide linked above).
+After apply, use the output `load_balancer.vpc_endpoint_service_name` when creating Databricks private endpoint rules (see Databricks guide linked above).
 Also, make sure to add a domain of your choice as private endpoint rule on your NCC that you could use for [troubleshooting](../README.md#troubleshooting) purposes.
 
 ---
@@ -72,9 +77,11 @@ These variables define what the proxy should do (listeners, health port, image t
 
 | Variable | Type | Default | Description |
 |---|---:|---:|---|
-| `dbx_proxy_image_version` | `string` | `"0.1.0"` | Docker image tag/version of `dbx-proxy` to deploy. |
+| `dbx_proxy_image_version` | `string` | `"0.2.0"` | Docker image tag/version of `dbx-proxy` to deploy. |
 | `dbx_proxy_health_port` | `number` | `8080` | Health port exposed by `dbx-proxy` (HTTP `GET /status`). Also used for NLB target group health checks. |
+| `dbx_proxy_max_connections` | `number` | `null` | Optional HAProxy `maxconn` override. If unset, the AWS module derives a value from vCPU and memory of the selected instance-type. |
 | `dbx_proxy_listener` | `list(object)` | `[]` | Listener configuration (ports/modes/routes/destinations). See **Listener configuration** below. |
+| `deployment_mode` | `string` | `"bootstrap"` | Controls whether the module bootstraps networking/NLB (`bootstrap`) or attaches to existing infrastructure (`proxy-only`). See **Deployment mode behavior** below. |
 
 #### AWS-specific variables (`terraform/aws`)
 
@@ -83,26 +90,54 @@ These variables define what the proxy should do (listeners, health port, image t
 | `region` | `string` | (required) | AWS region to deploy to. |
 | `prefix` | `string` | `null` | Optional naming prefix. A randomized suffix is always appended to avoid collisions. |
 | `tags` | `map(string)` | `{}` | Extra tags applied to AWS resources (also used as provider default tags). |
-| `instance_type` | `string` | `"t3.medium"` | EC2 instance type for proxy nodes. |
-| `vpc_id` | `string` | `null` | Existing VPC ID. If `null`, the module bootstraps a VPC. |
-| `subnet_ids` | `list(string)` | `[]` | Existing private subnet IDs for the NLB + ASG. If empty, subnets are created. |
-| `vpc_cidr` | `string` | `"10.0.0.0/16"` | VPC CIDR (only used when creating a VPC). |
-| `subnet_cidrs` | `list(string)` | `["10.0.1.0/24","10.0.2.0/24"]` | Private subnet CIDRs (only used when creating subnets). |
-| `enable_nat_gateway` | `bool` | `true` | Whether to create NAT (and related IGW/public subnet) for outbound internet access (only when creating networking). |
-| `public_subnet_cidr` | `string` | `"10.0.0.0/24"` | Public subnet CIDR for the NAT gateway (only used when creating networking). |
+| `instance_type` | `string` | `"t4g.medium"` | EC2 instance type for proxy instances. |
+| `min_capacity` | `number` | `1` | Minimum number of dbx-proxy instances. |
+| `max_capacity` | `number` | `1` | Maximum number of dbx-proxy instances. |
+| `vpc_id` | `string` | `null` | Existing VPC ID. Required for `proxy-only` mode. If `null`, a VPC can be bootstrapped in `bootstrap` mode. |
+| `subnet_ids` | `list(string)` | `[]` | Existing private subnet IDs for the NLB + ASG. Required for `proxy-only` mode. If empty, subnets can be created in `bootstrap` mode. |
+| `vpc_cidr` | `string` | `"10.0.0.0/16"` | VPC CIDR (only used when creating a VPC in `bootstrap`). |
+| `subnet_cidrs` | `list(string)` | `["10.0.1.0/24","10.0.2.0/24"]` | Private subnet CIDRs (only used when creating subnets in `bootstrap` mode). |
+| `enable_nat_gateway` | `bool` | `true` | Whether to create IGW + NAT for outbound internet access (only when creating networking in `bootstrap` mode). |
+| `nat_subnet_cidr` | `string` | `"10.0.0.0/24"` | Public subnet CIDR for the NAT gateway (only used when creating networking in `bootstrap` mode). |
+| `nlb_arn` | `string` | `null` | Existing NLB ARN to attach listeners/target groups to in `proxy-only` mode. |
+
+#### Deployment mode behavior
+
+- **`bootstrap`** (default)
+  - Creates an internal NLB and a PrivateLink endpoint service.
+  - If **`vpc_id` + `subnet_ids` are provided**, the module uses existing networking.
+  - If **`vpc_id` and `subnet_ids` are not provided**, the module creates a VPC + subnets (and optionally IGW/NAT based on `enable_nat_gateway`).
+- **`proxy-only`**
+  - Requires **`vpc_id` + `subnet_ids`** and **`nlb_arn`**.
+  - Does **not** create a new NLB or PrivateLink endpoint service; it attaches listeners/target groups to the existing NLB and deploys the proxy only (ec2, security-group, NLB listener & target-groups)
 
 ---
 
 ### Outputs (AWS)
 
-- `nlb_arn`: ARN of the internal NLB
-- `vpc_endpoint_service_name`: **input** for Databricks private endpoint rules
-- `vpc_endpoint_service_arn`: ARN of the endpoint service
-- `nlb_dns_name`: internal NLB DNS name
-- `nlb_zone_id`: Route53 hosted zone id for NLB aliases
-- `autoscaling_group_name`: ASG name
-- `security_group_id`: Security group ID attached to the proxy instances
-- `target_group_arns`: listener target groups keyed by listener name
+- `networking`: object with
+  - `vpc_id`
+  - `subnet_ids`
+  - `subnet_cidrs`
+  - `internet_gateway_id`
+  - `nat_gateway_id`
+  - `nat_subnet_id`
+  - `nat_subnet_cidr`
+- `load_balancer`: object with
+  - `nlb_arn`
+  - `nlb_dns_name`
+  - `nlb_target_group_arns`
+  - `vpc_endpoint_service_arn`
+  - `vpc_endpoint_service_name`
+- `proxy`: object with
+  - `iam_role_name`
+  - `iam_role_arn`
+  - `instance_profile_name`
+  - `instance_profile_arn`
+  - `security_group_id`
+  - `autoscaling_group_name`
+  - `launch_template_name`
+  - `dbx_proxy_cfg`
 
 ---
 

terraform/aws/.tfvars

@@ -2,10 +2,12 @@ prefix="dnks"
 region="eu-central-1"
 tags={}
 
+deployment_mode="bootstrap"
+
 # AWS resources
-instance_type="t3.medium"
+instance_type="t4g.medium"
 
 # dbx-proxy configuration
-dbx_proxy_image_version="0.1.0"
+dbx_proxy_image_version="0.1.1"
 dbx_proxy_health_port=8080
 dbx_proxy_listener=[]

terraform/aws/local.tf

@@ -9,78 +9,13 @@ locals {
     var.tags,
   )
 
-  vpc_id             = var.vpc_id != null ? var.vpc_id : aws_vpc.this[0].id
-  subnet_ids         = length(var.subnet_ids) > 0 ? var.subnet_ids : [for s in aws_subnet.this : s.id]
-  subnet_cidr_blocks = length(var.subnet_ids) > 0 ? [for s in values(data.aws_subnet.this) : s.cidr_block] : [for s in aws_subnet.this : s.cidr_block]
+  bootstrap_networking    = var.deployment_mode == "bootstrap" && (var.vpc_id == null && length(var.subnet_ids) == 0)
+  bootstrap_load_balancer = var.deployment_mode == "bootstrap"
 
-  allowed_principals = [
-    "arn:aws:iam::565502421330:role/private-connectivity-role-${var.region}"
-  ]
+  vpc_id        = module.networking.vpc_id
+  subnet_ids    = module.networking.subnet_ids
+  subnet_cidrs  = module.networking.subnet_cidrs
 
-  cloud_config = {
-    write_files = [
-      {
-        path        = "/dbx-proxy/conf/dbx-proxy.cfg"
-        owner       = "root:root"
-        permissions = "0644"
-        content     = module.common.dbx_proxy_cfg
-      },
-      {
-        path        = "/dbx-proxy/docker-compose.yaml"
-        owner       = "root:root"
-        permissions = "0644"
-        content     = module.common.docker_compose
-      },
-      {
-        path        = "/etc/systemd/system/dbx-proxy.service"
-        owner       = "root:root"
-        permissions = "0644"
-        content     = <<-EOT
-        [Unit]
-        Description=dbx-proxy (docker compose)
-        Requires=docker.service
-        After=docker.service network-online.target
-        Wants=network-online.target
+  nlb_target_group_arns = module.load_balancer.nlb_target_group_arns
 
-        [Service]
-        Type=oneshot
-        RemainAfterExit=yes
-        WorkingDirectory=/dbx-proxy
-        ExecStart=/usr/bin/docker compose --file /dbx-proxy/docker-compose.yaml up --detach
-        ExecStop=/usr/bin/docker compose --file /dbx-proxy/docker-compose.yaml down
-        TimeoutStartSec=0
-
-        [Install]
-        WantedBy=multi-user.target
-        EOT
-      },
-    ]
-    runcmd = [
-      [
-        "bash",
-        "-c",
-        "set -euxo pipefail; sudo dnf update -y || sudo yum update -y || sudo apt-get update -y",
-      ],
-      [
-        "bash",
-        "-c",
-        "set -euxo pipefail; (sudo dnf install -y docker || sudo yum install -y docker || sudo apt-get install -y docker.io)",
-      ],
-      [
-        "bash",
-        "-c",
-        "set -euxo pipefail; sudo systemctl enable docker; sudo systemctl start docker",
-      ],
-      [
-        "bash",
-        "-c",
-        "set -euxo pipefail; sudo mkdir -p /usr/libexec/docker/cli-plugins; sudo curl -sSL \"https://github.com/docker/compose/releases/latest/download/docker-compose-linux-$(uname -m)\" -o /usr/libexec/docker/cli-plugins/docker-compose; sudo chmod +x /usr/libexec/docker/cli-plugins/docker-compose; sudo systemctl restart docker",
-      ],
-      [
-        "bash",
-        "-c",
-        "set -euxo pipefail; sudo systemctl daemon-reload; sudo systemctl enable --now dbx-proxy.service",
-      ],
-    ]
-  }
 }

terraform/aws/main.tf

@@ -4,11 +4,62 @@ resource "random_string" "this" {
   length  = 10
 }
 
-# Common config module: renders docker-compose.yaml
-module "common" {
-  source = "../common"
+module "networking" {
+  source = "./modules/networking"
+
+  bootstrap_networking  = local.bootstrap_networking
+
+  prefix                = local.prefix
+  tags                  = local.tags
+
+  vpc_id                = var.vpc_id
+  vpc_cidr              = var.vpc_cidr
+
+  subnet_ids            = var.subnet_ids
+  subnet_cidrs          = var.subnet_cidrs
+
+  nat_subnet_cidr       = var.nat_subnet_cidr
+  enable_nat_gateway    = var.enable_nat_gateway
+
+}
+
+module "load_balancer" {
+  source = "./modules/load-balancer"
+
+  bootstrap_load_balancer = local.bootstrap_load_balancer
+
+  prefix                  = local.prefix
+  region                  = var.region
+  tags                    = local.tags
+
+  nlb_arn                 = var.nlb_arn
+
+  vpc_id                  = local.vpc_id
+  subnet_ids              = local.subnet_ids
 
-  dbx_proxy_image_version = var.dbx_proxy_image_version
   dbx_proxy_health_port   = var.dbx_proxy_health_port
   dbx_proxy_listener      = var.dbx_proxy_listener
+
+}
+
+module "proxy" {
+  source = "./modules/proxy"
+
+  prefix                    = local.prefix
+  tags                      = local.tags
+
+  vpc_id                    = local.vpc_id
+  subnet_ids                = local.subnet_ids
+  subnet_cidrs              = local.subnet_cidrs
+
+  instance_type             = var.instance_type
+  min_capacity              = var.min_capacity
+  max_capacity              = var.max_capacity
+  nlb_target_group_arns     = local.nlb_target_group_arns
+
+  dbx_proxy_image_version   = var.dbx_proxy_image_version
+  dbx_proxy_health_port     = var.dbx_proxy_health_port
+  dbx_proxy_max_connections = var.dbx_proxy_max_connections
+  dbx_proxy_listener        = var.dbx_proxy_listener
+
 }

terraform/aws/modules/load-balancer/data.tf

@@ -0,0 +1,4 @@
+data "aws_lb" "this" {
+  count = var.bootstrap_load_balancer == false ? 1 : 0
+  arn   = var.nlb_arn
+}

terraform/aws/modules/load-balancer/local.tf

@@ -0,0 +1,11 @@
+locals {
+
+  nlb_arn      = var.bootstrap_load_balancer ? aws_lb.this[0].arn : data.aws_lb.this[0].arn
+  nlb_dns_name = var.bootstrap_load_balancer ? aws_lb.this[0].dns_name : data.aws_lb.this[0].dns_name
+  nlb_zone_id  = var.bootstrap_load_balancer ? aws_lb.this[0].zone_id : data.aws_lb.this[0].zone_id
+
+  allowed_principals = [
+    "arn:aws:iam::565502421330:role/private-connectivity-role-${var.region}"
+  ]
+
+}