Merge pull request #15 from dnks0/feature/azure-terraform-support

* added terraform module for Azure
* updated README's
* aligned AWS TF module behavior to match Azure

Author: dnks0

README.md

@@ -19,22 +19,23 @@ Connectivity to your custom resources can be configured via a dedicated Private
 
 - **Forwarding of L4 & L7 network traffic** based on your configuration
   - L4 (TCP): forwarding of plain TCP traffic, e.g. for databases
-  - L7 (HTTP) forwarding of HTTP(s) traffic with **SNI-based routing**, e.g. for applications/APIS
-- **Terraform module** ready to use (currently **AWS only**)
+  - L7 (HTTP) forwarding of HTTP(s) traffic with **SNI-based routing**, e.g. for applications/API's
+- **Terraform module** ready to use for **AWS and Azure**
 - No TLS termination, only passthrough!
 
 
 ### High availability
 
-`dbx-proxy` is placed behind an AWS Network Load Balancer, which spreads connections across the instances in the Auto Scaling Group. Availability depends on how many instances you run and whether your subnets span multiple AZs. See the Terraform module details for configuration and behavior: [High availability (AWS)](terraform/README.md#high-availability-aws).
+High Availability depends on how many instances you run and whether your deployment spans multiple availability zones in your cloud. See the Terraform module details for configuration and behavior: [Terraform module documentation](terraform/README.md).
 
 ### Deployment (Terraform) / How to use
 
-`dbx-proxy` essentially provides Steps 1 and 2 when following the official Databricks documentation for private connectivity to resources in your own networks:
+`dbx-proxy` provides the customer-side components (Step 1 and 2) when following the official Databricks documentation for private connectivity to resources in your own networks:
 - [(AWS) Configure private connectivity to resources in your VPC](https://docs.databricks.com/aws/en/security/network/serverless-network-security/pl-to-internal-network)
+- [(Azure) Configure private connectivity to resources in your Vnet](https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/pl-to-internal-network)
 
 
-Include the module in your Terraform stack:
+Include the module in your Terraform stack (example for AWS):
 ```hcl
 module "dbx_proxy" {
 
@@ -69,15 +70,15 @@ module "dbx_proxy" {
 }
 ```
 
-More details about the Terraform module and configurations can be found [here](terraform/README.md).
+More details about the Terraform module (including Azure) can be found [here](terraform/README.md).
 
 You will still need to configure the Databricks-side objects like NCC, private endpoint rules and accept the connection on your endpoint-service.
 
-By default the module runs in `deployment_mode = "bootstrap"` and can create networking, NLB + endpoint service. If you already have a VPC/subnets, keep `deployment_mode = "bootstrap"` and provide `vpc_id` and `subnet_ids`. If you already have an NLB as well, set `deployment_mode = "proxy-only"` and provide `vpc_id`, `subnet_ids`, and `nlb_arn` (see Terraform docs for details).
+By default the module runs in `deployment_mode = "bootstrap"` and can create networking, an internal load balancer (NLB/SLB), and a private endpoint service (Private Link). If you already have networking, keep `deployment_mode = "bootstrap"` and provide the network IDs. If you already have a load balancer as well, set `deployment_mode = "proxy-only"` and provide the load balancer ID/ARN (see Terraform docs for details).
 
 ### Troubleshooting
 
-To validate that the proxy is up and reachable,run the following from a serverless notebook:
+To validate that the proxy is up and reachable,run the following from e.g. a serverless notebook:
 
 ```bash
 %sh

resources/img/azure-architecture.png

@@ -0,0 +1,99 @@
+## AWS Terraform module: `dbx-proxy`
+
+This module deploys `dbx-proxy` on AWS, using an internal Network Load Balancer (NLB) and a VPC Endpoint Service (PrivateLink) for Databricks Serverless private connectivity.
+
+For common concepts (listener config, deployment modes, overall limitations), see the global module documentation in `terraform/README.md`.
+
+#### Architecture
+
+![AWS dbx-proxy architecture](../../resources/img/aws-architecture.png)
+
+This module provisions a private Network-Load-Balancer with target groups, an endpoint service for Private Link communication from Databricks serverless, and an autoscaling-group of `dbx-proxy` instances inside your VPC.
+In bootstrap-mode, the default subnets are created across availability-zones. The autoscaling-group automatically tries to balance instances across subnets and therefore availability-zones to achieve robustness.
+In proxy-only mode, it is your responsibility to configure subnets accordingly.
+Optional bootstrap networking creates the VPC, subnets, and NAT/IGW when not provided.
+
+---
+
+### Quick start
+
+In your existing Terraform stack, add:
+
+```hcl
+module "dbx_proxy" {
+  source = "github.com/dnks0/dbx-proxy//terraform/aws?ref=v<release>"
+
+  # AWS config
+  region = "eu-central-1"
+  tags   = {}
+
+  # dbx-proxy config
+  dbx_proxy_image_version = "<release>"
+  dbx_proxy_health_port   = 8080
+  dbx_proxy_listener      = []
+}
+```
+
+Make sure to replace `<release>` with the actual release version!
+
+Then run:
+
+```bash
+terraform init
+terraform apply
+```
+
+After apply, use the output `load_balancer.vpc_endpoint_service_name` when creating Databricks private endpoint rules in your NCC. Also, add a domain of your choice as private endpoint rule on your NCC that you can use for troubleshooting.
+
+---
+
+### AWS-specific variables
+
+| Variable | Type | Default | Description |
+|---|---:|---:|---|
+| `region` | `string` | (required) | AWS region to deploy to. |
+| `vpc_id` | `string` | `null` | Existing VPC ID. Required for `proxy-only` mode. If `null`, a VPC can be bootstrapped in `bootstrap` mode. |
+| `subnet_ids` | `list(string)` | `[]` | Existing private subnet IDs for the NLB + ASG. Required for `proxy-only` mode. If empty, subnets can be created in `bootstrap` mode. |
+| `vpc_cidr` | `string` | `"10.0.0.0/16"` | VPC CIDR (only used when creating a VPC in `bootstrap`). |
+| `subnet_cidrs` | `list(string)` | `["10.0.1.0/24", "10.0.2.0/24"]` | Private subnet CIDRs (only used when creating subnets in `bootstrap` mode). |
+| `nat_subnet_cidr` | `string` | `"10.0.0.0/24"` | Public subnet CIDR for the NAT gateway (only used when creating networking in `bootstrap` mode). |
+| `nlb_arn` | `string` | `null` | Existing NLB ARN to attach listeners/target groups to in `proxy-only` mode. |
+
+Common variables are documented in `terraform/README.md`.
+
+---
+
+### Outputs
+
+- `networking`: object with
+  - `vpc_id`
+  - `vpc_cidr`
+  - `subnet_ids`
+  - `subnet_cidrs`
+  - `nat_gateway_id`
+  - `nat_subnet_id`
+  - `nat_subnet_cidr`
+  - `internet_gateway_id`
+
+- `load_balancer`: object with
+  - `nlb_arn`
+  - `nlb_dns_name`
+  - `nlb_target_group_arns`
+  - `nlb_security_group_ids`
+  - `vpc_endpoint_service_arn`
+  - `vpc_endpoint_service_name`
+
+- `proxy`: object with
+  - `iam_role_name`
+  - `iam_role_arn`
+  - `instance_profile_name`
+  - `instance_profile_arn`
+  - `security_group_id`
+  - `autoscaling_group_name`
+  - `launch_template_name`
+  - `dbx_proxy_cfg`
+
+---
+### Notes for AWS users
+
+- Multi availability-zone resilience can be achieved by providing subnets across multiple availability-zones. By default, the autoscaling-group tries to spread dbx-proxy instances across subnets eavenly. In `proxy-only` mode, you are responsible to configure subnets accordingly. In `bootstrap` mode, default subnets are created across multiple availaiblity-zones in the selected region.

terraform/README.md

@@ -12,9 +12,9 @@ locals {
   bootstrap_networking    = var.deployment_mode == "bootstrap" && (var.vpc_id == null && length(var.subnet_ids) == 0)
   bootstrap_load_balancer = var.deployment_mode == "bootstrap"
 
-  vpc_id        = module.networking.vpc_id
-  subnet_ids    = module.networking.subnet_ids
-  subnet_cidrs  = module.networking.subnet_cidrs
+  vpc_id       = module.networking.vpc_id
+  subnet_ids   = module.networking.subnet_ids
+  subnet_cidrs = module.networking.subnet_cidrs
 
   nlb_target_group_arns = module.load_balancer.nlb_target_group_arns
 

terraform/aws/README.md

@@ -7,19 +7,19 @@ resource "random_string" "this" {
 module "networking" {
   source = "./modules/networking"
 
-  bootstrap_networking  = local.bootstrap_networking
+  bootstrap_networking = local.bootstrap_networking
 
-  prefix                = local.prefix
-  tags                  = local.tags
+  prefix = local.prefix
+  tags   = local.tags
 
-  vpc_id                = var.vpc_id
-  vpc_cidr              = var.vpc_cidr
+  vpc_id   = var.vpc_id
+  vpc_cidr = var.vpc_cidr
 
-  subnet_ids            = var.subnet_ids
-  subnet_cidrs          = var.subnet_cidrs
+  subnet_ids   = var.subnet_ids
+  subnet_cidrs = var.subnet_cidrs
 
-  nat_subnet_cidr       = var.nat_subnet_cidr
-  enable_nat_gateway    = var.enable_nat_gateway
+  nat_subnet_cidr    = var.nat_subnet_cidr
+  enable_nat_gateway = var.enable_nat_gateway
 
 }
 
@@ -28,35 +28,35 @@ module "load_balancer" {
 
   bootstrap_load_balancer = local.bootstrap_load_balancer
 
-  prefix                  = local.prefix
-  region                  = var.region
-  tags                    = local.tags
+  prefix = local.prefix
+  region = var.region
+  tags   = local.tags
 
-  nlb_arn                 = var.nlb_arn
+  nlb_arn = var.nlb_arn
 
-  vpc_id                  = local.vpc_id
-  subnet_ids              = local.subnet_ids
-  subnet_cidrs            = local.subnet_cidrs
+  vpc_id       = local.vpc_id
+  subnet_ids   = local.subnet_ids
+  subnet_cidrs = local.subnet_cidrs
 
-  dbx_proxy_health_port   = var.dbx_proxy_health_port
-  dbx_proxy_listener      = var.dbx_proxy_listener
+  dbx_proxy_health_port = var.dbx_proxy_health_port
+  dbx_proxy_listener    = var.dbx_proxy_listener
 
 }
 
 module "proxy" {
   source = "./modules/proxy"
 
-  prefix                    = local.prefix
-  tags                      = local.tags
+  prefix = local.prefix
+  tags   = local.tags
 
-  vpc_id                    = local.vpc_id
-  subnet_ids                = local.subnet_ids
-  subnet_cidrs              = local.subnet_cidrs
+  vpc_id       = local.vpc_id
+  subnet_ids   = local.subnet_ids
+  subnet_cidrs = local.subnet_cidrs
 
-  instance_type             = var.instance_type
-  min_capacity              = var.min_capacity
-  max_capacity              = var.max_capacity
-  nlb_target_group_arns     = local.nlb_target_group_arns
+  instance_type         = var.instance_type
+  min_capacity          = var.min_capacity
+  max_capacity          = var.max_capacity
+  nlb_target_group_arns = local.nlb_target_group_arns
 
   dbx_proxy_image_version   = var.dbx_proxy_image_version
   dbx_proxy_health_port     = var.dbx_proxy_health_port

terraform/aws/local.tf

@@ -2,7 +2,6 @@ locals {
 
   nlb_arn      = var.bootstrap_load_balancer ? aws_lb.this[0].arn : data.aws_lb.this[0].arn
   nlb_dns_name = var.bootstrap_load_balancer ? aws_lb.this[0].dns_name : data.aws_lb.this[0].dns_name
-  nlb_zone_id  = var.bootstrap_load_balancer ? aws_lb.this[0].zone_id : data.aws_lb.this[0].zone_id
 
   nlb_security_group_ids = var.bootstrap_load_balancer ? aws_lb.this[0].security_groups : data.aws_lb.this[0].security_groups
 

terraform/aws/main.tf

@@ -9,7 +9,7 @@ resource "aws_lb" "this" {
   security_groups    = [aws_security_group.this[0].id]
 
   enable_cross_zone_load_balancing = true
-  enable_deletion_protection = false
+  enable_deletion_protection       = false
 
   # PrivateLink traffic bypasses NLB SG ingress when this is off. We use off
   # because the service owner cannot restrict ingress by endpoint SG/CIDR without

terraform/aws/modules/load-balancer/local.tf

@@ -1,7 +1,7 @@
 locals {
 
-  vpc_id        = var.bootstrap_networking ? aws_vpc.this[0].id : var.vpc_id
-  subnet_ids    = length(var.subnet_ids) > 0 ? var.subnet_ids : [for s in aws_subnet.this : s.id]
-  subnet_cidrs  = length(var.subnet_ids) > 0 ? [for s in values(data.aws_subnet.this) : s.cidr_block] : [for s in aws_subnet.this : s.cidr_block]
+  vpc_id       = var.bootstrap_networking ? aws_vpc.this[0].id : var.vpc_id
+  subnet_ids   = length(var.subnet_ids) > 0 ? var.subnet_ids : [for s in aws_subnet.this : s.id]
+  subnet_cidrs = length(var.subnet_ids) > 0 ? [for s in values(data.aws_subnet.this) : s.cidr_block] : [for s in aws_subnet.this : s.cidr_block]
 
 }

terraform/aws/modules/load-balancer/nlb.tf

@@ -1,5 +1,6 @@
 resource "aws_vpc" "this" {
-  count                = var.bootstrap_networking ? 1 : 0
+  count = var.bootstrap_networking ? 1 : 0
+
   cidr_block           = var.vpc_cidr
   enable_dns_hostnames = true
   enable_dns_support   = true