Merge pull request #51 from stackitcloud/23-implement-vpn

Add VPN + OPNSense firewall config

Author: mahauber

.gitignore

@@ -50,3 +50,10 @@ openai.token
 # Local persistent E2E runtime variables (canonical for this troubleshooting phase)
 src/config/e2e.tfvars
 src/config/e2e-bootstrap.override.tfvars
+
+# Cached OPNsense API key created by the firewall bootstrap
+.firewall-api-credentials.json
+src/.firewall-api-credentials.json
+
+# Ignore QEMU disk images (firewall)
+*.qcow2

docs/architecture.md

@@ -33,6 +33,7 @@ All modules live under `src/modules/`. The root `src/main.tf` calls them in depe
 | `devops` | `src/modules/devops/` | Optional DevOps project with managed Git instance |
 | `landing-zone` | `src/modules/landing-zone/` | Per-workload project: network, RBAC, Secrets Manager, object storage, DNS child zone, routing |
 | `sandboxes` | `src/modules/sandboxes/` | Lightweight sandbox projects for experimentation |
+| `firewall-config` | `src/modules/firewall-config/` | Optional: rules, NAT and internet hardening pushed to the OPNsense appliance through its API |
 
 ### Governance
 
@@ -103,9 +104,12 @@ When `connectivity.firewall` is set, a VM running OPNsense (provided as a `.qcow
 | `vtnet0` (WAN) | `wan_network`: attached to the WAN routing table | Outbound internet egress, assigned a static public IP |
 | `vtnet1` (LAN) | `lan_network`: a dedicated private subnet | Internal next-hop for all corporate landing zone traffic |
 
-The firewall's LAN IP is exported as `firewall_next_hop_ip` and passed to every corporate landing zone so they can point their default route at it.
+The firewall's LAN IP is exported as `firewall_next_hop_ip` and passed to every corporate landing zone so they can point their default route at it. Everything the platform sends outwards therefore crosses the appliance.
 
-Source: `src/modules/connectivity/`
+Both interfaces are DHCP clients and pick up the fixed addresses STACKIT assigns them. The image boots with almost no policy: only the two default `allow LAN to any` rules (IPv4 and IPv6), no gateways, no static routes, no IPsec, and outbound NAT on `automatic`. It filters nothing until a policy is pushed.
+
+> [!WARNING]
+> After the first apply the web GUI answers on the public IP to the whole internet, and it still carries the `root` password shipped in the image, which is identical on every copy. Port 443 is open, 80 and 22 are not, and there is no security group in front of the WAN interface. Pushing the policy closes it, so do not leave a deployment sitting between the two applies — see [Configure OPNsense firewall](getting-started.md#configure-opnsense-firewall). Binding the GUI to the LAN interface only (**System → Settings → Administration → Listen Interfaces**) is the alternative that does not depend on the ruleset.
 
 ### Landing Zone
 
@@ -219,6 +223,39 @@ Traffic flow for a corporate landing zone (firewall flavor):
 
 East-west traffic between corporate LZs stays within the Network Area and can be permitted or denied by firewall policies.
 
+## Site-to-Site VPN (optional)
+
+Any hub-spoke flavor can terminate an IPsec VPN in the connectivity project, connecting the Network Area to on-premises or another cloud. Enable it with the `connectivity.vpn` block — see the commented example in `src/config/hub-and-spoke.tfvars`.
+
+The gateway is highly available: it runs two tunnels in separate availability zones, each with its own public IP. Because both sides need the other's address, roll it out in two applies — provision the gateway with `connections = {}`, read `tofu output connectivity_vpn_public_ips`, configure the remote peer, then add the connection.
+
+Pre-shared keys are kept out of the config object in the separate `vpn_pre_shared_keys` variable, so the tunnel topology stays committable:
+
+```bash
+export TF_VAR_vpn_pre_shared_keys='{"onprem"={"tunnel1"="<20+ chars>","tunnel2"="<20+ chars>"}}'
+```
+
+Routes to the remote prefixes are distributed through the Network Area automatically, so every corporate landing zone reaches the remote site without per-spoke configuration.
+
+### What goes through the firewall
+
+In the firewall flavor, two of the four traffic directions run through the appliance cleanly. Neither VPN direction can be filtered with the managed gateway: one breaks if you try, the other is not steerable at all.
+
+| Direction | Through the firewall? | Mechanism |
+|---|---|---|
+| LZ → Internet | Yes, by default | Default route of the landing zone routing table points at the firewall LAN address. Needs an `outbound_nat` entry too, otherwise traffic reaches the appliance and stops there. |
+| LZ → LZ | Yes, by default | `system_routes = false` suppresses project-to-project routes, so spoke-to-spoke falls to the default route. Needs a static route back into the Network Area, otherwise the traffic leaves through WAN and the egress NAT rewrites the source address. |
+| LZ → on-premises (VPN) | **No** | The VPN prefix beats `0.0.0.0/0`, so it bypasses the appliance. |
+| on-premises → LZ (VPN) | **No** | Delivered straight to the spoke through the STACKIT-managed routing table. Not steerable. |
+
+Inbound VPN traffic bypasses the appliance because `static_routes` of a VPN connection are distributed as *dynamic* routes across the Network Area, and a remote prefix is always more specific than `0.0.0.0/0`. The VPN gateway itself forwards through a STACKIT-managed routing table that cannot be modified or reassigned.
+
+> [!WARNING]
+> Setting `dynamic_routes = false` does not fix the outbound direction, it breaks it. Outbound then goes through the firewall while inbound still bypasses it, and the stateful filter drops the half-flow it sees.
+
+If VPN traffic has to be inspected, terminate the tunnel on the OPNsense appliance itself instead of using the managed gateway — it is already the default route for every corporate landing zone, so both directions stay symmetric. The trade-off is losing the managed gateway's active-active HA.
+
+
 ## Resource Naming
 
 All resources follow a consistent convention driven by `company_code`:

docs/getting-started.md

@@ -27,6 +27,10 @@ Three ready-to-use configurations are provided in `src/config/`:
 
 Choose the flavour that matches your requirements and adjust the corresponding `.tfvars` file before deployment (step 7). At a minimum, update `owner_email`, `organization_id`, `company_name`, and `company_code`.
 
+The firewall flavour takes one extra step: the appliance boots unconfigured and its policy is pushed in a second apply, from the `firewall_config` block that ships commented out in the same `.tfvars` file. Until then it filters nothing and its web GUI is reachable from the internet — see [Configure OPNsense firewall](#configure-opnsense-firewall).
+
+Both hub-spoke flavours can additionally terminate a site-to-site IPsec VPN in the hub. It is disabled by default — see the commented `connectivity.vpn` block in the `.tfvars` file and [Site-to-Site VPN](architecture.md#site-to-site-vpn-optional). If you deploy the firewall flavour, read [what traffic the firewall actually sees](architecture.md#what-goes-through-the-firewall) before relying on it for VPN inspection.
+
 > [!NOTE]
 > This single-root-module approach works well for smaller environments. At larger scale — typically beyond 10 landing zones — you may encounter STACKIT API rate limits during applies and slower plan/refresh cycles due to a growing state file. Tools like [Terragrunt](https://terragrunt.gruntwork.io/), [Terramate](https://terramate.io/), or [Spacelift](https://spacelift.io/) can help by splitting landing zones into isolated state files and orchestrating root module calls with proper concurrency controls. If you are planning a larger enterprise deployment, reach out to [STACKIT](https://stackit.de) or a partner offering a verified landing zone solution via the [STACKIT Marketplace](https://marketplace.stackit.cloud/de/catalog?marketplaceFilters=industries:Service%20%26%20IT%20Provider,deliveryMethod:PROFESSIONAL_SERVICE,categories:DevOps).
 
@@ -124,6 +128,17 @@ tofu init
 
 ### 9. Deploy the landing zone
 
+If you enabled the `connectivity.vpn` block, export the pre-shared keys as an environment variable first. They are kept out of the `tfvars` on purpose:
+
+```bash
+export TF_VAR_vpn_pre_shared_keys='{"onprem"={"tunnel1"="<20+ chars>","tunnel2"="<20+ chars>"}}'
+```
+
+> [!NOTE]
+> If you deploy the provided OPNsense firewall make sure to follow [Configure OPNsense firewall](#configure-opnsense-firewall) afterwards. Until then the appliance filters nothing and its web GUI answers on the public IP.
+
+In any case run opentofu to deploy the infrastructure:
+
 ```bash
 tofu apply
 ```
@@ -221,11 +236,84 @@ stackit project delete --project-id <BOOTSTRAP_PROJECT_ID>
 
 ## Post-Deployment (Optional)
 
-### DNS automation for Gateway API resources
+### Configure OPNsense firewall
+
+Step 9 boots the appliance but leaves it unconfigured: it filters nothing and its web GUI answers on the public IP. Pushing a policy is a **second apply**, because the API key is derived from the appliance login and a provider configuration has to resolve before any resource is planned, so a key created during an apply cannot configure the provider in that same run.
+
+What the policy does, and which traffic directions it can actually see, is in [What goes through the firewall](architecture.md#what-goes-through-the-firewall).
+
+> [!NOTE]
+> The policy is pushed with the [`browningluke/opnsense`](https://registry.terraform.io/providers/browningluke/opnsense/latest/docs) provider. It maps OPNsense's expanded read format back to what was written, so in-place updates and drift detection work — the generic `Mastercard/restapi` provider cannot do this and makes objects effectively write-once. It is community maintained and its author advises against production use, so weigh that before relying on it.
+
+**1. Enable the policy**
+
+Uncomment the `firewall_config` block in `config/hub-and-spoke-firewall.tfvars`. It comes with default rules: internet egress with NAT, spoke-to-spoke limited to HTTPS and ICMP, and the two floating rules that take the web GUI off the internet.
+
+Two entries need your values:
+
+```hcl
+# Public: OpenTofu runs outside the Network Area* — a workstation, or a CI runner on the public internet. This is the normal case for a first deployment, because nothing is inside the area yet. Traffic goes over the appliance's public address:
+
+endpoint      = "https://<connectivity_firewall_public_ip>" # get it from running "tofu output connectivity_firewall_public_ip"
+fw_management = {
+  content = ["10.0.0.0/16", "<your public address>/32"] # get it eg by running "curl -s https://ifconfig.me" or better use a fixed ip range
+}
+
+
+# Private: OpenTofu runs inside the Network Area* — a CI runner in a landing zone, a jumphost, or an established site-to-site VPN. Nothing to look up: the LAN address is the default, and your source address is already covered by `10.0.0.0/16`:
+
+# omit endpoint
+fw_management = {
+  content = ["10.0.0.0/16"]
+}
+```
+
+> [!WARNING]
+> Whatever runs OpenTofu must be covered by the `fw_management` alias before this apply. Otherwise the same run that closes the GUI cuts off its own path to the API, and recovery goes through the serial console (`stackit server console <server-id> --project-id <connectivity-project-id>`).
+>
+> A dynamic home or office address will eventually change and lock out a later run.
+
+**2. Bootstrap the API key, then push the policy**
+
+```bash
+tofu apply -var firewall_bootstrap=true && tofu apply -var firewall_bootstrap=true
+```
+
+The first `tofu apply` derives the API key (waiting for the appliance to finish booting, which takes a few minutes) and caches it in `src/.firewall-api-credentials.json`, gitignored. The second `tofu apply` stores it in the management Secrets Manager and pushes aliases, static routes, rules and NAT. 
+
+Two runs are unavoidable: the provider needs the key at plan time, and a value created during an apply cannot configure a provider in that same run.
+
+The appliance login it uses comes from `firewall_admin_password`, which defaults to the `root` password baked into the STACKIT OPNsense image, so nothing has to be exported for a fresh deployment.
+
+> [!IMPORTANT]
+> Change the password on the appliance (**System → Access → Users**). The password is only used to derive the API key. Once the key exists, it is no longer read at all.
+
+**3. Drop the bootstrap variable and delete the cache file**
+
+From here on every apply is a plain `tofu apply` with no variables:
+
+```bash
+rm src/.firewall-api-credentials.json
+tofu apply
+```
+
+**Rotating the key**
+
+Re-running the bootstrap revokes every existing API key on the appliance and mints a fresh one, which immediately invalidates the copy in the Secrets Manager. The follow-up is therefore not optional: bump `firewall_api_secret_version` by one, so the new key is written through. Forgetting the bump fails with a 401 instead of silently keeping a dead key.
+
+```bash
+rm -f src/.firewall-api-credentials.json
+tofu apply -var firewall_bootstrap=true   # mints the new key
+# bump firewall_api_secret_version in your tfvars, then
+tofu apply -var firewall_bootstrap=true   # writes it to the Secrets Manager
+rm src/.firewall-api-credentials.json
+```
+
+### Kubernetes: DNS automation for Gateway API resources
 
 For Gateway API resources (for example Envoy Gateway with `Gateway` + `HTTPRoute`), use DNS records directly via `stackit_dns_record_set` until native provider support for `extensions.dns.gatewayApi` is available.
 
-For the existing sample content in this repository (`landing_zone_sample_gateway` + `landing_zone_sample_http_route` in `src/namespace-service.tf`), the DNS record is created automatically based on the Envoy Gateway LoadBalancer endpoint discovered via `kubernetes_resources`.
+For the existing sample content in this repository (`landing_zone_sample_gateway` + `landing_zone_sample_http_route` in `src/_landing-zone-kubernetes.tf`), the DNS record is created automatically based on the Envoy Gateway LoadBalancer endpoint discovered via `kubernetes_resources`.
 
 Implementation pattern:
 
@@ -260,8 +348,4 @@ resource "stackit_dns_record_set" "landing_zone_sample_gateway" {
 }
 ```
 
-This ensures a stable, Terraform-managed DNS path without external scripts until provider-native `gatewayApi` DNS extension support is available.
-
-### Configure OPNsense firewall
-
-If you deployed the Hub-Spoke + Firewall flavour, configure the OPNsense. Guidance will be available soon in the STACKIT docs.
+This ensures a stable, Terraform-managed DNS path without external scripts until provider-native `gatewayApi` DNS extension support is available.

mise.lock

@@ -0,0 +1,97 @@
+########################
+## FIREWALL BOOTSTRAP ##
+########################
+
+locals {
+  firewall_endpoint = try(
+    coalesce(
+      try(var.firewall_config.endpoint, null),
+      "https://${coalesce(
+        var.connectivity.firewall.lan_ip,
+        cidrhost(var.connectivity.firewall.lan_network_range, 4)
+      )}"
+    ),
+    "https://127.0.0.1"
+  )
+  firewall_bootstrapped_credentials = try(jsondecode(file("${path.root}/.firewall-api-credentials.json")), null)
+
+  firewall_secretsmanager_credentials = try({
+    api_key    = ephemeral.vault_kv_secret_v2.firewall_api[0].data.api_key
+    api_secret = ephemeral.vault_kv_secret_v2.firewall_api[0].data.api_secret
+  }, null)
+
+  firewall_api_credentials = coalesce(
+    var.firewall_api_credentials,
+    local.firewall_bootstrapped_credentials,
+    local.firewall_secretsmanager_credentials,
+    { api_key = "unset", api_secret = "unset" },
+  )
+
+  firewall_config_enabled = var.firewall_config != null && (
+    var.firewall_api_credentials != null ||
+    !var.firewall_bootstrap ||
+    local.firewall_bootstrapped_credentials != null
+  )
+}
+
+############################################
+## FIREWALL API KEY SECRET  - FIRST APPLY ##
+############################################
+
+resource "terraform_data" "firewall_api_bootstrap" {
+  count = (
+    try(var.connectivity.firewall, null) != null &&
+    var.firewall_bootstrap &&
+    var.firewall_api_credentials == null &&
+    local.firewall_bootstrapped_credentials == null
+  ) ? 1 : 0
+
+  triggers_replace = [
+    local.firewall_endpoint,
+    try(module.connectivity[0].firewall_public_ip, ""),
+  ]
+
+  provisioner "local-exec" {
+    command     = "'${path.module}/modules/firewall-config/scripts/bootstrap-api-key.sh' '${local.firewall_endpoint}' '${var.firewall_admin_username}'"
+    interpreter = ["/usr/bin/env", "bash", "-c"]
+
+    environment = {
+      OPNSENSE_PASSWORD = var.firewall_admin_password
+      OUTPUT_FILE       = "${path.root}/.firewall-api-credentials.json"
+    }
+  }
+}
+
+resource "vault_kv_secret_v2" "firewall_api_credentials" {
+  count = (
+    try(var.connectivity.firewall, null) != null && (
+      local.firewall_bootstrapped_credentials != null ||
+      (var.firewall_config != null && !var.firewall_bootstrap)
+    )
+  ) ? 1 : 0
+
+  mount               = module.management.secretsmanager_instance_id
+  name                = "firewall_api_${replace(var.company_code, "-", "_")}_pltfm_hub_prod"
+  delete_all_versions = true
+  data_json_wo = jsonencode(
+    local.firewall_bootstrapped_credentials != null
+    ? local.firewall_bootstrapped_credentials
+    : local.firewall_secretsmanager_credentials
+  )
+  data_json_wo_version = var.firewall_api_secret_version
+}
+
+#############################################
+## FIREWALL API KEY SECRET  - SECOND APPLY ##
+#############################################
+
+ephemeral "vault_kv_secret_v2" "firewall_api" {
+  count = (
+    try(var.connectivity.firewall, null) != null &&
+    var.firewall_config != null &&
+    !var.firewall_bootstrap
+  ) ? 1 : 0
+
+  mount = module.management.secretsmanager_instance_id
+  name  = "firewall_api_${replace(var.company_code, "-", "_")}_pltfm_hub_prod"
+}