feat(firewall-config): enhance firewall configuration documentation and bootstrap logic

Author: mahauber

docs/getting-started.md

@@ -128,6 +128,14 @@ tofu init
 
 ### 9. Deploy the landing zone
 
+If you enabled the `connectivity.vpn` block, export the pre-shared keys as an environment variable first. They are kept out of the `tfvars` on purpose:
+
+```bash
+export TF_VAR_vpn_pre_shared_keys='{"onprem"={"tunnel1"="<20+ chars>","tunnel2"="<20+ chars>"}}'
+```
+
+In any case run opentofu to deploy the infrastructure:
+
 ```bash
 tofu apply
 ```
@@ -225,11 +233,93 @@ stackit project delete --project-id <BOOTSTRAP_PROJECT_ID>
 
 ## Post-Deployment (Optional)
 
+### Configure OPNsense firewall
+
+Step 9 boots the appliance but leaves it unconfigured: it filters nothing and its web GUI answers on the public IP. Pushing a policy is a **second apply**, because the API key is derived from the appliance login and a provider configuration has to resolve before any resource is planned, so a key created during an apply cannot configure the provider in that same run.
+
+What the policy does, and which traffic directions it can actually see, is in [What goes through the firewall](architecture.md#what-goes-through-the-firewall).
+
+> [!NOTE]
+> The policy is pushed with the [`browningluke/opnsense`](https://registry.terraform.io/providers/browningluke/opnsense/latest/docs) provider. It maps OPNsense's expanded read format back to what was written, so in-place updates and drift detection work — the generic `Mastercard/restapi` provider cannot do this and makes objects effectively write-once. It is community maintained and its author advises against production use, so weigh that before relying on it.
+
+**1. Decide which firewall interface OpenTofu should talk to**
+
+*Public: OpenTofu runs outside the Network Area* — a workstation, or a CI runner on the public internet. This is the normal case for a first deployment, because nothing is inside the area yet. Traffic goes over the appliance's public address:
+
+```bash
+tofu output connectivity_firewall_public_ip   # the endpoint
+curl -s https://ifconfig.me                   # your own public address
+```
+
+*Private: OpenTofu runs inside the Network Area* — a CI runner in a landing zone, a jumphost, or an established site-to-site VPN. Nothing to look up: the LAN address is the default, and your source address is already covered by `10.0.0.0/16`.
+
+**2. Enable the policy**
+
+Uncomment the `firewall_config` block in `config/hub-and-spoke-firewall.tfvars`. It comes with default rules: internet egress with NAT, spoke-to-spoke limited to HTTPS and ICMP, and the two floating rules that take the web GUI off the internet.
+
+Two entries need your values, and what you put there follows from step 1:
+
+```hcl
+# Example for public
+
+endpoint      = "https://<connectivity_firewall_public_ip>"
+fw_management = {
+  content = ["10.0.0.0/16", "<your public address>/32"]
+}
+
+
+# Example for private
+
+# omit endpoint
+fw_management = {
+  content = ["10.0.0.0/16"]
+}
+```
+
+> [!WARNING]
+> Whatever runs OpenTofu must be covered by the `fw_management` alias before this apply. Otherwise the same run that closes the GUI cuts off its own path to the API, and recovery goes through the serial console (`stackit server console <server-id> --project-id <connectivity-project-id>`).
+>
+> A dynamic home or office address will eventually change and lock out a later run.
+
+**3. Bootstrap the API key, then push the policy**
+
+```bash
+tofu apply -var firewall_bootstrap=true && tofu apply -var firewall_bootstrap=true
+```
+
+The first pass derives the API key (waiting for the appliance to finish booting, which takes a few minutes) and caches it in `src/.firewall-api-credentials.json`, gitignored. The second stores it in the management Secrets Manager and pushes aliases, static routes, rules and NAT. Two passes are unavoidable: the provider needs the key at plan time, and a value created during an apply cannot configure a provider in that same run, which is why they chain safely as one command.
+
+The appliance login it uses comes from `firewall_admin_password`, which defaults to the `root` password baked into the STACKIT OPNsense image, so nothing has to be exported for a fresh deployment.
+
+> [!IMPORTANT]
+> That default is baked into the image, and between the two applies the web GUI is reachable from the internet. Change the password on the appliance (**System → Access → Users**). The password is only used to derive the API key. Once the key exists, it is no longer read at all.
+
+**4. Drop the bootstrap variable and delete the cache file**
+
+From here on every apply is a plain `tofu apply` with no variables:
+
+```bash
+rm src/.firewall-api-credentials.json
+tofu apply
+```
+
+**Rotating the key**
+
+Re-running the bootstrap revokes every existing API key on the appliance and mints a fresh one, which immediately invalidates the copy in the Secrets Manager. The follow-up is therefore not optional: bump `firewall_api_secret_version` by one, so the new key is written through. Forgetting the bump fails loudly with a 401 instead of silently keeping a dead key.
+
+```bash
+rm -f src/.firewall-api-credentials.json
+tofu apply -var firewall_bootstrap=true   # mints the new key
+# bump firewall_api_secret_version in your tfvars, then
+tofu apply -var firewall_bootstrap=true   # writes it to the Secrets Manager
+rm src/.firewall-api-credentials.json
+```
+
 ### DNS automation for Gateway API resources
 
 For Gateway API resources (for example Envoy Gateway with `Gateway` + `HTTPRoute`), use DNS records directly via `stackit_dns_record_set` until native provider support for `extensions.dns.gatewayApi` is available.
 
-For the existing sample content in this repository (`landing_zone_sample_gateway` + `landing_zone_sample_http_route` in `src/namespace-service.tf`), the DNS record is created automatically based on the Envoy Gateway LoadBalancer endpoint discovered via `kubernetes_resources`.
+For the existing sample content in this repository (`landing_zone_sample_gateway` + `landing_zone_sample_http_route` in `src/_landing-zone-kubernetes.tf`), the DNS record is created automatically based on the Envoy Gateway LoadBalancer endpoint discovered via `kubernetes_resources`.
 
 Implementation pattern:
 

src/_firewall-bootstrap.tf

@@ -63,7 +63,12 @@ resource "terraform_data" "firewall_api_bootstrap" {
 }
 
 resource "vault_kv_secret_v2" "firewall_api_credentials" {
-  count = (try(var.connectivity.firewall, null) != null && (local.firewall_bootstrapped_credentials != null || !var.firewall_bootstrap)) ? 1 : 0
+  count = (
+    try(var.connectivity.firewall, null) != null && (
+      local.firewall_bootstrapped_credentials != null ||
+      (var.firewall_config != null && !var.firewall_bootstrap)
+    )
+  ) ? 1 : 0
 
   mount               = module.management.secretsmanager_instance_id
   name                = "firewall_api_${replace(var.company_code, "-", "_")}_pltfm_hub_prod"
@@ -81,7 +86,11 @@ resource "vault_kv_secret_v2" "firewall_api_credentials" {
 #############################################
 
 ephemeral "vault_kv_secret_v2" "firewall_api" {
-  count = (try(var.connectivity.firewall, null) != null && !var.firewall_bootstrap) ? 1 : 0
+  count = (
+    try(var.connectivity.firewall, null) != null &&
+    var.firewall_config != null &&
+    !var.firewall_bootstrap
+  ) ? 1 : 0
 
   mount = module.management.secretsmanager_instance_id
   name  = "firewall_api_${replace(var.company_code, "-", "_")}_pltfm_hub_prod"

src/config/hub-and-spoke-firewall.tfvars

@@ -117,31 +117,72 @@ connectivity = {
 ## FIREWALL POLICY ##
 #####################
 
-# Rules and NAT pushed to the OPNsense appliance through its API.
+# Rules, routes and NAT pushed to the OPNsense appliance through its API.
 #
-# Setup: export TF_VAR_firewall_admin_password='...' and run `tofu apply` twice. The
-# first pass creates the appliance and derives the API key, the second pushes the
-# policy. A provider config must resolve at plan time, so the key cannot be created
-# and used in the same run.
-#
-# The API listens on the firewall's LAN address, so OpenTofu has to run inside the network area or site-to-site VPN
-# That source must be in fw_management before the block rule below, or you lock yourself out
+# The firewall needs to exist before configuring it. Bootstrapping is described here:
+# docs/getting-started.md#configure-opnsense-firewall 
 #
 # firewall_config = {
+#   endpoint = "https://198.51.100.20" # check bootstrap docs
+#
 #   aliases = {
+#     # Everything routed inside the network area: all landing zones plus the appliance's
+#     # own LAN and WAN prefixes. Keep in sync with connectivity.network_area.ranges.
+#     network_area = {
+#       description = "All prefixes routed inside the STACKIT network area"
+#       content     = ["10.0.0.0/16"]
+#     }
+#
+#     # Sources allowed to reach the web GUI and the API. The whole network area is the
+#     # loosest setting that still keeps the GUI off the public internet. Narrow it to the
+#     # subnet the OpenTofu runner and the operators sit in once that is known.
 #     fw_management = {
 #       description = "May administer the firewall"
-#       content     = ["10.0.0.0/16", "203.0.113.10/32"]
+#       content     = ["10.0.0.0/16", "203.0.113.10/32"] # check bootstrap docs
 #     }
-#     trusted_internal = {
-#       description = "Network area plus VPN remote prefixes"
-#       content     = ["10.0.0.0/16", "192.0.2.0/24"]
+#
+#     # Resolvers handed to the landing zones through network_area.default_nameservers.
+#     # The landing zones reach them through this appliance, so they need an explicit rule
+#     # as soon as the blanket internet rule is switched off.
+#     platform_dns = {
+#       type        = "host"
+#       description = "Resolvers the landing zones are pointed at"
+#       content     = ["1.1.1.1", "1.0.0.1"]
+#     }
+#
+#     # Example of a domain based allow rule. A host alias accepts FQDNs and OPNsense re-resolves them on a timer
+#     ubuntu_update_servers = {
+#       type        = "host"
+#       description = "Canonical archive, security and changelog mirrors"
+#       content = [
+#         "archive.ubuntu.com",
+#         "security.ubuntu.com",
+#         "changelogs.ubuntu.com",
+#         "esm.ubuntu.com",
+#       ]
+#     }
+#
+#     ubuntu_update_ports = {
+#       type        = "port"
+#       description = "Ports the Ubuntu mirrors are served on"
+#       content     = ["80", "443"]
+#     }
+#   }
+#
+#   # Both appliance interfaces are DHCP clients on a /28 and the default route leaves
+#   # through WAN, so without this the landing zone prefixes are only reachable the long way
+#   # round: spoke bound traffic would be sent to the WAN gateway and re-enter the network
+#   # area from the outside. That also drags LZ-to-LZ traffic through the WAN interface,
+#   # where the egress NAT rule would rewrite it and hide the real source address.
+#   routes = {
+#     network-area-via-lan = {
+#       network     = "10.0.0.0/16"
+#       gateway     = "LAN_DHCP"
+#       description = "Landing zones sit behind the LAN interface"
 #     }
 #   }
 #
 #   rules = {
-#     # The image ships a FLOATING rule passing TCP/443 from any source to the firewall,
-#     # so the GUI is public after a plain deploy. These rules lock it down
 #     allow-webgui-from-management = {
 #       sequence         = 10
 #       action           = "pass"
@@ -162,34 +203,87 @@ connectivity = {
 #       destination_port = "443"
 #       log              = true
 #     }
+
+#     allow-lz-to-dns = {
+#       sequence         = 100
+#       action           = "pass"
+#       interfaces       = ["lan"]
+#       protocol         = "TCP/UDP"
+#       source_net       = "network_area"
+#       destination_net  = "platform_dns"
+#       destination_port = "53"
+#     }
 #
-#     allow-icmp-from-trusted = {
-#       sequence   = 100
-#       action     = "pass"
-#       interfaces = ["lan"]
-#       protocol   = "ICMP"
-#       source_net = "trusted_internal"
+#     allow-lz-to-lz-https = {
+#       sequence         = 110
+#       action           = "pass"
+#       interfaces       = ["lan"]
+#       protocol         = "TCP"
+#       source_net       = "network_area"
+#       destination_net  = "network_area"
+#       destination_port = "443"
+#     }
+#
+#     allow-lz-to-lz-icmp = {
+#       sequence        = 120
+#       action          = "pass"
+#       interfaces      = ["lan"]
+#       protocol        = "ICMP"
+#       source_net      = "network_area"
+#       destination_net = "network_area"
+#     }
+#
+#     block-lz-to-lz = {
+#       sequence        = 130
+#       action          = "block"
+#       interfaces      = ["lan"]
+#       protocol        = "any"
+#       source_net      = "network_area"
+#       destination_net = "network_area"
+#       log             = true
+#     }
+#
+#     allow-lz-to-ubuntu-updates = {
+#       sequence         = 160
+#       action           = "pass"
+#       interfaces       = ["lan"]
+#       protocol         = "TCP"
+#       source_net       = "network_area"
+#       destination_net  = "ubuntu_update_servers"
+#       destination_port = "ubuntu_update_ports"
+#     }
+#
+#     allow-lz-to-internet = {
+#       sequence        = 200
+#       action          = "pass"
+#       interfaces      = ["lan"]
+#       protocol        = "any"
+#       source_net      = "network_area"
+#       destination_net = "any"
+#       description     = "Blanket egress. Disable this to switch to default-deny."
 #     }
 #   }
 #
 #   # Without an entry the landing zones reach nothing, since their default route ends here.
-#   # target_ip takes an address, an alias, or <int>ip.
+#   # target_ip takes an address, an alias, or <int>ip. These are applied regardless of the
+#   # appliance's Outbound NAT mode, which the API cannot change and which stays automatic.
 #   outbound_nat = {
+#     no-nat-inside-network-area = {
+#       sequence        = 10
+#       interface       = "wan"
+#       source_net      = "network_area"
+#       destination_net = "network_area"
+#       disable_nat     = true
+#     }
+#
+#     # The public IP is bound to the WAN interface address, so egress has to be translated
+#     # to exactly that address for the platform to reach the internet at all.
 #     network-area-egress = {
 #       sequence   = 100
 #       interface  = "wan"
-#       source_net = "10.0.0.0/16"
+#       source_net = "network_area"
 #       target_ip  = "wanip"
 #     }
-
-#     # Leave traffic untranslated, e.g. towards a site-to-site peer.
-#     # no-nat-towards-onprem = {
-#     #   sequence        = 10
-#     #   interface       = "wan"
-#     #   source_net      = "10.0.0.0/16"
-#     #   destination_net = "192.0.2.0/24"
-#     #   disable_nat     = true
-#     # }
 #   }
 #
 #   # Inbound NAT. Only rewrites the destination — unlike the GUI the provider adds no