WIP: Add ensure-integration-network job and terraform for GCP integration subnet

Introduces a manually-triggered Concourse job, ensure-integration-network,
that idempotently creates the per-branch GCP subnetwork consumed by the
test-stemcells-ipv4 and bats jobs (deploy-director, cleanup-bats-vms,
prepare-bats) under the bosh-concourse VPC.
Why
---
The test-stemcells-ipv4 and bats jobs in ci/pipelines/builder.yml assume a
subnetwork named stemcell-builder-integration-<subnet_int> exists in the
projects/cloud-foundry-310819/global/networks/bosh-concourse VPC, with a
/24 at 10.100.<subnet_int>.0/24, gateway .1, private Google access, and
IPV4_ONLY stack type. Until now this subnet had to be created and
maintained out of band; this change captures it as code so it can be
recreated reproducibly per branch (subnet_int is set per branch in
ci/pipelines/vars.yml).
What
----
* ci/tasks/ensure-integration-network/{input,network,output}.tf
  - hashicorp/google ~> 5.0
  - google_compute_subnetwork "integration":
      name                     = stemcell-builder-integration-<subnet_int>
      ip_cidr_range            = 10.100.<subnet_int>.0/24
      region                   = europe-north2 (matches GCP_ZONE
                                  europe-north2-a in deploy-director)
      network                  = bosh-concourse (configurable)
      private_ip_google_access = true
      purpose                  = PRIVATE
      stack_type               = IPV4_ONLY
  - subnet_int is taken as input so the same module produces the
    appropriate subnet for any branch.
* ci/pipelines/builder.yml
  - New `infrastructure` group containing the new job.
  - New resource_type `terraform_type` (ljfranklin/terraform-resource).
  - New resource `integration-network-environment` (GCS backend). The
    bucket is referenced via the Concourse credential
    ((integration_network_terraform_state_bucket)); pick a bucket name
    and add the credential before flying. A TODO comment marks this.
  - New job `ensure-integration-network`:
      * serial: true, manual trigger only (no `trigger: true` on get).
      * Puts to integration-network-environment with
        env_name = stemcell-builder-integration-<subnet_int>, so the
        state file is deterministically named and re-discovered on
        subsequent runs (no recreate-on-rerun).
      * Apply only — never destroyed by this job, since the subnet is
        long-lived shared infrastructure.
      * No `passed:` constraint on the existing test/bats jobs to avoid
        coupling; the job is intended to be run on demand when the
        subnet needs to be created or reconciled.
Verification
------------
* `ytt -f ci/pipelines/builder.yml -f ci/pipelines/vars.yml` renders
  successfully.
* `fly validate-pipeline -c <rendered>` reports "looks good".
Follow-ups
----------
* Create the GCS bucket that will hold terraform state and set the
  ((integration_network_terraform_state_bucket)) Concourse credential.
* If the existing subnetwork in cloud-foundry-310819 is to be adopted
  rather than recreated, run `terraform import` once before flying the
  job (or delete the existing subnet first).

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: aramprice

ci/pipelines/builder.yml

@@ -44,6 +44,9 @@ groups:
 - name: docker
   jobs:
   - build-os-image-stemcell-builder
+- name: infrastructure
+  jobs:
+  - ensure-integration-network
 
 #@yaml/text-templated-strings
 jobs:
@@ -89,6 +92,23 @@ jobs:
       get_params:
         skip_download: true
 
+#! Manually triggered job that idempotently creates (or updates) the GCP
+#! subnetwork consumed by deploy-director / cleanup-bats-vms / prepare-bats
+#! in the test-stemcells-ipv4 and bats jobs below. The terraform state lives
+#! in GCS, keyed by env_name = stemcell-builder-integration-<subnet_int>, so
+#! the same subnetwork is reused across pipeline runs and is never destroyed
+#! by this job.
+- name: ensure-integration-network
+  serial: true
+  plan:
+  - get: bosh-stemcells-ci
+  - put: integration-network-environment
+    params:
+      env_name: stemcell-builder-integration-(@= data.values.stemcell_details.subnet_int @)
+      terraform_source: bosh-stemcells-ci/ci/tasks/ensure-integration-network/
+      vars:
+        subnet_int: (@= data.values.stemcell_details.subnet_int @)
+
 - name: process-high-critical-cves
   serial_groups: [log-cves]
   plan:
@@ -885,6 +905,10 @@ resource_types:
   type: registry-image
   source:
     repository: frodenas/gcs-resource
+- name: terraform_type
+  type: registry-image
+  source:
+    repository: ljfranklin/terraform-resource
 
 #@yaml/text-templated-strings
 resources:
@@ -1193,3 +1217,23 @@ resources:
   source:
     regexp: azure-storage-cli-(.*)-linux-amd64
     bucket: bosh-azure-storage-cli-artifacts
+
+#! Terraform state for the per-branch GCP subnetwork used by the
+#! test-stemcells-ipv4 and bats jobs. The env_name passed to `put` is derived
+#! from data.values.stemcell_details.subnet_int so the state file is
+#! deterministically named (e.g. stemcell-builder-integration-22) and is
+#! re-discovered on subsequent runs without recreating the subnetwork.
+#!
+#! TODO: replace ((integration_network_terraform_state_bucket)) with the
+#! agreed-upon GCS bucket name (the bucket itself must be created out of band).
+- name: integration-network-environment
+  type: terraform_type
+  source:
+    backend_type: gcs
+    backend_config:
+      credentials: ((gcp_json_key))
+      bucket: ((integration_network_terraform_state_bucket))
+      prefix: stemcell-builder-integration-network/
+    vars:
+      gce_credentials_json: ((gcp_json_key))
+      gce_project_id: ((gcp_project_id))

ci/tasks/ensure-integration-network/input.tf

@@ -0,0 +1,27 @@
+variable "gce_project_id" {
+  type        = string
+  description = "GCP project that owns the bosh-concourse VPC."
+}
+
+variable "gce_credentials_json" {
+  type        = string
+  description = "JSON-encoded service account credentials with Compute Network Admin on the project."
+  sensitive   = true
+}
+
+variable "gce_region" {
+  type        = string
+  description = "Region to create the integration subnetwork in. Must match the region of the GCP_ZONE used by deploy-director."
+  default     = "europe-north2"
+}
+
+variable "network_name" {
+  type        = string
+  description = "Name of the VPC network the subnetwork will be attached to."
+  default     = "bosh-concourse"
+}
+
+variable "subnet_int" {
+  type        = string
+  description = "Integer (as string) used as both the third octet of the /24 CIDR and the suffix of the subnet name. Matches data.values.stemcell_details.subnet_int in ci/pipelines/vars.yml."
+}

ci/tasks/ensure-integration-network/network.tf

@@ -0,0 +1,46 @@
+terraform {
+  required_providers {
+    google = {
+      source  = "hashicorp/google"
+      version = "~> 5.0"
+    }
+  }
+}
+
+provider "google" {
+  project     = var.gce_project_id
+  region      = var.gce_region
+  credentials = var.gce_credentials_json
+}
+
+# Subnetwork used by the test-stemcells-ipv4 and bats jobs in
+# ci/pipelines/builder.yml. The reserved ranges in those jobs assume a /24
+# starting at 10.100.<subnet_int>.0 with the gateway at .1.
+resource "google_compute_subnetwork" "integration" {
+  name                     = "stemcell-builder-integration-${var.subnet_int}"
+  ip_cidr_range            = "10.100.${var.subnet_int}.0/24"
+  region                   = var.gce_region
+  network                  = "projects/${var.gce_project_id}/global/networks/${var.network_name}"
+  private_ip_google_access = true
+  purpose                  = "PRIVATE"
+  stack_type               = "IPV4_ONLY"
+}
+
+# Firewall rule that allows all traffic originating from within the integration
+# subnetwork to reach VMs tagged test-stemcells-bats or bat. This is required
+# so that compilation VMs and BAT deployment VMs can reach the BOSH director's
+# NATS server (port 4222) and other internal services on the same /24.
+# Mirrors the stemcell-builder-integration-22 rule in the bosh-concourse VPC.
+resource "google_compute_firewall" "integration" {
+  name    = "stemcell-builder-integration-${var.subnet_int}"
+  network = "projects/${var.gce_project_id}/global/networks/${var.network_name}"
+
+  direction     = "INGRESS"
+  priority      = 1000
+  source_ranges = [google_compute_subnetwork.integration.ip_cidr_range]
+  target_tags   = ["test-stemcells-bats", "bat"]
+
+  allow {
+    protocol = "all"
+  }
+}

ci/tasks/ensure-integration-network/output.tf

@@ -0,0 +1,29 @@
+output "subnetwork_name" {
+  description = "Name of the integration subnetwork. Matches GCP_SUBNET_NAME in deploy-director / cleanup-bats-vms."
+  value       = google_compute_subnetwork.integration.name
+}
+
+output "subnetwork_self_link" {
+  description = "Self link to the integration subnetwork."
+  value       = google_compute_subnetwork.integration.self_link
+}
+
+output "ip_cidr_range" {
+  description = "Primary IPv4 CIDR range allocated to the subnetwork."
+  value       = google_compute_subnetwork.integration.ip_cidr_range
+}
+
+output "gateway_address" {
+  description = "Gateway address of the subnetwork. Matches INTERNAL_GW in deploy-director."
+  value       = google_compute_subnetwork.integration.gateway_address
+}
+
+output "region" {
+  description = "Region the subnetwork lives in."
+  value       = google_compute_subnetwork.integration.region
+}
+
+output "firewall_rule_name" {
+  description = "Name of the ingress firewall rule that allows intra-subnet traffic to reach BOSH-managed VMs."
+  value       = google_compute_firewall.integration.name
+}