Add job to ensure expected GCP integration network is created

Why
---
The bats and test-stemcells-ipv4 jobs assume a GCP subnetwork named
stemcell-builder-integration-<subnet_int> exists in the bosh-concourse
VPC, with a /24 at 10.100.<subnet_int>.0/24, private Google access, and
IPV4_ONLY stack type. They also require a matching ingress firewall rule
(all-protocol, source CIDR → tags test-stemcells-bats/bat) so that
compilation VMs and BAT deployment VMs can reach the BOSH director's
NATS server. Until now both resources had to be created and maintained
out of band; their absence caused consistent compilation-VM agent
timeouts (builds 466–475).

What
----
* ci/tasks/gcp/ensure-integration-network.sh
  - Authenticates via GCP_JSON_KEY service account.
  - Derives SUBNET_NAME and SUBNET_CIDR from SUBNET_INT.
  - For the subnetwork: calls gcloud describe once, capturing exit code
    for existence and attributes for drift detection in a single API
    call. Checks network, ipCidrRange, privateIpGoogleAccess, stackType
    and exits non-zero with a clear error on any mismatch; creates the
    subnet if absent.
  - For the firewall rule: same single-call pattern, checking network,
    direction, and sourceRanges[0]; creates the rule if absent.
* ci/tasks/gcp/ensure-integration-network.yml
  - Concourse task definition. All params (GCP_JSON_KEY, GCP_PROJECT_ID,
    GCP_REGION, GCP_NETWORK_NAME, SUBNET_INT) are required; no defaults,
    values are provided explicitly by the pipeline.
* ci/pipelines/builder.yml
  - New infrastructure group containing the new job.
  - New job ensure-integration-network:
      * serial: true, manual trigger only.
      * Gets bosh-stemcells-ci and bosh-integration-image, then runs the
        task with GCP_REGION=europe-north2 and
        GCP_NETWORK_NAME=bosh-concourse passed explicitly.
      * No passed: constraint on existing jobs; run on demand when the
        subnet/firewall needs to be created or reconciled.

Verification
------------
* ytt -f ci/pipelines/builder.yml -f ci/pipelines/vars.yml renders
  successfully.
* fly validate-pipeline -c <rendered> reports "looks good".

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: aramprice

ci/pipelines/builder.yml

@@ -44,6 +44,9 @@ groups:
 - name: docker
   jobs:
   - build-os-image-stemcell-builder
+- name: infrastructure
+  jobs:
+  - ensure-integration-network
 
 #@yaml/text-templated-strings
 jobs:
@@ -89,6 +92,25 @@ jobs:
       get_params:
         skip_download: true
 
+#! Manually triggered job that idempotently ensures the GCP subnetwork and
+#! firewall rule consumed by deploy-director / cleanup-bats-vms / prepare-bats
+#! in the test-stemcells-ipv4 and bats jobs below exist. GCP is the source of
+#! truth — no state file is required.
+- name: ensure-integration-network
+  serial: true
+  plan:
+  - get: bosh-stemcells-ci
+  - get: bosh-integration-image
+  - task: ensure-integration-network
+    file: bosh-stemcells-ci/ci/tasks/gcp/ensure-integration-network.yml
+    image: bosh-integration-image
+    params:
+      GCP_JSON_KEY: ((gcp_json_key))
+      GCP_PROJECT_ID: ((gcp_project_id))
+      GCP_REGION: europe-north2
+      GCP_NETWORK_NAME: bosh-concourse
+      SUBNET_INT: (@= data.values.stemcell_details.subnet_int @)
+
 - name: process-high-critical-cves
   serial_groups: [log-cves]
   plan:
@@ -885,7 +907,6 @@ resource_types:
   type: registry-image
   source:
     repository: frodenas/gcs-resource
-
 #@yaml/text-templated-strings
 resources:
 - name: daily

ci/tasks/gcp/ensure-integration-network.sh

@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+set -eu -o pipefail
+
+: "${GCP_JSON_KEY:?}"
+: "${GCP_PROJECT_ID:?}"
+: "${GCP_REGION:?}"
+: "${GCP_NETWORK_NAME:?}"
+: "${SUBNET_INT:?}"
+
+echo "${GCP_JSON_KEY}" | gcloud auth activate-service-account --key-file - --project "${GCP_PROJECT_ID}"
+
+SUBNET_NAME="stemcell-builder-integration-${SUBNET_INT}"
+SUBNET_CIDR="10.100.${SUBNET_INT}.0/24"
+
+# Ensure subnet exists and is correctly configured
+echo "Checking for subnet '${SUBNET_NAME}' in region '${GCP_REGION}'..."
+if current_subnet="$(gcloud compute networks subnets describe "${SUBNET_NAME}" \
+    --region="${GCP_REGION}" \
+    --project="${GCP_PROJECT_ID}" \
+    --format='csv[no-heading](network.basename(),ipCidrRange,privateIpGoogleAccess,stackType)' 2>/dev/null)"; then
+  expected_subnet="${GCP_NETWORK_NAME},${SUBNET_CIDR},True,IPV4_ONLY"
+  if [[ "${current_subnet}" != "${expected_subnet}" ]]; then
+    echo "ERROR: Subnet '${SUBNET_NAME}' exists but is misconfigured."
+    echo "  Expected: ${expected_subnet}"
+    echo "  Actual:   ${current_subnet}"
+    exit 1
+  fi
+  echo "Subnet '${SUBNET_NAME}' already exists and matches expected configuration."
+else
+  echo "Creating subnet '${SUBNET_NAME}'..."
+  gcloud compute networks subnets create "${SUBNET_NAME}" \
+    --network="${GCP_NETWORK_NAME}" \
+    --region="${GCP_REGION}" \
+    --range="${SUBNET_CIDR}" \
+    --enable-private-ip-google-access \
+    --stack-type=IPV4_ONLY \
+    --project="${GCP_PROJECT_ID}"
+  echo "Subnet '${SUBNET_NAME}' created."
+fi
+
+# Ensure firewall rule exists and is correctly configured
+echo "Checking for firewall rule '${SUBNET_NAME}'..."
+if current_fw="$(gcloud compute firewall-rules describe "${SUBNET_NAME}" \
+    --project="${GCP_PROJECT_ID}" \
+    --format='csv[no-heading](network.basename(),direction,sourceRanges[0])' 2>/dev/null)"; then
+  expected_fw="${GCP_NETWORK_NAME},INGRESS,${SUBNET_CIDR}"
+  if [[ "${current_fw}" != "${expected_fw}" ]]; then
+    echo "ERROR: Firewall rule '${SUBNET_NAME}' exists but is misconfigured."
+    echo "  Expected: ${expected_fw}"
+    echo "  Actual:   ${current_fw}"
+    exit 1
+  fi
+  echo "Firewall rule '${SUBNET_NAME}' already exists and matches expected configuration."
+else
+  echo "Creating firewall rule '${SUBNET_NAME}'..."
+  gcloud compute firewall-rules create "${SUBNET_NAME}" \
+    --network="${GCP_NETWORK_NAME}" \
+    --project="${GCP_PROJECT_ID}" \
+    --direction=INGRESS \
+    --priority=1000 \
+    --allow=all \
+    --source-ranges="${SUBNET_CIDR}" \
+    --target-tags=test-stemcells-bats,bat
+  echo "Firewall rule '${SUBNET_NAME}' created."
+fi
+
+echo "Integration network '${SUBNET_NAME}' is ready."

ci/tasks/gcp/ensure-integration-network.yml

@@ -0,0 +1,15 @@
+---
+platform: linux
+
+inputs:
+  - name: bosh-stemcells-ci
+
+params:
+  GCP_JSON_KEY:
+  GCP_PROJECT_ID:
+  GCP_REGION:
+  GCP_NETWORK_NAME:
+  SUBNET_INT:
+
+run:
+  path: bosh-stemcells-ci/ci/tasks/gcp/ensure-integration-network.sh