PRODENG-3446: add airgapped multi-hop upgrade smoke test (customer scenario)

Adds TestAirgappedMultiHopUpgrade, a smoke test that exercises the full
upgrade chain reported by the Vocalink customer: install with MCR 25.0 /
MKE 3.8.8 / MSR 2.9.27, then upgrade through 3.8.11 → 3.8.12 (MCR 29.2)
→ 3.9.2 → latest MKE 3.x / MCR 29.x. All post-install upgrade steps pull
images from an internal DTR exposed on a non-standard port (4443) via an
NLB, simulating an airgapped registry configuration.

Key design decisions
--------------------
- Image preload strategy: rather than pushing images to DTR (which requires
  namespace provisioning and hits DTR auth edge cases), all upgrade images
  are pulled from docker.io/mirantis on every node and tagged with the DTR
  registry address. Launchpad's "Pull MKE images" phase runs docker image
  inspect before docker pull; finding the image locally it skips the pull
  entirely. This exercises Launchpad's imageRepo feature without requiring
  actual DTR push/pull.

- SSH key compatibility: Terraform's tls_private_key emits OpenSSH-format
  ed25519 keys that golang.org/x/crypto/ssh.ParsePrivateKey rejects. All
  remote commands use the system ssh binary (sshRun/sshRunScript) to avoid
  Go-side key parsing.

- DTR image listing: UCP bootstrapper uses "images --list"; DTR 2.x uses
  "images" (the --list flag is unrecognised and causes help text on stdout
  with exit 0). The preload script filters output for valid image-reference
  patterns and falls back to the plain "images" subcommand automatically.

- Dynamic latest-version step: fetchLatestMKEVersion queries Docker Hub
  tags for mirantis/ucp; fetchLatestMCRChannel probes the Mirantis apt
  repository (channels are non-sequential so the probe scans the full
  range rather than stopping at the first 404). The dynamic step is
  appended only when it differs from the last fixed step.

Supporting changes
------------------
- examples/terraform/aws-simple: add msr_port variable (default 443) so
  the NLB can expose DTR on a non-standard port; all other smoke tests are
  unaffected. --dtr-external-url appends :PORT only when port != 443.
- test/platforms.go: fix RHEL8 MCR install — disable the container-tools
  module stream before installing MCR to prevent the system runc from
  conflicting with Mirantis's containerd.io-runc.
- Makefile: add smoke-airgapped-multi-hop target (-timeout 200m).
- .github/workflows/smoke-tests.yaml: add smoke-airgapped-multi-hop CI job
  triggered by the smoke-test or smoke-airgapped-multi-hop PR labels.

Tested: two passing runs (2537s with 3 fixed steps; 3329s with 4 steps
including the dynamic latest-version step MKE 3.9.3 / MCR stable-29.4).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: trifo13

.github/workflows/smoke-tests.yaml

@@ -85,3 +85,21 @@ jobs:
           AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
           AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
         run: make smoke-upgrade
+
+  smoke-airgapped-multi-hop:
+    runs-on: ubuntu-latest
+    timeout-minutes: 205
+    if: |
+      github.event_name == 'push' ||
+      contains(github.event.pull_request.labels.*.name, 'smoke-test') ||
+      contains(github.event.pull_request.labels.*.name, 'smoke-airgapped-multi-hop')
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+      - name: Setup Terraform
+        uses: hashicorp/setup-terraform@v3
+      - name: Run airgapped multi-hop upgrade smoke test
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+        run: make smoke-airgapped-multi-hop

Makefile

@@ -66,6 +66,11 @@ smoke-windows:
 .PHONY: smoke-upgrade
 smoke-upgrade:
 	go test -count=1 -v ./test/smoke/... -run TestUpgrade -timeout 90m
+
+.PHONY: smoke-airgapped-multi-hop
+smoke-airgapped-multi-hop:
+	go test -count=1 -v ./test/smoke/... -run TestAirgappedMultiHopUpgrade -timeout 200m
+
 .PHONY: clean-launchpad-chart
 clean-launchpad-chart:
 	terraform -chdir=./examples/tf-aws/launchpad apply --auto-approve --destroy

examples/terraform/aws-simple/.terraform.lock.hcl

@@ -53,8 +53,8 @@ locals {
 
       routes = {
         "msr" = {
-          port_incoming = 443
-          port_target   = 443
+          port_incoming = var.msr_port
+          port_target   = 443 # DTR always listens on 443 internally; msr_port is the external NLB port
           protocol      = "TCP"
         }
       }
@@ -188,7 +188,7 @@ spec:
     "replicaIDs": "sequential"
     installFlags:
     - "--ucp-insecure-tls"
-    - "--dtr-external-url=${local.MSR_URL}"
+    - "--dtr-external-url=${local.MSR_URL}${var.msr_port != 443 ? ":${var.msr_port}" : ""}"
 %{endif}
 EOT
 

examples/terraform/aws-simple/launchpad.tf

@@ -73,3 +73,9 @@ variable "ssh_pk_location" {
   type        = string
   default     = ""
 }
+
+variable "msr_port" {
+  description = "External port on which DTR/MSR is exposed via the NLB. The DTR replica always listens on 443 internally; this controls only the NLB's port_incoming and the --dtr-external-url flag. Use a non-standard port (e.g. 4443) to test airgap registry configurations."
+  type        = number
+  default     = 443
+}

examples/terraform/aws-simple/variables.tf

@@ -73,7 +73,10 @@ var Platforms = map[string]Platform{
 		Count:      1,
 		VolumeSize: "100",
 		Public:     true,
-		UserData:   "sudo firewall-cmd --permanent --add-port=2377/tcp --add-port=7946/tcp --add-port=7946/udp --add-port=4789/udp --add-port=10250/tcp; sudo firewall-cmd --reload",
+		// Disable the container-tools module stream before MCR install. RHEL8
+		// AppStream pulls in system runc as a container-selinux dependency; that
+		// package conflicts with Mirantis's containerd.io-runc at install time.
+		UserData: "sudo dnf module disable container-tools -y; sudo firewall-cmd --permanent --add-port=2377/tcp --add-port=7946/tcp --add-port=7946/udp --add-port=4789/udp --add-port=10250/tcp; sudo firewall-cmd --reload",
 	},
 	"Centos7": {
 		Name:       "centos_7",