|
| 1 | +--- |
| 2 | +title: Provision a Dual-Arch GKE Cluster and Publish Multi-Arch Images |
| 3 | +weight: 4 |
| 4 | + |
| 5 | +### FIXED, DO NOT MODIFY |
| 6 | +layout: learningpathall |
| 7 | +--- |
| 8 | + |
| 9 | +Now create a **GKE cluster** with **two node pools** (amd64 & arm64), then build and push multi-arch images natively on those node pools. Each architecture uses its own BuildKit pod, and no QEMU emulation is involved. |
| 10 | + |
| 11 | +#### Networking (VPC-native / IP aliasing) |
| 12 | + |
| 13 | +GKE uses **VPC-native (IP aliasing)** and requires **two secondary ranges** on the chosen subnet: one for **Pods** and one for **Services**. |
| 14 | +- **Default VPC:** Skip this step. GKE will create the secondary ranges automatically. |
| 15 | +- **Custom VPC/subnet:** Set variables and add/verify secondary ranges: |
| 16 | + |
| 17 | +```bash |
| 18 | +# Set/confirm network variables (adjust to your environment) |
| 19 | +export REGION="${REGION:-us-central1}" |
| 20 | +export NETWORK="dev-eco-nw-pb" # your VPC |
| 21 | +export SUBNET="dev-eco-nw-subnet" # your subnet |
| 22 | +export POD_RANGE_NAME="gke-boutique-pods" |
| 23 | +export SVC_RANGE_NAME="gke-boutique-svcs" |
| 24 | + |
| 25 | +# Inspect the subnet and existing ranges |
| 26 | +gcloud compute networks subnets list --network "${NETWORK}" --regions "${REGION}" \ |
| 27 | + --format="table(name,region,ipCidrRange,secondaryIpRanges.list())" |
| 28 | + |
| 29 | +# If missing, add two secondary ranges (example CIDRs; ensure no overlap) |
| 30 | +gcloud compute networks subnets update "${SUBNET}" --region "${REGION}" --add-secondary-ranges ${POD_RANGE_NAME}=10.8.0.0/14,${SVC_RANGE_NAME}=10.4.0.0/20 |
| 31 | +``` |
| 32 | +This avoids users on default VPC accidentally setting NETWORK/SUBNET and passing the wrong flags later. |
| 33 | + |
| 34 | +### Create the GKE cluster |
| 35 | + |
| 36 | +Create a GKE Standard cluster with VPC-native (IP aliasing) enabled and no default node pool (you'll add amd64 and arm64 pools next). The command below works for both default and custom VPCs: if NETWORK, SUBNET, and the secondary range variables are unset, GKE uses the default VPC and manages ranges automatically. |
| 37 | + |
| 38 | +Create the cluster with no default node pool and add node pools explicitly. |
| 39 | + |
| 40 | +```bash |
| 41 | +# Cluster vars (reuses earlier PROJECT_ID/REGION/ZONE) |
| 42 | +export CLUSTER_NAME="${CLUSTER_NAME:-gke-multi-arch-cluster}" |
| 43 | + |
| 44 | +# If using the default VPC, you can omit --network/--subnetwork. |
| 45 | +# If using a custom VPC, include them and pass the secondary range names you set above. |
| 46 | +gcloud container clusters create "${CLUSTER_NAME}" --region "${REGION}" --enable-ip-alias --num-nodes "1" --machine-type "e2-standard-2" ${NETWORK:+--network "${NETWORK}"} ${SUBNET:+--subnetwork "${SUBNET}"} ${POD_RANGE_NAME:+--cluster-secondary-range-name "${POD_RANGE_NAME}"} ${SVC_RANGE_NAME:+--services-secondary-range-name "${SVC_RANGE_NAME}"} |
| 47 | +``` |
| 48 | + |
| 49 | +Create an x86 (amd64) pool and an Arm (arm64) pool. Use machine types available in your region (e.g., c4-standard-* for x86 and c4a-standard-* for Axion). |
| 50 | + |
| 51 | +```bash |
| 52 | +# amd64 pool (x86) |
| 53 | +gcloud container node-pools create amd64-pool --cluster="${CLUSTER_NAME}" --region="${REGION}" --machine-type="c4-standard-16" --num-nodes="1" --image-type="COS_CONTAINERD" --quiet |
| 54 | + |
| 55 | +# arm64 pool (Axion) |
| 56 | +gcloud container node-pools create arm64-pool --cluster="${CLUSTER_NAME}" --region="${REGION}" --machine-type="c4a-standard-16" --num-nodes="1" --image-type="COS_CONTAINERD" --quiet |
| 57 | + |
| 58 | +# delete the tiny default pool |
| 59 | +gcloud container node-pools delete default-pool --cluster="${CLUSTER_NAME}" --region="${REGION}" --quiet |
| 60 | +``` |
| 61 | + |
| 62 | +Connect kubectl and confirm node architectures: |
| 63 | + |
| 64 | +```bash |
| 65 | +gcloud container clusters get-credentials "${CLUSTER_NAME}" --region "${REGION}" |
| 66 | +kubectl config current-context |
| 67 | +kubectl get nodes -o wide |
| 68 | +kubectl get nodes -L kubernetes.io/arch |
| 69 | +``` |
| 70 | +You should see nodes for both architectures. In zonal clusters (or when a pool has --num-nodes=1 in a single zone), expect one amd64 and one arm64 node. In regional clusters, --num-nodes is per zone, with three zones you'll see three amd64 and three arm64 nodes. |
| 71 | + |
| 72 | +### Create the Buildx builder on GKE (native, one pod per arch) |
| 73 | + |
| 74 | +Now run a BuildKit pod on an amd64 node and another on an arm64 node. Buildx will route each platform's build to the matching pod - native builds, no emulation. |
| 75 | + |
| 76 | +```bash |
| 77 | + |
| 78 | +# Namespace for BuildKit pods |
| 79 | +kubectl create ns buildkit --dry-run=client -o yaml | kubectl apply -f - |
| 80 | + |
| 81 | +# Create the builder (amd64 node) |
| 82 | +docker buildx create --driver kubernetes --name gke-native --use --driver-opt namespace=buildkit,replicas=1,loadbalance=sticky,nodeselector=kubernetes.io/arch=amd64 --platform linux/amd64 |
| 83 | + |
| 84 | +# Append an arm64 node to the same builder |
| 85 | +docker buildx create --driver kubernetes --append --name gke-native --driver-opt namespace=buildkit,replicas=1,loadbalance=sticky,nodeselector=kubernetes.io/arch=arm64 --platform linux/arm64 |
| 86 | + |
| 87 | +# Bootstrap and verify pods |
| 88 | +docker buildx inspect gke-native --bootstrap |
| 89 | +kubectl -n buildkit get pods -o wide |
| 90 | + |
| 91 | +``` |
| 92 | +{{% notice Note %}} |
| 93 | +You will now have a multi-node Buildx builder named gke-native. Each BuildKit pod is pinned to a specific CPU arch via nodeselector. |
| 94 | +{{% /notice %}} |
| 95 | + |
| 96 | +### Build & push all services (multi-arch manifest lists) |
| 97 | +Build every service for linux/amd64 and linux/arm64 using the GKE-backed builder: |
| 98 | +```bash |
| 99 | +cat << 'EOF' > build-all-multiarch.sh |
| 100 | +#!/usr/bin/env bash |
| 101 | +set -euo pipefail |
| 102 | +
|
| 103 | +: "${GAR:?Set GAR like REGION-docker.pkg.dev/PROJECT/REPO first}" |
| 104 | +
|
| 105 | +services=( |
| 106 | + adservice |
| 107 | + cartservice # special context below |
| 108 | + checkoutservice |
| 109 | + currencyservice |
| 110 | + emailservice |
| 111 | + frontend |
| 112 | + paymentservice |
| 113 | + productcatalogservice |
| 114 | + recommendationservice |
| 115 | + shippingservice |
| 116 | + loadgenerator |
| 117 | +) |
| 118 | +
|
| 119 | +for svc in "${services[@]}"; do |
| 120 | + # cartservice Dockerfile path differs |
| 121 | + if [ "$svc" = "cartservice" ] && [ -d "src/cartservice/src" ]; then |
| 122 | + ctx="src/cartservice/src" |
| 123 | + else |
| 124 | + ctx="src/${svc}" |
| 125 | + fi |
| 126 | +
|
| 127 | + echo ">>> Building ${svc} for amd64+arm64..." |
| 128 | + docker buildx build --builder gke-native --platform linux/amd64,linux/arm64 --provenance=false -t "${GAR}/${svc}:v1" "${ctx}" --push |
| 129 | +done |
| 130 | +EOF |
| 131 | + |
| 132 | +chmod +x build-all-multiarch.sh |
| 133 | +./build-all-multiarch.sh |
| 134 | +``` |
| 135 | +Each :v1 you push is a manifest list that points to two images (one per arch). |
| 136 | + |
| 137 | +### Verify manifest lists and per-arch pulls |
| 138 | + |
| 139 | +List pushed images: |
| 140 | + |
| 141 | +```bash |
| 142 | +gcloud artifacts docker images list "${GAR}" |
| 143 | +``` |
| 144 | + |
| 145 | +Inspect one tag (should show both platforms): |
| 146 | +```bash |
| 147 | +docker buildx imagetools inspect "${GAR}/adservice:v1" |
| 148 | +``` |
| 149 | + |
| 150 | +Expected Output: |
| 151 | +``` |
| 152 | +Platform: linux/amd64 |
| 153 | +Platform: linux/arm64 |
| 154 | +``` |
0 commit comments