docs: Update migration guide with GitOps approach and clarify node provisioning

- Replace all kubectl apply commands with GitOps/FluxCD workflow
- Add Prerequisites section emphasizing FluxCD and GitOps repository requirements
- Add detailed explanation of node provisioning trigger mechanism with real example
- Show step-by-step process: Terraform scale-down → pod eviction → unschedulable pods → Karpenter provisioning
- Include FluxCD reconciliation commands and HelmRelease example
- Clean up corrupted content from previous complex migration approach
- Emphasize GitOps best practices throughout the migration process

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

Author: haimari

docs/migration-zero-downtime.md

@@ -13,6 +13,19 @@ This guide provides a **simple, safe approach** for migrating from Terraform-man
 
 ---
 
+## 🔧 **Prerequisites**
+
+Before starting the migration, ensure you have:
+
+- **Oracle Kubernetes Engine (OKE)** cluster running
+- **OCI credentials** configured (CLI or Instance Principal)
+- **FluxCD v2.x** installed and managing your cluster
+- **GitOps repository** where Kubernetes manifests are stored
+- **`flux` CLI** installed locally for manual reconciliation
+- **Terraform** managing your current node pools
+
+---
+
 ## 🏗️ **Pre-Migration Assessment**
 
 ### **1. Inventory Current State**
@@ -111,20 +124,42 @@ The key to zero-downtime migration is installing Karpenter alongside existing in
 
 #### **Step 1.1: Install Karpenter (Non-Disruptive)**
 
+```yaml
+# Add to your GitOps repository: karpenter/karpenter-release.yaml
+apiVersion: helm.toolkit.fluxcd.io/v2beta1
+kind: HelmRelease
+metadata:
+  name: karpenter-oci
+  namespace: karpenter
+spec:
+  interval: 15m
+  chart:
+    spec:
+      chart: karpenter-oci
+      version: "0.1.57"  # Use latest version
+      sourceRef:
+        kind: HelmRepository
+        name: karpenter-oci
+        namespace: karpenter
+  values:
+    oci:
+      region: us-ashburn-1
+      compartmentId: "ocid1.compartment.oc1..."
+      clusterId: "ocid1.cluster.oc1..."
+      existingSecret: "oci-config"
+```
+
 ```bash
-# 1. Create Karpenter namespace and install
-kubectl create namespace karpenter
-
-# 2. Install Karpenter with careful configuration
-helm install karpenter karpenter-oci/karpenter-oci \
-  --namespace karpenter \
-  --set oci.region=$OCI_REGION \
-  --set oci.compartmentId=$OCI_COMPARTMENT_ID \
-  --set oci.clusterId=$OCI_CLUSTER_ID \
-  --set oci.existingSecret="oci-config" \
-  --wait
-
-# 3. Verify Karpenter is running but not managing anything yet
+# Commit and deploy via GitOps
+git add karpenter/karpenter-release.yaml
+git commit -m "Install Karpenter OCI Provider for migration"
+git push origin main
+
+# Trigger FluxCD reconciliation
+flux reconcile source git flux-system
+flux reconcile helmrelease karpenter-oci -n karpenter
+
+# Verify Karpenter is running
 kubectl get deployment -n karpenter
 kubectl get nodepools -A  # Should be empty initially
 ```
@@ -191,12 +226,16 @@ spec:
 ```
 
 ```bash
-# Apply Karpenter NodePools (ready to take over)
-kubectl apply -f kafka-nodepool.yaml
-kubectl apply -f rabbitmq-nodepool.yaml  
-kubectl apply -f redis-nodepool.yaml
+# Commit NodePool manifests to your GitOps repository
+git add kafka-nodepool.yaml rabbitmq-nodepool.yaml redis-nodepool.yaml
+git commit -m "Add Karpenter NodePools for migration handoff"
+git push origin main
 
-# Verify NodePools are ready
+# Wait for FluxCD to reconcile
+flux reconcile source git flux-system
+flux reconcile kustomization karpenter
+
+# Verify NodePools are deployed
 kubectl get nodepools -A
 # No immediate provisioning will happen unless pods become unschedulable
 ```
@@ -248,9 +287,39 @@ resource "oci_containerengine_node_pool" "kafka_pool" {
 }
 ```
 
-### **Step 2.3: Monitor Karpenter Response**
+### **Step 2.3: Understanding the Node Provisioning Trigger**
+
+**Here's exactly how new Karpenter nodes get provisioned during migration:**
 
-**Karpenter should automatically provision nodes if pods become unschedulable:**
+#### **🎯 The Triggering Mechanism**
+1. **Terraform Scale-Down**: When Terraform reduces node pool size (e.g., 3→2 nodes)
+2. **Node Termination**: OCI terminates one of the existing nodes
+3. **Pod Eviction**: Pods on the terminated node are evicted by Kubernetes  
+4. **Rescheduling**: Kubernetes scheduler tries to reschedule evicted pods
+5. **Unschedulable State**: If remaining nodes lack capacity, pods become "Pending"
+6. **Karpenter Trigger**: Karpenter detects unschedulable pods and provisions new nodes
+7. **New Node**: Karpenter creates a new OCI instance matching NodePool requirements
+8. **Pod Scheduling**: Pending pods are scheduled on the new Karpenter-managed node
+
+#### **📋 Real Example**
+```bash
+# Before: 3 Terraform nodes, 0 Karpenter nodes
+kubectl get nodes | grep -E "(kafka-pool|karpenter)"
+# kafka-pool-terraform-node-1   Ready   <none>   1d   v1.28.2
+# kafka-pool-terraform-node-2   Ready   <none>   1d   v1.28.2  
+# kafka-pool-terraform-node-3   Ready   <none>   1d   v1.28.2
+
+# Apply Terraform scale-down (3→2)
+terraform apply -target=oci_containerengine_node_pool.kafka_pool
+
+# After: 2 Terraform nodes, 1 Karpenter node (automatically provisioned)
+kubectl get nodes | grep -E "(kafka-pool|karpenter)"
+# kafka-pool-terraform-node-1   Ready   <none>   1d   v1.28.2
+# kafka-pool-terraform-node-2   Ready   <none>   1d   v1.28.2
+# kafka-pool-karpenter-abcd123   Ready   <none>   5m   v1.28.2   # <- New Karpenter node
+```
+
+#### **🔍 Monitor the Process**
 
 ```bash
 # Monitor Karpenter's response to the Terraform scale-down
@@ -353,6 +422,7 @@ kubectl get pods -A | grep -E "(Pending|Failed)"
 Now that migration is complete, enable cost optimization features:
 
 ```yaml
+# kafka-nodepool-optimized.yaml
 # Enable flexible shape selection for cost optimization
 apiVersion: karpenter.sh/v1
 kind: NodePool
@@ -379,53 +449,36 @@ spec:
     consolidateAfter: "30s"  # Enable aggressive consolidation
 ```
 
-```yaml
-# kafka-test-replica.yaml - CREATE ADDITIONAL REPLICA FOR TESTING
-apiVersion: v1
-kind: Pod
-metadata:
-  name: kafka-test-replica
-  namespace: kafka
-  labels:
-    app.kubernetes.io/name: kafka
-    test-migration: "true"
-spec:
-  # Force scheduling on Karpenter node
-  nodeSelector:
-    oci.oraclecloud.com/node-pool: kafka-pool-karpenter
-  tolerations:
-    - key: node_pool
-      value: kafka
-      effect: NoSchedule
-  containers:
-  - name: kafka
-    image: confluentinc/cp-kafka:latest
-    resources:
-      # Match StatefulSet resource requirements exactly
-      requests:
-        cpu: "2000m"
-        memory: "8Gi"
-      limits:
-        cpu: "2000m"  
-        memory: "8Gi"
-    env:
-    - name: KAFKA_ZOOKEEPER_CONNECT
-      value: "zookeeper-service:2181"
-    - name: KAFKA_ADVERTISED_LISTENERS
-      value: "PLAINTEXT://kafka-test-replica:9092"
+```bash
+# Update NodePool configurations in GitOps repository
+git add kafka-nodepool-optimized.yaml
+git commit -m "Enable cost optimization features post-migration"
+git push origin main
+
+# Trigger FluxCD reconciliation
+flux reconcile source git flux-system
+flux reconcile kustomization karpenter
 ```
 
+### **Step 4.2: Final Validation**
+
 ```bash
-# Deploy test replica on Karpenter node
-kubectl apply -f kafka-test-replica.yaml
+# Comprehensive final validation
+echo "=== Migration Validation Report ==="
 
-# Monitor scheduling and startup
-kubectl get pod kafka-test-replica -n kafka -o wide
-kubectl logs kafka-test-replica -n kafka
+# 1. Node distribution
+echo "Current nodes:"
+kubectl get nodes -o custom-columns="NAME:.metadata.name,POOL:.metadata.labels.oci\.oraclecloud\.com/node-pool,STATUS:.status.conditions[?(@.type=='Ready')].status"
 
-# Verify Kafka connectivity from test replica
-kubectl exec kafka-test-replica -n kafka -- kafka-topics --bootstrap-server localhost:9092 --list
-```
+# 2. StatefulSet health
+echo -e "\nStatefulSet status:"
+kubectl get statefulsets -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:.status.readyReplicas,DESIRED:.spec.replicas"
+
+# 3. Critical workload validation
+echo -e "\nCritical workload validation:"
+kubectl exec -n kafka kafka-cluster-0 -- kafka-topics --bootstrap-server localhost:9092 --list | head -5
+kubectl exec -n rabbitmq rabbitmq-cluster-0 -- rabbitmqctl node_health_check
+kubectl exec -n redis redis-cluster-0 -- redis-cli info replication
 
 ---