Troubleshooting Guide

This guide covers common issues and their solutions when deploying and operating the HA RKE2 cluster.

Deployment Issues
Cluster Access Issues
Node Issues
etcd Issues
Networking Issues
Pod Issues
Useful Commands Reference
Log Locations

Deployment Issues

Terraform Init Fails

Symptom:

Error: Failed to install provider

Solutions:

# Clear cache and retry
rm -rf .terraform
rm .terraform.lock.hcl
terraform init

# If behind proxy
export HTTP_PROXY=http://proxy:port
export HTTPS_PROXY=http://proxy:port
terraform init

Terraform Apply Fails on VPC

Symptom:

Error: Error creating VPC: VpcLimitExceeded

Solutions:

# Check current VPC count
aws ec2 describe-vpcs --query 'Vpcs[].VpcId' --output table

# Delete unused VPCs or request limit increase
aws ec2 delete-vpc --vpc-id vpc-xxx

# Request limit increase via AWS console:
# Service Quotas → Amazon VPC → VPCs per Region

EC2 Instance Launch Fails

Symptom:

Error: Error launching source instance: InsufficientInstanceCapacity

Solutions:

# Option 1: Try different instance type
# terraform.tfvars
control_plane_instance_type = "t3.large"  # Instead of t3.medium

# Option 2: Try different AZs
# terraform.tfvars
public_subnet_cidrs = ["10.0.10.0/24", "10.0.11.0/24", "10.0.12.0/24"]

# Option 3: Wait and retry (capacity may free up)
terraform apply

SSH Key Permission Error

Symptom:

Error: Invalid key_name

Solutions:

# Check if key exists
aws ec2 describe-key-pairs --key-names rke2-cluster

# Verify public key format
cat ~/.ssh/rke2-cluster-key.pub
# Should start with: ssh-ed25519 or ssh-rsa

# Recreate key if corrupted
ssh-keygen -t ed25519 -f ~/.ssh/rke2-cluster-key -N ""

Node Bootstrap Timeout

Symptom:

# In EC2 console: Instance running but RKE2 not starting
# In user-data log: Timeout waiting for control plane

Solutions:

# SSH to the failing node
ssh -i ~/.ssh/key.pem ubuntu@<NODE_IP>

# Check user-data execution
sudo cat /var/log/cloud-init-output.log

# Check RKE2 service status
sudo systemctl status rke2-server  # or rke2-agent for workers

# Check RKE2 logs
sudo journalctl -u rke2-server -f

# Manually restart if needed
sudo systemctl restart rke2-server

Cluster Access Issues

Cannot Connect to API Server

Symptom:

Unable to connect to the server: dial tcp: lookup xxx on xxx: no such host

Solutions:

# Verify NLB is ready
aws elbv2 describe-load-balancers \
  --names "rke2-ha-cluster-nlb" \
  --query 'LoadBalancers[].State.Code'

# Check NLB DNS resolution
nslookup $(terraform output -raw nlb_dns_name)

# Verify target health
aws elbv2 describe-target-health \
  --target-group-arn $(terraform output -raw k8s_api_target_group_arn)

# Test connectivity
curl -k https://$(terraform output -raw nlb_dns_name):6443/healthz

Kubeconfig Issues

Symptom:

error: You must be logged in to the server (Unauthorized)

Solutions:

# Regenerate kubeconfig
$(terraform output -raw kubeconfig_command)

# Verify kubeconfig content
cat kubeconfig.yaml | grep server

# Check certificate expiration
openssl s_client -connect $(terraform output -raw nlb_dns_name):6443 2>/dev/null | openssl x509 -noout -dates

# Set correct permissions
chmod 600 kubeconfig.yaml

# Verify KUBECONFIG env var
echo $KUBECONFIG
export KUBECONFIG=$(pwd)/kubeconfig.yaml

TLS Certificate Errors

Symptom:

x509: certificate is valid for 127.0.0.1, not nlb-xxx.amazonaws.com

Solutions:

# Verify TLS-SAN in RKE2 config on control plane
ssh -i ~/.ssh/key.pem ubuntu@<CP_IP>
sudo cat /etc/rancher/rke2/config.yaml | grep -A5 tls-san

# If NLB DNS is missing, add it and restart
sudo vim /etc/rancher/rke2/config.yaml
# Add NLB DNS to tls-san list
sudo systemctl restart rke2-server

# Regenerate kubeconfig after fix

Node Issues

Node Not Ready

Symptom:

NAME         STATUS     ROLES    AGE   VERSION
node-xxx     NotReady   <none>   10m   v1.34.6+rke2r1

Solutions:

# Check node conditions
kubectl describe node <NODE_NAME> | grep -A20 Conditions

# Common causes:
# 1. CNI not ready - Check Cilium pods
kubectl get pods -n kube-system -l k8s-app=cilium

# 2. Disk pressure - Check disk usage
ssh ubuntu@<NODE_IP> df -h

# 3. Memory pressure - Check memory
ssh ubuntu@<NODE_IP> free -m

# 4. kubelet issues - Check kubelet
ssh ubuntu@<NODE_IP> sudo journalctl -u kubelet -f
# For RKE2:
ssh ubuntu@<NODE_IP> sudo journalctl -u rke2-agent -f

Worker Cannot Join Cluster

Symptom:

# Worker user-data log:
Waiting for control plane to be ready...
Retry 60/60: Control plane not ready

Solutions:

# On the worker node:
ssh -i ~/.ssh/key.pem ubuntu@<WORKER_IP>

# Test connectivity to control plane via NLB
curl -sk https://<NLB_DNS>:9345/ping
curl -sk https://<NLB_DNS>:6443/healthz

# Check security group allows worker → CP traffic
aws ec2 describe-security-groups \
  --group-ids <WORKER_SG_ID> \
  --query 'SecurityGroups[].IpPermissionsEgress'

# Check RKE2 agent config
sudo cat /etc/rancher/rke2/config.yaml

# Manually start agent
sudo systemctl restart rke2-agent
sudo journalctl -u rke2-agent -f

Node Disk Pressure

Symptom:

kubectl describe node <NODE_NAME>
# Conditions:
#   DiskPressure     True

Solutions:

# SSH to node
ssh ubuntu@<NODE_IP>

# Check disk usage
df -h

# Clean up container images
sudo crictl rmi --prune

# Clean up RKE2 cache
sudo rm -rf /var/lib/rancher/rke2/agent/images/*.tar

# Clean up old logs
sudo journalctl --vacuum-time=2d

# If still full, consider increasing volume size
# (requires Terraform change and instance replacement)

etcd Issues

etcd Health Check Failed

Symptom:

{"level":"warn","msg":"health check failed","error":"context deadline exceeded"}

Solutions:

# On control plane node:
ssh -i ~/.ssh/key.pem ubuntu@<CP_IP>

# Check etcd status
sudo /var/lib/rancher/rke2/bin/kubectl \
  --kubeconfig /etc/rancher/rke2/rke2.yaml \
  -n kube-system exec -it etcd-<NODE_NAME> -- \
  etcdctl endpoint health --cluster

# Check etcd member list
sudo /var/lib/rancher/rke2/bin/kubectl \
  --kubeconfig /etc/rancher/rke2/rke2.yaml \
  -n kube-system exec -it etcd-<NODE_NAME> -- \
  etcdctl member list

# Check etcd logs
sudo journalctl -u rke2-server | grep etcd

etcd Lost Quorum

Symptom:

etcdserver: publish error: etcdserver: request timed out

Solutions:

# Check how many etcd members are healthy
# On any CP node:
sudo /var/lib/rancher/rke2/bin/kubectl \
  --kubeconfig /etc/rancher/rke2/rke2.yaml \
  get pods -n kube-system -l component=etcd

# If 2+ members are down, follow disaster recovery:
# https://docs.rke2.io/backup_restore

# Restore from snapshot (if available):
sudo rke2 server \
  --cluster-reset \
  --cluster-reset-restore-path=/path/to/snapshot

etcd Database Size Warning

Symptom:

etcdserver: mvcc: database space exceeded

Solutions:

# Check database size
sudo du -sh /var/lib/rancher/rke2/server/db/etcd/

# Compact and defragment
sudo /var/lib/rancher/rke2/bin/kubectl \
  --kubeconfig /etc/rancher/rke2/rke2.yaml \
  -n kube-system exec -it etcd-<NODE_NAME> -- \
  etcdctl compact $(etcdctl endpoint status --write-out="json" | jq -r '.[0].Status.header.revision')

sudo /var/lib/rancher/rke2/bin/kubectl \
  --kubeconfig /etc/rancher/rke2/rke2.yaml \
  -n kube-system exec -it etcd-<NODE_NAME> -- \
  etcdctl defrag --cluster

Networking Issues

Pods Cannot Communicate

Symptom:

# Pod A cannot reach Pod B
kubectl exec -it pod-a -- ping <POD_B_IP>
# No response

Solutions:

# Check Cilium agent status
kubectl -n kube-system get pods -l k8s-app=cilium
kubectl -n kube-system logs -l k8s-app=cilium --tail=50

# Check Cilium connectivity
kubectl -n kube-system exec -it cilium-xxxxx -- cilium status
kubectl -n kube-system exec -it cilium-xxxxx -- cilium connectivity test

# Check security group allows VXLAN (UDP 8472)
aws ec2 describe-security-groups --group-ids <SG_ID> \
  --query 'SecurityGroups[].IpPermissions[?FromPort==`8472`]'

# Check node can reach other nodes
ssh ubuntu@<NODE_IP> ping <OTHER_NODE_IP>

DNS Not Working

Symptom:

kubectl exec -it pod -- nslookup kubernetes
# Server:    10.43.0.10
# ** server can't find kubernetes: SERVFAIL

Solutions:

# Check CoreDNS pods
kubectl -n kube-system get pods -l k8s-app=kube-dns

# Check CoreDNS logs
kubectl -n kube-system logs -l k8s-app=kube-dns

# Verify CoreDNS service
kubectl -n kube-system get svc kube-dns

# Test DNS from a debug pod
kubectl run -it --rm debug --image=busybox -- nslookup kubernetes.default

# Restart CoreDNS if needed
kubectl -n kube-system rollout restart deployment coredns

NodePort Not Accessible

Symptom:

curl http://<WORKER_IP>:30080
# curl: (7) Failed to connect

Solutions:

# Verify service exists
kubectl get svc <SERVICE_NAME>

# Check NodePort range in security group
aws ec2 describe-security-groups --group-ids <WORKER_SG> \
  --query 'SecurityGroups[].IpPermissions[?FromPort<=`30080` && ToPort>=`30080`]'

# Verify pod is running and healthy
kubectl get pods -l app=<APP_LABEL>

# Test from inside cluster
kubectl run -it --rm debug --image=busybox -- wget -qO- <SERVICE_NAME>:80

# Check kube-proxy/Cilium service handling
kubectl -n kube-system logs -l k8s-app=cilium | grep -i <SERVICE_NAME>

Pod Issues

Pod Stuck in Pending

Symptom:

NAME      READY   STATUS    RESTARTS   AGE
nginx     0/1     Pending   0          10m

Solutions:

# Check events
kubectl describe pod <POD_NAME> | tail -20

# Common causes:

# 1. Insufficient resources
kubectl describe nodes | grep -A10 "Allocated resources"

# 2. Node selector mismatch
kubectl get pod <POD_NAME> -o yaml | grep nodeSelector

# 3. Taints not tolerated
kubectl describe nodes | grep Taints
kubectl get pod <POD_NAME> -o yaml | grep -A10 tolerations

# 4. PVC not bound
kubectl get pvc

Pod Stuck in ContainerCreating

Symptom:

NAME      READY   STATUS              RESTARTS   AGE
nginx     0/1     ContainerCreating   0          10m

Solutions:

# Check events
kubectl describe pod <POD_NAME> | grep -A20 Events

# Common causes:

# 1. Image pull failure
kubectl describe pod <POD_NAME> | grep -i "image"

# 2. CNI issue - Check Cilium
kubectl -n kube-system get pods -l k8s-app=cilium
kubectl -n kube-system logs -l k8s-app=cilium | tail -50

# 3. Volume mount issue
kubectl describe pod <POD_NAME> | grep -i volume

# Check containerd on the node
ssh ubuntu@<NODE_IP> sudo crictl ps
ssh ubuntu@<NODE_IP> sudo crictl logs <CONTAINER_ID>

Pod CrashLoopBackOff

Symptom:

NAME      READY   STATUS             RESTARTS   AGE
nginx     0/1     CrashLoopBackOff   5          10m

Solutions:

# Check pod logs
kubectl logs <POD_NAME>
kubectl logs <POD_NAME> --previous  # Previous instance

# Check container exit code
kubectl describe pod <POD_NAME> | grep -A5 "Last State"

# Common exit codes:
# 0: Success (check restartPolicy)
# 1: Application error
# 137: OOMKilled (increase memory limits)
# 139: Segfault (application bug)

# Check events
kubectl get events --field-selector involvedObject.name=<POD_NAME>

ImagePullBackOff

Symptom:

NAME      READY   STATUS             RESTARTS   AGE
nginx     0/1     ImagePullBackOff   0          10m

Solutions:

# Check image name/tag
kubectl describe pod <POD_NAME> | grep Image

# Test image pull manually on node
ssh ubuntu@<NODE_IP>
sudo crictl pull <IMAGE_NAME>

# Check registry accessibility
curl -I https://registry-1.docker.io/v2/

# For private registries, check ImagePullSecrets
kubectl get pod <POD_NAME> -o yaml | grep imagePullSecrets
kubectl get secret <SECRET_NAME> -o yaml

Useful Commands Reference

Cluster Health

# Overall cluster status
kubectl cluster-info
kubectl get nodes -o wide
kubectl get pods -A | grep -v Running

# Component status (deprecated but still useful)
kubectl get componentstatuses

# Check API server health
curl -k https://<NLB_DNS>:6443/healthz
curl -k https://<NLB_DNS>:6443/livez
curl -k https://<NLB_DNS>:6443/readyz

RKE2 Specific

# On control plane node:

# Check RKE2 version
rke2 --version

# Check RKE2 config
sudo cat /etc/rancher/rke2/config.yaml

# Check RKE2 service
sudo systemctl status rke2-server

# RKE2 logs
sudo journalctl -u rke2-server -f

# kubectl via RKE2
sudo /var/lib/rancher/rke2/bin/kubectl \
  --kubeconfig /etc/rancher/rke2/rke2.yaml \
  get nodes

etcd Commands

# On control plane node:

# etcd member list
sudo /var/lib/rancher/rke2/bin/kubectl \
  --kubeconfig /etc/rancher/rke2/rke2.yaml \
  -n kube-system exec -it etcd-<NODE_NAME> -- \
  etcdctl member list

# etcd endpoint status
sudo /var/lib/rancher/rke2/bin/kubectl \
  --kubeconfig /etc/rancher/rke2/rke2.yaml \
  -n kube-system exec -it etcd-<NODE_NAME> -- \
  etcdctl endpoint status --cluster --write-out=table

# etcd alarm list
sudo /var/lib/rancher/rke2/bin/kubectl \
  --kubeconfig /etc/rancher/rke2/rke2.yaml \
  -n kube-system exec -it etcd-<NODE_NAME> -- \
  etcdctl alarm list

Networking Debug

# Cilium status
kubectl -n kube-system exec -it cilium-xxxxx -- cilium status

# Cilium connectivity test
kubectl -n kube-system exec -it cilium-xxxxx -- cilium connectivity test

# Check endpoints
kubectl -n kube-system exec -it cilium-xxxxx -- cilium endpoint list

# Check service mapping
kubectl -n kube-system exec -it cilium-xxxxx -- cilium service list

Log Locations

On Nodes

Log	Location	Command
User Data	`/var/log/cloud-init-output.log`	`cat /var/log/cloud-init-output.log`
RKE2 Server	journald	`journalctl -u rke2-server`
RKE2 Agent	journald	`journalctl -u rke2-agent`
containerd	journald	`journalctl -u containerd`
System	`/var/log/syslog`	`tail -f /var/log/syslog`

In Cluster

Log	Command
All pod logs	`kubectl logs <POD_NAME>`
Previous container	`kubectl logs <POD_NAME> --previous`
All containers in pod	`kubectl logs <POD_NAME> --all-containers`
Follow logs	`kubectl logs <POD_NAME> -f`
Cilium logs	`kubectl -n kube-system logs -l k8s-app=cilium`
CoreDNS logs	`kubectl -n kube-system logs -l k8s-app=kube-dns`

Collecting Debug Bundle

# Create debug bundle
kubectl cluster-info dump > cluster-dump.txt

# More comprehensive
kubectl cluster-info dump --all-namespaces --output-directory=./cluster-dump/

# RKE2 specific debug
ssh ubuntu@<CP_IP> sudo rke2 server --help | grep -i debug

FilesExpand file tree

TROUBLESHOOTING.md

Latest commit

History

TROUBLESHOOTING.md

File metadata and controls

Troubleshooting Guide

Table of Contents

Deployment Issues

Terraform Init Fails

Terraform Apply Fails on VPC

EC2 Instance Launch Fails

SSH Key Permission Error

Node Bootstrap Timeout

Cluster Access Issues

Cannot Connect to API Server

Kubeconfig Issues

TLS Certificate Errors

Node Issues

Node Not Ready

Worker Cannot Join Cluster

Node Disk Pressure

etcd Issues

etcd Health Check Failed

etcd Lost Quorum

etcd Database Size Warning

Networking Issues

Pods Cannot Communicate

DNS Not Working

NodePort Not Accessible

Pod Issues

Pod Stuck in Pending

Pod Stuck in ContainerCreating

Pod CrashLoopBackOff

ImagePullBackOff

Useful Commands Reference

Cluster Health

RKE2 Specific

etcd Commands

Networking Debug

Log Locations

On Nodes

In Cluster

Collecting Debug Bundle

Getting Help

Resources

Support Channels