Skip to content

troubleshooting.md

Latest commit

 

History

History
2857 lines (2299 loc) · 64.8 KB

troubleshooting.md

File metadata and controls

2857 lines (2299 loc) · 64.8 KB
Raw

AWS Neuron on EKS: Comprehensive Troubleshooting Guide

Complete troubleshooting reference for AWS Trainium (Trn1), Inferentia2 (Inf2), and vLLM-Neuron deployments on Amazon EKS.

Table of Contents

  1. Trainium Training Issues
  2. Inferentia2 Inference Issues
  3. Kubernetes/EKS Issues
  4. Validation Checklist
  5. Quick Reference

Trainium Training Issues

Issue 1: Neuron Device Not Found / Driver Not Loaded

Symptom:

RuntimeError: No Neuron devices found
Error: Could not find NeuronCore devices
Traceback: neuron_cc: command not found

Diagnosis:

Check if Neuron driver is loaded:

# SSH into the Trn1 node
kubectl get nodes -l node.kubernetes.io/instance-type=trn1.32xlarge
NODE_NAME=$(kubectl get nodes -l node.kubernetes.io/instance-type=trn1.32xlarge -o jsonpath='{.items[0].metadata.name}')
kubectl debug node/$NODE_NAME -it --image=ubuntu

# In the debug container
chroot /host bash

# Check driver
ls /dev/neuron*
# Expected: /dev/neuron0 /dev/neuron1 ...

# Check kernel module
lsmod | grep neuron
# Expected: neuron 1234567 0

# Check Neuron runtime version
neuron-ls
# Expected: Neuron runtime version X.Y.Z

Solution:

Option A: Driver not installed (most common)

# Install Neuron driver on the node
# Add to EC2NodeClass userData or AMI

apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: trainium-nodeclass
spec:
  userData: |
    #!/bin/bash
    # Install Neuron driver
    sudo yum install aws-neuronx-dkms -y
    sudo yum install aws-neuronx-collectives -y
    sudo yum install aws-neuronx-runtime-lib -y
    
    # Load kernel module
    sudo modprobe neuron
    
    # Verify
    ls /dev/neuron*

Option B: Use AWS Deep Learning AMI (recommended)

# Get latest DL AMI ID
aws ec2 describe-images \
  --owners amazon \
  --filters "Name=name,Values=Deep Learning AMI Neuron PyTorch*" \
  --query 'Images | sort_by(@, &CreationDate) | [-1].ImageId' \
  --output text

# Update EC2NodeClass
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: trainium-nodeclass
spec:
  amiFamily: AL2
  amiSelectorTerms:
    - id: ami-0123456789abcdef0  # DL AMI

Option C: Fix missing kernel module

# On the node
sudo modprobe neuron

# Make persistent
echo "neuron" | sudo tee -a /etc/modules-load.d/neuron.conf

# Verify
lsmod | grep neuron
ls -la /dev/neuron*

Prevention:

  1. Use AWS Deep Learning AMI with Neuron pre-installed
  2. Add Neuron driver installation to EC2NodeClass userData
  3. Validate nodes before scheduling training jobs:
# Add node selector to training pod
spec:
  nodeSelector:
    aws.amazon.com/neuron-device-count: "32"
  tolerations:
  - key: "aws.amazon.com/neuron"
    operator: "Exists"

Issue 2: Training Job OOM (Out of Memory)

Symptom:

RuntimeError: NEURON_RT_EXEC_ERROR: Out of memory
NeuronCore memory allocation failed
torch.cuda.OutOfMemoryError (even though not using CUDA)
Pod killed by OOMKiller (exit code 137)

Diagnosis:

Check memory usage:

# Get pod name
POD=$(kubectl get pods -l job-name=llama-training -o jsonpath='{.items[0].metadata.name}')

# Check pod memory limits
kubectl describe pod $POD | grep -A 5 "Limits:"

# Check NeuronCore memory usage
kubectl exec $POD -- neuron-top

# Check host memory
kubectl exec $POD -- free -h

# Check logs for memory allocation failures
kubectl logs $POD | grep -i "memory\|oom"

Solution:

Option A: Increase batch size / enable gradient checkpointing

# training_config.yaml
training_args:
  per_device_train_batch_size: 1  # Reduce from 4 to 1
  gradient_accumulation_steps: 16  # Increase to maintain effective batch size
  gradient_checkpointing: true     # Enable to save memory

# Or in Python
from transformers import TrainingArguments

training_args = TrainingArguments(
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,
    gradient_checkpointing=True,
    fp16=True,  # Use mixed precision
)

Option B: Enable Neuron memory optimization

# Enable tensor parallelism to distribute model across NeuronCores
import torch_neuronx
from neuronx_distributed.parallel_layers import parallel_state

# Initialize tensor parallel group
parallel_state.initialize_model_parallel(tensor_model_parallel_size=8)

# Enable activation checkpointing
model.gradient_checkpointing_enable()

Option C: Increase pod memory limits

# llama-training-job.yaml
spec:
  containers:
  - name: training
    resources:
      limits:
        memory: 256Gi           # Increase from 128Gi
        aws.amazon.com/neuroncore: 32
      requests:
        memory: 256Gi
        aws.amazon.com/neuroncore: 32

Option D: Use larger instance type

# NodePool update - move from trn1.2xlarge to trn1.32xlarge
spec:
  template:
    spec:
      requirements:
      - key: node.kubernetes.io/instance-type
        operator: In
        values:
        - trn1.32xlarge  # 32 cores, 512GB memory

Prevention:

  1. Profile model memory before full training:
# memory_profiling.py
import torch
import torch_neuronx

model = YourModel()
model.to('xla')

# Dry run with profiling
with torch.no_grad():
    dummy_input = torch.randn(1, 512).to('xla')
    output = model(dummy_input)
    
# Check memory
print(torch_neuronx.memory_summary())
  1. Start with small batch size and increase gradually
  2. Use gradient checkpointing for models >7B parameters
  3. Monitor memory with Neuron Monitor dashboard

Issue 3: Distributed Training Hangs (EFA Issues)

Symptom:

Distributed training stuck at initialization
Timeout waiting for rendezvous
nccl.so: cannot open shared object file
EFA device not found
AllReduce operation timeout

Diagnosis:

Check EFA adapter:

# Get training pod
POD=$(kubectl get pods -l job-name=distributed-training -o jsonpath='{.items[0].metadata.name}')

# Check EFA interface
kubectl exec $POD -- fi_info -p efa

# Expected output:
# provider: efa
# version: X.Y.Z

# Check Neuron Collective Communication Library
kubectl exec $POD -- ls -la /opt/aws/neuron/lib | grep nccl

# Check EFA network interface
kubectl exec $POD -- ip link show | grep efa
# Expected: efa0: <BROADCAST,MULTICAST,UP,LOWER_UP>

# Test EFA connectivity between pods
kubectl exec $POD -- efa-test

Solution:

Option A: Enable EFA in NodePool

# karpenter/nodepool-trainium.yaml
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: trainium-distributed
spec:
  template:
    spec:
      requirements:
      - key: node.kubernetes.io/instance-type
        operator: In
        values:
        - trn1.32xlarge
      kubelet:
        systemReserved:
          cpu: "1"
          memory: 2Gi
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: trainium-efa-nodeclass
spec:
  networkInterfaces:
  - deviceIndex: 0
    interfaceType: efa  # Enable EFA
  securityGroupSelectorTerms:
  - tags:
      karpenter.sh/discovery: my-cluster
  subnetSelectorTerms:
  - tags:
      karpenter.sh/discovery: my-cluster

Option B: Fix Pod networking for EFA

# distributed-training-job.yaml
apiVersion: v1
kind: Pod
metadata:
  name: training-worker
  annotations:
    k8s.amazonaws.com/efa: "1"  # Request EFA device
spec:
  hostNetwork: true  # Required for EFA
  dnsPolicy: ClusterFirstWithHostNet
  containers:
  - name: training
    image: your-training-image
    securityContext:
      capabilities:
        add:
        - IPC_LOCK  # Required for EFA
    resources:
      limits:
        vpc.amazonaws.com/efa: 1
        aws.amazon.com/neuroncore: 32
    volumeMounts:
    - name: efa-device
      mountPath: /dev/infiniband
  volumes:
  - name: efa-device
    hostPath:
      path: /dev/infiniband

Option C: Install EFA driver in container

# Dockerfile
FROM pytorch/pytorch:2.0-cuda11.8-cudnn8-runtime

# Install Neuron SDK
RUN pip install torch-neuronx neuronx-cc --extra-index-url=https://pip.repos.neuron.amazonaws.com

# Install EFA driver
RUN apt-get update && apt-get install -y \
    libfabric-dev \
    libhwloc-dev \
    && rm -rf /var/lib/apt/lists/*

# Install Neuron Collective Communication
RUN pip install neuronx-distributed

Option D: Fix security group for EFA traffic

# Allow all traffic within security group (required for EFA)
SECURITY_GROUP_ID=$(aws eks describe-cluster \
  --name my-cluster \
  --query 'cluster.resourcesVpcConfig.clusterSecurityGroupId' \
  --output text)

aws ec2 authorize-security-group-ingress \
  --group-id $SECURITY_GROUP_ID \
  --protocol all \
  --source-group $SECURITY_GROUP_ID

Prevention:

  1. Always use hostNetwork for distributed training
  2. Enable EFA in EC2NodeClass from the start
  3. Test EFA connectivity before multi-node training:
# test_efa.py
import torch
import torch_neuronx
import torch.distributed as dist

dist.init_process_group(backend='xla')
rank = dist.get_rank()
world_size = dist.get_world_size()

print(f"Rank {rank}/{world_size} - EFA initialized")

# Test collective operation
tensor = torch.ones(1).to('xla')
dist.all_reduce(tensor)
print(f"Rank {rank} - AllReduce result: {tensor.item()}")
  1. Use placement groups for low-latency EFA communication

Issue 4: Slow Throughput vs Expected

Symptom:

Training speed: 100 tokens/s (expected: 500+ tokens/s)
Model compilation taking >30 minutes
Repeated graph recompilation during training
XLA HLO compilation warnings in logs

Diagnosis:

Check performance metrics:

# Monitor NeuronCore utilization
kubectl exec $POD -- neuron-top

# Expected: NeuronCore 0-31 at 90%+ utilization

# Check compilation logs
kubectl logs $POD | grep "compilation\|XLA"

# Check if model is recompiling
kubectl logs $POD | grep "Compiling" | wc -l
# Should be 1 (initial compilation only)

# Monitor throughput
kubectl exec $POD -- cat /tmp/training_metrics.txt

Solution:

Option A: Enable graph caching to avoid recompilation

# training_script.py
import torch_neuronx
import os

# Enable persistent graph cache
os.environ['NEURON_CC_FLAGS'] = '--cache_dir=/tmp/neuron_cache'
os.environ['NEURON_COMPILE_CACHE_URL'] = '/tmp/neuron_cache'

# Persist cache across runs via PVC
# See volume mount below
# Mount persistent cache volume
spec:
  containers:
  - name: training
    volumeMounts:
    - name: neuron-cache
      mountPath: /tmp/neuron_cache
  volumes:
  - name: neuron-cache
    persistentVolumeClaim:
      claimName: neuron-compile-cache

Option B: Optimize data loading

# training_script.py
from torch.utils.data import DataLoader

# Use multiple workers for data loading
train_dataloader = DataLoader(
    train_dataset,
    batch_size=8,
    num_workers=8,      # Increase from 0
    pin_memory=True,    # Enable pin memory
    prefetch_factor=2   # Prefetch batches
)

Option C: Enable mixed precision training

# training_script.py
import torch
from transformers import TrainingArguments

training_args = TrainingArguments(
    fp16=True,              # Enable FP16
    bf16=False,             # Or use BF16
    fp16_opt_level='O1',    # Optimization level
)

# For Neuron-specific optimization
import torch_neuronx
torch_neuronx.amp.autocast(enabled=True)

Option D: Profile and optimize hotspots

# Enable Neuron profiling
kubectl exec $POD -- neuron-profile -p $TRAINING_PID -o /tmp/profile.json

# Analyze profile
kubectl cp $POD:/tmp/profile.json ./profile.json

# View profile (requires Neuron Tools)
neuron-profile view profile.json

Prevention:

  1. Always use persistent compilation cache
  2. Pre-compile models before training:
# pre_compile.py
import torch
import torch_neuronx

model = YourModel()
dummy_input = torch.randn(1, 512)

# Compile model
traced_model = torch_neuronx.trace(model, dummy_input)
torch.jit.save(traced_model, 'compiled_model.pt')
  1. Monitor NeuronCore utilization - should be >80%
  2. Use Neuron Monitor dashboard to track throughput trends

Issue 5: Model Compilation Failures

Symptom:

NeuronCompilerError: Compilation failed
RuntimeError: Graph execution failed
UnsupportedOperator: aten::some_operation
ERROR: Maximum supported tensor size exceeded
Segmentation fault during compilation

Diagnosis:

Check compilation logs:

# Get detailed compilation logs
kubectl logs $POD --tail=1000 | grep -A 20 "Compilation"

# Check Neuron compiler version
kubectl exec $POD -- neuron-cc --version

# Check for unsupported operations
kubectl logs $POD | grep "UnsupportedOperator"

# Check model architecture
kubectl exec $POD -- python -c "from transformers import AutoModel; \
  model = AutoModel.from_pretrained('meta-llama/Llama-2-7b-hf'); \
  print(model)"

Solution:

Option A: Update Neuron SDK to latest version

# Dockerfile
FROM public.ecr.aws/neuron/pytorch-training-neuronx:1.13.1-neuronx-py310-sdk2.16.0-ubuntu20.04

# Or update in running environment
RUN pip install --upgrade torch-neuronx neuronx-cc

Option B: Replace unsupported operations

# model_fixes.py
import torch
import torch.nn as nn

# Example: Replace unsupported layer norm
class NeuronLayerNorm(nn.Module):
    def __init__(self, normalized_shape, eps=1e-5):
        super().__init__()
        self.weight = nn.Parameter(torch.ones(normalized_shape))
        self.bias = nn.Parameter(torch.zeros(normalized_shape))
        self.eps = eps
    
    def forward(self, x):
        return torch.nn.functional.layer_norm(
            x, self.weight.shape, self.weight, self.bias, self.eps
        )

# Patch model
def patch_model_for_neuron(model):
    for name, module in model.named_modules():
        if isinstance(module, nn.LayerNorm):
            setattr(model, name, NeuronLayerNorm(
                module.normalized_shape, module.eps
            ))
    return model

model = patch_model_for_neuron(model)

Option C: Reduce model size to fit compiler limits

# training_script.py
from transformers import AutoConfig, AutoModelForCausalLM

# Load config and modify
config = AutoConfig.from_pretrained('meta-llama/Llama-2-7b-hf')
config.max_position_embeddings = 2048  # Reduce from 4096
config.num_hidden_layers = 24          # Reduce from 32

model = AutoModelForCausalLM.from_config(config)

Option D: Split large model across multiple NeuronCores

# distributed_model.py
from neuronx_distributed.parallel_layers import parallel_state
from neuronx_distributed.parallel_layers import ColumnParallelLinear

# Enable tensor parallelism
parallel_state.initialize_model_parallel(tensor_model_parallel_size=8)

# Replace large layers with parallel versions
model.lm_head = ColumnParallelLinear(
    in_features=4096,
    out_features=50257,
    gather_output=True
)

Prevention:

  1. Test model compilation before full training:
# test_compilation.py
import torch
import torch_neuronx

model = YourModel()
model.eval()

# Test compile with small input
dummy_input = torch.randn(1, 128)
try:
    traced = torch_neuronx.trace(model, dummy_input)
    print("Compilation successful!")
except Exception as e:
    print(f"Compilation failed: {e}")

  1. Check Neuron SDK compatibility matrix for your model architecture

  2. Use reference implementations from AWS Neuron Samples:

  3. Enable verbose compilation logs:

import os
os.environ['NEURON_CC_FLAGS'] = '--verbose ALL'

Inferentia2 Inference Issues

Issue 1: Model Loading Fails / NeuronCore Allocation Error

Symptom:

RuntimeError: Failed to load model on NeuronCore
Error: No available NeuronCores for model allocation
NeuronCoreAllocationError: Insufficient NeuronCore memory
vllm.engine: Cannot allocate model across available devices

Diagnosis:

Check NeuronCore availability:

# Get inference pod
POD=$(kubectl get pods -l app=vllm-llama3 -o jsonpath='{.items[0].metadata.name}')

# Check NeuronCore allocation
kubectl exec $POD -- neuron-ls

# Expected output:
# +--------+--------+--------+
# | Device | Status | Memory |
# +--------+--------+--------+
# | 0      | In-use | 16 GB  |
# | 1      | Free   | 16 GB  |
# ...

# Check container resource limits
kubectl describe pod $POD | grep -A 10 "Limits:"

# Check for resource allocation errors
kubectl describe pod $POD | grep -i "neuron"

Solution:

Option A: Request correct number of NeuronCores

# vllm-llama3-inf2.yaml
spec:
  containers:
  - name: vllm-server
    image: vllm/vllm-neuron:latest
    resources:
      limits:
        aws.amazon.com/neuroncore: 2  # Llama 3 8B needs 2 cores
        memory: 32Gi
      requests:
        aws.amazon.com/neuroncore: 2
        memory: 32Gi

Model size to NeuronCore mapping:

  • 7-8B models: 2 NeuronCores
  • 13B models: 4 NeuronCores
  • 70B models: 24 NeuronCores

Option B: Reduce model size to fit available cores

# Use smaller model or quantized version
kubectl set env deployment/vllm-llama3 \
  MODEL_NAME=meta-llama/Meta-Llama-3-8B-Instruct \
  QUANTIZATION=awq  # Enable quantization

Option C: Fix NeuronCore fragmentation

# Restart pod to reset NeuronCore allocation
kubectl delete pod $POD

# Or drain node and restart
NODE=$(kubectl get pod $POD -o jsonpath='{.spec.nodeName}')
kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data
kubectl uncordon $NODE

Option D: Verify Device Plugin is registering cores correctly

# Check device plugin logs
kubectl logs -n kube-system -l app.kubernetes.io/name=neuron-device-plugin

# Check node capacity
kubectl get node -o json | jq '.items[].status.capacity["aws.amazon.com/neuroncore"]'

# Should show total NeuronCores (e.g., "2" for Inf2.xlarge)

Prevention:

  1. Always specify exact NeuronCore requirements in pod spec
  2. Use pod anti-affinity to prevent core fragmentation:
spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - vllm-llama3
          topologyKey: kubernetes.io/hostname
  1. Monitor NeuronCore allocation with Neuron Monitor
  2. Test model loading before production deployment:
# test_model_load.py
from vllm import LLM

try:
    llm = LLM(
        model="meta-llama/Meta-Llama-3-8B",
        device="neuron",
        tensor_parallel_size=2
    )
    print("Model loaded successfully!")
except Exception as e:
    print(f"Model loading failed: {e}")

Issue 2: Low Inference Throughput (<500 tokens/s, expected 1100+)

Symptom:

Observed throughput: 300 tokens/s
Expected throughput: 1100+ tokens/s for Llama 3 8B on Inf2.xlarge
High latency per request (>500ms P99)
vLLM dashboard showing low batch utilization

Diagnosis:

Check throughput metrics:

# Check vLLM metrics
kubectl exec $POD -- curl http://localhost:8000/metrics | grep throughput

# Check batch size
kubectl logs $POD | grep "batch_size"

# Check NeuronCore utilization
kubectl exec $POD -- neuron-top
# Should show >80% utilization

# Run benchmark
kubectl exec $POD -- python3 /app/benchmark.py \
  --model meta-llama/Meta-Llama-3-8B \
  --num-prompts 100

Solution:

Option A: Increase batch size for better throughput

# vllm-llama3-inf2.yaml
spec:
  containers:
  - name: vllm-server
    env:
    - name: MAX_BATCH_SIZE
      value: "32"  # Increase from default 8
    - name: MAX_NUM_SEQS
      value: "256"  # Increase max concurrent sequences
    command:
    - python3
    - -m
    - vllm.entrypoints.openai.api_server
    - --model
    - meta-llama/Meta-Llama-3-8B
    - --device
    - neuron
    - --tensor-parallel-size
    - "2"
    - --max-model-len
    - "2048"
    - --max-num-batched-tokens
    - "4096"  # Increase batching

Option B: Enable continuous batching

# vllm_config.py
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Meta-Llama-3-8B",
    device="neuron",
    tensor_parallel_size=2,
    max_num_batched_tokens=4096,
    max_num_seqs=256,
    enable_chunked_prefill=True  # Enable continuous batching
)

Option C: Optimize model compilation for inference

# Set Neuron compiler flags for inference optimization
kubectl set env deployment/vllm-llama3 \
  NEURON_CC_FLAGS="--model-type=transformer-inference --auto-cast=matmult --auto-cast-type=bf16"

Option D: Use larger instance type for higher throughput

# Update NodePool to use Inf2.48xlarge
spec:
  template:
    spec:
      requirements:
      - key: node.kubernetes.io/instance-type
        operator: In
        values:
        - inf2.48xlarge  # 12 NeuronCores, higher memory bandwidth
# Update deployment for inf2.48xlarge
spec:
  containers:
  - name: vllm-server
    resources:
      limits:
        aws.amazon.com/neuroncore: 12  # Use all 12 cores
        memory: 192Gi
    env:
    - name: TENSOR_PARALLEL_SIZE
      value: "12"

Prevention:

  1. Always benchmark with realistic traffic patterns:
# benchmark_inference.py
import asyncio
import aiohttp
import time

async def send_request(session, prompt):
    async with session.post(
        'http://localhost:8000/v1/completions',
        json={
            'model': 'meta-llama/Meta-Llama-3-8B',
            'prompt': prompt,
            'max_tokens': 100
        }
    ) as response:
        return await response.json()

async def benchmark(num_requests=100):
    start = time.time()
    async with aiohttp.ClientSession() as session:
        tasks = [
            send_request(session, f"Prompt {i}")
            for i in range(num_requests)
        ]
        responses = await asyncio.gather(*tasks)
    
    duration = time.time() - start
    tokens_generated = num_requests * 100
    throughput = tokens_generated / duration
    print(f"Throughput: {throughput:.2f} tokens/s")

asyncio.run(benchmark())
  1. Monitor throughput continuously with Prometheus:
# prometheus-rule.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: vllm-throughput-alert
spec:
  groups:
  - name: vllm
    rules:
    - alert: LowThroughput
      expr: rate(vllm_tokens_generated_total[5m]) < 500
      annotations:
        summary: "vLLM throughput below 500 tokens/s"
  1. Use appropriate batch sizes based on model size
  2. Enable PagedAttention for memory efficiency

Issue 3: High P99 Latency

Symptom:

P99 latency: >500ms (expected: <200ms)
Inconsistent response times
Long tail latencies affecting user experience
Timeout errors for some requests

Diagnosis:

Check latency metrics:

# Get latency distribution from vLLM metrics
kubectl exec $POD -- curl http://localhost:8000/metrics | grep latency

# Check request queue depth
kubectl logs $POD | grep "queue"

# Monitor with custom script
kubectl exec $POD -- python3 - <<EOF
import time
import requests

latencies = []
for i in range(100):
    start = time.time()
    resp = requests.post('http://localhost:8000/v1/completions', json={
        'model': 'meta-llama/Meta-Llama-3-8B',
        'prompt': 'Hello',
        'max_tokens': 50
    })
    latency = time.time() - start
    latencies.append(latency)

latencies.sort()
print(f"P50: {latencies[50]*1000:.2f}ms")
print(f"P99: {latencies[99]*1000:.2f}ms")
EOF

Solution:

Option A: Reduce KV cache size to avoid memory pressure

spec:
  containers:
  - name: vllm-server
    env:
    - name: VLLM_GPU_MEMORY_UTILIZATION  # Also applies to Neuron
      value: "0.8"  # Reduce from 0.9 to avoid OOM spikes
    command:
    - python3
    - -m
    - vllm.entrypoints.openai.api_server
    - --model
    - meta-llama/Meta-Llama-3-8B
    - --max-model-len
    - "2048"  # Reduce from 4096
    - --gpu-memory-utilization
    - "0.8"

Option B: Enable speculative decoding

# vllm_server.py
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Meta-Llama-3-8B",
    device="neuron",
    tensor_parallel_size=2,
    speculative_model="facebook/opt-125m",  # Small draft model
    num_speculative_tokens=5
)

Option C: Implement request prioritization

# Create separate deployments for latency-sensitive traffic
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-low-latency
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: vllm-server
        env:
        - name: MAX_NUM_SEQS
          value: "64"  # Lower concurrency for lower latency
        - name: MAX_BATCH_SIZE
          value: "8"   # Smaller batches

Option D: Add HPA for scaling under load

# hpa-vllm.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: vllm-llama3-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama3
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: vllm_request_latency_seconds_p99
      target:
        type: Value
        value: "200m"  # 200ms

Prevention:

  1. Set appropriate timeouts:
# client_config.py
import requests

response = requests.post(
    'http://vllm-service:8000/v1/completions',
    json={'model': 'meta-llama/Meta-Llama-3-8B', 'prompt': 'Hello'},
    timeout=5.0  # 5 second timeout
)
  1. Monitor P99 latency continuously:
# grafana-dashboard.json (excerpt)
{
  "targets": [{
    "expr": "histogram_quantile(0.99, rate(vllm_request_duration_seconds_bucket[5m]))",
    "legendFormat": "P99 Latency"
  }]
}
  1. Use pod disruption budgets to prevent latency spikes during updates:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: vllm-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: vllm-llama3

Issue 4: vLLM-Neuron Not Utilizing PagedAttention

Symptom:

High memory usage for KV cache
OOM errors with multiple concurrent requests
vLLM logs not showing PagedAttention initialization
Memory usage not scaling efficiently with request count

Diagnosis:

Check PagedAttention status:

# Check vLLM logs for PagedAttention
kubectl logs $POD | grep -i "paged\|attention"

# Expected: "PagedAttention enabled"

# Check memory usage pattern
kubectl exec $POD -- neuron-top -t memory

# Check vLLM version (PagedAttention requires v0.3.0+)
kubectl exec $POD -- pip show vllm-neuron

Solution:

Option A: Upgrade to vLLM-Neuron with PagedAttention support

# Dockerfile
FROM public.ecr.aws/neuron/pytorch-inference-neuronx:2.1.2-neuronx-py310-sdk2.16.0-ubuntu20.04

# Install vLLM-Neuron with PagedAttention
RUN pip install vllm-neuron>=0.3.0

Option B: Explicitly enable PagedAttention

# vllm_server.py
from vllm import LLM

llm = LLM(
    model="meta-llama/Meta-Llama-3-8B",
    device="neuron",
    tensor_parallel_size=2,
    block_size=16,           # Enable PagedAttention
    max_num_batched_tokens=2048,
    enable_paged_kv_cache=True  # Explicit flag
)

Option C: Configure KV cache block size

spec:
  containers:
  - name: vllm-server
    command:
    - python3
    - -m
    - vllm.entrypoints.openai.api_server
    - --model
    - meta-llama/Meta-Llama-3-8B
    - --device
    - neuron
    - --block-size
    - "16"  # PagedAttention block size
    - --max-num-batched-tokens
    - "2048"

Option D: Verify Neuron SDK version supports PagedAttention

# Check Neuron SDK version (requires 2.16.0+)
kubectl exec $POD -- python3 -c "import torch_neuronx; print(torch_neuronx.__version__)"

# Upgrade if needed
kubectl set image deployment/vllm-llama3 \
  vllm-server=public.ecr.aws/neuron/pytorch-inference-neuronx:2.1.2-neuronx-py310-sdk2.16.0-ubuntu20.04

Prevention:

  1. Always use latest vLLM-Neuron release
  2. Verify PagedAttention in logs on startup:
kubectl logs -f $POD | grep -A 5 "Initializing"
# Should show: "Using PagedAttention with block_size=16"
  1. Monitor memory efficiency:
# memory_efficiency_check.py
# Before PagedAttention: ~10GB per request
# After PagedAttention: ~500MB per request shared pool
  1. Test with multiple concurrent requests:
# concurrent_test.sh
for i in {1..10}; do
  curl -X POST http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "meta-llama/Meta-Llama-3-8B",
      "prompt": "Hello",
      "max_tokens": 100
    }' &
done
wait

Issue 5: Pod Stuck in ContainerCreating

Symptom:

Pod status: ContainerCreating for >5 minutes
Events: "Failed to attach NeuronDevice"
Events: "Insufficient aws.amazon.com/neuroncore"
Node has NeuronCores but pod not scheduling

Diagnosis:

Check pod status and events:

# Get pod details
POD=$(kubectl get pods -l app=vllm-llama3 --field-selector=status.phase=Pending -o jsonpath='{.items[0].metadata.name}')

kubectl describe pod $POD

# Look for:
# - "Insufficient aws.amazon.com/neuroncore"
# - "FailedScheduling"
# - "FailedMount"

# Check node capacity
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, capacity: .status.capacity, allocatable: .status.allocatable}'

# Check device plugin
kubectl get daemonset -n kube-system neuron-device-plugin-daemonset

Solution:

Option A: Device plugin not running on target nodes

# Check device plugin pods
kubectl get pods -n kube-system -l app.kubernetes.io/name=neuron-device-plugin -o wide

# Redeploy device plugin
kubectl delete daemonset -n kube-system neuron-device-plugin-daemonset
kubectl apply -f https://raw.githubusercontent.com/aws-neuron/aws-neuron-sdk/main/src/k8s/k8s-neuron-device-plugin.yml

# Verify
kubectl wait --for=condition=ready pod -n kube-system -l app.kubernetes.io/name=neuron-device-plugin --timeout=120s

Option B: Insufficient NeuronCores on available nodes

# Check current allocation
kubectl describe nodes | grep -A 5 "aws.amazon.com/neuroncore"

# Scale up node pool
kubectl scale nodepool inferentia2-pool --replicas=3

# Or use Karpenter to provision more nodes (automatic)

Option C: Volume mount issues

# Check for volume mount errors
kubectl describe pod $POD | grep -A 10 "Volumes:"

# Common issue: PVC not bound
kubectl get pvc

# Delete and recreate pod
kubectl delete pod $POD

Option D: Node taints preventing scheduling

# Check node taints
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, taints: .spec.taints}'

# Add toleration to pod spec if needed
kubectl patch deployment vllm-llama3 --type=json -p='[
  {
    "op": "add",
    "path": "/spec/template/spec/tolerations",
    "value": [{
      "key": "aws.amazon.com/neuron",
      "operator": "Exists",
      "effect": "NoSchedule"
    }]
  }
]'

Prevention:

  1. Always verify device plugin is running before deploying workloads:
# validate-neuron.sh
#!/bin/bash
kubectl get daemonset -n kube-system neuron-device-plugin-daemonset || {
  echo "Device plugin not found. Installing..."
  kubectl apply -f https://raw.githubusercontent.com/aws-neuron/aws-neuron-sdk/main/src/k8s/k8s-neuron-device-plugin.yml
}
  1. Set appropriate resource requests:
spec:
  containers:
  - name: vllm-server
    resources:
      requests:  # Requests == Limits for Neuron resources
        aws.amazon.com/neuroncore: 2
        memory: 32Gi
      limits:
        aws.amazon.com/neuroncore: 2
        memory: 32Gi
  1. Use pod affinity to prefer nodes with available cores:
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: aws.amazon.com/neuron-device-count
            operator: Gt
            values: ["0"]

Kubernetes/EKS Issues

Issue 1: Neuron Device Plugin Not Running

Symptom:

DaemonSet not creating pods on Inf2/Trn1 nodes
Resource "aws.amazon.com/neuroncore" not showing in node capacity
Device plugin logs showing "Failed to discover Neuron devices"
Pods failing with "UnexpectedAdmissionError"

Diagnosis:

Check device plugin status:

# Check DaemonSet
kubectl get daemonset -n kube-system neuron-device-plugin-daemonset

# Check device plugin pods
kubectl get pods -n kube-system -l app.kubernetes.io/name=neuron-device-plugin -o wide

# Check logs
kubectl logs -n kube-system -l app.kubernetes.io/name=neuron-device-plugin --tail=100

# Check if device plugin registered with kubelet
kubectl get nodes -o json | jq '.items[0].status.capacity'
# Should include "aws.amazon.com/neuroncore" and "aws.amazon.com/neurondevice"

Solution:

Option A: Install device plugin

# Install from official repository
kubectl apply -f https://raw.githubusercontent.com/aws-neuron/aws-neuron-sdk/main/src/k8s/k8s-neuron-device-plugin.yml

# Verify installation
kubectl wait --for=condition=ready pod \
  -n kube-system \
  -l app.kubernetes.io/name=neuron-device-plugin \
  --timeout=120s

# Check registration
kubectl get nodes -o json | jq '.items[].status.capacity["aws.amazon.com/neuroncore"]'

Option B: Fix DaemonSet node selector

# Check if DaemonSet has restrictive node selectors
kubectl get daemonset -n kube-system neuron-device-plugin-daemonset -o yaml | grep -A 5 nodeSelector

# Remove node selector if too restrictive
kubectl patch daemonset -n kube-system neuron-device-plugin-daemonset --type=json -p='[
  {
    "op": "remove",
    "path": "/spec/template/spec/nodeSelector"
  }
]'

# Or update to match your nodes
kubectl patch daemonset -n kube-system neuron-device-plugin-daemonset --type=json -p='[
  {
    "op": "replace",
    "path": "/spec/template/spec/nodeSelector",
    "value": {"node.kubernetes.io/instance-type": "inf2.xlarge"}
  }
]'

Option C: Fix device plugin permissions

# device-plugin-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: neuron-device-plugin
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: neuron-device-plugin
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "watch", "patch", "update"]
- apiGroups: [""]
  resources: ["nodes/status"]
  verbs: ["patch", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: neuron-device-plugin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: neuron-device-plugin
subjects:
- kind: ServiceAccount
  name: neuron-device-plugin
  namespace: kube-system
kubectl apply -f device-plugin-rbac.yaml

Option D: Restart device plugin pods

# Delete pods to force recreation
kubectl delete pods -n kube-system -l app.kubernetes.io/name=neuron-device-plugin

# Wait for new pods
kubectl wait --for=condition=ready pod \
  -n kube-system \
  -l app.kubernetes.io/name=neuron-device-plugin \
  --timeout=120s

Prevention:

  1. Add device plugin to cluster bootstrap process
  2. Monitor device plugin health:
# prometheus-alert.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: neuron-device-plugin-alert
spec:
  groups:
  - name: neuron-device-plugin
    rules:
    - alert: DevicePluginDown
      expr: kube_daemonset_status_number_ready{daemonset="neuron-device-plugin-daemonset"} == 0
      for: 5m
      annotations:
        summary: "Neuron device plugin is down"
  1. Validate device plugin after node addition:
# validate-node.sh
#!/bin/bash
NODE=$1
kubectl wait --for=condition=ready node $NODE --timeout=300s
kubectl get node $NODE -o json | jq '.status.capacity["aws.amazon.com/neuroncore"]' | grep -q "[0-9]" || {
  echo "NeuronCores not registered on node $NODE"
  exit 1
}

Issue 2: Karpenter Not Provisioning Inf2/Trn1 Nodes

Symptom:

Pods pending with "Insufficient aws.amazon.com/neuroncore"
Karpenter logs showing "No viable instance types"
Nodes not launching despite pending pods
NodePool not selecting Inf2/Trn1 instance types

Diagnosis:

Check Karpenter status:

# Check Karpenter logs
kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter --tail=200

# Look for:
# - "No viable instance types"
# - "Launch failed"
# - "Insufficient capacity"

# Check NodePool configuration
kubectl get nodepool -o yaml

# Check EC2NodeClass
kubectl get ec2nodeclass -o yaml

# Check pending pods
kubectl get pods --field-selector=status.phase=Pending
kubectl describe pod <pending-pod> | grep -A 10 "Events:"

# Check AWS capacity
aws ec2 describe-instance-type-offerings \
  --location-type availability-zone \
  --filters Name=instance-type,Values=inf2.xlarge \
  --region us-east-1

Solution:

Option A: Fix NodePool instance type requirements

# karpenter/nodepool-inferentia2.yaml
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: inferentia2-pool
spec:
  template:
    spec:
      requirements:
      - key: karpenter.k8s.aws/instance-family
        operator: In
        values:
        - inf2  # Ensure Inf2 family is included
      - key: node.kubernetes.io/instance-type
        operator: In
        values:
        - inf2.xlarge
        - inf2.8xlarge
        - inf2.24xlarge
        - inf2.48xlarge
      - key: karpenter.sh/capacity-type
        operator: In
        values:
        - on-demand  # Use on-demand for ML workloads
      nodeClassRef:
        name: inferentia2-nodeclass
  limits:
    cpu: "1000"
    memory: 1000Gi

Option B: Fix EC2NodeClass subnet/security group selection

# karpenter/ec2nodeclass-neuron.yaml
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: inferentia2-nodeclass
spec:
  amiFamily: AL2
  role: KarpenterNodeRole-my-cluster
  subnetSelectorTerms:
  - tags:
      karpenter.sh/discovery: my-cluster  # Match your cluster name
      kubernetes.io/role/internal-elb: "1"  # Private subnets
  securityGroupSelectorTerms:
  - tags:
      karpenter.sh/discovery: my-cluster
  tags:
    Name: karpenter-inferentia2-node
    Environment: production
  blockDeviceMappings:
  - deviceName: /dev/xvda
    ebs:
      volumeSize: 100Gi
      volumeType: gp3
      encrypted: true
  userData: |
    #!/bin/bash
    # Optional: Install Neuron driver if not using DL AMI
    # yum install -y aws-neuronx-dkms aws-neuronx-runtime-lib

Option C: Check IAM permissions for Karpenter

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:RunInstances",
        "ec2:CreateFleet",
        "ec2:CreateTags",
        "ec2:DescribeInstances",
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeInstanceTypeOfferings",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeImages",
        "ec2:DescribeSubnets",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeLaunchTemplates",
        "iam:PassRole",
        "pricing:GetProducts",
        "ssm:GetParameter"
      ],
      "Resource": "*"
    }
  ]
}
# Verify Karpenter IAM role has permissions
aws iam get-role-policy \
  --role-name KarpenterControllerRole-my-cluster \
  --policy-name KarpenterControllerPolicy

Option D: Check availability zone capacity

# List AZs with Inf2 capacity
aws ec2 describe-instance-type-offerings \
  --filters Name=instance-type,Values=inf2.xlarge \
  --location-type availability-zone \
  --query 'InstanceTypeOfferings[].Location' \
  --output table

# Update NodePool to use AZs with capacity
kubectl patch nodepool inferentia2-pool --type=json -p='[
  {
    "op": "add",
    "path": "/spec/template/spec/requirements/-",
    "value": {
      "key": "topology.kubernetes.io/zone",
      "operator": "In",
      "values": ["us-east-1a", "us-east-1b", "us-east-1c"]
    }
  }
]'

Prevention:

  1. Test NodePool configuration before deploying workloads:
# Create test pod that requires NeuronCores
kubectl run neuron-test \
  --image=public.ecr.aws/neuron/pytorch-inference-neuronx:latest \
  --limits="aws.amazon.com/neuroncore=2" \
  --command -- sleep infinity

# Watch Karpenter provision node
kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter -f

# Cleanup
kubectl delete pod neuron-test
  1. Monitor Karpenter metrics:
# grafana-dashboard.json (excerpt)
{
  "targets": [{
    "expr": "karpenter_nodes_created_total",
    "legendFormat": "Nodes Created"
  }]
}
  1. Set up alerts for provisioning failures:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: karpenter-alerts
spec:
  groups:
  - name: karpenter
    rules:
    - alert: KarpenterProvisioningFailed
      expr: rate(karpenter_provisioning_errors_total[5m]) > 0
      annotations:
        summary: "Karpenter failing to provision nodes"

Issue 3: Resource "aws.amazon.com/neuroncore" Not Available

Symptom:

Pod scheduling error: "Insufficient aws.amazon.com/neuroncore"
Node capacity doesn't show aws.amazon.com/neuroncore
Device plugin running but resources not registered
kubectl describe node shows 0 NeuronCores

Diagnosis:

Check resource availability:

# Check node capacity
kubectl get nodes -o json | jq '.items[] | {
  name: .metadata.name,
  capacity: .status.capacity,
  allocatable: .status.allocatable
}'

# Should show:
# "aws.amazon.com/neuroncore": "2"
# "aws.amazon.com/neurondevice": "1"

# Check device plugin pods on node
NODE_NAME=$(kubectl get nodes -l node.kubernetes.io/instance-type=inf2.xlarge -o jsonpath='{.items[0].metadata.name}')
kubectl get pods -n kube-system -o wide | grep $NODE_NAME | grep neuron

# Check device plugin logs for that node
DEVICE_PLUGIN_POD=$(kubectl get pods -n kube-system -l app.kubernetes.io/name=neuron-device-plugin --field-selector spec.nodeName=$NODE_NAME -o jsonpath='{.items[0].metadata.name}')
kubectl logs -n kube-system $DEVICE_PLUGIN_POD

Solution:

Option A: Verify Neuron devices exist on node

# Access node
kubectl debug node/$NODE_NAME -it --image=ubuntu

# In debug container
chroot /host bash
ls -la /dev/neuron*

# Expected: /dev/neuron0 (for Inf2.xlarge with 2 cores)

# If devices missing, check driver
lsmod | grep neuron
dmesg | grep -i neuron

Option B: Restart device plugin on specific node

# Delete device plugin pod on problematic node
kubectl delete pod -n kube-system $DEVICE_PLUGIN_POD

# Wait for new pod
kubectl wait --for=condition=ready pod \
  -n kube-system \
  -l app.kubernetes.io/name=neuron-device-plugin \
  --field-selector spec.nodeName=$NODE_NAME \
  --timeout=60s

# Verify resources registered
kubectl get node $NODE_NAME -o json | jq '.status.capacity'

Option C: Check kubelet configuration

# SSH to node
kubectl debug node/$NODE_NAME -it --image=ubuntu
chroot /host bash

# Check kubelet config
cat /var/lib/kubelet/config.yaml | grep -i plugin

# Ensure device plugin registration path exists
ls -la /var/lib/kubelet/device-plugins/

# Check kubelet logs
journalctl -u kubelet -n 100 | grep -i "device plugin\|neuron"

Option D: Manually patch node capacity (temporary workaround)

# This is a workaround, fix root cause instead
kubectl proxy &

curl --header "Content-Type: application/json-patch+json" \
  --request PATCH \
  --data '[{
    "op": "add",
    "path": "/status/capacity/aws.amazon.com~1neuroncore",
    "value": "2"
  }]' \
  http://localhost:8001/api/v1/nodes/$NODE_NAME/status

Prevention:

  1. Validate node readiness with custom checks:
# node-validation-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: neuron-node-validator
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: neuron-validator
  template:
    metadata:
      labels:
        name: neuron-validator
    spec:
      nodeSelector:
        aws.amazon.com/neuron: "true"
      containers:
      - name: validator
        image: ubuntu:22.04
        command:
        - /bin/bash
        - -c
        - |
          apt-get update && apt-get install -y jq curl
          while true; do
            if [ ! -e /dev/neuron0 ]; then
              echo "ERROR: Neuron device not found!"
            fi
            sleep 60
          done
        volumeMounts:
        - name: dev
          mountPath: /dev
      volumes:
      - name: dev
        hostPath:
          path: /dev
  1. Add node labels after validation:
# validate-and-label.sh
#!/bin/bash
NODE=$1
NEURON_CORES=$(kubectl get node $NODE -o json | jq -r '.status.capacity["aws.amazon.com/neuroncore"]' || echo "0")

if [ "$NEURON_CORES" -gt "0" ]; then
  kubectl label node $NODE aws.amazon.com/neuron-validated=true
  echo "Node $NODE validated: $NEURON_CORES cores"
else
  echo "ERROR: Node $NODE has no NeuronCores!"
  kubectl label node $NODE aws.amazon.com/neuron-validated=false
fi
  1. Use node selectors to avoid unvalidated nodes:
spec:
  nodeSelector:
    aws.amazon.com/neuron-validated: "true"

Issue 4: EFA Not Working for Distributed Training

Symptom:

Distributed training falling back to TCP
EFA device not detected in container
fi_info command fails
NCCL timeout errors in multi-node training
Poor distributed training performance

Diagnosis:

Check EFA configuration:

# Check if node supports EFA
kubectl get nodes -o json | jq '.items[] | {
  name: .metadata.name,
  instanceType: .metadata.labels["node.kubernetes.io/instance-type"]
}'

# Trn1 instances support EFA by default

# Check EFA device plugin
kubectl get daemonset -n kube-system aws-efa-k8s-device-plugin

# Check pod EFA allocation
kubectl describe pod $TRAINING_POD | grep -A 5 "vpc.amazonaws.com/efa"

# Test EFA in pod
kubectl exec $TRAINING_POD -- fi_info -p efa

Solution:

Option A: Install EFA device plugin

# Install EFA device plugin
kubectl apply -f https://raw.githubusercontent.com/aws/eks-charts/master/stable/aws-efa-k8s-device-plugin/deploy-all.yaml

# Verify
kubectl get daemonset -n kube-system aws-efa-k8s-device-plugin

Option B: Enable EFA in EC2NodeClass

# karpenter/ec2nodeclass-trainium.yaml
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: trainium-efa-nodeclass
spec:
  amiFamily: AL2
  role: KarpenterNodeRole-my-cluster
  networkInterfaces:
  - deviceIndex: 0
    interfaceType: efa  # Enable EFA
    groups:
    - sg-0123456789abcdef0  # Security group with all-all rule
  subnetSelectorTerms:
  - tags:
      karpenter.sh/discovery: my-cluster
  userData: |
    #!/bin/bash
    # Install EFA driver (if not in AMI)
    curl -O https://efa-installer.amazonaws.com/aws-efa-installer-latest.tar.gz
    tar -xf aws-efa-installer-latest.tar.gz
    cd aws-efa-installer
    sudo ./efa_installer.sh -y

Option C: Configure pod for EFA

# distributed-training-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: distributed-training
spec:
  template:
    metadata:
      annotations:
        k8s.amazonaws.com/efa: "1"  # Request EFA
    spec:
      hostNetwork: true  # Required for EFA
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: training
        image: your-training-image
        securityContext:
          capabilities:
            add:
            - IPC_LOCK  # Required for EFA
            - SYS_PTRACE
        resources:
          limits:
            vpc.amazonaws.com/efa: 1
            aws.amazon.com/neuroncore: 32
          requests:
            vpc.amazonaws.com/efa: 1
            aws.amazon.com/neuroncore: 32
        volumeMounts:
        - name: efa-device
          mountPath: /dev/infiniband
        - name: shm
          mountPath: /dev/shm
      volumes:
      - name: efa-device
        hostPath:
          path: /dev/infiniband
      - name: shm
        emptyDir:
          medium: Memory
          sizeLimit: 64Gi  # Large shared memory for EFA

Option D: Fix security group for EFA

# EFA requires all-all traffic within security group
SECURITY_GROUP_ID=$(kubectl get ec2nodeclass trainium-efa-nodeclass -o jsonpath='{.spec.securityGroupSelectorTerms[0].id}')

# Add all traffic rule
aws ec2 authorize-security-group-ingress \
  --group-id $SECURITY_GROUP_ID \
  --protocol all \
  --source-group $SECURITY_GROUP_ID

# Verify
aws ec2 describe-security-groups \
  --group-ids $SECURITY_GROUP_ID \
  --query 'SecurityGroups[0].IpPermissions'

Prevention:

  1. Test EFA before distributed training:
# test_efa.py
import subprocess
import sys

# Check EFA device
result = subprocess.run(['fi_info', '-p', 'efa'], capture_output=True)
if result.returncode != 0:
    print("ERROR: EFA not available")
    sys.exit(1)

print("EFA device found:")
print(result.stdout.decode())

# Test NCCL with EFA
import torch
import torch.distributed as dist

dist.init_process_group(backend='xla')
print(f"Distributed initialized with EFA: rank {dist.get_rank()}")
  1. Enable EFA from the start for distributed workloads
  2. Use placement groups for co-located nodes:
# ec2nodeclass-with-placement-group.yaml
spec:
  launchTemplate:
    placement:
      groupName: distributed-training-pg

Issue 5: Neuron Monitor Metrics Not Appearing in Prometheus

Symptom:

Neuron metrics missing from Prometheus
Grafana dashboard showing "No Data"
neuron_execution_latency metrics not found
Neuron Monitor pod running but no metrics
ServiceMonitor not scraping

Diagnosis:

Check Neuron Monitor status:

# Check Neuron Monitor DaemonSet
kubectl get daemonset -n kube-system neuron-monitor

# Check Neuron Monitor pods
kubectl get pods -n kube-system -l app=neuron-monitor -o wide

# Check Neuron Monitor logs
kubectl logs -n kube-system -l app=neuron-monitor --tail=50

# Check metrics endpoint
MONITOR_POD=$(kubectl get pods -n kube-system -l app=neuron-monitor -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n kube-system $MONITOR_POD -- curl http://localhost:8000/metrics

# Check ServiceMonitor
kubectl get servicemonitor -n kube-system neuron-monitor

# Check Prometheus targets
kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090 &
curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.labels.job == "neuron-monitor")'

Solution:

Option A: Install Neuron Monitor DaemonSet

# monitoring/neuron-monitor-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: neuron-monitor
  namespace: kube-system
  labels:
    app: neuron-monitor
spec:
  selector:
    matchLabels:
      app: neuron-monitor
  template:
    metadata:
      labels:
        app: neuron-monitor
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
        prometheus.io/path: "/metrics"
    spec:
      nodeSelector:
        aws.amazon.com/neuron-device-count: "2"  # Only on Neuron nodes
      containers:
      - name: neuron-monitor
        image: public.ecr.aws/neuron/neuron-monitor:latest
        ports:
        - containerPort: 8000
          name: metrics
        securityContext:
          privileged: true
        volumeMounts:
        - name: neuron-devices
          mountPath: /dev
      volumes:
      - name: neuron-devices
        hostPath:
          path: /dev
---
apiVersion: v1
kind: Service
metadata:
  name: neuron-monitor
  namespace: kube-system
  labels:
    app: neuron-monitor
spec:
  ports:
  - port: 8000
    name: metrics
  selector:
    app: neuron-monitor
  type: ClusterIP
kubectl apply -f monitoring/neuron-monitor-daemonset.yaml

Option B: Create ServiceMonitor for Prometheus Operator

# monitoring/neuron-monitor-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: neuron-monitor
  namespace: kube-system
  labels:
    app: neuron-monitor
spec:
  selector:
    matchLabels:
      app: neuron-monitor
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics
kubectl apply -f monitoring/neuron-monitor-servicemonitor.yaml

# Verify ServiceMonitor is created
kubectl get servicemonitor -n kube-system neuron-monitor

Option C: Add Prometheus scrape configuration (if not using Operator)

# prometheus-additional-scrape-configs.yaml
- job_name: 'neuron-monitor'
  kubernetes_sd_configs:
  - role: pod
    namespaces:
      names:
      - kube-system
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_app]
    action: keep
    regex: neuron-monitor
  - source_labels: [__meta_kubernetes_pod_ip]
    action: replace
    target_label: __address__
    replacement: $1:8000
  - source_labels: [__meta_kubernetes_pod_node_name]
    action: replace
    target_label: node
# Update Prometheus configuration
kubectl create secret generic prometheus-additional-scrape-configs \
  --from-file=prometheus-additional-scrape-configs.yaml \
  --dry-run=client -o yaml | kubectl apply -n monitoring -f -

# Reload Prometheus
kubectl rollout restart statefulset/prometheus-prometheus -n monitoring

Option D: Check Prometheus RBAC permissions

# prometheus-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring
kubectl apply -f prometheus-rbac.yaml

Prevention:

  1. Deploy Neuron Monitor as part of cluster setup
  2. Validate metrics after deployment:
# validate-metrics.sh
#!/bin/bash
kubectl wait --for=condition=ready pod -n kube-system -l app=neuron-monitor --timeout=120s

MONITOR_POD=$(kubectl get pods -n kube-system -l app=neuron-monitor -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n kube-system $MONITOR_POD -- curl -s http://localhost:8000/metrics | grep -q "neuron_" || {
  echo "ERROR: Neuron metrics not available"
  exit 1
}

echo "Neuron metrics validated successfully"
  1. Set up alerts for missing metrics:
# prometheus-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: neuron-monitor-alerts
  namespace: monitoring
spec:
  groups:
  - name: neuron-monitor
    rules:
    - alert: NeuronMetricsMissing
      expr: absent(neuron_execution_latency_seconds_count)
      for: 5m
      annotations:
        summary: "Neuron metrics missing from Prometheus"
  1. Import Grafana dashboard for Neuron metrics:
# Import official Neuron dashboard
kubectl apply -f https://raw.githubusercontent.com/aws-neuron/aws-neuron-sdk/main/src/k8s/neuron-monitor-grafana-dashboard.json

Validation Checklist

Use this checklist to validate your AWS Neuron on EKS deployment.

1. Neuron Driver Installed

# On each Neuron node
kubectl debug node/<node-name> -it --image=ubuntu
chroot /host bash

# Check driver
ls /dev/neuron* | wc -l
# Should be > 0

# Check kernel module
lsmod | grep neuron
# Should show: neuron <size> 0

# Check runtime
neuron-ls
# Should show list of NeuronCores

Expected Result: Neuron devices present, kernel module loaded, runtime responding.

2. Device Plugin Running

# Check DaemonSet
kubectl get daemonset -n kube-system neuron-device-plugin-daemonset
# Expected: DESIRED == READY

# Check pods on Neuron nodes
kubectl get pods -n kube-system -l app.kubernetes.io/name=neuron-device-plugin -o wide

# Verify no errors in logs
kubectl logs -n kube-system -l app.kubernetes.io/name=neuron-device-plugin --tail=50 | grep -i error
# Expected: No errors

Expected Result: Device plugin pod on every Neuron node, no error logs.

3. NeuronCores Advertised as Resources

# Check node capacity
kubectl get nodes -o json | jq '.items[] | {
  name: .metadata.name,
  neuronCores: .status.capacity["aws.amazon.com/neuroncore"],
  neuronDevices: .status.capacity["aws.amazon.com/neurondevice"]
}'

# Expected for Inf2.xlarge: "neuronCores": "2", "neuronDevices": "1"
# Expected for Trn1.32xlarge: "neuronCores": "32", "neuronDevices": "16"

Expected Result: All Neuron nodes show correct neuroncore count.

4. Models Loading Correctly

For Inference (Inferentia2):

# Test model loading
kubectl run neuron-test --rm -it \
  --image=public.ecr.aws/neuron/pytorch-inference-neuronx:latest \
  --limits="aws.amazon.com/neuroncore=2" \
  --command -- python3 -c "
import torch
import torch_neuronx
from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    'gpt2',
    torchscript=True
)
tokenizer = AutoTokenizer.from_pretrained('gpt2')

# Trace model
sample_input = tokenizer('Hello world', return_tensors='pt')
traced = torch_neuronx.trace(model, sample_input['input_ids'])

print('Model loaded and traced successfully!')
"

Expected Result: Model loads without errors, tracing completes.

For Training (Trainium):

# Test training setup
kubectl run training-test --rm -it \
  --image=public.ecr.aws/neuron/pytorch-training-neuronx:latest \
  --limits="aws.amazon.com/neuroncore=2" \
  --command -- python3 -c "
import torch
import torch_neuronx
import torch.nn as nn

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(10, 10)
    
    def forward(self, x):
        return self.linear(x)

model = SimpleModel()
device = 'xla'
model.to(device)

# Test forward pass
x = torch.randn(1, 10).to(device)
y = model(x)

print('Training model initialized successfully!')
print(f'Device: {device}')
"

Expected Result: Model initializes on XLA device, forward pass completes.

5. Inference/Training Working

Inference Test (Full Pipeline):

# Deploy vLLM inference server
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: vllm-test
spec:
  containers:
  - name: vllm
    image: vllm/vllm-neuron:latest
    command:
    - python3
    - -m
    - vllm.entrypoints.openai.api_server
    - --model
    - gpt2
    - --device
    - neuron
    - --tensor-parallel-size
    - "2"
    ports:
    - containerPort: 8000
    resources:
      limits:
        aws.amazon.com/neuroncore: 2
        memory: 16Gi
EOF

# Wait for pod ready
kubectl wait --for=condition=ready pod/vllm-test --timeout=300s

# Test inference
kubectl exec vllm-test -- curl -s http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt2",
    "prompt": "Hello, my name is",
    "max_tokens": 50
  }' | jq '.choices[0].text'

# Expected: Returns generated text

# Cleanup
kubectl delete pod vllm-test

Training Test (Simple Job):

# Run simple training job
kubectl apply -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: training-test
spec:
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: training
        image: public.ecr.aws/neuron/pytorch-training-neuronx:latest
        command:
        - python3
        - -c
        - |
          import torch
          import torch_neuronx
          import torch.nn as nn
          
          class SimpleModel(nn.Module):
              def __init__(self):
                  super().__init__()
                  self.linear = nn.Linear(100, 10)
              
              def forward(self, x):
                  return self.linear(x)
          
          model = SimpleModel().to('xla')
          optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
          
          for step in range(10):
              x = torch.randn(32, 100).to('xla')
              y = torch.randint(0, 10, (32,)).to('xla')
              
              output = model(x)
              loss = nn.functional.cross_entropy(output, y)
              
              optimizer.zero_grad()
              loss.backward()
              optimizer.step()
              
              print(f'Step {step}, Loss: {loss.item()}')
          
          print('Training completed successfully!')
        resources:
          limits:
            aws.amazon.com/neuroncore: 2
            memory: 16Gi
EOF

# Watch job
kubectl wait --for=condition=complete job/training-test --timeout=600s
kubectl logs job/training-test

# Expected: Training completes with loss decreasing

# Cleanup
kubectl delete job training-test

Expected Result: Both inference and training complete without errors.

Complete Validation Script

#!/bin/bash
# validate-neuron-deployment.sh

set -e

echo "=== AWS Neuron on EKS Validation ==="

# 1. Check Neuron nodes
echo -e "\n[1/5] Checking Neuron nodes..."
NEURON_NODES=$(kubectl get nodes -l node.kubernetes.io/instance-type=inf2.xlarge -o name | wc -l)
if [ "$NEURON_NODES" -eq 0 ]; then
  echo "ERROR: No Inf2 nodes found"
  exit 1
fi
echo "✓ Found $NEURON_NODES Neuron nodes"

# 2. Check Device Plugin
echo -e "\n[2/5] Checking Device Plugin..."
kubectl wait --for=condition=ready pod \
  -n kube-system \
  -l app.kubernetes.io/name=neuron-device-plugin \
  --timeout=60s > /dev/null
echo "✓ Device Plugin running"

# 3. Check NeuronCore resources
echo -e "\n[3/5] Checking NeuronCore resources..."
CORES=$(kubectl get nodes -o json | jq -r '.items[0].status.capacity["aws.amazon.com/neuroncore"]')
if [ "$CORES" == "null" ] || [ "$CORES" == "" ]; then
  echo "ERROR: NeuronCores not registered"
  exit 1
fi
echo "✓ NeuronCores registered: $CORES per node"

# 4. Test inference
echo -e "\n[4/5] Testing inference..."
kubectl run neuron-inference-test --rm -i \
  --image=public.ecr.aws/neuron/pytorch-inference-neuronx:latest \
  --limits="aws.amazon.com/neuroncore=2" \
  --restart=Never \
  --command -- python3 -c "
import torch
import torch_neuronx
print('Inference test passed')
" 2>/dev/null
echo "✓ Inference test passed"

# 5. Test training
echo -e "\n[5/5] Testing training..."
kubectl run neuron-training-test --rm -i \
  --image=public.ecr.aws/neuron/pytorch-training-neuronx:latest \
  --limits="aws.amazon.com/neuroncore=2" \
  --restart=Never \
  --command -- python3 -c "
import torch
import torch_neuronx
device = 'xla'
x = torch.ones(1).to(device)
print('Training test passed')
" 2>/dev/null
echo "✓ Training test passed"

echo -e "\n=== All validation checks passed! ==="

Usage:

chmod +x validate-neuron-deployment.sh
./validate-neuron-deployment.sh

Quick Reference

Common Commands

Check Neuron Resources:

# Node capacity
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, neuronCores: .status.capacity["aws.amazon.com/neuroncore"]}'

# Pod allocation
kubectl describe pod <pod-name> | grep -A 5 "Limits:"

# Available cores on node
kubectl describe node <node-name> | grep -A 5 "aws.amazon.com/neuroncore"

Monitor NeuronCores:

# In pod
kubectl exec <pod-name> -- neuron-top

# From node
kubectl debug node/<node-name> -it --image=ubuntu
chroot /host bash
neuron-top

Check Device Plugin:

# Status
kubectl get daemonset -n kube-system neuron-device-plugin-daemonset

# Logs
kubectl logs -n kube-system -l app.kubernetes.io/name=neuron-device-plugin --tail=100

Restart Components:

# Restart device plugin
kubectl rollout restart daemonset -n kube-system neuron-device-plugin-daemonset

# Restart Neuron Monitor
kubectl rollout restart daemonset -n kube-system neuron-monitor

# Restart inference deployment
kubectl rollout restart deployment vllm-llama3

Instance Type to NeuronCore Mapping

Instance Type NeuronCores Memory Use Case
inf2.xlarge 2 16 GB Small models (7-8B)
inf2.8xlarge 2 32 GB Small models (7-8B)
inf2.24xlarge 6 192 GB Medium models (13B)
inf2.48xlarge 12 384 GB Large models (70B)
trn1.2xlarge 2 32 GB Small training
trn1.32xlarge 32 512 GB Large-scale training

Model to NeuronCore Requirements

Model NeuronCores Memory Instance Type
Llama 3 8B 2 16 GB inf2.xlarge
Llama 3 70B 24 384 GB 2× inf2.48xlarge
Mistral 7B 2 16 GB inf2.xlarge
GPT-J 6B 2 16 GB inf2.xlarge
Llama 2 13B 4 32 GB inf2.24xlarge

Useful Log Patterns

Compilation Logs:

kubectl logs <pod-name> | grep "Compiling\|compilation"

Memory Errors:

kubectl logs <pod-name> | grep -i "oom\|memory\|allocation"

EFA Issues:

kubectl logs <pod-name> | grep -i "efa\|fabric\|rdma"

Device Errors:

kubectl logs <pod-name> | grep -i "neuron\|device\|core"

Support Resources

AWS Documentation:

GitHub:

Community:

For additional support: Open an issue at https://github.com/aws-neuron/aws-neuron-sdk/issues