03-vllm-deployment.md

vLLM-Neuron Deployment Guide: High-Performance LLM Serving on Inferentia2

Comprehensive guide to deploying vLLM with AWS Neuron on Amazon EKS for production LLM inference.

---

Table of Contents

1. Introduction to vLLM-Neuron
2. PagedAttention Architecture
3. vLLM-Neuron Setup
4. Deployment on EKS
5. Multi-Model Serving
6. Performance Optimization
7. Token Streaming
8. Autoscaling Patterns
9. Ray Serve Integration
10. Monitoring and Observability
11. Troubleshooting

---

Introduction to vLLM-Neuron

What is vLLM?

vLLM (Very Large Language Model serving) is an open-source library for fast LLM inference and serving, developed by UC Berkeley. Key features:

• PagedAttention: Memory-efficient attention mechanism
• Continuous batching: Dynamic request batching
• OpenAI-compatible API: Drop-in replacement for OpenAI API
• High throughput: 3-4× faster than standard HuggingFace inference

vLLM-Neuron: AWS Neuron Integration

vLLM-Neuron is the AWS Neuron backend for vLLM, enabling high-performance LLM serving on Inferentia2:

Feature	Standard vLLM (GPU)	vLLM-Neuron (Inferentia2)
Backend	CUDA (NVIDIA)	Neuron SDK (AWS)
PagedAttention	✅ Supported	✅ Supported (optimized for HBM)
Continuous batching	✅ Supported	✅ Supported
Tensor parallelism	✅ NCCL	✅ NeuronLink
FP8 quantization	✅ CUDA kernels	✅ Neuron kernels
Cost (Llama 3 8B)	$1.01/hr (A10G)	$0.76/hr (inf2.xlarge)

Why vLLM-Neuron?

Performance Comparison (Llama 3 8B on inf2.xlarge):

Serving Method	Throughput	Latency P99	Concurrent Requests	Cost/1M tokens
HuggingFace Transformers	850 tokens/s	95ms	16	$0.89
Transformers-NeuronX	1,100 tokens/s	78ms	32	$0.69
vLLM-Neuron	3,200 tokens/s	52ms	128	$0.24

Key Benefits:

1. 3× throughput improvement via PagedAttention and continuous batching
2. 65% cost reduction compared to GPU inference
3. OpenAI API compatibility (no code changes for clients)
4. Automatic batching (no manual batch management)

---

PagedAttention Architecture

Traditional Attention Memory Layout

Traditional LLM serving allocates fixed-size KV cache per request:

┌─────────────────────────────────────────────────────────────┐
│           Traditional KV Cache (Static Allocation)           │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  Request 1: Allocated 2048 tokens                            │
│  [████████████████████████░░░░░░░░]                          │
│   Used: 1536 tokens (75%)                                    │
│   Wasted: 512 tokens (25%)                                   │
│                                                               │
│  Request 2: Allocated 2048 tokens                            │
│  [██████░░░░░░░░░░░░░░░░░░░░░░░░░░]                          │
│   Used: 512 tokens (25%)                                     │
│   Wasted: 1536 tokens (75%)                                  │
│                                                               │
│  Request 3: Allocated 2048 tokens                            │
│  [████████████████████████████████]                          │
│   Used: 2048 tokens (100%)                                   │
│   Wasted: 0 tokens (0%)                                      │
│                                                               │
│  Total: 6144 tokens allocated, 4096 used                     │
│  Memory waste: 33%                                            │
│  Max concurrent requests: 3                                   │
└─────────────────────────────────────────────────────────────┘

PagedAttention: Virtual Memory for KV Cache

PagedAttention treats KV cache like OS virtual memory:

┌─────────────────────────────────────────────────────────────┐
│              PagedAttention Memory Layout                    │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  Physical Memory: Pool of 64-token pages                     │
│  ┌────┐┌────┐┌────┐┌────┐┌────┐┌────┐┌────┐┌────┐          │
│  │ P0 ││ P1 ││ P2 ││ P3 ││ P4 ││ P5 ││ P6 ││ P7 │ ...      │
│  └────┘└────┘└────┘└────┘└────┘└────┘└────┘└────┘          │
│                                                               │
│  Request 1 (1536 tokens):                                    │
│  Logical: [Token 0-1535]                                     │
│  Physical: P0, P1, P2, ..., P23  (24 pages × 64 = 1536)    │
│                                                               │
│  Request 2 (512 tokens):                                     │
│  Logical: [Token 0-511]                                      │
│  Physical: P24, P25, P26, P27, P28, P29, P30, P31          │
│            (8 pages × 64 = 512)                              │
│                                                               │
│  Request 3 (2048 tokens):                                    │
│  Logical: [Token 0-2047]                                     │
│  Physical: P32, P33, ..., P63  (32 pages × 64 = 2048)      │
│                                                               │
│  Total: 64 pages = 4096 tokens (100% utilization)            │
│  Memory waste: 0%                                             │
│  Max concurrent requests: Depends on actual usage, not max   │
└─────────────────────────────────────────────────────────────┘

Key Advantages:

1. No fragmentation: Pages allocated on-demand
2. Memory sharing: Prefix sharing for common prompts
3. Dynamic allocation: Pages freed immediately when request completes
4. Higher concurrency: 2-3× more concurrent requests vs static allocation

Continuous Batching

Traditional batching waits for batch to fill before processing:

Traditional Static Batching:
─────────────────────────────────────────────────────────────
Time  0ms:  [Req1 arrives]  Wait...
Time 10ms:  [Req2 arrives]  Wait...
Time 20ms:  [Req3 arrives]  Wait...
Time 30ms:  [Req4 arrives]  Batch full! Process [Req1,2,3,4]
Time 130ms: All complete, start new batch
─────────────────────────────────────────────────────────────
Average latency: 65ms (wait) + 25ms (process) = 90ms
Throughput: 4 requests / 130ms = 30.8 req/s

Continuous batching processes requests as they arrive:

vLLM Continuous Batching:
─────────────────────────────────────────────────────────────
Time  0ms:  [Req1 arrives]  Immediately start processing Req1
Time 10ms:  [Req2 arrives]  Add to batch: [Req1, Req2]
Time 20ms:  [Req3 arrives]  Add to batch: [Req1, Req2, Req3]
Time 25ms:  [Req1 done]     Remove: [Req2, Req3]
Time 30ms:  [Req4 arrives]  Add to batch: [Req2, Req3, Req4]
Time 35ms:  [Req2 done]     Remove: [Req3, Req4]
─────────────────────────────────────────────────────────────
Average latency: 0ms (wait) + 25ms (process) = 25ms
Throughput: 4 requests / 35ms = 114.3 req/s

Performance Gain: 3.7× higher throughput, 3.6× lower latency

---

vLLM-Neuron Setup

Installation

# On inf2 instance (Ubuntu 20.04)

# 1. Install Neuron driver and runtime
sudo apt-get update
sudo apt-get install aws-neuronx-dkms aws-neuronx-runtime-lib -y

# 2. Install vLLM-Neuron
pip install vllm-neuron==0.6.3
pip install torch-neuronx==2.1.2.1.0 --extra-index-url=https://pip.repos.neuron.amazonaws.com
pip install transformers-neuronx==0.9.474

# 3. Verify installation
python -c "import vllm; print(vllm.__version__)"
python -c "from vllm import LLM; print('vLLM-Neuron installed successfully')"

Quick Start: Local Inference

# simple_inference.py
from vllm import LLM, SamplingParams

# Initialize model (compiles on first run)
llm = LLM(
    model="meta-llama/Meta-Llama-3-8B",
    tensor_parallel_size=2,  # Use 2 NeuronCores
    max_model_len=4096,
    dtype="bfloat16",
)

# Define sampling parameters
sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=100
)

# Single prompt inference
prompts = ["Explain AWS Neuron in simple terms."]
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(f"Prompt: {output.prompt}")
    print(f"Output: {output.outputs[0].text}")
    print(f"Tokens: {len(output.outputs[0].token_ids)}")

# Batch inference (automatic batching)
prompts = [
    "What is machine learning?",
    "Explain quantum computing.",
    "What are transformers in NLP?",
    "How does PagedAttention work?"
]

outputs = llm.generate(prompts, sampling_params)
for i, output in enumerate(outputs):
    print(f"\nPrompt {i+1}: {output.prompt}")
    print(f"Output: {output.outputs[0].text}")

First run (compilation):

python simple_inference.py

# Output:
Compiling model layers... (5-10 minutes)
INFO: Model compiled successfully
INFO: Loaded model in 8.2s

Prompt: Explain AWS Neuron in simple terms.
Output: AWS Neuron is a software development kit that enables machine learning frameworks to run efficiently on AWS Trainium and Inferentia accelerators...

Subsequent runs (instant):

python simple_inference.py

# Output:
INFO: Using cached compiled model
INFO: Loaded model in 2.1s

OpenAI-Compatible API Server

# Start vLLM API server
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-8B \
    --tensor-parallel-size 2 \
    --max-model-len 4096 \
    --dtype bfloat16 \
    --port 8000 \
    --host 0.0.0.0

# Server starts at http://0.0.0.0:8000

Client usage (OpenAI Python SDK):

from openai import OpenAI

# Point to vLLM server
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"
)

# Chat completion (OpenAI API format)
response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B",
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant."},
        {"role": "user", "content": "What is AWS Inferentia2?"}
    ],
    temperature=0.7,
    max_tokens=150
)

print(response.choices[0].message.content)

# Streaming response
stream = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B",
    messages=[{"role": "user", "content": "Count from 1 to 10."}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

cURL usage:

# Chat completion
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3-8B",
    "messages": [
      {"role": "user", "content": "What is vLLM?"}
    ],
    "temperature": 0.7,
    "max_tokens": 100
  }'

# Text completion
curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3-8B",
    "prompt": "Once upon a time",
    "max_tokens": 50
  }'

---

Deployment on EKS

Basic Deployment (Single Model)

# vllm-llama3-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3
  namespace: ml-inference
spec:
  replicas: 2  # High availability
  selector:
    matchLabels:
      app: vllm-llama3
  template:
    metadata:
      labels:
        app: vllm-llama3
        model: llama3-8b
    spec:
      # Node selection
      tolerations:
        - key: aws.amazon.com/neuron
          value: inferentia2
          effect: NoSchedule
      
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node.kubernetes.io/instance-type
                    operator: In
                    values:
                      - inf2.xlarge
                      - inf2.8xlarge
        
        # Spread across availability zones
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: vllm-llama3
                topologyKey: topology.kubernetes.io/zone
      
      containers:
        - name: vllm-server
          image: public.ecr.aws/neuron/vllm:0.6.3-neuronx-py310-sdk2.18.1-ubuntu20.04
          
          command:
            - python3
            - -m
            - vllm.entrypoints.openai.api_server
          
          args:
            - --model=meta-llama/Meta-Llama-3-8B
            - --tensor-parallel-size=2
            - --max-model-len=4096
            - --max-num-seqs=16
            - --dtype=bfloat16
            - --disable-log-requests
            - --port=8000
            - --host=0.0.0.0
          
          ports:
            - containerPort: 8000
              name: http
              protocol: TCP
          
          resources:
            requests:
              aws.amazon.com/neuroncore: "2"
              memory: 16Gi
              cpu: "4"
            limits:
              aws.amazon.com/neuroncore: "2"
              memory: 24Gi
              cpu: "8"
          
          env:
            # Neuron runtime
            - name: NEURON_RT_NUM_CORES
              value: "2"
            - name: NEURON_RT_VISIBLE_CORES
              value: "0,1"
            
            # vLLM configuration
            - name: VLLM_NEURON_PARALLEL_SIZE
              value: "2"
            - name: VLLM_NEURON_MAX_MODEL_LEN
              value: "4096"
            
            # Model caching
            - name: TRANSFORMERS_CACHE
              value: /tmp/transformers_cache
            - name: HF_HOME
              value: /tmp/huggingface
            
            # Performance tuning
            - name: NEURON_CC_FLAGS
              value: "--model-type=transformer --auto-cast=bf16"
          
          # Health checks
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 300  # Allow time for model compilation
            periodSeconds: 30
            timeoutSeconds: 5
            failureThreshold: 3
          
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 240
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 3
          
          volumeMounts:
            - name: tmp
              mountPath: /tmp
            - name: shm
              mountPath: /dev/shm
      
      volumes:
        - name: tmp
          emptyDir:
            sizeLimit: 50Gi
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: 16Gi

---
apiVersion: v1
kind: Service
metadata:
  name: vllm-llama3
  namespace: ml-inference
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
spec:
  type: LoadBalancer
  selector:
    app: vllm-llama3
  ports:
    - port: 80
      targetPort: 8000
      protocol: TCP
      name: http
  sessionAffinity: None

---
apiVersion: v1
kind: Service
metadata:
  name: vllm-llama3-headless
  namespace: ml-inference
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    app: vllm-llama3
  ports:
    - port: 8000
      targetPort: 8000
      name: http

Deploy:

# Create namespace
kubectl create namespace ml-inference

# Deploy vLLM
kubectl apply -f vllm-llama3-deployment.yaml

# Wait for pods to be ready (5-10 minutes for first compilation)
kubectl wait --for=condition=ready pod -l app=vllm-llama3 -n ml-inference --timeout=600s

# Get service endpoint
kubectl get svc vllm-llama3 -n ml-inference

# Test inference
LB_URL=$(kubectl get svc vllm-llama3 -n ml-inference -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

curl http://$LB_URL/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3-8B",
    "prompt": "AWS Inferentia2 is",
    "max_tokens": 50
  }'

Large Model Deployment (Llama 2 70B)

For 70B models, use tensor parallelism across more NeuronCores:

# vllm-llama70b-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama70b
  namespace: ml-inference
spec:
  replicas: 1  # 70B is expensive, start with 1 replica
  template:
    spec:
      tolerations:
        - key: aws.amazon.com/neuron
          value: inferentia2
          effect: NoSchedule
      
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node.kubernetes.io/instance-type
                    operator: In
                    values:
                      - inf2.48xlarge  # Need 12 NeuronCores
      
      containers:
        - name: vllm-server
          image: public.ecr.aws/neuron/vllm:0.6.3-neuronx-py310-sdk2.18.1-ubuntu20.04
          
          command:
            - python3
            - -m
            - vllm.entrypoints.openai.api_server
          
          args:
            - --model=meta-llama/Llama-2-70b-hf
            - --tensor-parallel-size=12  # 12-way TP
            - --max-model-len=2048       # Reduce for memory
            - --max-num-seqs=8           # Lower concurrency
            - --dtype=bfloat16
            - --port=8000
          
          resources:
            requests:
              aws.amazon.com/neuroncore: "12"  # All 12 cores
              memory: 150Gi
              cpu: "48"
            limits:
              aws.amazon.com/neuroncore: "12"
              memory: 192Gi
              cpu: "64"
          
          env:
            - name: NEURON_RT_NUM_CORES
              value: "12"
            - name: VLLM_NEURON_PARALLEL_SIZE
              value: "12"
            - name: VLLM_NEURON_MAX_MODEL_LEN
              value: "2048"
          
          volumeMounts:
            - name: shm
              mountPath: /dev/shm
      
      volumes:
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: 64Gi

Performance expectations:

• Throughput: 520 tokens/s
• Latency P50: 128ms, P99: 165ms
• Cost: $12.98/hour (vs $256/hour on 8× A100)

---

Multi-Model Serving

Strategy 1: Multiple Deployments (Isolated)

# One deployment per model
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-mistral
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-codellama

Pros:

• Full isolation (one model crash doesn't affect others)
• Independent scaling per model
• Simple deployment

Cons:

• Higher cost (each model needs dedicated instance)
• Resource underutilization if traffic is low

Strategy 2: Single Pod, Multiple Containers

# multi-model-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: multi-model-server
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node.kubernetes.io/instance-type
                operator: In
                values:
                  - inf2.48xlarge
  
  containers:
    # Model 1: Llama 3 8B (cores 0-1)
    - name: llama3
      image: public.ecr.aws/neuron/vllm:latest
      command:
        - python3
        - -m
        - vllm.entrypoints.openai.api_server
        - --model=meta-llama/Meta-Llama-3-8B
        - --tensor-parallel-size=2
        - --port=8001
      env:
        - name: NEURON_RT_VISIBLE_CORES
          value: "0,1"
      resources:
        limits:
          aws.amazon.com/neuroncore: "2"
      ports:
        - containerPort: 8001
    
    # Model 2: Mistral 7B (cores 2-3)
    - name: mistral
      image: public.ecr.aws/neuron/vllm:latest
      command:
        - python3
        - -m
        - vllm.entrypoints.openai.api_server
        - --model=mistralai/Mistral-7B-v0.1
        - --tensor-parallel-size=2
        - --port=8002
      env:
        - name: NEURON_RT_VISIBLE_CORES
          value: "2,3"
      resources:
        limits:
          aws.amazon.com/neuroncore: "2"
      ports:
        - containerPort: 8002
    
    # Model 3: CodeLlama 7B (cores 4-5)
    - name: codellama
      image: public.ecr.aws/neuron/vllm:latest
      command:
        - python3
        - -m
        - vllm.entrypoints.openai.api_server
        - --model=codellama/CodeLlama-7b-hf
        - --tensor-parallel-size=2
        - --port=8003
      env:
        - name: NEURON_RT_VISIBLE_CORES
          value: "4,5"
      resources:
        limits:
          aws.amazon.com/neuroncore: "2"
      ports:
        - containerPort: 8003
    
    # Nginx router (on CPU)
    - name: router
      image: nginx:latest
      ports:
        - containerPort: 80
      volumeMounts:
        - name: nginx-config
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
  
  volumes:
    - name: nginx-config
      configMap:
        name: multi-model-nginx-config

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: multi-model-nginx-config
data:
  nginx.conf: |
    events {
      worker_connections 1024;
    }
    
    http {
      # Model routing based on URL path
      upstream llama3 {
        server localhost:8001;
      }
      
      upstream mistral {
        server localhost:8002;
      }
      
      upstream codellama {
        server localhost:8003;
      }
      
      server {
        listen 80;
        
        # Route /llama3/* to Llama 3 8B
        location /llama3/ {
          rewrite ^/llama3/(.*) /$1 break;
          proxy_pass http://llama3;
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
        }
        
        # Route /mistral/* to Mistral 7B
        location /mistral/ {
          rewrite ^/mistral/(.*) /$1 break;
          proxy_pass http://mistral;
          proxy_set_header Host $host;
        }
        
        # Route /code/* to CodeLlama 7B
        location /code/ {
          rewrite ^/code/(.*) /$1 break;
          proxy_pass http://codellama;
          proxy_set_header Host $host;
        }
        
        # Health check endpoint
        location /health {
          access_log off;
          return 200 "healthy\n";
          add_header Content-Type text/plain;
        }
      }
    }

Usage:

# Deploy
kubectl apply -f multi-model-pod.yaml

# Test each model
kubectl port-forward pod/multi-model-server 8080:80

# Llama 3
curl http://localhost:8080/llama3/v1/completions -d '{"model": "meta-llama/Meta-Llama-3-8B", "prompt": "Hello"}'

# Mistral
curl http://localhost:8080/mistral/v1/completions -d '{"model": "mistralai/Mistral-7B-v0.1", "prompt": "Hello"}'

# CodeLlama
curl http://localhost:8080/code/v1/completions -d '{"model": "codellama/CodeLlama-7b-hf", "prompt": "def fibonacci"}'

---

Performance Optimization

1. Batch Size Tuning

# benchmark_batch_size.py
import time
import requests
from concurrent.futures import ThreadPoolExecutor, as_completed

def send_request(prompt):
    response = requests.post(
        "http://localhost:8000/v1/completions",
        json={
            "model": "meta-llama/Meta-Llama-3-8B",
            "prompt": prompt,
            "max_tokens": 50
        }
    )
    return response.json()

def benchmark(num_concurrent):
    prompts = [f"Prompt {i}" for i in range(num_concurrent)]
    
    start = time.time()
    with ThreadPoolExecutor(max_workers=num_concurrent) as executor:
        futures = [executor.submit(send_request, p) for p in prompts]
        results = [f.result() for f in as_completed(futures)]
    end = time.time()
    
    throughput = (num_concurrent * 50) / (end - start)
    latency = (end - start) / num_concurrent
    
    return throughput, latency

# Test different concurrency levels
for concurrency in [1, 5, 10, 20, 50, 100]:
    throughput, latency = benchmark(concurrency)
    print(f"Concurrency {concurrency:3d}: {throughput:6.0f} tok/s, {latency*1000:6.1f}ms latency")

# Expected output:
# Concurrency   1:   1850 tok/s,   27.0ms latency
# Concurrency   5:   3100 tok/s,   80.6ms latency
# Concurrency  10:   3200 tok/s,  156.3ms latency
# Concurrency  20:   3200 tok/s,  312.5ms latency
# Concurrency  50:   3200 tok/s,  781.3ms latency (saturated)
# Concurrency 100:   3200 tok/s, 1562.5ms latency (overloaded)

Optimal batch size: Concurrency = 10-20 (maximize throughput while P99 < 200ms)

2. Sequence Length Optimization

# Configure max_model_len based on use case

# Short generation (chatbots, Q&A)
--max-model-len=2048  # Lower memory, higher concurrency

# Medium generation (summarization)
--max-model-len=4096  # Balanced

# Long generation (document generation)
--max-model-len=8192  # Higher memory, lower concurrency

Memory vs Concurrency Trade-off (Llama 3 8B on inf2.xlarge):

max_model_len	KV Cache per Request	Max Concurrent	Throughput
1024	540 MB	40	3,500 tok/s
2048	1.07 GB	20	3,200 tok/s
4096	2.15 GB	10	2,800 tok/s
8192	4.29 GB	5	2,000 tok/s

3. Prefill Chunking

For long prompts, split prefill phase into chunks:

# Enable prefill chunking (vLLM 0.6+)
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-8B \
    --tensor-parallel-size 2 \
    --enable-chunked-prefill \
    --max-num-batched-tokens 8192  # Chunk size

# Benefits:
# - Lower latency for long prompts (10K+ tokens)
# - Better batching (prefill + decode in same batch)
# - 15-20% throughput improvement

---

Token Streaming

Server-Side Streaming

vLLM natively supports Server-Sent Events (SSE) for token streaming:

# client_streaming.py
import requests
import json

def stream_completion(prompt):
    response = requests.post(
        "http://localhost:8000/v1/completions",
        json={
            "model": "meta-llama/Meta-Llama-3-8B",
            "prompt": prompt,
            "max_tokens": 100,
            "stream": True  # Enable streaming
        },
        stream=True
    )
    
    for line in response.iter_lines():
        if line:
            line = line.decode('utf-8')
            if line.startswith('data: '):
                data = line[6:]  # Remove 'data: ' prefix
                if data == '[DONE]':
                    break
                
                chunk = json.loads(data)
                token = chunk['choices'][0]['text']
                print(token, end='', flush=True)

# Usage
stream_completion("Once upon a time in a land far away,")

Chat Streaming (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"
)

# Stream chat completion
stream = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B",
    messages=[
        {"role": "user", "content": "Write a short story about AI."}
    ],
    stream=True,
    max_tokens=200
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

WebSocket Streaming (Advanced)

For bidirectional communication (e.g., chatbot with interruptions):

# websocket_server.py (FastAPI + vLLM)
from fastapi import FastAPI, WebSocket
from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams

app = FastAPI()

# Initialize vLLM async engine
engine_args = AsyncEngineArgs(
    model="meta-llama/Meta-Llama-3-8B",
    tensor_parallel_size=2
)
engine = AsyncLLMEngine.from_engine_args(engine_args)

@app.websocket("/ws/chat")
async def websocket_chat(websocket: WebSocket):
    await websocket.accept()
    
    try:
        while True:
            # Receive message from client
            prompt = await websocket.receive_text()
            
            # Generate response with streaming
            sampling_params = SamplingParams(temperature=0.7, max_tokens=100)
            
            async for output in engine.generate_stream(prompt, sampling_params):
                token = output.outputs[0].text
                
                # Send token to client
                await websocket.send_text(token)
                
                # Check for client interruption
                if await websocket.receive_text() == "STOP":
                    break
            
            # Send completion marker
            await websocket.send_text("[DONE]")
    
    except Exception as e:
        print(f"WebSocket error: {e}")
    finally:
        await websocket.close()

# Run: uvicorn websocket_server:app --host 0.0.0.0 --port 8000

Client:

# websocket_client.py
import asyncio
import websockets

async def chat():
    uri = "ws://localhost:8000/ws/chat"
    
    async with websockets.connect(uri) as websocket:
        # Send prompt
        await websocket.send("Explain quantum computing in simple terms.")
        
        # Receive streaming response
        while True:
            token = await websocket.recv()
            
            if token == "[DONE]":
                break
            
            print(token, end="", flush=True)

asyncio.run(chat())

---

Autoscaling Patterns

Horizontal Pod Autoscaler (HPA)

# vllm-hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: vllm-llama3-hpa
  namespace: ml-inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama3
  
  minReplicas: 2
  maxReplicas: 20
  
  metrics:
    # Scale based on custom metric: queue depth
    - type: Pods
      pods:
        metric:
          name: vllm_queue_depth
        target:
          type: AverageValue
          averageValue: "10"
    
    # Also scale on CPU (backup metric)
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # Wait 5min before scaling down
      policies:
        - type: Percent
          value: 50
          periodSeconds: 60
    
    scaleUp:
      stabilizationWindowSeconds: 0  # Scale up immediately
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30
        - type: Pods
          value: 2
          periodSeconds: 30
      selectPolicy: Max

Custom metrics (Prometheus adapter):

# prometheus-adapter-config.yaml
rules:
  - seriesQuery: 'vllm_num_requests_waiting'
    resources:
      overrides:
        namespace: {resource: "namespace"}
        pod: {resource: "pod"}
    name:
      matches: "^(.*)$"
      as: "vllm_queue_depth"
    metricsQuery: 'avg_over_time(vllm_num_requests_waiting[1m])'

Karpenter Scaling (Node-level)

# karpenter-inferentia2-nodepool.yaml
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: inferentia2-inference
spec:
  template:
    spec:
      requirements:
        - key: node.kubernetes.io/instance-type
          operator: In
          values:
            - inf2.xlarge
            - inf2.8xlarge
        
        - key: karpenter.sh/capacity-type
          operator: In
          values:
            - spot        # Use spot instances (60-70% discount)
            - on-demand   # Fallback to on-demand
      
      taints:
        - key: aws.amazon.com/neuron
          value: inferentia2
          effect: NoSchedule
  
  limits:
    cpu: "1000"  # Max 30 nodes (32 vCPU per inf2.xlarge)
  
  disruption:
    consolidationPolicy: WhenUnderutilized
    consolidateAfter: 30s
    
    # Handle spot interruptions gracefully
    budgets:
      - nodes: "10%"
        schedule: "0 9-17 * * MON-FRI"  # Business hours: disrupt max 10%
      - nodes: "50%"
        schedule: "0 0-8,18-23 * * *"   # Off-hours: disrupt up to 50%

Benefits:

• Auto-scale from 2 to 20 pods based on load
• Use spot instances for 60-70% cost savings
• Graceful handling of spot interruptions

---

Ray Serve Integration

For advanced deployment patterns (model composition, A/B testing), use Ray Serve:

Setup

# Install Ray
pip install ray[serve]==2.9.0

# Deploy Ray cluster on EKS (using KubeRay operator)
kubectl create -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/default/operator.yaml

# Create Ray cluster
kubectl apply -f ray-cluster-neuron.yaml

Ray Cluster for Neuron

# ray-cluster-neuron.yaml
apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: ray-neuron-cluster
spec:
  rayVersion: '2.9.0'
  
  headGroupSpec:
    rayStartParams:
      dashboard-host: '0.0.0.0'
    template:
      spec:
        containers:
          - name: ray-head
            image: rayproject/ray:2.9.0-py310
            resources:
              limits:
                cpu: "4"
                memory: "16Gi"
            ports:
              - containerPort: 6379  # Redis
              - containerPort: 8265  # Dashboard
              - containerPort: 10001 # Client
  
  workerGroupSpecs:
    - replicas: 3
      minReplicas: 1
      maxReplicas: 10
      groupName: neuron-workers
      rayStartParams: {}
      template:
        spec:
          tolerations:
            - key: aws.amazon.com/neuron
              value: inferentia2
              effect: NoSchedule
          
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                  - matchExpressions:
                      - key: node.kubernetes.io/instance-type
                        operator: In
                        values:
                          - inf2.xlarge
          
          containers:
            - name: ray-worker
              image: rayproject/ray:2.9.0-py310
              
              resources:
                limits:
                  aws.amazon.com/neuroncore: "2"
                  memory: "16Gi"
                  cpu: "4"
              
              env:
                - name: NEURON_RT_NUM_CORES
                  value: "2"

Multi-Model Deployment with Ray Serve

# ray_serve_deployment.py
from ray import serve
from vllm import LLM, SamplingParams

@serve.deployment(
    name="llama3",
    num_replicas=2,
    ray_actor_options={
        "num_cpus": 4,
        "resources": {"neuroncore": 2}
    }
)
class LlamaModel:
    def __init__(self):
        self.llm = LLM(
            model="meta-llama/Meta-Llama-3-8B",
            tensor_parallel_size=2
        )
        self.sampling_params = SamplingParams(temperature=0.7)
    
    async def __call__(self, request):
        prompt = request.query_params["prompt"]
        outputs = self.llm.generate(prompt, self.sampling_params)
        return {"output": outputs[0].outputs[0].text}

@serve.deployment(
    name="mistral",
    num_replicas=1,
    ray_actor_options={
        "num_cpus": 4,
        "resources": {"neuroncore": 2}
    }
)
class MistralModel:
    def __init__(self):
        self.llm = LLM(
            model="mistralai/Mistral-7B-v0.1",
            tensor_parallel_size=2
        )
        self.sampling_params = SamplingParams(temperature=0.7)
    
    async def __call__(self, request):
        prompt = request.query_params["prompt"]
        outputs = self.llm.generate(prompt, self.sampling_params)
        return {"output": outputs[0].outputs[0].text}

# Router with A/B testing
@serve.deployment(route_prefix="/")
class Router:
    def __init__(self, llama_handle, mistral_handle):
        self.llama = llama_handle
        self.mistral = mistral_handle
    
    async def __call__(self, request):
        model = request.query_params.get("model", "llama3")
        
        if model == "llama3":
            return await self.llama.remote(request)
        elif model == "mistral":
            return await self.mistral.remote(request)
        else:
            return {"error": "Invalid model"}

# Deploy
llama = LlamaModel.bind()
mistral = MistralModel.bind()
router = Router.bind(llama, mistral)

serve.run(router, host="0.0.0.0", port=8000)

Deploy on Ray cluster:

# Connect to Ray cluster
ray attach ray-neuron-cluster

# Submit deployment
ray serve deploy ray_serve_deployment.py

# Test
curl "http://<ray-head-ip>:8000/?model=llama3&prompt=Hello"
curl "http://<ray-head-ip>:8000/?model=mistral&prompt=Hello"

---

Monitoring and Observability

Prometheus Metrics

vLLM exposes Prometheus metrics at /metrics:

# prometheus-scrape-config.yaml
scrape_configs:
  - job_name: 'vllm'
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
            - ml-inference
    
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: vllm.*
        action: keep
      
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
      
      - source_labels: [__address__]
        target_label: __address__
        regex: ([^:]+)(?::\d+)?
        replacement: $1:8000

Key Metrics:

# Request rate
rate(vllm_request_success_total[1m])

# Request latency (P50, P99)
histogram_quantile(0.50, rate(vllm_request_duration_seconds_bucket[1m]))
histogram_quantile(0.99, rate(vllm_request_duration_seconds_bucket[1m]))

# Queue depth (for autoscaling)
vllm_num_requests_waiting

# Throughput (tokens/s)
rate(vllm_total_output_tokens[1m])

# Model utilization
rate(vllm_total_generation_time_seconds[1m]) / rate(vllm_request_success_total[1m])

# KV cache usage
vllm_gpu_kv_cache_usage_perc  # Actually HBM, not GPU

Grafana Dashboard

{
  "dashboard": {
    "title": "vLLM-Neuron Inference",
    "panels": [
      {
        "title": "Requests per Second",
        "targets": [
          {
            "expr": "sum(rate(vllm_request_success_total[1m]))"
          }
        ]
      },
      {
        "title": "Latency Percentiles",
        "targets": [
          {
            "expr": "histogram_quantile(0.50, rate(vllm_request_duration_seconds_bucket[1m]))",
            "legendFormat": "P50"
          },
          {
            "expr": "histogram_quantile(0.95, rate(vllm_request_duration_seconds_bucket[1m]))",
            "legendFormat": "P95"
          },
          {
            "expr": "histogram_quantile(0.99, rate(vllm_request_duration_seconds_bucket[1m]))",
            "legendFormat": "P99"
          }
        ]
      },
      {
        "title": "Throughput (tokens/s)",
        "targets": [
          {
            "expr": "sum(rate(vllm_total_output_tokens[1m]))"
          }
        ]
      },
      {
        "title": "Queue Depth",
        "targets": [
          {
            "expr": "vllm_num_requests_waiting"
          }
        ]
      }
    ]
  }
}

Logging

# Configure structured logging
import logging
import json

class JSONFormatter(logging.Formatter):
    def format(self, record):
        log_data = {
            "timestamp": self.formatTime(record),
            "level": record.levelname,
            "message": record.getMessage(),
            "request_id": getattr(record, 'request_id', None),
            "model": getattr(record, 'model', None),
            "latency_ms": getattr(record, 'latency_ms', None),
            "tokens": getattr(record, 'tokens', None),
        }
        return json.dumps(log_data)

# Apply to vLLM logger
logger = logging.getLogger("vllm")
handler = logging.StreamHandler()
handler.setFormatter(JSONFormatter())
logger.addHandler(handler)

---

Troubleshooting

Issue 1: Slow First Request

Symptom:

First request: 5000ms
Subsequent requests: 50ms

Cause: Model not compiled or not warmed up

Solution:

# Add warmup to readiness probe
def readiness_check():
    if not is_warmed_up:
        # Send dummy request to trigger compilation
        llm.generate("warmup prompt")
        is_warmed_up = True
    return {"status": "ready"}

Issue 2: High Memory Usage

Symptom:

HBM usage: 31.5 GB / 32 GB (98%)
OOM errors when adding new requests

Cause: KV cache too large

Solution:

# Reduce max_num_seqs or max_model_len
--max-num-seqs=8     # Was 16
--max-model-len=2048  # Was 4096

Issue 3: Low Throughput

Diagnosis:

# Check NeuronCore utilization
neuron-top

# Low utilization (<60%)?
# → Increase batch size or check for CPU bottleneck

Solution:

# Increase max_num_seqs
--max-num-seqs=32  # Was 16

# Enable prefill chunking
--enable-chunked-prefill
--max-num-batched-tokens=8192

---

Additional Resources

• vLLM GitHub Repository
• vLLM-Neuron Documentation
• Deployment Examples
• Benchmarking Guide

---

Next Steps:

• Explore Benchmarking Guide to measure your deployment
• Review Inferentia2 Architecture for hardware optimizations
• Check Troubleshooting Guide for common issues