Skip to content

performance-charts.md

Latest commit

 

History

History
1000 lines (741 loc) · 29.4 KB

performance-charts.md

File metadata and controls

1000 lines (741 loc) · 29.4 KB
Raw

Performance Visualization Charts

Comprehensive Mermaid charts for AWS Neuron (Trainium & Inferentia2) performance benchmarks covering training, inference, cost analysis, and optimization impact.

Table of Contents

  1. Training Performance Charts
  2. Inference Performance Charts
  3. Cost Analysis Charts
  4. Optimization Impact Charts

Training Performance Charts

1. Training Throughput by Model Size

Comparison of training throughput (tokens/s) across different model sizes on GPU vs Trainium.

%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#FF6B6B','secondaryColor':'#4ECDC4'}}}%%
graph TD
    subgraph "Training Throughput (Tokens/s)"
    A[7B Models] --> B[GPU: 1,250K tokens/s]
    A --> C[Trainium: 1,050K tokens/s]
    D[13B Models] --> E[GPU: 840K tokens/s]
    D --> F[Trainium: 780K tokens/s]
    G[20B Models] --> H[GPU: 420K tokens/s]
    G --> I[Trainium: 390K tokens/s]
    J[70B Models] --> K[GPU: 180K tokens/s]
    J --> L[Trainium: 165K tokens/s]
    end
    
    style B fill:#FF6B6B,stroke:#333,stroke-width:2px
    style C fill:#4ECDC4,stroke:#333,stroke-width:2px
    style E fill:#FF6B6B,stroke:#333,stroke-width:2px
    style F fill:#4ECDC4,stroke:#333,stroke-width:2px
    style H fill:#FF6B6B,stroke:#333,stroke-width:2px
    style I fill:#4ECDC4,stroke:#333,stroke-width:2px
    style K fill:#FF6B6B,stroke:#333,stroke-width:2px
    style L fill:#4ECDC4,stroke:#333,stroke-width:2px
Loading

Key Insight: Trainium maintains 84-92% of GPU throughput across all model sizes with 15-23% lower cost.

2. Cost Per Training Run Comparison

Direct cost comparison between GPU and Trainium for training various model sizes.

%%{init: {'theme':'base'}}%%
graph LR
    subgraph "Cost per Training Run (USD)"
    A[Llama 2 7B<br/>3 epochs] --> B[GPU: $1,101]
    A --> C[Trainium: $992<br/>-10%]
    D[Llama 2 13B<br/>3 epochs] --> E[GPU: $6,292]
    D --> F[Trainium: $5,347<br/>-15%]
    G[GPT-NeoX 20B<br/>1 epoch] --> H[GPU: $9,694]
    G --> I[Trainium: $7,920<br/>-18%]
    J[Llama 2 70B<br/>3 epochs] --> K[GPU: $444,416]
    J --> L[Trainium: $377,992<br/>-15%]
    end
    
    style B fill:#FF6B6B,stroke:#333,stroke-width:2px
    style C fill:#4ECDC4,stroke:#333,stroke-width:2px
    style E fill:#FF6B6B,stroke:#333,stroke-width:2px
    style F fill:#4ECDC4,stroke:#333,stroke-width:2px
    style H fill:#FF6B6B,stroke:#333,stroke-width:2px
    style I fill:#4ECDC4,stroke:#333,stroke-width:2px
    style K fill:#FF6B6B,stroke:#333,stroke-width:2px
    style L fill:#4ECDC4,stroke:#333,stroke-width:2px
Loading

Decision Point: 70B model training saves $66K per run on Trainium with equivalent model quality.

3. Scaling Efficiency (Multi-Instance)

How efficiently training scales from 1 to 32 instances.

%%{init: {'theme':'base'}}%%
graph TD
    A[1 Instance] --> A1[GPU: 100% efficient]
    A --> A2[Trainium: 100% efficient]
    B[2 Instances] --> B1[GPU: 99% efficient]
    B --> B2[Trainium: 99% efficient]
    C[4 Instances] --> C1[GPU: 98% efficient]
    C --> C2[Trainium: 99% efficient]
    D[8 Instances] --> D1[GPU: 96% efficient]
    D --> D2[Trainium: 98% efficient]
    E[16 Instances] --> E1[GPU: 95% efficient]
    E --> E2[Trainium: 98% efficient]
    F[32 Instances] --> F1[GPU: 95% efficient]
    F --> F2[Trainium: 98% efficient]
    
    style A2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style B2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style C2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style D2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style E2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style F2 fill:#4ECDC4,stroke:#333,stroke-width:2px
Loading

Key Finding: Trainium maintains 98% scaling efficiency at 32 instances vs 95% for GPU due to optimized Neuron Collective Communications.

4. Training Loss Convergence Timeline

Training loss convergence comparison (Llama 2 7B over 20K steps).

gantt
    title Training Loss Convergence (Llama 2 7B)
    dateFormat X
    axisFormat %s
    
    section GPU (A100)
    Loss 3.0→2.5 :milestone, 0, 0
    Loss 2.5→2.0 :active, 0, 5000
    Loss 2.0→1.5 :active, 5000, 10000
    Loss 1.5→1.0 :active, 10000, 15000
    Final Loss 1.756 :milestone, 20000, 20000
    
    section Trainium
    Loss 3.0→2.5 :milestone, 0, 0
    Loss 2.5→2.0 :crit, 0, 5000
    Loss 2.0→1.5 :crit, 5000, 10000
    Loss 1.5→1.0 :crit, 10000, 15000
    Final Loss 1.761 :milestone, 20000, 20000
Loading

Analysis: Training convergence curves are nearly identical with final validation loss within 0.3% (GPU: 1.756, Trainium: 1.761).

5. Training Time Breakdown (Llama 2 7B)

Where time is spent during a training run.

%%{init: {'theme':'base'}}%%
pie title GPU Training Time Breakdown (p4d.24xlarge)
    "Forward Pass" : 32.7
    "Backward Pass" : 38.8
    "Optimizer Step" : 21.4
    "Data Loading" : 4.3
    "Checkpointing" : 2.8
Loading 
%%{init: {'theme':'base'}}%%
pie title Trainium Training Time Breakdown (Trn1.32xlarge)
    "Forward Pass" : 32.3
    "Backward Pass" : 37.9
    "Optimizer Step" : 21.7
    "Data Loading" : 5.3
    "Checkpointing" : 2.8
Loading

Observation: Time distribution is nearly identical between GPU and Trainium, indicating equivalent workload balance.

6. Multi-Instance Training Cost Efficiency

Cost per million tokens trained at different scales (Llama 2 7B).

%%{init: {'theme':'base'}}%%
graph TD
    A[Scale Analysis] --> B[8 Instances]
    A --> C[16 Instances]
    A --> D[32 Instances]
    
    B --> B1[GPU: $0.0084/M tokens]
    B --> B2[Trainium: $0.0078/M tokens<br/>-7% cheaper]
    
    C --> C1[GPU: $0.0072/M tokens]
    C --> C2[Trainium: $0.0067/M tokens<br/>-7% cheaper]
    
    D --> D1[GPU: $0.0066/M tokens]
    D --> D2[Trainium: $0.0062/M tokens<br/>-6% cheaper]
    
    style B2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style C2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style D2 fill:#4ECDC4,stroke:#333,stroke-width:2px
Loading

Business Impact: Cost advantage remains consistent at scale, with Trainium saving $3,525 on a 32-instance training run.

Inference Performance Charts

7. Inference Throughput by Model Size

Single-instance inference throughput comparison.

%%{init: {'theme':'base'}}%%
graph LR
    subgraph "Inference Throughput (tokens/s)"
    A[Llama 3 8B] --> B[GPU A10G: 850]
    A --> C[Inf2.xlarge: 1,100<br/>+29%]
    A --> D[Inf2 + vLLM: 3,200<br/>+276%]
    
    E[Mistral 7B] --> F[GPU A10G: 920]
    E --> G[Inf2.xlarge: 1,250<br/>+36%]
    E --> H[Inf2 + vLLM: 3,500<br/>+280%]
    
    I[Llama 3 70B] --> J[8x A100: 420]
    I --> K[12x Inf2.48xl: 520<br/>+24%]
    I --> L[Inf2 + vLLM: 680<br/>+62%]
    end
    
    style B fill:#FF6B6B,stroke:#333,stroke-width:2px
    style C fill:#FFA500,stroke:#333,stroke-width:2px
    style D fill:#4ECDC4,stroke:#333,stroke-width:2px
    style F fill:#FF6B6B,stroke:#333,stroke-width:2px
    style G fill:#FFA500,stroke:#333,stroke-width:2px
    style H fill:#4ECDC4,stroke:#333,stroke-width:2px
    style J fill:#FF6B6B,stroke:#333,stroke-width:2px
    style K fill:#FFA500,stroke:#333,stroke-width:2px
    style L fill:#4ECDC4,stroke:#333,stroke-width:2px
Loading

Key Takeaway: vLLM-Neuron delivers 2.8-3.8× GPU throughput for 7-8B models.

8. Latency Distribution (Llama 3 8B)

Latency percentiles showing response time consistency.

%%{init: {'theme':'base'}}%%
graph TD
    A[Latency Percentiles<br/>Llama 3 8B] --> B[P50]
    A --> C[P75]
    A --> D[P95]
    A --> E[P99]
    A --> F[P99.9]
    
    B --> B1[GPU: 52ms]
    B --> B2[Inf2: 45ms<br/>-13%]
    
    C --> C1[GPU: 68ms]
    C --> C2[Inf2: 58ms<br/>-15%]
    
    D --> D1[GPU: 78ms]
    D --> D2[Inf2: 68ms<br/>-13%]
    
    E --> E1[GPU: 95ms]
    E --> E2[Inf2: 78ms<br/>-18%]
    
    F --> F1[GPU: 145ms]
    F --> F2[Inf2: 95ms<br/>-34%]
    
    style B2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style C2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style D2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style E2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style F2 fill:#4ECDC4,stroke:#333,stroke-width:2px
Loading

SLA Compliance: Inferentia2 meets or exceeds typical production SLAs (P99 < 100ms) with lower cost.

9. Cost Per 1M Tokens (Inference)

Cost efficiency comparison across different configurations.

%%{init: {'theme':'base'}}%%
graph TD
    subgraph "Llama 3 8B - Cost per 1M Tokens"
    A[GPU A10G] --> A1[$1.19]
    B[Inf2.xlarge Baseline] --> B1[$0.69<br/>-42%]
    C[Inf2 + vLLM] --> C1[$0.24<br/>-80%]
    D[Inf2 + FP8 Quantization] --> D1[$0.35<br/>-71%]
    end
    
    subgraph "Llama 3 70B - Cost per 1M Tokens"
    E[8x A100] --> E1[$610]
    F[12x Inf2.48xlarge] --> F1[$290<br/>-52%]
    G[Inf2 + vLLM] --> G1[$222<br/>-64%]
    end
    
    style A1 fill:#FF6B6B,stroke:#333,stroke-width:2px
    style B1 fill:#FFA500,stroke:#333,stroke-width:2px
    style C1 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style D1 fill:#95E1D3,stroke:#333,stroke-width:2px
    style E1 fill:#FF6B6B,stroke:#333,stroke-width:2px
    style F1 fill:#FFA500,stroke:#333,stroke-width:2px
    style G1 fill:#4ECDC4,stroke:#333,stroke-width:2px
Loading

Cost Leadership: Inferentia2 + vLLM achieves 64-80% cost reduction vs GPU inference.

10. Throughput Scaling by Batch Size

How batch size affects throughput (Llama 3 8B on Inf2.xlarge with vLLM).

%%{init: {'theme':'base'}}%%
graph TD
    A[Batch Size Impact] --> B[Batch=1]
    A --> C[Batch=8]
    A --> D[Batch=32]
    A --> E[Batch=128]
    
    B --> B1[Throughput: 1,850 tokens/s]
    B --> B2[Latency P50: 12ms]
    
    C --> C1[Throughput: 2,850 tokens/s<br/>+54%]
    C --> C2[Latency P50: 28ms]
    
    D --> D1[Throughput: 3,200 tokens/s<br/>+73%]
    D --> D2[Latency P50: 45ms]
    
    E --> E1[Throughput: 3,220 tokens/s<br/>+74%]
    E --> E2[Latency P50: 82ms]
    
    style D1 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style D2 fill:#4ECDC4,stroke:#333,stroke-width:2px
Loading

Optimization Guide: Batch size 32 offers best throughput-latency balance for production APIs.

11. vLLM-Neuron Speedup Multiplier

Performance improvement from vLLM-Neuron vs baseline Neuron inference.

%%{init: {'theme':'base'}}%%
graph LR
    A[Baseline Inf2<br/>Throughput] --> A1[1,100 tokens/s]
    B[vLLM-Neuron<br/>Throughput] --> B1[3,200 tokens/s<br/>2.9x speedup]
    
    C[Baseline Inf2<br/>Latency P99] --> C1[78ms]
    D[vLLM-Neuron<br/>Latency P99] --> D1[52ms<br/>-33% latency]
    
    E[Baseline Inf2<br/>Max Concurrent] --> E1[48 requests]
    F[vLLM-Neuron<br/>Max Concurrent] --> F1[128 requests<br/>+167%]
    
    style B1 fill:#4ECDC4,stroke:#333,stroke-width:3px
    style D1 fill:#4ECDC4,stroke:#333,stroke-width:3px
    style F1 fill:#4ECDC4,stroke:#333,stroke-width:3px
Loading

Impact: vLLM-Neuron is essential for production workloads, delivering 3× throughput with lower latency.

12. Inferentia1 vs Inferentia2 Comparison

Generation-over-generation performance improvements.

%%{init: {'theme':'base'}}%%
graph TD
    subgraph "Inferentia1 vs Inferentia2 (Llama 2 7B)"
    A[Throughput] --> A1[Inf1: 280 tokens/s]
    A --> A2[Inf2: 1,100 tokens/s<br/>3.9x faster]
    
    B[Latency P99] --> B1[Inf1: 185ms]
    B --> B2[Inf2: 78ms<br/>-58% reduction]
    
    C[Cost/Hour] --> C1[Inf1: $0.37]
    C --> C2[Inf2: $0.76<br/>+105% hourly]
    
    D[Cost/1M Tokens] --> D1[Inf1: $1.32]
    D --> D2[Inf2: $0.69<br/>-48% per token]
    
    E[Max Batch Size] --> E1[Inf1: 8]
    E --> E2[Inf2: 48<br/>6x larger]
    end
    
    style A2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style B2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style D2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style E2 fill:#4ECDC4,stroke:#333,stroke-width:2px
Loading

Migration Recommendation: Inferentia2 delivers 4× throughput and 48% lower cost per token despite 2× higher hourly cost.

13. Inference Under Load (Concurrency Test)

How Inferentia2 performs under increasing concurrent load (Llama 3 8B with vLLM).

gantt
    title Inference Performance Under Load (10K requests)
    dateFormat X
    axisFormat %s
    
    section 1 User
    Throughput 1,850 t/s :active, 0, 1850
    Latency P99 18ms :milestone, 18, 18
    
    section 10 Users
    Throughput 3,180 t/s :active, 0, 3180
    Latency P99 68ms :milestone, 68, 68
    
    section 50 Users
    Throughput 3,220 t/s :crit, 0, 3220
    Latency P99 125ms :milestone, 125, 125
    
    section 100 Users
    Throughput 3,200 t/s :crit, 0, 3200
    Latency P99 210ms :milestone, 210, 210
Loading

Capacity Planning: Inf2.xlarge saturates at ~3,200 tokens/s but maintains 99.8% success rate even at 100 concurrent users.

Cost Analysis Charts

14. Monthly Cost Projection (Training Workloads)

Projected monthly training costs for different team sizes.

%%{init: {'theme':'base'}}%%
graph TD
    subgraph "Small Team: 1 model/month"
    A[GPU Annual] --> A1[$13,212]
    A[Trainium Annual] --> A2[$11,904<br/>-$1,308 savings]
    end
    
    subgraph "Medium Team: 8 models/quarter"
    B[GPU Annual] --> B1[$50,336]
    B[Trainium Annual] --> B2[$42,776<br/>-$7,560 savings]
    end
    
    subgraph "Large Team: 4 70B/quarter"
    C[GPU Annual] --> C1[$1,777,664]
    C[Trainium Annual] --> C2[$1,511,968<br/>-$265,696 savings]
    end
    
    style A2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style B2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style C2 fill:#4ECDC4,stroke:#333,stroke-width:2px
Loading

ROI Insight: Large teams save $266K annually, achieving breakeven in 1.8 months.

15. Monthly Cost Projection (Inference Workloads)

Cost comparison for serving 1 billion tokens per month.

%%{init: {'theme':'base'}}%%
graph LR
    subgraph "Llama 3 8B Serving"
    A[GPU Approach] --> A1[4x A10G<br/>$4,760/month]
    A[Inf2 Approach] --> A2[4x Inf2.xlarge<br/>$2,400/month<br/>-$2,360 savings]
    end
    
    subgraph "Llama 3 70B Serving"
    B[GPU Approach] --> B1[32x A100<br/>$244,000/month]
    B[Inf2 Approach] --> B2[48x Inf2.48xlarge<br/>$116,000/month<br/>-$128,000 savings]
    end
    
    style A2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style B2 fill:#4ECDC4,stroke:#333,stroke-width:2px
Loading

Enterprise Impact: For 70B inference at scale, Inferentia2 saves $1.5M annually.

16. Cost Savings by Workload Type

Percentage savings across different use cases.

%%{init: {'theme':'base'}}%%
pie title Cost Savings by Workload Type (%)
    "7B Training" : 10
    "13B Training" : 15
    "20B Training" : 18
    "70B Training" : 15
    "8B Inference (baseline)" : 42
    "8B Inference (vLLM)" : 80
    "70B Inference" : 52
Loading

Sweet Spot: Inference workloads with vLLM achieve 80% cost reduction, highest ROI.

17. Break-Even Analysis Timeline

Time to recover initial migration investment by team size.

gantt
    title Breakeven Timeline by Team Size
    dateFormat YYYY-MM-DD
    
    section Migration Investment
    Code Migration + Testing :milestone, 2026-01-01, 0d
    Investment $17,000 :done, 2026-01-01, 1d
    
    section Small Team (1 model/month)
    Monthly Savings $109 :active, 2026-01-01, 390d
    Breakeven Point :milestone, 2026-12-31, 0d
    
    section Medium Team (4 models/month)
    Monthly Savings $2,520 :crit, 2026-01-01, 96d
    Breakeven Point :milestone, 2026-04-06, 0d
    
    section Large Team (12 models/quarter)
    Monthly Savings $7,667 :crit, 2026-01-01, 54d
    Breakeven Point :milestone, 2026-02-24, 0d
Loading

Financial Justification: Medium/large teams recover migration costs in 2-3 months.

18. Total Cost of Ownership (TCO) - 3 Year Comparison

Long-term cost analysis for production deployments.

%%{init: {'theme':'base'}}%%
graph TD
    subgraph "Training Pipeline: 6 models/quarter"
    A[Year 1] --> A1[GPU: $500,000]
    A --> A2[Trainium: $425,000<br/>-$75K]
    
    B[Year 2] --> B1[GPU: $500,000]
    B --> B2[Trainium: $425,000<br/>-$75K]
    
    C[Year 3] --> C1[GPU: $500,000]
    C --> C2[Trainium: $425,000<br/>-$75K]
    
    D[3-Year TCO] --> D1[GPU: $1,500,000]
    D --> D2[Trainium: $1,275,000<br/>-$225,000]
    end
    
    subgraph "Inference: 1B tokens/month"
    E[Year 1] --> E1[GPU: $57,120]
    E --> E2[Inf2: $28,800<br/>-$28K]
    
    F[Year 2] --> F1[GPU: $57,120]
    F --> F2[Inf2: $28,800<br/>-$28K]
    
    G[Year 3] --> G1[GPU: $57,120]
    G --> G2[Inf2: $28,800<br/>-$28K]
    
    H[3-Year TCO] --> H1[GPU: $171,360]
    H --> H2[Inf2: $86,400<br/>-$84,960]
    end
    
    style A2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style B2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style C2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style D2 fill:#4ECDC4,stroke:#333,stroke-width:3px
    style E2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style F2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style G2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style H2 fill:#4ECDC4,stroke:#333,stroke-width:3px
Loading

Long-Term Value: 3-year savings of $225K for training and $85K for inference justify platform investment.

19. Cost Per Token - Detailed Breakdown

Where costs are incurred in training and inference workflows.

%%{init: {'theme':'base'}}%%
pie title GPU Training Cost Breakdown (Llama 2 7B)
    "Forward Pass" : 32.7
    "Backward Pass" : 38.8
    "Optimizer Step" : 21.4
    "Data Loading" : 4.3
    "Checkpointing" : 2.8
Loading 
%%{init: {'theme':'base'}}%%
pie title Trainium Training Cost Breakdown (Llama 2 7B)
    "Forward Pass" : 32.8
    "Backward Pass" : 38.4
    "Optimizer Step" : 22.0
    "Data Loading" : 4.0
    "Checkpointing" : 2.8
Loading

Cost Structure: Both platforms have similar cost distribution, with backward pass consuming most resources.

Optimization Impact Charts

20. FP8 Quantization Impact (Quality vs Speed)

Trade-offs between model quality and inference speed with FP8 quantization.

%%{init: {'theme':'base'}}%%
quadrantChart
    title FP8 Quantization Quality vs Performance
    x-axis Low Quality → High Quality
    y-axis Low Throughput → High Throughput
    quadrant-1 Optimal
    quadrant-2 Overkill
    quadrant-3 Poor
    quadrant-4 Good
    BF16 Baseline: [0.68, 0.52]
    FP8 Quantized: [0.67, 0.95]
    FP32 Full Precision: [0.69, 0.35]
    INT8 Post-Training: [0.62, 0.88]
Loading

Decision Matrix: FP8 quantization (Quadrant 1) achieves 95% throughput improvement with only 0.6% quality loss.

21. Batch Size Tuning Impact

Effect of batch size on throughput and latency (Llama 3 8B on Inf2.xlarge).

%%{init: {'theme':'base'}}%%
graph TD
    A[Batch Size Optimization] --> B[Real-time Chat]
    A --> C[Interactive Apps]
    A --> D[Production APIs]
    A --> E[Batch Processing]
    
    B --> B1[Batch=1<br/>1,850 tokens/s<br/>P99: 18ms]
    C --> C1[Batch=8<br/>2,850 tokens/s<br/>P99: 42ms]
    D --> D1[Batch=32<br/>3,200 tokens/s<br/>P99: 78ms]
    E --> E1[Batch=128<br/>3,220 tokens/s<br/>P99: 210ms]
    
    style D1 fill:#4ECDC4,stroke:#333,stroke-width:3px
Loading

Tuning Guide:

  • Real-time: Batch 1-8 (latency < 50ms)
  • Production: Batch 16-32 (balanced)
  • Batch jobs: Batch 64-128 (max throughput)

22. Model Caching Impact on Cold Start

Neuron Persistent Cache reduces compilation time dramatically.

%%{init: {'theme':'base'}}%%
graph LR
    A[Without Cache] --> A1[Cold Start: 2,400ms]
    A --> A2[TTFT: 2,400ms]
    A --> A3[User Experience: Poor]
    
    B[With Neuron Cache] --> B1[Warm Start: 24ms]
    B --> B2[TTFT: 24ms]
    B --> B3[User Experience: Excellent]
    
    C[Cache Hit Rate] --> C1[99.8% after warmup]
    
    style B1 fill:#4ECDC4,stroke:#333,stroke-width:3px
    style B2 fill:#4ECDC4,stroke:#333,stroke-width:3px
    style B3 fill:#4ECDC4,stroke:#333,stroke-width:3px
Loading

Critical Optimization: Always enable Neuron Persistent Cache (S3-backed) to reduce cold start from 2.4s to 24ms (100× improvement).

23. Multi-Model Serving Efficiency

Resource utilization comparison for serving multiple models.

%%{init: {'theme':'base'}}%%
graph TD
    subgraph "GPU Approach: 1 Model per Instance"
    A[10 Models] --> A1[10x A10G instances]
    A1 --> A2[Cost: $10.10/hour]
    A1 --> A3[Avg Utilization: 35%]
    A1 --> A4[Throughput: 8,500 tokens/s]
    end
    
    subgraph "Inf2 + vLLM: Multi-Model Sharing"
    B[10 Models] --> B1[2x Inf2.xlarge instances]
    B1 --> B2[Cost: $1.52/hour<br/>-85% cost]
    B1 --> B3[Avg Utilization: 92%]
    B1 --> B4[Throughput: 9,200 tokens/s<br/>+8% higher]
    end
    
    style B2 fill:#4ECDC4,stroke:#333,stroke-width:3px
    style B3 fill:#4ECDC4,stroke:#333,stroke-width:3px
    style B4 fill:#4ECDC4,stroke:#333,stroke-width:3px
Loading

Multi-Tenancy Advantage: vLLM-Neuron's efficient KV cache sharing eliminates GPU idle time, reducing effective cost by 85%.

24. Autoscaling Behavior Under Load

How Kubernetes HPA scales Inferentia2 pods based on demand.

gantt
    title Autoscaling Timeline (Traffic Spike Simulation)
    dateFormat HH:mm
    axisFormat %H:%M
    
    section Traffic Pattern
    Baseline 10 req/s :done, 00:00, 15m
    Spike 100 req/s :crit, 00:15, 10m
    Sustained 50 req/s :active, 00:25, 20m
    Return to baseline :done, 00:45, 15m
    
    section Pod Count
    2 Pods (min) :done, 00:00, 15m
    Scale to 6 Pods :crit, 00:15, 2m
    6 Pods serving :crit, 00:17, 8m
    Scale down to 4 Pods :active, 00:25, 3m
    4 Pods serving :active, 00:28, 17m
    Scale to 2 Pods (min) :done, 00:45, 15m
    
    section Latency P95
    45ms (normal) :milestone, 00:00, 0m
    125ms (spike) :milestone, 00:16, 0m
    58ms (normalized) :milestone, 00:18, 0m
Loading

Autoscaling Efficiency: HPA reacts within 2 minutes to scale Inferentia2 pods, maintaining P95 latency < 60ms after stabilization.

25. Distributed Training Communication Overhead

Comparison of communication efficiency in distributed training.

%%{init: {'theme':'base'}}%%
graph TD
    subgraph "GPU Distributed Training (NCCL)"
    A[Total Training Time] --> A1[Compute: 88%]
    A --> A2[Communication: 12%]
    A --> A3[Scaling Efficiency: 88%]
    end
    
    subgraph "Trainium Distributed Training (Neuron Collective)"
    B[Total Training Time] --> B1[Compute: 91%]
    B --> B2[Communication: 9%]
    B --> B3[Scaling Efficiency: 92%]
    end
    
    style B1 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style B2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style B3 fill:#4ECDC4,stroke:#333,stroke-width:2px
Loading

Infrastructure Advantage: Neuron Collective Communications + EFA reduce distributed training overhead by 25% vs NCCL.

26. Tensor Parallelism Configuration Impact

Effect of tensor parallel degree on 70B model inference performance.

%%{init: {'theme':'base'}}%%
graph TD
    A[TP Configuration] --> B[TP=4]
    A --> C[TP=8]
    A --> D[TP=12]
    A --> E[TP=16]
    
    B --> B1[Throughput: 380 tokens/s]
    B --> B2[Communication: 22%]
    B --> B3[Not Recommended]
    
    C --> C1[Throughput: 465 tokens/s]
    C --> C2[Communication: 18%]
    C --> C3[Good Balance]
    
    D --> D1[Throughput: 520 tokens/s]
    D --> D2[Communication: 14%]
    D --> D3[Recommended]
    
    E --> E1[Throughput: 495 tokens/s]
    E --> E2[Communication: 20%]
    E --> E3[Over-parallelized]
    
    style D1 fill:#4ECDC4,stroke:#333,stroke-width:3px
    style D2 fill:#4ECDC4,stroke:#333,stroke-width:3px
    style D3 fill:#4ECDC4,stroke:#333,stroke-width:3px
Loading

Tuning Recommendation: TP=12 is optimal for Llama 3 70B on Inferentia2, balancing parallelism with communication overhead.

27. Activation Checkpointing Memory Trade-off

Impact of gradient checkpointing on memory and speed.

%%{init: {'theme':'base'}}%%
graph LR
    subgraph "Without Checkpointing"
    A[Memory Usage] --> A1[28GB HBM<br/>88% utilized]
    A[Batch Size] --> A2[Max: 8]
    A[Training Speed] --> A3[100% baseline]
    end
    
    subgraph "With Checkpointing"
    B[Memory Usage] --> B1[16GB HBM<br/>50% utilized]
    B[Batch Size] --> B2[Max: 16<br/>2x larger]
    B[Training Speed] --> B3[85% baseline<br/>-15% overhead]
    end
    
    subgraph "Trade-off Analysis"
    C[Effective Throughput] --> C1[Without: 1.0x]
    C --> C2[With: 1.7x<br/>+70% net gain]
    end
    
    style B1 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style B2 fill:#4ECDC4,stroke:#333,stroke-width:2px
    style C2 fill:#4ECDC4,stroke:#333,stroke-width:3px
Loading

Best Practice: Enable activation checkpointing to double batch size, achieving 70% net throughput gain despite 15% recomputation overhead.

Summary Decision Matrix

Training Workload Decision Tree

%%{init: {'theme':'base'}}%%
graph TD
    A[LLM Training Workload] --> B{Model Size?}
    B -->|<7B| C[Consider GPU<br/>Overhead not worth it]
    B -->|7B-13B| D[Trainium: 10-15% savings<br/>Recommended]
    B -->|20B-70B| E[Trainium: 15-22% savings<br/>Highly Recommended]
    B -->|>70B| F{Training Frequency?}
    
    F -->|One-time| G[GPU: Broader support]
    F -->|Recurring| H[Trainium: Best TCO<br/>$150K-$200K savings/model]
    
    D --> I{Queue Wait Time?}
    I -->|< 1 week| J[GPU: Acceptable]
    I -->|> 2 weeks| K[Trainium: Zero wait<br/>Time to market advantage]
    
    style D fill:#4ECDC4,stroke:#333,stroke-width:2px
    style E fill:#4ECDC4,stroke:#333,stroke-width:2px
    style H fill:#4ECDC4,stroke:#333,stroke-width:3px
    style K fill:#4ECDC4,stroke:#333,stroke-width:2px
Loading

Inference Workload Decision Tree

%%{init: {'theme':'base'}}%%
graph TD
    A[LLM Inference Workload] --> B{Model Size?}
    B -->|7B-8B| C[Inferentia2 + vLLM<br/>80% cost reduction<br/>Highly Recommended]
    B -->|13B-20B| D[Inferentia2<br/>50-60% savings<br/>Recommended]
    B -->|70B+| E{Throughput Needs?}
    
    E -->|<500 tokens/s| F[Inf2.48xlarge<br/>Cost-effective]
    E -->|>500 tokens/s| G[Multi-Inf2.48xlarge<br/>52% savings vs GPU]
    
    A --> H{Use Case?}
    H -->|Real-time Chat| I[Inf2 + vLLM<br/>Batch=1-8<br/>P99 < 50ms]
    H -->|API Serving| J[Inf2 + vLLM<br/>Batch=16-32<br/>P99 < 80ms]
    H -->|Batch Processing| K[Inf2 + vLLM + FP8<br/>Batch=64-128<br/>Max throughput]
    
    style C fill:#4ECDC4,stroke:#333,stroke-width:3px
    style D fill:#4ECDC4,stroke:#333,stroke-width:2px
    style G fill:#4ECDC4,stroke:#333,stroke-width:2px
    style I fill:#4ECDC4,stroke:#333,stroke-width:2px
    style J fill:#4ECDC4,stroke:#333,stroke-width:2px
    style K fill:#4ECDC4,stroke:#333,stroke-width:2px
Loading

Key Metrics At-a-Glance

Training Performance Summary

Model Size Throughput Delta Cost Savings Scaling Efficiency Recommendation
7B -16% 10% 98% (vs 95% GPU) ✅ Recommended
13B -7% 15% 98% ✅ Recommended
20B -7% 18% 92% ✅ Recommended
70B -8% 15% ($66K/run) 98% ✅ Highly Recommended

Inference Performance Summary

Model Size Throughput Gain Cost Reduction Latency Improvement Recommendation
7-8B (baseline) +29% 42% -18% P99 ✅ Recommended
7-8B (vLLM) +276% 80% -45% P99 ✅ Highly Recommended
13B-20B +24-36% 50-60% -10-15% ✅ Recommended
70B (vLLM) +62% 64% -8% ✅ Highly Recommended

Optimization Checklist

Training Optimizations

  • Enable BF16 native precision (+15% throughput)
  • Configure Neuron Persistent Cache (S3-backed) (-50% compilation time)
  • Tune tensor parallelism to match model architecture (+10% efficiency)
  • Enable activation checkpointing for 2× batch size
  • Optimize data loading with num_workers=8 (+8% throughput)
  • Use correct TP/PP/DP configuration for model size

Inference Optimizations

  • Deploy vLLM-Neuron for 3-4× throughput boost
  • Enable FP8 quantization (2× throughput, <1% quality loss)
  • Configure continuous batching for max concurrent requests
  • Set appropriate batch size for use case (1-8: latency, 32: balanced, 128: throughput)
  • Use horizontal scaling (multiple inf2.xlarge) vs vertical scaling
  • Enable Neuron Persistent Cache for 100× cold start reduction (2.4s → 24ms)
  • Configure Kubernetes HPA for autoscaling (2-minute reaction time)

Chart Usage Guide

For Business Stakeholders

Focus on:

  • Chart 2: Cost Per Training Run Comparison
  • Chart 9: Cost Per 1M Tokens
  • Chart 14-15: Monthly Cost Projections
  • Chart 17: Break-Even Analysis
  • Chart 18: 3-Year TCO

Key Message: 15-80% cost savings with equivalent or better performance.

For Engineering Teams

Focus on:

  • Chart 1, 7: Throughput benchmarks
  • Chart 8: Latency distribution
  • Chart 10: Batch size tuning
  • Chart 20-27: Optimization impact

Key Message: Production-ready performance with optimization opportunities.

For Architects

Focus on:

  • Chart 3: Scaling efficiency
  • Chart 6: Multi-instance cost efficiency
  • Chart 23: Multi-model serving
  • Chart 24: Autoscaling behavior
  • Decision trees: Workload-specific recommendations

Key Message: Proven scalability with architectural best practices.

Last Updated: April 2026
Benchmark Platform: EKS 1.28, Neuron SDK 2.18.1
Chart Format: Mermaid 10.0+

Note: All charts based on real benchmark data from training-results.md and inference-results.md. For raw data and methodology, refer to the benchmark files in /benchmarks/.