Skip to content

SCRIPTS_MONITORING.md

Latest commit

 

History

History
460 lines (347 loc) · 10.4 KB

SCRIPTS_MONITORING.md

File metadata and controls

460 lines (347 loc) · 10.4 KB
Raw

Scripts and Monitoring Quick Reference

This document provides a quick reference for the operational scripts and monitoring configurations in this repository.

Table of Contents

Scripts Overview

1. setup-neuron-device-plugin.sh

Purpose: Install AWS Neuron Device Plugin on EKS

Quick Start:

./scripts/setup-neuron-device-plugin.sh

Common Options:

# With specific version
./scripts/setup-neuron-device-plugin.sh --version v2.0.0

# Extended timeout for large clusters
./scripts/setup-neuron-device-plugin.sh --wait 600

What it does:

  • Downloads device plugin manifest
  • Applies to kube-system namespace
  • Verifies DaemonSet is running
  • Confirms NeuronCore resources are advertised

2. validate-neuron.sh

Purpose: End-to-end validation of Neuron infrastructure

Quick Start:

./scripts/validate-neuron.sh --cleanup --verbose

Common Options:

# Skip inference test (faster)
./scripts/validate-neuron.sh --skip-inference

# Target specific namespace
./scripts/validate-neuron.sh --namespace production --cleanup

Exit Codes:

  • 0: All tests passed
  • 1: One or more tests failed
  • 2: Prerequisites not met

Tests Performed:

  1. Device plugin status check
  2. NeuronCore availability verification
  3. Model loading test
  4. Inference capability test

3. benchmark-training.sh

Purpose: Training performance benchmarks

Quick Start:

./scripts/benchmark-training.sh --model gpt2 --batch-size 16

Common Options:

# Full benchmark with output
./scripts/benchmark-training.sh \
  --model gpt2-medium \
  --batch-size 32 \
  --iterations 200 \
  --output training-results.json

# Quick test
./scripts/benchmark-training.sh \
  --model gpt2 \
  --batch-size 8 \
  --iterations 50

Output Metrics:

  • Throughput (tokens/second)
  • Average time per iteration
  • Total cost (USD)
  • Cost per token
  • Cost per 1M tokens

4. benchmark-inference.sh

Purpose: Inference performance benchmarks

Quick Start:

./scripts/benchmark-inference.sh --model gpt2 --requests 5000

Common Options:

# High-load test
./scripts/benchmark-inference.sh \
  --model gpt2 \
  --requests 10000 \
  --concurrency 50 \
  --output inference-results.json

# Quick test
./scripts/benchmark-inference.sh \
  --model gpt2 \
  --requests 1000 \
  --concurrency 10

Output Metrics:

  • Throughput (requests/second)
  • Latency (P50, P95, P99)
  • Total cost (USD)
  • Cost per request
  • Cost per 1K requests

5. cleanup.sh

Purpose: Remove Neuron resources from cluster

Quick Start:

# Safe cleanup (workloads only)
./scripts/cleanup.sh --force

# Full cleanup (including monitoring)
./scripts/cleanup.sh --all --monitoring --force

# Complete cleanup (including infrastructure)
./scripts/cleanup.sh --all --node-pools --node-classes --force

Safety Levels:

  1. Default: Removes workload resources only
  2. --all: Adds device plugin and monitoring
  3. --node-pools: Includes Karpenter NodePools
  4. --node-classes: Includes EC2NodeClass (destructive!)

Monitoring Setup

Deploy Monitoring Stack

# 1. Deploy Neuron Monitor DaemonSet
kubectl apply -f monitoring/neuron-monitor-daemonset.yaml

# 2. Deploy Prometheus Rules
kubectl apply -f monitoring/prometheus-rules.yaml

# 3. Import Grafana Dashboard
# Upload monitoring/grafana-dashboard.json via Grafana UI

Verify Monitoring

# Check DaemonSet
kubectl get daemonset neuron-monitor -n kube-system

# View logs
kubectl logs -n kube-system -l app=neuron-monitor -f

# Test metrics endpoint
kubectl port-forward -n kube-system svc/neuron-monitor 8000:8000
curl http://localhost:8000/metrics | grep neuron_core_utilization

Quick Commands

Device Plugin Status

# Check DaemonSet
kubectl get daemonset neuron-device-plugin-daemonset -n kube-system

# View logs
kubectl logs -n kube-system -l name=neuron-device-plugin-ds

# Check node resources
kubectl describe nodes | grep -A 5 "aws.amazon.com/neuron"

NeuronCore Resources

# List nodes with NeuronCores
kubectl get nodes -o json | jq -r '.items[] | select(.status.allocatable["aws.amazon.com/neuron"] != null) | "\(.metadata.name): \(.status.allocatable["aws.amazon.com/neuron"]) cores"'

# Check allocatable resources
kubectl get nodes -o custom-columns=NAME:.metadata.name,NEURON:.status.allocatable.aws\.amazon\.com/neuron

Pod Validation

# Check pod with Neuron resources
kubectl get pods -o json | jq -r '.items[] | select(.spec.containers[].resources.limits["aws.amazon.com/neuron"] != null) | .metadata.name'

# Exec into pod and check Neuron runtime
kubectl exec -it <pod-name> -- neuron-ls
kubectl exec -it <pod-name> -- neuron-top

Monitoring Metrics

# Port-forward to Neuron Monitor
kubectl port-forward -n kube-system svc/neuron-monitor 8000:8000

# Get current utilization
curl -s http://localhost:8000/metrics | grep neuron_core_utilization_percent

# Get memory usage
curl -s http://localhost:8000/metrics | grep neuron_device_memory

Troubleshooting

Device Plugin Not Running

Symptoms: No NeuronCore resources advertised

Check:

kubectl get daemonset neuron-device-plugin-daemonset -n kube-system
kubectl describe daemonset neuron-device-plugin-daemonset -n kube-system
kubectl logs -n kube-system -l name=neuron-device-plugin-ds

Fix:

# Reinstall device plugin
./scripts/setup-neuron-device-plugin.sh

Pod Stuck in Pending

Symptoms: Pod waiting for NeuronCore resources

Check:

kubectl describe pod <pod-name>
# Look for: "Insufficient aws.amazon.com/neuron"

Fix:

# Check available resources
kubectl get nodes -o custom-columns=NAME:.metadata.name,NEURON:.status.allocatable.aws\.amazon\.com/neuron

# Scale up or wait for Karpenter to provision nodes
kubectl get nodepools

Model Loading Failures

Symptoms: Pod crashes during model loading

Check:

kubectl logs <pod-name>
kubectl describe pod <pod-name>

Common Issues:

  1. Insufficient memory - increase pod memory limits
  2. Model incompatible - check Neuron SDK version
  3. Timeout - increase readiness probe timeout

Fix:

# Increase memory in deployment
resources:
  limits:
    memory: 32Gi
  requests:
    memory: 16Gi

High Latency

Symptoms: Inference slower than expected

Check:

# Check NeuronCore utilization
kubectl exec <pod-name> -- neuron-top

# Check metrics
kubectl port-forward -n kube-system svc/neuron-monitor 8000:8000
curl http://localhost:8000/metrics | grep neuron_core_utilization

Common Issues:

  1. NeuronCore oversubscribed - scale horizontally
  2. Batch size too small - increase batch size
  3. Model not compiled - ensure model is compiled for Neuron

Monitoring Not Working

Symptoms: No metrics in Grafana

Check:

# Check Neuron Monitor DaemonSet
kubectl get daemonset neuron-monitor -n kube-system

# Check if metrics are exposed
kubectl port-forward -n kube-system svc/neuron-monitor 8000:8000
curl http://localhost:8000/metrics

# Check Prometheus scrape config
kubectl get servicemonitor -A

Fix:

# Redeploy monitoring
kubectl delete -f monitoring/neuron-monitor-daemonset.yaml
kubectl apply -f monitoring/neuron-monitor-daemonset.yaml

Instance Type Reference

Inferentia2 (Inference)

Instance Type NeuronCores Memory Cost/Hour
inf2.xlarge 2 8 GB $0.76
inf2.8xlarge 2 32 GB $2.39
inf2.24xlarge 6 96 GB $7.09
inf2.48xlarge 12 192 GB $14.18

Trainium (Training)

Instance Type NeuronCores Memory Cost/Hour
trn1.2xlarge 2 32 GB $1.34
trn1.32xlarge 32 512 GB $21.50
trn1n.32xlarge 32 512 GB $24.78

Key Metrics to Monitor

Performance

  • neuron_core_utilization_percent: Target 60-90%
  • neuron_inference_latency_seconds: Watch P95/P99
  • neuron_inference_throughput_requests_per_second: Monitor for drops

Reliability

  • neuron_execution_errors_total: Should be 0
  • neuron_runtime_errors_total: Should be 0
  • Device plugin up/down status

Cost Optimization

  • Average utilization < 20% = consider downsizing
  • Idle instances (0% for 2+ hours) = terminate
  • Memory utilization > 90% = increase pod limits

Alert Severity Guide

Critical (Page immediately)

  • Device plugin down
  • Execution errors > 0
  • P99 latency > 5s
  • Memory usage > 95%

Warning (Review within 30m)

  • High utilization > 95%
  • High latency P95 > 1s
  • Memory usage > 90%
  • Throughput drop > 30%

Info (Review daily)

  • Low utilization < 10%
  • Idle instances
  • Cost optimization opportunities

Automation Examples

Daily Cost Report

#!/bin/bash
# Get average utilization and estimate savings
UTILIZATION=$(kubectl exec -n kube-system daemonset/neuron-monitor -- curl -s localhost:8000/metrics | grep neuron_core_utilization_percent | awk '{sum+=$2; count++} END {print sum/count}')

echo "Average NeuronCore Utilization: ${UTILIZATION}%"

if (( $(echo "$UTILIZATION < 20" | bc -l) )); then
    echo "WARNING: Consider downsizing - low utilization detected"
fi

Automated Validation (CI/CD)

#!/bin/bash
# Run validation and fail pipeline if tests don't pass
./scripts/validate-neuron.sh --cleanup

if [ $? -ne 0 ]; then
    echo "Validation failed - check logs above"
    exit 1
fi

echo "Validation passed - infrastructure ready"

Weekly Benchmark

#!/bin/bash
# Weekly inference benchmark with result archival
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

./scripts/benchmark-inference.sh \
  --model gpt2 \
  --requests 5000 \
  --concurrency 20 \
  --output "benchmarks/results_${TIMESTAMP}.json"

# Upload to S3 for long-term storage
aws s3 cp "benchmarks/results_${TIMESTAMP}.json" s3://my-benchmarks/

Additional Resources

Last Updated: 2026-04-04