Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
520 lines (393 loc) · 11.6 KB

CONTRIBUTING.md

File metadata and controls

520 lines (393 loc) · 11.6 KB
Raw

Contributing to AWS Neuron on EKS Guide

Thank you for considering contributing to this project! This guide will help you get started.

Table of Contents

Code of Conduct

This project adheres to a Code of Conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to the project maintainers.

Expected behavior:

  • Be respectful and inclusive
  • Welcome newcomers and help them learn
  • Focus on what is best for the community
  • Show empathy towards other community members

Unacceptable behavior:

  • Harassment, discrimination, or trolling
  • Publishing others' private information
  • Personal or political attacks
  • Unprofessional conduct

How Can I Contribute?

1. Reporting Bugs

Before creating bug reports, please check existing issues to avoid duplicates.

Good bug reports include:

  • Title: Clear and descriptive
  • Description: Detailed explanation of the issue
  • Steps to reproduce: Numbered list of steps
  • Expected behavior: What should happen
  • Actual behavior: What actually happens
  • Environment: EKS version, Neuron SDK version, instance type
  • Logs: Relevant error messages
  • Screenshots: If applicable

Example:

## Bug: vLLM-Neuron fails to load Llama 3 8B on Inf2.xlarge

### Description
When deploying vLLM with Llama 3 8B on Inf2.xlarge, the pod fails with "No Neuron devices found".

### Steps to Reproduce
1. Apply `kubectl apply -f deployments/vllm/vllm-llama3-inf2.yaml`
2. Wait for pod to start
3. Check logs: `kubectl logs vllm-llama3-xxx`

### Expected Behavior
Pod should load model and start serving on port 8000.

### Actual Behavior
Pod crashes with error:

RuntimeError: No Neuron devices found. Ensure Neuron driver is installed.


### Environment
- EKS version: 1.28
- Neuron SDK: 2.18.1
- Instance type: inf2.xlarge
- Device plugin: neuron-device-plugin-daemonset v1.2.3

### Logs
[Attach full logs]

2. Suggesting Enhancements

Enhancement suggestions should include:

  • Use case: What problem does this solve?
  • Proposed solution: How should it work?
  • Alternatives considered: What other options exist?
  • Implementation details: Any technical considerations

Example:

## Enhancement: Add support for Mistral 7B×8 (Mixtral)

### Use Case
Many users want to deploy Mixtral (sparse MoE) models on Inferentia2.

### Proposed Solution
Add example deployment YAML for Mixtral 8x7B with:
- Correct resource allocation (NeuronCore requirements)
- vLLM-Neuron configuration for MoE
- Benchmark results for throughput/latency

### Alternatives
- Document manual configuration steps (less user-friendly)
- Wait for official vLLM-Neuron Mixtral support

### Implementation
1. Test Mixtral 8x7B on Inf2.48xlarge
2. Create `deployments/vllm/vllm-mixtral-8x7b-inf2.yaml`
3. Add benchmark to `benchmarks/inference-results.md`
4. Update README with Mixtral example

3. Contributing Code

Areas where contributions are especially welcome:

  • New model examples: Falcon, Qwen, Yi, CodeLlama, etc.
  • Benchmark results: From your own deployments
  • Optimization guides: FP8 quantization, tensor parallelism
  • Deployment patterns: Ray Serve, KServe, custom pipelines
  • Monitoring: Additional Grafana dashboards, Prometheus rules
  • Documentation: Tutorials, troubleshooting, best practices

4. Contributing Documentation

Documentation contributions are highly valued:

  • Architecture diagrams: New Mermaid diagrams
  • Performance charts: Additional benchmark visualizations
  • Troubleshooting guides: Common issues and solutions
  • Tutorials: Step-by-step guides for specific use cases
  • Blog posts: Link to your articles using this guide

5. Contributing Benchmark Results

Share your performance data:

  • Training benchmarks: Throughput, cost, convergence on Trainium
  • Inference benchmarks: Latency, throughput, cost on Inferentia2
  • Cost analysis: Real-world TCO from production deployments
  • Optimization results: Impact of tuning parameters

Format (add to benchmarks/community-results.md):

## Contributor: [Your Name/Company]

### Workload
- **Model**: Llama 3 70B
- **Hardware**: 8× Inf2.48xlarge
- **vLLM-Neuron**: v0.6.3
- **Optimization**: FP8 quantization

### Results
- **Throughput**: 4,200 tokens/s
- **P99 latency**: 180ms
- **Cost per 1M tokens**: $2.10
- **Comparison**: 55% cheaper than 8× A100

### Configuration
[Link to YAML or paste config]

### Notes
Multi-tenant serving with 3 models (Llama 3 70B, Mistral 7B, CodeLlama 34B).

Development Setup

Prerequisites

# Install tools
brew install kubectl eksctl helm  # macOS
# or
sudo apt-get install kubectl      # Linux

# AWS CLI
pip install awscli

# Mermaid CLI (for diagram exports)
npm install -g @mermaid-js/mermaid-cli

Local Testing

  1. Fork and clone:
git clone https://github.com/YOUR_USERNAME/aws-neuron-eks-guide.git
cd aws-neuron-eks-guide
  1. Create feature branch:
git checkout -b feature/add-mixtral-example
  1. Make changes:
# Edit files
vim deployments/vllm/vllm-mixtral-8x7b-inf2.yaml

# Test YAML syntax
yamllint deployments/vllm/vllm-mixtral-8x7b-inf2.yaml

# Test deployment (optional, if you have EKS cluster)
kubectl apply --dry-run=client -f deployments/vllm/vllm-mixtral-8x7b-inf2.yaml
  1. Test documentation:
# Render Mermaid diagrams locally
mmdc -i docs/architecture-diagrams.md -o /tmp/test.png

# Check markdown links
markdown-link-check README.md
  1. Commit and push:
git add .
git commit -m "Add Mixtral 8x7B deployment example"
git push origin feature/add-mixtral-example
  1. Create pull request on GitHub

Contribution Guidelines

Code Style

YAML files:

  • Use 2-space indentation
  • Follow Kubernetes naming conventions
  • Include comments for complex configurations
  • Validate with yamllint

Example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-mixtral-8x7b  # Descriptive name
  labels:
    app: vllm-mixtral
    model: mixtral-8x7b
spec:
  replicas: 1
  # ... rest of spec

Python scripts:

  • Follow PEP 8
  • Use type hints
  • Include docstrings
  • Add tests

Example:

def calculate_cost_per_token(
    tokens_generated: int,
    runtime_hours: float,
    instance_cost_per_hour: float
) -> float:
    """
    Calculate cost per token for inference workload.
    
    Args:
        tokens_generated: Total tokens generated
        runtime_hours: Runtime in hours
        instance_cost_per_hour: Instance cost ($/hour)
    
    Returns:
        Cost per token in dollars
    """
    total_cost = runtime_hours * instance_cost_per_hour
    return total_cost / tokens_generated

Bash scripts:

  • Use #!/bin/bash shebang
  • Enable error handling: set -e
  • Add usage documentation
  • Validate with shellcheck

Example:

#!/bin/bash
set -e

# validate-neuron.sh - Validate Neuron deployment on EKS
#
# Usage:
#   ./validate-neuron.sh [--nodepool NODEPOOL_NAME]
#
# Options:
#   --nodepool    NodePool name to validate (default: inferentia2-pool)

NODEPOOL="inferentia2-pool"

while [[ $# -gt 0 ]]; do
  case $1 in
    --nodepool)
      NODEPOOL="$2"
      shift 2
      ;;
    *)
      echo "Unknown option: $1"
      exit 1
      ;;
  esac
done

# ... rest of script

Documentation Style

Markdown:

  • Use clear, descriptive headings
  • Include code examples
  • Add command outputs (expected results)
  • Link to related documentation

Mermaid diagrams:

  • Keep diagrams focused (one concept per diagram)
  • Use consistent color coding (see DIAGRAMS.md)
  • Include legends for complex diagrams
  • Test rendering on https://mermaid.live

Example:

graph LR
    A[Neuron SDK] -->|Compile| B[NEFF Binary]
    B -->|Load| C[Neuron Runtime]
    C -->|Execute| D[NeuronCore]
    
    style A fill:#9370db
    style D fill:#ff9900
    
    %% Legend
    subgraph Legend
        SDK[Neuron SDK]
        HW[Hardware]
    end
Loading

Commit Messages

Follow conventional commits:

Format:

<type>(<scope>): <subject>

<body>

<footer>

Types:

  • feat: New feature
  • fix: Bug fix
  • docs: Documentation
  • style: Formatting
  • refactor: Code restructuring
  • test: Adding tests
  • chore: Maintenance

Examples:

feat(vllm): Add Mixtral 8x7B deployment example

- Add vllm-mixtral-8x7b-inf2.yaml
- Include resource requirements for MoE model
- Add benchmark results to inference-results.md

Closes #42

fix(karpenter): Correct NeuronCore limit in Inf2 NodePool

The NodePool was requesting 24 cores but Inf2.48xlarge only has 12.
Fixed limit to match hardware specification.

Fixes #56

Reporting Issues

Security Issues

Do NOT open public GitHub issues for security vulnerabilities.

Instead, email security details to: [your-email@example.com]

Include:

  • Description of the vulnerability
  • Steps to reproduce
  • Potential impact
  • Suggested fix (if any)

Bug Reports

Use the bug report template on GitHub.

Before submitting:

  1. Search existing issues
  2. Check troubleshooting guide
  3. Verify on latest version
  4. Collect relevant logs

Feature Requests

Use the feature request template on GitHub.

Before submitting:

  1. Search existing requests
  2. Check roadmap (if available)
  3. Describe use case clearly
  4. Propose implementation approach

Pull Request Process

1. Preparation

  • Fork the repository
  • Create a feature branch
  • Make your changes
  • Test locally
  • Write/update documentation
  • Commit with descriptive messages

2. Submission

Pull request should include:

  • Title: Clear description of changes
  • Description: What and why
  • Testing: How you tested changes
  • Checklist: All items checked
  • Screenshots: If UI changes
  • Related issues: Link to issues

Example description:

## Description
Add support for Mixtral 8x7B deployment on Inferentia2.

## Changes
- Created `deployments/vllm/vllm-mixtral-8x7b-inf2.yaml`
- Updated `benchmarks/inference-results.md` with Mixtral benchmarks
- Added Mixtral example to README.md

## Testing
- ✅ Tested deployment on EKS 1.28 with Inf2.48xlarge
- ✅ Validated throughput (2,800 tokens/s)
- ✅ Verified P99 latency (<200ms)
- ✅ YAML passes `yamllint`

## Checklist
- [x] My code follows the style guidelines
- [x] I have performed a self-review
- [x] I have commented my code
- [x] I have updated documentation
- [x] My changes generate no new warnings
- [x] I have tested on real infrastructure

## Related Issues
Closes #42

3. Review Process

  1. Automated checks: CI/CD must pass
  2. Maintainer review: Code review by maintainers
  3. Feedback: Address review comments
  4. Approval: Once approved, PR will be merged

Response time:

  • First response: Within 7 days
  • Review completion: Within 14 days

4. After Merge

  • Delete feature branch
  • Pull latest main
  • Celebrate! 🎉

Recognition

Contributors will be:

  • Listed in CHANGELOG.md
  • Mentioned in release notes (if applicable)
  • Added to contributors list (if significant contribution)

Questions?

  • GitHub Discussions: For general questions
  • GitHub Issues: For bugs and feature requests
  • Email: [your-email@example.com] for private inquiries

License

By contributing, you agree that your contributions will be licensed under the MIT License.

Thank you for contributing to AWS Neuron on EKS Guide! 🚀