diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index c88664c..ac0e602 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -18,6 +18,12 @@ Serverless AutoML platform with **split architecture**: **Key insight:** Containers ONLY for training - ML deps (265MB) exceed Lambda's 250MB limit. +**Critical architectural principle:** The training container is **fully autonomous** and stateless. It: +- Never calls the backend API +- Receives ALL context via environment variables only +- Writes results directly to DynamoDB and S3 +- Can be tested locally with `docker-compose --profile training run training` + ## Critical: Environment Variable Cascade Training container is **autonomous** - receives ALL context via environment variables, never calls the API: @@ -74,26 +80,75 @@ Located in `backend/training/`, runs as Docker container in AWS Batch: ## Development Commands +### Local Development (Docker Compose) + +```powershell +# 1. Configure backend environment +cp backend/.env.example backend/.env +# Edit backend/.env with values from: terraform output + +# 2. Start API (connects to dev AWS services) +docker-compose up + +# 3. Frontend (separate terminal) +cd frontend +cp .env.local.example .env.local +# Edit .env.local with NEXT_PUBLIC_API_URL +pnpm install && pnpm dev +``` + +### Backend Development + +```powershell +# Run API locally without Docker +cd backend +python -m venv venv +.\venv\Scripts\Activate.ps1 # Linux/Mac: source venv/bin/activate +pip install -r requirements.txt +uvicorn api.main:app --reload # API at http://localhost:8000/docs +``` + +### Training Container Testing + ```powershell -# Backend (local) - http://localhost:8000/docs -cd backend; uvicorn api.main:app --reload +# Test training locally with uploaded dataset +DATASET_ID=xxx TARGET_COLUMN=price docker-compose --profile training run training -# Frontend (local) - Set NEXT_PUBLIC_API_URL=http://localhost:8000 in .env.local -cd frontend; pnpm dev +# Or use helper script (requires dataset uploaded to dev) +python scripts/run-training-local.py --dataset-id xxx --target-column price --time-budget 120 +``` + +### Deployment -# Test training container locally (requires uploaded dataset) -docker-compose --profile training run training +```powershell +# Full infrastructure +cd infrastructure/terraform +terraform apply -# Deploy Lambda only -cd infrastructure/terraform; terraform apply -target=aws_lambda_function.api +# Deploy Lambda API only (fast iteration) +terraform apply -target=aws_lambda_function.api -# Deploy training container +# Build and deploy training container $EcrUrl = terraform output -raw ecr_repository_url +$Region = terraform output -raw aws_region +aws ecr get-login-password --region $Region | docker login --username AWS --password-stdin $($EcrUrl.Split('/')[0]) docker build -t automl-training:latest backend/training -docker tag automl-training:latest "$EcrUrl:latest"; docker push "$EcrUrl:latest" +docker tag automl-training:latest "$EcrUrl:latest" +docker push "$EcrUrl:latest" + +# Verify image in ECR +aws ecr describe-images --repository-name automl-lite-dev-training --image-ids imageTag=latest +``` +### Utilities + +```powershell # Generate architecture diagrams (requires: pip install diagrams + Graphviz) python scripts/generate_architecture_diagram.py + +# Make predictions with trained model +docker build -f scripts/Dockerfile.predict -t automl-predict . +docker run --rm -v ${PWD}:/data automl-predict /data/model.pkl --info ``` ## Common Pitfalls @@ -138,6 +193,58 @@ Backend Pydantic and Frontend TypeScript schemas must match. When adding fields: | `scripts/predict.py` | Make predictions with trained models (Docker) | | `scripts/generate_architecture_diagram.py` | Generate AWS architecture diagrams | +## Testing & Validation + +### API Testing + +```powershell +# Health check +curl $API_URL/health + +# Test upload endpoint +curl -X POST $API_URL/upload -H "Content-Type: application/json" -d '{"filename": "test.csv"}' + +# View API docs (Swagger UI) +# http://localhost:8000/docs (local) or $API_URL/docs (deployed) +``` + +### Container Testing + +```powershell +# Build training container locally +docker build -t automl-training:latest backend/training + +# Test with environment variables +docker run --rm \ + -e DATASET_ID=xxx \ + -e TARGET_COLUMN=price \ + -e JOB_ID=test-123 \ + -e TIME_BUDGET=60 \ + -e S3_BUCKET_DATASETS=automl-lite-dev-datasets-XXX \ + -e DYNAMODB_JOBS_TABLE=automl-lite-dev-training-jobs \ + -e REGION=us-east-1 \ + -v ~/.aws:/root/.aws:ro \ + automl-training:latest +``` + +### Frontend Testing + +```powershell +cd frontend +pnpm dev # Development server at http://localhost:3000 +pnpm build # Test production build +pnpm lint # ESLint check +``` + +### Terraform Validation + +```powershell +cd infrastructure/terraform +terraform fmt # Format files +terraform validate # Syntax check +terraform plan # Preview changes +``` + ## CI/CD Workflows (`.github/workflows/`) | Workflow | Trigger | Purpose | @@ -145,7 +252,14 @@ Backend Pydantic and Frontend TypeScript schemas must match. When adding fields: | `deploy-lambda-api.yml` | Push to main/dev | Deploy FastAPI to Lambda | | `deploy-training-container.yml` | Push to main/dev | Build & push training image to ECR | | `deploy-infrastructure.yml` | Manual | Terraform apply | +| `deploy-frontend.yml` | Push to main/dev (via Amplify) | Auto-deploy Next.js frontend | | `ci-terraform.yml` | PR | Terraform validate & plan | +| `destroy-environment.yml` | Manual | Destroy all infrastructure (requires confirmation) | + +**Branch Strategy:** +- `dev` โ†’ Deploy to dev environment (automl-lite-dev-*) +- `main` โ†’ Deploy to prod environment (automl-lite-prod-*) +- Feature branches โ†’ CI validation only (no deployment) ## Key Docs diff --git a/.github/git-commit-messages-instructions.md b/.github/git-commit-messages-instructions.md index fec4144..373fc83 100644 --- a/.github/git-commit-messages-instructions.md +++ b/.github/git-commit-messages-instructions.md @@ -4,6 +4,34 @@ Generate [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) **Tech Stack:** Next.js 16 + FastAPI + AWS Lambda + Terraform + AWS Batch + DynamoDB + S3 +## ๐Ÿ“‘ Table of Contents + +- [Commit Structure](#commit-structure) +- [Commit Types](#commit-types) + - [feat - New Functionality](#feat---new-functionality) + - [fix - Bug Resolution](#fix---bug-resolution) + - [refactor - Code Improvement](#refactor---code-improvement-same-behavior) + - [perf - Performance Optimization](#perf---performance-optimization) + - [docs - Documentation](#docs---documentation-only) + - [test - Test Code](#test---test-code) + - [build - Build System & Infrastructure](#build---build-system--infrastructure) + - [ci - CI/CD Pipelines](#ci---cicd-pipelines) + - [style - Formatting](#style---formatting-only) + - [chore - Miscellaneous](#chore---miscellaneous) + - [revert - Revert Commit](#revert---revert-commit) +- [Scopes](#scopes-project-specific) +- [Body Guidelines](#body-guidelines) +- [Breaking Changes](#breaking-changes) +- [Quick Decision Tree](#quick-decision-tree) +- [File Type Reference](#file-type-reference) +- [Examples](#examples) +- [Common Mistakes](#common-mistakes) +- [Checklist Before Committing](#checklist-before-committing) +- [Key Principles](#key-principles) +- [AWS AutoML Lite Specific Notes](#aws-automl-lite-specific-notes) + +--- + ## Commit Structure ``` @@ -156,51 +184,119 @@ Reason: Fargate doesn't support GPU, reverting to CPU. ## Scopes (Project-Specific) -**Backend layers:** -- `api`: FastAPI routers and main application -- `services`: S3, DynamoDB, Batch service layers -- `models`: Pydantic schemas -- `training`: ML training container code -- `utils`: Helper functions - -**Frontend:** -- `upload`: File upload flow -- `configure`: Training configuration page -- `history`: Training jobs history -- `results`: Model results display -- `components`: Reusable UI components -- `lib`: API client and utilities - -**Infrastructure:** -- `terraform`: Infrastructure as code -- `lambda`: Lambda function configuration -- `batch`: AWS Batch job definitions -- `dynamodb`: DynamoDB table schemas -- `s3`: S3 bucket configurations -- `iam`: IAM roles and policies -- `ecr`: Container registry - -**Cross-cutting:** -- `api/upload`: Upload router specifically -- `api/training`: Training router -- `training/eda`: EDA generation logic +**IMPORTANT:** Use full path context for clarity. Format: `area/component` or just `component` for obvious cases. + +### Backend Scopes + +**API Layer (`backend/api/`):** +- `backend/api` or `api`: General API changes +- `api/routers`: All routers +- `api/routers/upload`: Upload router specifically +- `api/routers/training`: Training router +- `api/routers/datasets`: Datasets router +- `api/routers/models`: Models router +- `api/services`: All services +- `api/services/s3`: S3 service only +- `api/services/dynamo`: DynamoDB service +- `api/services/batch`: Batch service +- `api/models`: Pydantic schemas +- `api/utils`: Helper functions + +**Training Container (`backend/training/`):** +- `backend/training` or `training`: General training changes +- `training/train`: Main training script - `training/preprocessor`: Data preprocessing - -Examples: +- `training/model_trainer`: FLAML AutoML trainer +- `training/eda`: EDA report generation +- `training/training_report`: Training report generation + +### Frontend Scopes + +**Pages (`frontend/app/`):** +- `frontend` or `frontend/app`: General frontend changes +- `frontend/upload`: Upload page (home) +- `frontend/configure`: Configuration page +- `frontend/training`: Training progress page +- `frontend/results`: Results page +- `frontend/history`: Training history page + +**Components & Utilities (`frontend/`):** +- `frontend/components`: All components +- `frontend/components/FileUpload`: FileUpload component +- `frontend/components/Header`: Header component +- `frontend/lib`: Utilities +- `frontend/lib/api`: API client +- `frontend/lib/utils`: Utility functions + +### Infrastructure Scopes + +**Terraform (`infrastructure/terraform/`):** +- `terraform`: General Terraform changes +- `terraform/lambda`: Lambda configuration +- `terraform/batch`: AWS Batch configuration +- `terraform/dynamodb`: DynamoDB tables +- `terraform/s3`: S3 buckets +- `terraform/iam`: IAM roles and policies +- `terraform/ecr`: ECR repository +- `terraform/amplify`: Amplify configuration +- `terraform/api-gateway`: API Gateway + +**CI/CD (`.github/workflows/`):** +- `ci`: General CI/CD changes +- `ci/deploy-infrastructure`: Infrastructure deployment workflow +- `ci/deploy-lambda`: Lambda deployment workflow +- `ci/deploy-training`: Training container workflow +- `ci/deploy-frontend`: Frontend deployment workflow +- `ci/terraform-validation`: Terraform validation workflow + +### Cross-Cutting Scopes + +When changes span multiple areas, use the most specific common scope: ``` feat(api/upload): add CSV validation before S3 upload fix(training/eda): resolve matplotlib memory leak -refactor(services/dynamo): consolidate query builders +refactor(api/services): consolidate query builders build(terraform/batch): increase job timeout to 60min +ci/deploy-lambda: add health check after deployment +``` + +### Scope Selection Guidelines + +**Use full path when:** +- Changes are specific to a subfolder +- Multiple similar components exist (e.g., multiple services) +- Context is needed for clarity + +**Examples:** ``` +โœ… feat(api/routers/upload): add file size validation +โœ… fix(training/preprocessor): handle missing values +โœ… refactor(frontend/lib/api): extract error handling +โœ… build(terraform/lambda): increase memory to 2GB +``` + +**Use short scope when:** +- Change affects entire area +- Context is obvious from description + +**Examples:** +``` +โœ… feat(api): add health check endpoint +โœ… fix(training): resolve memory overflow +โœ… refactor(frontend): standardize error messages +โœ… build(terraform): update to version 1.9 +``` + +### Omit Scope When -Omit scope for project-wide changes: +Changes affect the entire project: ``` -refactor: standardize error responses -style: format all Python files +refactor: standardize error responses across all services +style: format all Python files with Black docs: update architecture documentation +build: upgrade all dependencies to latest versions ``` --- @@ -307,21 +403,22 @@ All jobs created after this commit include new fields. ## File Type Reference -| File Pattern | Type | Example | -| -------------------------------------- | ------- | -------------------------------------- | -| `*.md` | `docs` | `docs: update QUICKSTART` | -| `.vscode/*`, `.editorconfig` | `chore` | `chore: configure Python linting` | -| `.github/workflows/*.yml` | `ci` | `ci: add Lambda deployment workflow` | -| `infrastructure/terraform/*.tf` | `build` | `build(terraform): add S3 bucket` | -| `backend/training/Dockerfile` | `build` | `build(docker): optimize image layers` | -| `backend/requirements.txt` | `build` | `build: upgrade FastAPI to 0.115` | -| `frontend/package.json` (deps) | `build` | `build: add chart.js for metrics` | -| `backend/api/routers/*.py` (new) | `feat` | `feat(api): add model download route` | -| `backend/training/*.py` (new logic) | `feat` | `feat(training): add FLAML estimator` | -| `frontend/app/**/page.tsx` (new) | `feat` | `feat(history): add jobs list page` | -| `frontend/components/*.tsx` (new) | `feat` | `feat(components): add FileUpload` | -| `backend/api/services/*.py` (bugfix) | `fix` | `fix(services): handle S3 exceptions` | -| `infrastructure/terraform/scripts/*.ps1` | `ci` | `ci: add automated deployment script` | +| File Pattern | Type | Example with Full Scope | +| -------------------------------------- | ------- | ------------------------------------------------------ | +| `*.md` | `docs` | `docs: update QUICKSTART` | +| `.vscode/*`, `.editorconfig` | `chore` | `chore: configure Python linting` | +| `.github/workflows/*.yml` | `ci` | `ci/deploy-lambda: add health check` | +| `infrastructure/terraform/*.tf` | `build` | `build(terraform/s3): add lifecycle policy` | +| `backend/training/Dockerfile` | `build` | `build(training): optimize Docker image layers` | +| `backend/requirements.txt` | `build` | `build(backend): upgrade FastAPI to 0.115` | +| `backend/training/requirements.txt` | `build` | `build(training): add sweetviz for EDA` | +| `frontend/package.json` (deps) | `build` | `build(frontend): add recharts for visualization` | +| `backend/api/routers/*.py` (new) | `feat` | `feat(api/routers/models): add download endpoint` | +| `backend/training/*.py` (new logic) | `feat` | `feat(training/model_trainer): add FLAML estimator` | +| `frontend/app/**/page.tsx` (new) | `feat` | `feat(frontend/history): add jobs list page` | +| `frontend/components/*.tsx` (new) | `feat` | `feat(frontend/components): add FileUpload component` | +| `backend/api/services/*.py` (bugfix) | `fix` | `fix(api/services/s3): handle upload exceptions` | +| `infrastructure/terraform/scripts/*.ps1` | `ci` | `ci: add automated deployment script` | --- @@ -330,24 +427,24 @@ All jobs created after this commit include new fields. ### Simple Commits ``` -feat(upload): add presigned S3 URL generation -feat(training): integrate FLAML AutoML -fix(batch): resolve container OOM errors -fix(api): correct DynamoDB pagination -refactor(services): extract S3 client logic -perf(lambda): reduce cold start time -test(training): add preprocessor tests +feat(api/routers/upload): add presigned S3 URL generation +feat(training/model_trainer): integrate FLAML AutoML +fix(terraform/batch): resolve container OOM errors +fix(api/services/dynamo): correct pagination logic +refactor(api/services): extract S3 client logic +perf(backend/api): reduce Lambda cold start time +test(training/preprocessor): add unit tests docs: update deployment guide -build(terraform): add ECR repository -build: upgrade FastAPI dependencies -ci: add ECR push automation -chore: update .gitignore for Python +build(terraform/ecr): add repository for training image +build(backend): upgrade FastAPI dependencies +ci/deploy-training: add ECR push automation +chore: update .gitignore for Python cache ``` ### With Body (Upload Feature) ``` -feat(upload): implement CSV upload with validation +feat(api/upload): implement CSV upload with validation Complete upload workflow with presigned URLs, validation, and metadata extraction for AutoML training. @@ -371,7 +468,7 @@ Features: ### Bug Fix (Training Container) ``` -fix(batch): resolve training container memory errors +fix(training): resolve training container memory errors Fixed OOM kills during FLAML training on large datasets. Increased Fargate memory from 2GB to 4GB and optimized @@ -394,7 +491,7 @@ Fixes #78 ### Performance (Lambda Optimization) ``` -perf(lambda): reduce package size by 40% +perf(backend/api): reduce Lambda package size by 40% Optimized Lambda deployment package by excluding training dependencies, reducing cold start from 3.5s to 2.1s. @@ -419,7 +516,7 @@ Refs: #45 ### Infrastructure (Batch Job) ``` -build(batch): configure Fargate Spot for training jobs +build(terraform/batch): configure Fargate Spot for training jobs Implemented AWS Batch with Fargate Spot compute for cost-effective AutoML training. 70% cost savings vs on-demand. @@ -455,30 +552,30 @@ Cost estimate: ~$0.34/month for 20 training jobs (Fargate compute only) ### Infrastructure ``` -โŒ feat(terraform): add S3 bucket โ†’ โœ… build(terraform): add S3 bucket -โŒ chore(batch): increase memory โ†’ โœ… build(batch): increase memory -โŒ fix(terraform): correct IAM policy โ†’ โœ… build(iam): fix Batch permissions +โŒ feat(terraform): add S3 bucket โ†’ โœ… build(terraform/s3): add datasets bucket +โŒ chore(batch): increase memory โ†’ โœ… build(terraform/batch): increase memory to 4GB +โŒ fix(terraform): correct IAM policy โ†’ โœ… build(terraform/iam): fix Batch execution permissions ``` ### CI/CD ``` -โŒ build(github-actions): add workflow โ†’ โœ… ci(github-actions): add workflow -โŒ feat(scripts): deployment script โ†’ โœ… ci(scripts): add deployment automation +โŒ build(github-actions): add workflow โ†’ โœ… ci/deploy-lambda: add deployment workflow +โŒ feat(scripts): deployment script โ†’ โœ… ci: add automated deployment script ``` ### Backend API ``` -โŒ refactor(api): add new endpoint โ†’ โœ… feat(api): add model download endpoint -โŒ feat: fix upload bug โ†’ โœ… fix(upload): resolve presigned URL expiry +โŒ refactor(api): add new endpoint โ†’ โœ… feat(api/routers/models): add download endpoint +โŒ feat: fix upload bug โ†’ โœ… fix(api/routers/upload): resolve presigned URL expiry ``` ### Training Code ``` -โŒ chore(training): add FLAML โ†’ โœ… feat(training): integrate FLAML AutoML -โŒ refactor: fix preprocessing โ†’ โœ… fix(training): correct target detection +โŒ chore(training): add FLAML โ†’ โœ… feat(training/model_trainer): integrate FLAML AutoML +โŒ refactor: fix preprocessing โ†’ โœ… fix(training/preprocessor): correct target detection logic ``` --- @@ -518,15 +615,15 @@ Cost estimate: ~$0.34/month for 20 training jobs (Fargate compute only) ### Lambda vs Batch Container -- **Lambda changes** (API, services, routers): Usually `feat` or `fix` -- **Training container** (Dockerfile, training/*.py): Usually `feat(training)` or `build(docker)` -- **Remember:** Lambda = direct code, Batch = Docker container +- **Lambda changes** (backend/api/): Usually `feat(api/...)` or `fix(api/...)` +- **Training container** (backend/training/): Usually `feat(training/...)` or `build(training)` +- **Remember:** Lambda = direct code (backend/api), Batch = Docker container (backend/training) ### Common Workflows **Upload flow:** ``` -feat(upload): add CSV upload workflow +feat(api/upload): add CSV upload workflow Modified files (4): - backend/api/routers/upload.py @@ -549,7 +646,7 @@ Modified files (5): **Frontend page:** ``` -feat(history): add training jobs history page +feat(frontend/history): add training jobs history page Modified files (3): - frontend/app/history/page.tsx @@ -567,7 +664,7 @@ Always mention cost impact for: Example: ``` -build(batch): increase training job timeout to 60min +build(terraform/batch): increase training job timeout to 60min Cost impact: ~$0.10/job โ†’ ~$0.15/job (longer runtime) Allows FLAML to explore more models for better accuracy. @@ -578,7 +675,7 @@ Allows FLAML to explore more models for better accuracy. When modifying environment variables: ``` -build(lambda): add S3_BUCKET_MODELS environment variable +build(terraform/lambda): add S3_BUCKET_MODELS environment variable Modified files (2): - infrastructure/terraform/lambda.tf: Added env var @@ -593,7 +690,7 @@ with default value fallback. When updating training container: ``` -build(docker): optimize training image size +build(training): optimize Docker image size Reduced image from 1.2GB to 850MB using multi-stage build and Alpine base image where possible. @@ -616,7 +713,7 @@ docker push $(terraform output -raw ecr_repository_url):latest When modifying API contracts: ``` -feat(api)!: add optional metadata to dataset schema +feat(api/models)!: add optional metadata to dataset schema BREAKING CHANGE: DatasetResponse now includes metadata field. @@ -640,7 +737,7 @@ Backward compatible: metadata defaults to null for old datasets. When adding new endpoints: ``` -feat(api): add model metrics endpoint +feat(api/routers/models): add model metrics endpoint Added /models/{model_id}/metrics for retrieving training performance metrics from DynamoDB. @@ -659,7 +756,7 @@ Returns: accuracy, precision, recall, f1_score, training_time When modifying table structures: ``` -build(terraform): add GSI for dataset queries by user +build(terraform/dynamodb): add GSI for dataset queries by user Added Global Secondary Index to support multi-tenant features for filtering datasets by user_id. diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..eccac92 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,82 @@ +# Changelog + +All notable changes to AWS AutoML Lite will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [Unreleased] + +### Changed +- Nothing yet - preparing for v1.1.0 + +## [1.0.0] - 2025-12-03 + +### Added +- Complete serverless AutoML platform on AWS +- FastAPI backend with Lambda deployment +- FLAML AutoML integration for automatic model training +- Next.js 16 frontend with SSR support via AWS Amplify +- AWS Batch + Fargate Spot for cost-effective training +- Automatic problem type detection (classification/regression) +- EDA report generation with Sweetviz +- Training history and job tracking with DynamoDB +- Model download and export (.pkl format) +- Docker-based prediction script +- CI/CD with GitHub Actions + OIDC +- Comprehensive documentation and architecture diagrams + +### Infrastructure +- S3 buckets for datasets, models, and reports +- DynamoDB tables for metadata and job tracking +- Lambda function for API endpoints (direct ZIP deployment) +- API Gateway for REST API +- AWS Batch with Fargate Spot for training jobs +- ECR repository for training container +- CloudWatch logging and monitoring +- AWS Amplify for frontend hosting + +### Features +- CSV file upload with drag & drop +- Auto-calculated time budget based on dataset size +- Smart column detection (ID columns automatically excluded) +- Feature importance visualization +- Training progress monitoring +- Portable model export for local use + +### Documentation +- Complete quickstart guide +- Architecture decision records +- Terraform best practices analysis +- Lessons learned document +- CI/CD setup guide with OIDC +- Git commit message conventions (713 lines) +- Table of contents for long documents +- CONTRIBUTING.md for collaboration +- CHANGELOG.md for version tracking +- Version badges in README + +### Cost Optimization +- ~$10-25/month total cost for moderate usage +- Fargate Spot pricing (70% discount) +- No always-on infrastructure +- Training cost: ~$0.02/job + +--- + +## Version History + +- **v1.0.0** (2025-12-03) - Initial release with full serverless architecture and comprehensive documentation + +--- + +## Contributing + +See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on how to contribute to this project. + +## Links + +- [Documentation](./docs/README.md) +- [Quickstart Guide](./docs/QUICKSTART.md) +- [Architecture Decisions](./infrastructure/terraform/ARCHITECTURE_DECISIONS.md) +- [Lessons Learned](./docs/LESSONS_LEARNED.md) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..8b3e1fe --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,146 @@ +# Contributing to AWS AutoML Lite + +Thanks for your interest in AWS AutoML Lite! This project is primarily educational and designed for individual use, but contributions are welcome. + +## ๐ŸŽฏ Project Purpose + +This is an **educational project** demonstrating serverless AutoML on AWS. Most users will: +- Clone and deploy to their own AWS account +- Use as a reference for their own projects +- Learn about serverless architecture patterns + +## ๐Ÿ”ง Quick Setup for Local Development + +### Prerequisites +- AWS Account with credentials configured +- Terraform >= 1.9 +- Docker installed +- Node.js 20+ +- Python 3.11+ + +### Local Development +```bash +# 1. Deploy infrastructure first +cd infrastructure/terraform +terraform init +terraform apply + +# 2. Run backend locally +cd ../../backend +cp .env.example .env +# Edit .env with your AWS resource names from terraform output +docker-compose up + +# 3. Run frontend locally (separate terminal) +cd frontend +cp .env.local.example .env.local +# Edit .env.local with API URL +pnpm install && pnpm dev +``` + +See [QUICKSTART.md](./docs/QUICKSTART.md) for full deployment instructions. + +## ๐Ÿ“ Commit Messages + +We use [Conventional Commits](https://www.conventionalcommits.org/). Please read [git-commit-messages-instructions.md](.github/git-commit-messages-instructions.md) for detailed guidelines. + +**Quick reference:** +```bash +feat(api): add new endpoint # New feature +fix(training): resolve memory issue # Bug fix +docs: update README # Documentation only +build(terraform): add S3 bucket # Infrastructure changes +ci: update GitHub Actions workflow # CI/CD changes +``` + +## ๐Ÿงช Testing Your Changes + +### Backend API +```bash +cd backend +pytest +# Or test locally with docker-compose +docker-compose up +``` + +### Frontend +```bash +cd frontend +pnpm lint +pnpm build +``` + +### Infrastructure +```bash +cd infrastructure/terraform +terraform fmt -recursive +terraform validate +terraform plan +``` + +## ๐Ÿ“‹ Pull Request Process + +1. **Fork the repository** +2. **Create a feature branch** from `dev` + ```bash + git checkout -b feat/your-feature-name + ``` +3. **Make your changes** following commit conventions +4. **Test locally** (deploy to your AWS account) +5. **Push to your fork** and create a PR to `dev` branch +6. **Wait for review** (this is a personal project, reviews may take time) + +## ๐Ÿšซ What We're NOT Looking For + +- Large architectural changes (this is an educational example) +- Additional ML frameworks (keep it simple with FLAML) +- Complex features that increase cost significantly +- Changes that require manual infrastructure setup + +## โœ… What We ARE Looking For + +- **Bug fixes** - especially in training container or data preprocessing +- **Documentation improvements** - typos, clarity, better examples +- **Cost optimizations** - ways to reduce AWS costs +- **Error handling** - better error messages and validation +- **Minor features** - small improvements that don't change architecture + +## ๐Ÿ“š Key Documentation + +Before contributing, please read: +- [copilot-instructions.md](.github/copilot-instructions.md) - Architecture patterns and guidelines +- [ARCHITECTURE_DECISIONS.md](infrastructure/terraform/ARCHITECTURE_DECISIONS.md) - Why containers only for training +- [LESSONS_LEARNED.md](docs/LESSONS_LEARNED.md) - Common pitfalls and solutions + +## ๐Ÿ› Reporting Issues + +If you find a bug: +1. Check [LESSONS_LEARNED.md](docs/LESSONS_LEARNED.md) first - it might be documented +2. Open an issue with: + - Clear description of the problem + - Steps to reproduce + - Expected vs actual behavior + - CloudWatch logs if available + - Your AWS region and Terraform version + +## ๐Ÿ’ก Suggesting Features + +For feature requests: +1. Open an issue first to discuss +2. Explain the use case +3. Consider the cost impact +4. Keep it aligned with the educational purpose + +## ๐Ÿ“„ License + +By contributing, you agree that your contributions will be licensed under the MIT License. + +## โ“ Questions? + +- Check [docs/README.md](docs/README.md) for documentation index +- Review [QUICKSTART.md](docs/QUICKSTART.md) for deployment help +- See [LESSONS_LEARNED.md](docs/LESSONS_LEARNED.md) for troubleshooting + +--- + +**Remember:** This is an educational project. The goal is simplicity and learning, not production-grade enterprise software. Keep contributions aligned with this philosophy. diff --git a/README.md b/README.md index 9452a25..42e7f28 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,12 @@ # AWS AutoML Lite +[![Terraform](https://img.shields.io/badge/Terraform-1.9+-623CE4?logo=terraform)](https://www.terraform.io/) +[![Python](https://img.shields.io/badge/Python-3.11+-3776AB?logo=python&logoColor=white)](https://www.python.org/) +[![Node.js](https://img.shields.io/badge/Node.js-20+-339933?logo=nodedotjs&logoColor=white)](https://nodejs.org/) +[![Next.js](https://img.shields.io/badge/Next.js-16-000000?logo=nextdotjs&logoColor=white)](https://nextjs.org/) +[![AWS](https://img.shields.io/badge/AWS-Serverless-FF9900?logo=amazonaws&logoColor=white)](https://aws.amazon.com/) +[![License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE) + A lightweight, cost-effective AutoML platform built on AWS serverless architecture. Upload CSV files, automatically detect problem types, and train machine learning models with just a few clicks. ## ๐Ÿ”„ CI/CD Status @@ -82,9 +89,9 @@ User โ†’ AWS Amplify (Frontend - Next.js SSR) - AWS Account - AWS CLI v2 configured -- Terraform >= 1.5 +- Terraform >= 1.9 - Docker installed -- Node.js 18+ (for frontend) +- Node.js 20+ (for frontend) - Python 3.11+ ## ๐Ÿš€ Quick Start @@ -128,6 +135,8 @@ terraform output api_gateway_url - [ARCHITECTURE_DECISIONS.md](./infrastructure/terraform/ARCHITECTURE_DECISIONS.md) - Container usage rationale - [LESSONS_LEARNED.md](./docs/LESSONS_LEARNED.md) - Challenges, solutions & best practices - [FRONTEND_DEPLOYMENT_ANALYSIS.md](./docs/FRONTEND_DEPLOYMENT_ANALYSIS.md) - Frontend deployment decision analysis +- [CONTRIBUTING.md](./CONTRIBUTING.md) - Contribution guidelines +- [CHANGELOG.md](./CHANGELOG.md) - Version history ## ๐Ÿ’ฐ Cost Estimation diff --git a/docs/LESSONS_LEARNED.md b/docs/LESSONS_LEARNED.md index 89ff452..d0076f2 100644 --- a/docs/LESSONS_LEARNED.md +++ b/docs/LESSONS_LEARNED.md @@ -1,5 +1,20 @@ # Lessons Learned +## ๐Ÿ“‘ Table of Contents + +- [Overview](#overview) +- [1. Docker & Container Management](#1-docker--container-management) +- [2. Environment Variables & Configuration](#2-environment-variables--configuration) +- [3. Machine Learning & Feature Engineering](#3-machine-learning--feature-engineering) +- [4. AWS Services Integration](#4-aws-services-integration) +- [5. Frontend Deployment Architecture](#5-frontend-deployment-architecture) +- [6. CI/CD & Automation](#6-cicd--automation) +- [7. Cost Optimization](#7-cost-optimization) +- [8. Development Workflow](#8-development-workflow) +- [Key Takeaways](#key-takeaways) + +--- + ## Overview This document captures key challenges, solutions, and architectural insights discovered during the development and enhancement of AWS AutoML Lite. These lessons are organized by category to help future developers avoid common pitfalls and understand critical design decisions. diff --git a/docs/PROJECT_REFERENCE.md b/docs/PROJECT_REFERENCE.md index cd69d99..15c2029 100644 --- a/docs/PROJECT_REFERENCE.md +++ b/docs/PROJECT_REFERENCE.md @@ -1,5 +1,22 @@ # AWS AutoML Lite - Project Reference +## ๐Ÿ“‘ Table of Contents + +- [Project Overview](#-project-overview) +- [Architecture](#๏ธ-architecture) +- [Technical Stack](#-technical-stack) +- [Project Structure](#-project-structure) +- [Data Flow](#-data-flow) +- [API Endpoints](#-api-endpoints) +- [Training Pipeline](#๏ธ-training-pipeline) +- [Frontend Pages](#-frontend-pages) +- [Infrastructure](#-infrastructure) +- [Cost Analysis](#-cost-analysis) +- [Development Status](#-development-status) +- [Next Steps](#-next-steps) + +--- + ## ๐Ÿ“‹ Project Overview **Goal:** Build a lightweight AutoML platform on AWS that allows users to upload CSV files, automatically detect problem types (classification/regression), perform EDA, train models, and maintain training history. diff --git a/docs/QUICKSTART.md b/docs/QUICKSTART.md index 85cb188..f9a4ea5 100644 --- a/docs/QUICKSTART.md +++ b/docs/QUICKSTART.md @@ -10,7 +10,7 @@ Before you begin, ensure you have: - โœ… **AWS Account** with administrative access - โœ… **AWS CLI** installed and configured ([Install Guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)) -- โœ… **Terraform** >= 1.5 installed ([Download](https://www.terraform.io/downloads)) +- โœ… **Terraform** >= 1.9 installed ([Download](https://www.terraform.io/downloads)) - โœ… **Docker** installed and running ([Get Docker](https://www.docker.com/get-started)) - *Only for training container* - โœ… **Git** installed diff --git a/docs/README.md b/docs/README.md index 1767b29..4ce7e22 100644 --- a/docs/README.md +++ b/docs/README.md @@ -22,6 +22,10 @@ Complete documentation for AWS AutoML Lite platform. - Data flows - Development status +### Project Management +- **[CHANGELOG.md](../CHANGELOG.md)** - Version history and notable changes +- **[CONTRIBUTING.md](../CONTRIBUTING.md)** - Contribution guidelines + ### Specialized Guides - **[SETUP_CICD.md](../.github/SETUP_CICD.md)** - CI/CD with GitHub Actions - OIDC setup diff --git a/frontend/README.md b/frontend/README.md index 7c261f4..368c9ab 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -6,7 +6,7 @@ Next.js 16 frontend for AWS AutoML Lite platform. ### Prerequisites -- Node.js 18+ installed +- Node.js 20+ installed - Backend API deployed and running ### Setup diff --git a/infrastructure/terraform/README.md b/infrastructure/terraform/README.md index 4eb6e0f..bc24bd1 100644 --- a/infrastructure/terraform/README.md +++ b/infrastructure/terraform/README.md @@ -4,7 +4,7 @@ Infrastructure as Code for AWS AutoML Lite platform. ## Prerequisites -- Terraform >= 1.5 +- Terraform >= 1.9 - AWS CLI configured - Docker (for training container) diff --git a/infrastructure/terraform/TERRAFORM_BEST_PRACTICES.md b/infrastructure/terraform/TERRAFORM_BEST_PRACTICES.md index f70aac8..ebb48a2 100644 --- a/infrastructure/terraform/TERRAFORM_BEST_PRACTICES.md +++ b/infrastructure/terraform/TERRAFORM_BEST_PRACTICES.md @@ -16,7 +16,7 @@ **Alignment with Workflows:** - โœ… All workflows correctly use workspace pattern: `select || new` -- โœ… Terraform version 1.5.0 consistent across all workflows +- โœ… Terraform version 1.9.8 consistent across all workflows - โœ… CI/CD validates format, init, and validate before deployment ---