diff --git a/docs/CLI_REFERENCE.md b/docs/CLI_REFERENCE.md new file mode 100644 index 0000000..d9edfbb --- /dev/null +++ b/docs/CLI_REFERENCE.md @@ -0,0 +1,723 @@ +# CLI Reference + +Complete reference for the `devstack` command-line interface. + +--- + +## Quick Reference Card + +```bash +# Lifecycle +./devstack start # Start all services +./devstack stop # Stop all services +./devstack restart # Restart services +./devstack reset # Delete everything + +# Status +./devstack status # VM and container status +./devstack health # Health check all services + +# Logs & Shell +./devstack logs [service] # View logs +./devstack logs -f postgres # Follow logs +./devstack shell mysql # Shell into container + +# Vault +./devstack vault-init # Initialize Vault +./devstack vault-bootstrap # Setup credentials +./devstack vault-status # Vault status +./devstack vault-show-password pg # Get password + +# Data +./devstack backup # Backup databases +./devstack restore # List/restore backups + +# Profiles +./devstack start --profile minimal # Minimal services +./devstack start --profile full # All services +./devstack profiles # List profiles +``` + +--- + +## Command Categories + +| Category | Commands | +|----------|----------| +| **Lifecycle** | `start`, `stop`, `restart`, `reset` | +| **Status** | `status`, `health`, `ip` | +| **Logs & Shell** | `logs`, `shell` | +| **Vault** | `vault-init`, `vault-unseal`, `vault-status`, `vault-token`, `vault-bootstrap`, `vault-ca-cert`, `vault-show-password` | +| **Data** | `backup`, `restore`, `verify` | +| **Setup** | `forgejo-init`, `redis-cluster-init`, `profiles` | + +--- + +## Lifecycle Commands + +### start + +Start Colima VM and Docker services. + +```bash +./devstack start [OPTIONS] +``` + +**Options:** +| Option | Description | Default | +|--------|-------------|---------| +| `--profile` | Service profile to start | `standard` | +| `--no-vm` | Don't start Colima VM (use existing) | false | + +**Examples:** +```bash +# Start with default profile (standard) +./devstack start + +# Start minimal services only +./devstack start --profile minimal + +# Start with monitoring +./devstack start --profile full + +# Combine profiles +./devstack start --profile standard --profile reference + +# Skip VM startup (already running) +./devstack start --no-vm +``` + +**What it does:** +1. Starts Colima VM (if not running) +2. Pulls Docker images (if needed) +3. Starts containers in dependency order +4. Waits for health checks + +--- + +### stop + +Stop Docker services and optionally Colima VM. + +```bash +./devstack stop [OPTIONS] +``` + +**Options:** +| Option | Description | Default | +|--------|-------------|---------| +| `--vm` | Also stop Colima VM | false | + +**Examples:** +```bash +# Stop containers only (VM keeps running) +./devstack stop + +# Stop everything including VM +./devstack stop --vm +``` + +**What it does:** +1. Stops all Docker containers +2. Preserves data volumes +3. Optionally stops Colima VM + +--- + +### restart + +Restart all Docker services without restarting VM. + +```bash +./devstack restart [OPTIONS] +``` + +**Options:** +| Option | Description | Default | +|--------|-------------|---------| +| `--profile` | Profile to restart with | current | + +**Examples:** +```bash +# Restart all services +./devstack restart + +# Restart with different profile +./devstack restart --profile minimal +``` + +--- + +### reset + +Completely reset and delete Colima VM. **DESTRUCTIVE!** + +```bash +./devstack reset [OPTIONS] +``` + +**Options:** +| Option | Description | Default | +|--------|-------------|---------| +| `--force` | Skip confirmation prompt | false | + +**Examples:** +```bash +# Reset with confirmation +./devstack reset + +# Reset without confirmation (scripts) +./devstack reset --force +``` + +**What it does:** +1. Stops all containers +2. Removes all containers +3. Removes all volumes (data deleted!) +4. Deletes Colima VM +5. Preserves Vault keys in `~/.config/vault/` + +**Warning:** This deletes all database data. Back up first with `./devstack backup`. + +--- + +## Status Commands + +### status + +Display Colima VM and service status. + +```bash +./devstack status +``` + +**Output shows:** +- Colima VM state (running/stopped) +- CPU, memory, disk allocation +- Container status for each service + +**Example output:** +``` +═══ DevStack Core - Status ═══ + +Colima VM Status: + Status: Running + CPU: 4 cores + Memory: 8 GB + Disk: 60 GB + +Container Status: + dev-vault running + dev-postgres running + dev-mysql running + ... +``` + +--- + +### health + +Check health status of all running services. + +```bash +./devstack health +``` + +**Output shows:** +- Service name +- Running status +- Health check result (healthy/unhealthy/starting) + +**Example output:** +``` + Service Health Status +╭───────────────┬─────────┬─────────╮ +│ Service │ Status │ Health │ +├───────────────┼─────────┼─────────┤ +│ vault │ running │ healthy │ +│ postgres │ running │ healthy │ +│ mysql │ running │ healthy │ +│ redis-1 │ running │ healthy │ +╰───────────────┴─────────┴─────────╯ +``` + +--- + +### ip + +Display Colima VM IP address. + +```bash +./devstack ip +``` + +**Example output:** +``` +192.168.106.2 +``` + +**Use case:** When you need to connect from another VM or container outside the Docker network. + +--- + +## Logs & Shell Commands + +### logs + +View logs for all services or a specific service. + +```bash +./devstack logs [SERVICE] [OPTIONS] +``` + +**Options:** +| Option | Description | Default | +|--------|-------------|---------| +| `-f`, `--follow` | Follow log output | false | +| `-n`, `--tail` | Number of lines | 100 | +| `--since` | Show logs since timestamp | all | + +**Examples:** +```bash +# View all service logs +./devstack logs + +# View specific service logs +./devstack logs postgres + +# Follow logs in real-time +./devstack logs -f postgres + +# Last 50 lines only +./devstack logs -n 50 vault + +# Logs from last hour +./devstack logs --since 1h redis-1 +``` + +--- + +### shell + +Open an interactive shell in a running container. + +```bash +./devstack shell SERVICE [OPTIONS] +``` + +**Options:** +| Option | Description | Default | +|--------|-------------|---------| +| `--user` | User to run shell as | container default | + +**Examples:** +```bash +# Shell into PostgreSQL container +./devstack shell postgres + +# Shell into MySQL as root +./devstack shell mysql --user root + +# Shell into Redis +./devstack shell redis-1 +``` + +**Common use cases:** +```bash +# PostgreSQL psql +./devstack shell postgres +$ psql -U devuser devdb + +# MySQL client +./devstack shell mysql +$ mysql -u devuser -p devdb + +# Redis CLI +./devstack shell redis-1 +$ redis-cli -a $REDIS_PASSWORD +``` + +--- + +## Vault Commands + +### vault-init + +Initialize and unseal Vault (first-time setup). + +```bash +./devstack vault-init +``` + +**What it does:** +1. Initializes Vault with 5 key shares, 3 threshold +2. Saves unseal keys to `~/.config/vault/keys.json` +3. Saves root token to `~/.config/vault/root-token` +4. Automatically unseals Vault + +**When to use:** First time after `./devstack reset` or fresh install. + +--- + +### vault-unseal + +Manually unseal Vault using stored keys. + +```bash +./devstack vault-unseal +``` + +**What it does:** +1. Reads keys from `~/.config/vault/keys.json` +2. Submits unseal keys until threshold reached +3. Vault becomes operational + +**When to use:** After Vault container restart if auto-unseal failed. + +--- + +### vault-status + +Display Vault seal status and token information. + +```bash +./devstack vault-status +``` + +**Example output:** +``` +═══ Vault Status ═══ + +Seal Status: + Sealed: false + Version: 1.18.5 + Cluster: vault-cluster-abc123 + +Root Token: + Stored at: ~/.config/vault/root-token + Token: hvs.CAESI... (truncated) +``` + +--- + +### vault-token + +Print Vault root token to stdout. + +```bash +./devstack vault-token +``` + +**Example output:** +``` +hvs.CAESIxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx +``` + +**Use case:** Piping to other commands: +```bash +export VAULT_TOKEN=$(./devstack vault-token) +vault kv list secret/ +``` + +--- + +### vault-bootstrap + +Bootstrap Vault with PKI and service credentials. + +```bash +./devstack vault-bootstrap [OPTIONS] +``` + +**Options:** +| Option | Description | Default | +|--------|-------------|---------| +| `--regenerate` | Regenerate all credentials | false | + +**What it does:** +1. Enables KV secrets engine +2. Sets up PKI (Root CA → Intermediate CA) +3. Creates certificate roles for each service +4. Generates and stores service passwords +5. Configures AppRole authentication +6. Exports CA certificates + +**When to use:** After `vault-init` or when credentials need regeneration. + +--- + +### vault-ca-cert + +Export Vault CA certificate chain to stdout. + +```bash +./devstack vault-ca-cert +``` + +**Output:** PEM-encoded certificate chain (Root CA + Intermediate CA) + +**Use case:** Adding to trust store: +```bash +./devstack vault-ca-cert > ca-chain.pem +``` + +--- + +### vault-show-password + +Retrieve and display service credentials from Vault. + +```bash +./devstack vault-show-password SERVICE +``` + +**Available services:** +- `postgres` or `postgresql` +- `mysql` +- `mongodb` or `mongo` +- `redis` +- `rabbitmq` + +**Examples:** +```bash +# Get PostgreSQL credentials +./devstack vault-show-password postgres + +# Get MySQL credentials +./devstack vault-show-password mysql + +# Get Redis password +./devstack vault-show-password redis +``` + +**Example output:** +``` +═══ PostgreSQL Credentials ═══ + +Username: devuser +Password: Hx7kL9mNpQr2sTuVwXyZ12345 +Database: devdb +Host: localhost +Port: 5432 + +Connection String: + postgresql://devuser:Hx7kL9mNpQr2sTuVwXyZ12345@localhost:5432/devdb +``` + +--- + +## Data Commands + +### backup + +Backup all service data to timestamped directory. + +```bash +./devstack backup [OPTIONS] +``` + +**Options:** +| Option | Description | Default | +|--------|-------------|---------| +| `--output` | Custom output directory | `./backups/YYYYMMDD_HHMMSS` | +| `--encrypt` | Encrypt backup with GPG | false | + +**Examples:** +```bash +# Create backup with timestamp +./devstack backup + +# Backup to specific directory +./devstack backup --output ./my-backup + +# Create encrypted backup +./devstack backup --encrypt +``` + +**What gets backed up:** +- PostgreSQL: Full database dump +- MySQL: Full database dump +- MongoDB: Archive dump +- Redis: RDB snapshot +- Forgejo: Git repositories +- Configuration files + +--- + +### restore + +Restore service data from backup. + +```bash +./devstack restore [BACKUP_ID] +``` + +**Examples:** +```bash +# List available backups +./devstack restore + +# Restore specific backup +./devstack restore 20250113_143022 +``` + +**List output:** +``` +═══ Available Backups ═══ + +╭─────────────────────┬──────────────┬──────────╮ +│ Backup ID │ Date │ Size │ +├─────────────────────┼──────────────┼──────────┤ +│ 20250113_143022 │ Jan 13, 2025 │ 156 MB │ +│ 20250112_091500 │ Jan 12, 2025 │ 148 MB │ +╰─────────────────────┴──────────────┴──────────╯ +``` + +--- + +### verify + +Verify backup integrity using checksums. + +```bash +./devstack verify BACKUP_ID +``` + +**Example:** +```bash +./devstack verify 20250113_143022 +``` + +**Output:** +``` +═══ Backup Verification ═══ + +Backup: 20250113_143022 +Status: Valid + +Files: + ✓ postgres_all.sql (sha256 verified) + ✓ mysql_all.sql (sha256 verified) + ✓ mongodb_dump.archive (sha256 verified) +``` + +--- + +## Setup Commands + +### forgejo-init + +Initialize Forgejo via automated bootstrap. + +```bash +./devstack forgejo-init +``` + +**What it does:** +1. Waits for Forgejo to be healthy +2. Creates initial admin user (if not exists) +3. Configures Forgejo settings + +**When to use:** After first start to set up Forgejo. + +--- + +### redis-cluster-init + +Initialize Redis cluster for standard/full profiles. + +```bash +./devstack redis-cluster-init +``` + +**What it does:** +1. Waits for all 3 Redis nodes +2. Creates cluster with `redis-cli --cluster create` +3. Assigns hash slots to each node + +**When to use:** After starting with `--profile standard` or `--profile full`. + +**Note:** Only needed once. Cluster configuration persists in volumes. + +--- + +### profiles + +List all available service profiles with details. + +```bash +./devstack profiles +``` + +**Output:** +``` +═══ Available Profiles ═══ + +╭───────────┬──────────┬─────────────────────────────────────╮ +│ Profile │ Services │ Description │ +├───────────┼──────────┼─────────────────────────────────────┤ +│ minimal │ 5 │ Core services (Vault, PG, Redis) │ +│ standard │ 10 │ Full dev stack + Redis cluster │ +│ full │ 18 │ Standard + observability │ +│ reference │ 5 │ Example APIs (combinable) │ +╰───────────┴──────────┴─────────────────────────────────────╯ +``` + +--- + +## Environment Variables + +The CLI respects these environment variables: + +| Variable | Description | Default | +|----------|-------------|---------| +| `COLIMA_PROFILE` | Colima profile name | `default` | +| `COLIMA_CPU` | VM CPU cores | `4` | +| `COLIMA_MEMORY` | VM memory (GB) | `8` | +| `COLIMA_DISK` | VM disk (GB) | `60` | +| `VAULT_ADDR` | Vault server address | `http://localhost:8200` | +| `VAULT_TOKEN` | Vault authentication token | (from file) | + +**Example:** +```bash +# Start with more resources +COLIMA_CPU=8 COLIMA_MEMORY=16 ./devstack start +``` + +--- + +## Exit Codes + +| Code | Meaning | +|------|---------| +| 0 | Success | +| 1 | General error | +| 2 | Invalid arguments | +| 3 | Service not running | +| 4 | Vault error | +| 5 | Docker error | + +--- + +## Getting Help + +```bash +# General help +./devstack --help + +# Command-specific help +./devstack start --help +./devstack vault-show-password --help + +# Version +./devstack --version +``` + +--- + +## See Also + +- [Getting Started](GETTING_STARTED.md) - Quick start guide +- [Quick Reference](QUICK_REFERENCE.md) - Cheat sheet +- [Troubleshooting](TROUBLESHOOTING.md) - Problem solving +- [Learning Paths](LEARNING_PATHS.md) - Guided learning diff --git a/docs/GETTING_STARTED.md b/docs/GETTING_STARTED.md new file mode 100644 index 0000000..f772466 --- /dev/null +++ b/docs/GETTING_STARTED.md @@ -0,0 +1,374 @@ +# Getting Started with DevStack Core + +> **Time to complete:** 5-10 minutes +> **Prerequisites:** macOS with Apple Silicon, terminal access +> **Result:** A complete local development infrastructure with databases, caching, and secrets management + +## What is DevStack Core? + +DevStack Core is a **local development infrastructure** that gives you production-like services on your Mac. Instead of installing databases and services individually, you get a complete stack with one command. + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ Your Development Mac │ +│ ┌───────────────────────────────────────────────────────────────┐ │ +│ │ DevStack Core (Colima VM) │ │ +│ │ │ │ +│ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ +│ │ │ Vault │ │Postgres │ │ MySQL │ │ MongoDB │ │ │ +│ │ │ Secrets │ │ DB │ │ DB │ │ DB │ │ │ +│ │ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │ │ +│ │ │ │ +│ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ +│ │ │ Redis │ │RabbitMQ │ │ Forgejo │ │ Grafana │ │ │ +│ │ │ Cluster │ │ Msgs │ │ Git │ │ Monitor │ │ │ +│ │ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │ │ +│ └───────────────────────────────────────────────────────────────┘ │ +│ │ +│ Your App ──────────────────────────────────────────────────────▶ │ +│ connects to localhost ports (5432, 3306, 6379, etc.) │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +### What You Get + +| Service | Purpose | Access | +|---------|---------|--------| +| **Vault** | Secrets & certificates | http://localhost:8200 | +| **PostgreSQL** | Relational database | localhost:5432 | +| **MySQL** | Relational database | localhost:3306 | +| **MongoDB** | Document database | localhost:27017 | +| **Redis Cluster** | Caching (3 nodes) | localhost:6379-6381 | +| **RabbitMQ** | Message queue | localhost:5672, http://localhost:15672 | +| **Forgejo** | Git hosting | http://localhost:3000 | +| **Grafana** | Monitoring dashboards | http://localhost:3001 | + +--- + +## Quick Start (5 Minutes) + +### Step 1: Install Prerequisites + +```bash +# Install Homebrew (if not installed) +/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" + +# Install required tools +brew install colima docker docker-compose uv +``` + +### Step 2: Clone and Setup + +```bash +# Clone the repository +git clone https://github.com/NormB/devstack-core.git +cd devstack-core + +# Create Python environment +uv venv && uv pip install -r scripts/requirements.txt + +# Copy environment configuration +cp .env.example .env +``` + +### Step 3: Start DevStack + +```bash +# Start all services (first time takes 2-3 minutes) +./devstack start +``` + +**What you should see:** +``` +═══ DevStack Core - Start Services ═══ + +Starting with profile(s): standard +✓ Colima VM started + +Starting Docker services... + Container dev-vault Started + Container dev-postgres Started + Container dev-mysql Started + Container dev-mongodb Started + Container dev-redis-1 Started + Container dev-redis-2 Started + Container dev-redis-3 Started + Container dev-rabbitmq Started + Container dev-forgejo Started + +✓ Services started successfully +``` + +### Step 4: Initialize Vault (First Time Only) + +```bash +# Initialize Vault (creates encryption keys) +./devstack vault-init + +# Bootstrap services with credentials +./devstack vault-bootstrap +``` + +**What you should see:** +``` +═══ Vault Initialization ═══ +✓ Vault initialized successfully +✓ Keys saved to ~/.config/vault/keys.json +✓ Root token saved to ~/.config/vault/root-token + +═══ Vault Bootstrap ═══ +✓ PKI engines configured +✓ Service credentials stored +✓ AppRole authentication configured +``` + +### Step 5: Verify Everything Works + +```bash +# Check all services are healthy +./devstack health +``` + +**What you should see:** +``` + Service Health Status +╭───────────────┬─────────┬─────────╮ +│ Service │ Status │ Health │ +├───────────────┼─────────┼─────────┤ +│ vault │ running │ healthy │ +│ postgres │ running │ healthy │ +│ mysql │ running │ healthy │ +│ mongodb │ running │ healthy │ +│ redis-1 │ running │ healthy │ +│ redis-2 │ running │ healthy │ +│ redis-3 │ running │ healthy │ +│ rabbitmq │ running │ healthy │ +│ forgejo │ running │ healthy │ +╰───────────────┴─────────┴─────────╯ +``` + +--- + +## Connect Your Application + +### Get Database Credentials + +DevStack stores all passwords in Vault. Get them with: + +```bash +# Get PostgreSQL password +./devstack vault-show-password postgres + +# Get all credentials as environment variables +source scripts/load-vault-env.sh +echo $POSTGRES_PASSWORD # Now available in your shell +``` + +### Connection Examples + +**PostgreSQL (Python)** +```python +import psycopg2 + +conn = psycopg2.connect( + host="localhost", + port=5432, + database="devdb", + user="devuser", + password="" +) +``` + +**MySQL (Node.js)** +```javascript +const mysql = require('mysql2'); + +const connection = mysql.createConnection({ + host: 'localhost', + port: 3306, + user: 'devuser', + password: '', + database: 'devdb' +}); +``` + +**Redis (Go)** +```go +import "github.com/redis/go-redis/v9" + +rdb := redis.NewClient(&redis.Options{ + Addr: "localhost:6379", + Password: "", +}) +``` + +**MongoDB (Rust)** +```rust +use mongodb::Client; + +let uri = "mongodb://devuser:@localhost:27017/devdb"; +let client = Client::with_uri_str(uri).await?; +``` + +--- + +## Daily Workflow + +### Starting Your Day +```bash +cd ~/devstack-core +./devstack start # Start all services +./devstack health # Verify everything is healthy +``` + +### During Development +```bash +./devstack logs postgres # View PostgreSQL logs +./devstack shell mysql # Get shell inside MySQL container +./devstack status # Quick status check +``` + +### End of Day +```bash +./devstack stop # Stop all services (data preserved) +``` + +--- + +## Choosing a Profile + +DevStack has different profiles for different needs: + +| Profile | RAM | Services | Use Case | +|---------|-----|----------|----------| +| **minimal** | 2GB | Vault, PostgreSQL, Redis, Forgejo | Light development | +| **standard** | 4GB | + MySQL, MongoDB, RabbitMQ, Redis cluster | Full development | +| **full** | 6GB | + Prometheus, Grafana, Loki, Vector | With monitoring | +| **reference** | +1GB | + Example APIs (5 languages) | Learning/testing | + +```bash +# Start with minimal profile +./devstack start --profile minimal + +# Start with full monitoring +./devstack start --profile full + +# Combine profiles +./devstack start --profile standard --profile reference +``` + +--- + +## Common Tasks + +### View Logs +```bash +./devstack logs # All services +./devstack logs postgres # Specific service +./devstack logs -f redis-1 # Follow logs in real-time +``` + +### Restart a Service +```bash +./devstack restart postgres +``` + +### Reset Everything +```bash +./devstack reset # Stops and removes all containers +./devstack start # Fresh start +./devstack vault-init # Re-initialize Vault +./devstack vault-bootstrap # Re-create credentials +``` + +### Backup Data +```bash +./devstack backup # Creates timestamped backup +./devstack restore # Lists available backups +./devstack restore 20250113_143022 # Restore specific backup +``` + +--- + +## Web Interfaces + +After starting DevStack, you can access these web UIs: + +| Service | URL | Default Login | +|---------|-----|---------------| +| **Vault** | http://localhost:8200 | Token from `~/.config/vault/root-token` | +| **RabbitMQ** | http://localhost:15672 | `./devstack vault-show-password rabbitmq` | +| **Forgejo** | http://localhost:3000 | Create account on first visit | +| **Grafana** | http://localhost:3001 | admin / admin | +| **Prometheus** | http://localhost:9090 | No auth required | + +--- + +## Troubleshooting + +### Services Won't Start + +```bash +# Check if Colima VM is running +colima status + +# If not running, start it +./devstack start + +# Check service logs for errors +./devstack logs vault +``` + +### Database Connection Refused + +```bash +# Verify service is healthy +./devstack health + +# Check the port is accessible +nc -zv localhost 5432 + +# Get the correct password +./devstack vault-show-password postgres +``` + +### Vault is Sealed + +```bash +# Check Vault status +./devstack vault-status + +# Unseal if needed +./devstack vault-unseal +``` + +### Need to Start Fresh + +```bash +# Complete reset (preserves Vault keys) +./devstack reset +./devstack start +./devstack vault-bootstrap +``` + +--- + +## Next Steps + +Now that DevStack is running: + +1. **Learn the CLI** - See [CLI Reference](CLI_REFERENCE.md) for all commands +2. **Understand the architecture** - Read [Architecture](ARCHITECTURE.md) +3. **Connect your app** - See [Use Cases](USE_CASES.md) for examples +4. **Customize profiles** - See [Service Profiles](SERVICE_PROFILES.md) +5. **Set up monitoring** - Start with `--profile full` and open Grafana + +--- + +## Getting Help + +- **Quick reference:** `./devstack --help` +- **Service status:** `./devstack health` +- **Logs:** `./devstack logs ` +- **Documentation:** See [docs/README.md](README.md) for full documentation +- **Issues:** https://github.com/NormB/devstack-core/issues diff --git a/docs/GLOSSARY.md b/docs/GLOSSARY.md new file mode 100644 index 0000000..63fa32d --- /dev/null +++ b/docs/GLOSSARY.md @@ -0,0 +1,346 @@ +# Glossary + +A plain-language explanation of terms and concepts used in DevStack Core. + +--- + +## Core Concepts + +### DevStack Core +A local development infrastructure that provides production-like services (databases, caching, secrets management) on your Mac using Docker containers managed by Colima. + +### Colima +A lightweight virtual machine (VM) manager for macOS that runs Docker containers. Think of it as a lightweight alternative to Docker Desktop that's optimized for Apple Silicon Macs. + +**Why Colima instead of Docker Desktop?** +- Lower memory usage +- Better performance on Apple Silicon +- Open source with no licensing restrictions +- Command-line focused + +### Container +A lightweight, isolated environment that runs a single application. Unlike virtual machines, containers share the host operating system kernel, making them faster to start and more efficient. + +**Example:** The `dev-postgres` container runs PostgreSQL in isolation from other services. + +### Docker Compose +A tool for defining and running multi-container applications. DevStack uses a `docker-compose.yml` file to define all services and their configurations. + +### Profile +A named group of services to start together. Profiles let you run only what you need: + +| Profile | What it includes | +|---------|------------------| +| minimal | Core services only (Vault, PostgreSQL, single Redis, Forgejo) | +| standard | All databases + Redis cluster + RabbitMQ | +| full | Standard + monitoring stack (Prometheus, Grafana, Loki) | +| reference | Example API applications in 5 languages | + +--- + +## Vault Concepts + +### HashiCorp Vault +A secrets management tool that securely stores and controls access to passwords, API keys, certificates, and other sensitive data. DevStack uses Vault instead of putting passwords in configuration files. + +**Why use Vault?** +- Passwords are never stored in plain text files +- Credentials can be rotated without changing config files +- Access can be audited and controlled +- Certificates can be automatically generated + +### Secrets Engine +A Vault component that stores or generates secrets. DevStack uses: + +| Engine | Purpose | +|--------|---------| +| **KV (Key-Value)** | Stores database passwords and credentials | +| **PKI** | Generates TLS certificates | + +### Unsealing +When Vault starts, it's in a "sealed" state and cannot read its data. Unsealing is the process of providing keys to decrypt Vault's storage. + +**Analogy:** Think of Vault as a bank vault. Even if you have the door key (root token), you first need to unlock multiple safety deposit boxes (unseal keys) before the vault door will open. + +``` +Sealed Vault ──[Unseal Keys]──▶ Unsealed Vault ──[Token]──▶ Access Secrets +``` + +### Root Token +The most powerful authentication credential for Vault. Like a master key, it can access everything. DevStack stores this in `~/.config/vault/root-token`. + +**Important:** The root token is for administrative tasks. Services use AppRole for day-to-day access. + +### AppRole +A Vault authentication method designed for machines and applications. Each service gets its own AppRole identity with limited permissions. + +**How it works:** +``` +┌─────────────┐ ┌──────────────┐ ┌─────────────┐ +│ Service │────▶│ role_id + │────▶│ Vault │ +│ (PostgreSQL)│ │ secret_id │ │ │ +└─────────────┘ └──────────────┘ └─────────────┘ + │ │ + │ ▼ + │ ┌─────────────┐ + └─────────────▶│ Temp Token │ + │ (1hr TTL) │ + └─────────────┘ +``` + +**Why AppRole instead of Root Token?** +- Limited permissions (can only access specific secrets) +- Temporary tokens that expire +- Auditable access +- No long-lived credentials to steal + +### PKI (Public Key Infrastructure) +A system for creating and managing digital certificates used for TLS/SSL encryption. DevStack uses a two-tier PKI: + +``` +┌─────────────────────────────────────────────────────┐ +│ Root CA (10 years) │ +│ Self-signed, stored securely │ +└────────────────────────┬────────────────────────────┘ + │ signs + ▼ +┌─────────────────────────────────────────────────────┐ +│ Intermediate CA (5 years) │ +│ Used for issuing service certs │ +└────────────────────────┬────────────────────────────┘ + │ signs + ┌────────────────┼────────────────┐ + ▼ ▼ ▼ +┌──────────────┐ ┌──────────────┐ ┌──────────────┐ +│ PostgreSQL │ │ MySQL │ │ Redis │ +│ Cert (1yr) │ │ Cert (1yr) │ │ Cert (1yr) │ +└──────────────┘ └──────────────┘ └──────────────┘ +``` + +--- + +## Database Concepts + +### PostgreSQL +An advanced open-source relational database. Often called "Postgres." Used for structured data with relationships (users, orders, products). + +**DevStack access:** `localhost:5432` + +### MySQL +A popular open-source relational database. Often used for web applications and content management systems. + +**DevStack access:** `localhost:3306` + +### MongoDB +A document database that stores data in flexible, JSON-like documents. Good for unstructured or rapidly changing data. + +**DevStack access:** `localhost:27017` + +### PgBouncer +A connection pooler for PostgreSQL. It maintains a pool of database connections and shares them among clients, reducing the overhead of opening new connections. + +**Why use PgBouncer?** +- Faster connection times +- Handles more concurrent connections +- Reduces database server load + +**DevStack access:** `localhost:6432` (connects to PostgreSQL on your behalf) + +--- + +## Redis Concepts + +### Redis +An in-memory data store used for caching, session storage, and message queuing. Extremely fast because data is stored in RAM. + +### Redis Cluster +Multiple Redis nodes working together to provide: +- **Sharding:** Data is automatically split across nodes +- **High availability:** If one node fails, others continue working +- **Scalability:** Add more nodes for more capacity + +DevStack runs 3 Redis nodes: +``` +┌─────────────────────────────────────────────────────┐ +│ Redis Cluster │ +│ ┌───────────┐ ┌───────────┐ ┌───────────┐ │ +│ │ redis-1 │ │ redis-2 │ │ redis-3 │ │ +│ │ :6379 │ │ :6380 │ │ :6381 │ │ +│ │ slots │ │ slots │ │ slots │ │ +│ │ 0-5460 │ │ 5461-10922│ │10923-16383│ │ +│ └───────────┘ └───────────┘ └───────────┘ │ +└─────────────────────────────────────────────────────┘ +``` + +### Hash Slots +Redis Cluster divides the keyspace into 16,384 hash slots. Each key belongs to a specific slot, and each node is responsible for a range of slots. + +**Example:** When you store a key, Redis calculates which slot it belongs to: +``` +key "user:1000" → slot 15495 → stored on redis-3 +key "session:abc" → slot 8234 → stored on redis-2 +``` + +--- + +## Messaging Concepts + +### RabbitMQ +A message broker that enables applications to communicate asynchronously. One application sends a message to a queue, and another application processes it later. + +**Use cases:** +- Background job processing +- Microservice communication +- Event-driven architectures + +**DevStack access:** +- AMQP: `localhost:5672` +- Management UI: `localhost:15672` + +### Queue +A buffer that holds messages until they're processed. Messages are typically processed in order (FIFO: First In, First Out). + +### Exchange +A RabbitMQ component that receives messages and routes them to queues based on routing rules. + +--- + +## Networking Concepts + +### TLS (Transport Layer Security) +Encryption protocol that protects data in transit. Previously known as SSL. When you see "HTTPS" or a padlock in your browser, that's TLS. + +DevStack can encrypt connections between services using TLS certificates generated by Vault. + +### Dual-Mode TLS +DevStack services accept both encrypted (TLS) and unencrypted connections. This makes development easier while still supporting secure connections. + +| Port | Type | +|------|------| +| 5432 | PostgreSQL (TLS optional) | +| 6379 | Redis (no TLS) | +| 6380 | Redis (TLS required) | + +### Network Segmentation +DevStack divides services into separate networks for security: + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ DevStack Networks │ +├─────────────────────────────────────────────────────────────────┤ +│ vault-network (172.20.1.0/24) │ +│ └── Vault server only │ +├─────────────────────────────────────────────────────────────────┤ +│ data-network (172.20.2.0/24) │ +│ └── PostgreSQL, MySQL, MongoDB, Redis, RabbitMQ │ +├─────────────────────────────────────────────────────────────────┤ +│ app-network (172.20.3.0/24) │ +│ └── Forgejo, Reference APIs │ +├─────────────────────────────────────────────────────────────────┤ +│ observability-network (172.20.4.0/24) │ +│ └── Prometheus, Grafana, Loki, Vector │ +└─────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Monitoring Concepts + +### Prometheus +A monitoring system that collects metrics from services. It scrapes (pulls) metrics from HTTP endpoints and stores them in a time-series database. + +**DevStack access:** `localhost:9090` + +### Grafana +A visualization platform for creating dashboards from metrics data. Connects to Prometheus (and other data sources) to display charts, graphs, and alerts. + +**DevStack access:** `localhost:3001` + +### Loki +A log aggregation system (like Prometheus, but for logs). Collects logs from all services for centralized searching and analysis. + +### Vector +A data pipeline tool that collects, transforms, and routes logs and metrics. In DevStack, Vector collects container logs and sends them to Loki. + +### Metrics +Numerical measurements collected over time. Examples: +- CPU usage percentage +- Memory consumption +- Request count +- Response time + +### cAdvisor +Container Advisor - a tool that monitors container resource usage and performance. Provides CPU, memory, and network statistics for each container. + +--- + +## Git & Development Concepts + +### Forgejo +A self-hosted Git server (like GitHub, but running locally). Fork of Gitea. Use it to store code repositories locally during development. + +**DevStack access:** `localhost:3000` + +### Reference Apps +Example API implementations in 5 programming languages (Python/FastAPI, Go, Node.js, Rust, TypeScript) that demonstrate how to integrate with DevStack services. + +--- + +## DevStack CLI Commands + +| Command | Description | +|---------|-------------| +| `./devstack start` | Start all services | +| `./devstack stop` | Stop all services | +| `./devstack restart` | Restart all services | +| `./devstack status` | Show VM and container status | +| `./devstack health` | Check service health | +| `./devstack logs ` | View service logs | +| `./devstack shell ` | Open shell in container | +| `./devstack vault-init` | Initialize Vault | +| `./devstack vault-bootstrap` | Configure Vault with credentials | +| `./devstack vault-show-password ` | Get service password | +| `./devstack backup` | Backup all databases | +| `./devstack restore` | Restore from backup | +| `./devstack reset` | Remove all containers | + +--- + +## File Locations + +| Path | Contents | +|------|----------| +| `~/.config/vault/` | Vault keys, tokens, certificates | +| `~/.config/vault/root-token` | Vault root token | +| `~/.config/vault/keys.json` | Vault unseal keys | +| `~/.config/vault/certs/` | Service TLS certificates | +| `~/devstack-core/.env` | Environment configuration | +| `~/devstack-core/configs/` | Service configuration files | +| `~/devstack-core/backups/` | Database backups | + +--- + +## Common Abbreviations + +| Abbreviation | Meaning | +|--------------|---------| +| **API** | Application Programming Interface | +| **CA** | Certificate Authority | +| **CLI** | Command Line Interface | +| **DB** | Database | +| **FQDN** | Fully Qualified Domain Name | +| **KV** | Key-Value (storage) | +| **PKI** | Public Key Infrastructure | +| **TLS** | Transport Layer Security | +| **TTL** | Time To Live | +| **UI** | User Interface | +| **VM** | Virtual Machine | + +--- + +## See Also + +- [Getting Started](GETTING_STARTED.md) - Quick start guide +- [Architecture](ARCHITECTURE.md) - System design +- [CLI Reference](CLI_REFERENCE.md) - All commands +- [Learning Paths](LEARNING_PATHS.md) - Guides by goal diff --git a/docs/LEARNING_PATHS.md b/docs/LEARNING_PATHS.md new file mode 100644 index 0000000..a1b5a65 --- /dev/null +++ b/docs/LEARNING_PATHS.md @@ -0,0 +1,470 @@ +# Learning Paths + +Choose a learning path based on your role or goal. Each path provides a structured reading order with the knowledge you'll gain at each step. + +--- + +## Quick Reference + +| Your Goal | Start Here | Time | +|-----------|------------|------| +| Get running ASAP | [Just Start Developing](#path-1-just-start-developing) | 15 min | +| Understand the system | [Understanding DevStack](#path-2-understanding-devstack) | 1 hour | +| Connect my app | [Application Developer](#path-3-application-developer) | 30 min | +| Learn Vault secrets | [Vault Deep Dive](#path-4-vault-deep-dive) | 45 min | +| Set up monitoring | [Observability](#path-5-observability) | 30 min | +| Debug issues | [Troubleshooting](#path-6-troubleshooting) | As needed | + +--- + +## Path 1: Just Start Developing + +**Goal:** Get DevStack running and connect your application as quickly as possible. + +**Time:** 15 minutes + +### Step 1: Install and Start (5 min) +📖 Read: [Getting Started](GETTING_STARTED.md) + +**You'll learn:** +- How to install prerequisites +- How to start all services +- How to verify everything works + +### Step 2: Get Credentials (2 min) +📖 Read: [Getting Started - Connect Your Application](GETTING_STARTED.md#connect-your-application) + +**You'll learn:** +- How to get database passwords +- Connection strings for each database + +### Step 3: Daily Commands (3 min) +📖 Read: [Quick Reference](QUICK_REFERENCE.md) (first page only) + +**You'll learn:** +- Start/stop commands +- How to view logs +- Basic troubleshooting + +### Step 4: Keep This Open +📖 Bookmark: [CLI Reference](CLI_REFERENCE.md) + +**For quick command lookup while developing.** + +--- + +## Path 2: Understanding DevStack + +**Goal:** Understand how DevStack works before using it. + +**Time:** 1 hour + +### Step 1: What It Is (10 min) +📖 Read: [Getting Started](GETTING_STARTED.md) +📖 Read: [Glossary](GLOSSARY.md) - scan key terms + +**You'll learn:** +- What DevStack provides +- Key terminology (Vault, AppRole, Colima) +- High-level architecture + +### Step 2: How It's Built (15 min) +📖 Read: [Architecture - Visual Overview](ARCHITECTURE.md#visual-architecture-overview) +📖 Read: [Architecture - Network Architecture](ARCHITECTURE.md#network-architecture) + +**You'll learn:** +- Service layout and dependencies +- Network segmentation (4 networks) +- How services communicate + +### Step 3: How Secrets Work (15 min) +📖 Read: [Architecture - Security Architecture](ARCHITECTURE.md#security-architecture) +📖 Read: [Glossary - Vault Concepts](GLOSSARY.md#vault-concepts) + +**You'll learn:** +- Why Vault instead of .env files +- How services get credentials +- Certificate management (PKI) + +### Step 4: Service Profiles (10 min) +📖 Read: [Service Profiles](SERVICE_PROFILES.md) + +**You'll learn:** +- Different profile options +- Resource requirements +- When to use each profile + +### Step 5: Hands-On (10 min) +📖 Follow: [Getting Started - Quick Start](GETTING_STARTED.md#quick-start-5-minutes) + +**You'll do:** +- Install and start DevStack +- Initialize Vault +- Verify services are healthy + +--- + +## Path 3: Application Developer + +**Goal:** Connect your application to DevStack services. + +**Time:** 30 minutes + +### Step 1: Get Running (10 min) +📖 Follow: [Getting Started - Quick Start](GETTING_STARTED.md#quick-start-5-minutes) + +**You'll do:** +- Start DevStack +- Initialize Vault + +### Step 2: Get Credentials (5 min) +📖 Read: [Getting Started - Get Database Credentials](GETTING_STARTED.md#get-database-credentials) + +**You'll learn:** +- How to get passwords from Vault +- How to load credentials into environment + +**Commands you'll use:** +```bash +./devstack vault-show-password postgres +source scripts/load-vault-env.sh +``` + +### Step 3: Connection Examples (10 min) +📖 Read: [Use Cases](USE_CASES.md) + +**You'll learn:** +- Connection strings for each language +- Code examples for Python, Go, Node.js, Rust +- How to test connections + +### Step 4: Reference Implementations (5 min) +📖 Explore: `reference-apps/` directory + +**You'll see:** +- Working examples in 5 languages +- Best practices for Vault integration +- Health check implementations + +**Explore in browser:** +- http://localhost:8000/docs (FastAPI docs) +- http://localhost:8000/health/all (Health check) + +--- + +## Path 4: Vault Deep Dive + +**Goal:** Understand how HashiCorp Vault manages secrets and certificates. + +**Time:** 45 minutes + +### Step 1: Vault Basics (10 min) +📖 Read: [Glossary - Vault Concepts](GLOSSARY.md#vault-concepts) + +**You'll learn:** +- What Vault is and why it's used +- Key concepts: unsealing, tokens, secrets engines +- AppRole authentication + +### Step 2: Vault Architecture (15 min) +📖 Read: [Vault](VAULT.md) + +**You'll learn:** +- PKI hierarchy (Root CA → Intermediate CA → Certs) +- Secrets engine structure +- How credentials are stored + +### Step 3: AppRole Deep Dive (10 min) +📖 Read: [Architecture - AppRole Authentication Flow](ARCHITECTURE.md#approle-authentication-flow) + +**You'll learn:** +- How services authenticate +- Role and secret IDs +- Token lifecycle + +### Step 4: Hands-On Exploration (10 min) +📖 Follow: These commands + +```bash +# View Vault status +./devstack vault-status + +# Get a password +./devstack vault-show-password postgres + +# Access Vault UI +open http://localhost:8200 +# Token: cat ~/.config/vault/root-token + +# List secrets +export VAULT_ADDR=http://localhost:8200 +export VAULT_TOKEN=$(cat ~/.config/vault/root-token) +vault kv list secret/ +``` + +### Step 5: Security Best Practices (Optional) +📖 Read: [Vault Security](VAULT_SECURITY.md) + +**You'll learn:** +- Production hardening recommendations +- Policy management +- Audit logging + +--- + +## Path 5: Observability + +**Goal:** Set up and use monitoring with Prometheus and Grafana. + +**Time:** 30 minutes + +### Step 1: Start with Full Profile (5 min) +📖 Follow: These commands + +```bash +# Start with observability services +./devstack start --profile full + +# Verify observability services are healthy +./devstack health | grep -E "(prometheus|grafana|loki)" +``` + +### Step 2: Explore Grafana (10 min) +📖 Open: http://localhost:3001 + +**Login:** admin / admin + +**Explore:** +1. **Dashboards** → Browse pre-configured dashboards +2. **Explore** → Query metrics with PromQL +3. **Explore** → Switch to Loki for logs + +### Step 3: Understand the Stack (10 min) +📖 Read: [Architecture - Observability Architecture](ARCHITECTURE.md#observability-architecture) + +**You'll learn:** +- Metrics flow (services → Prometheus → Grafana) +- Logs flow (containers → Promtail → Loki → Grafana) +- What metrics are available + +### Step 4: Query Examples (5 min) +📖 Try in Grafana: These PromQL queries + +```promql +# CPU usage by container +rate(container_cpu_usage_seconds_total[5m]) + +# Memory usage +container_memory_usage_bytes + +# HTTP request rate (FastAPI) +rate(http_requests_total[5m]) + +# Redis operations +rate(redis_commands_total[5m]) +``` + +### Step 5: Log Exploration +📖 Try in Grafana: Explore → Loki + +```logql +# All PostgreSQL logs +{container="dev-postgres"} + +# Errors only +{container="dev-reference-api"} |= "error" + +# JSON log parsing +{container="dev-reference-api"} | json | level="ERROR" +``` + +--- + +## Path 6: Troubleshooting + +**Goal:** Diagnose and fix common problems. + +**Time:** As needed + +### Quick Diagnostic Commands + +```bash +# Overall status +./devstack health + +# Service logs +./devstack logs + +# Vault status +./devstack vault-status + +# Container status +docker ps -a +``` + +### Problem-Specific Guides + +| Problem | Go To | +|---------|-------| +| Services won't start | [Troubleshooting - Services](TROUBLESHOOTING.md#services-wont-start) | +| Database connection failed | [Troubleshooting - Databases](TROUBLESHOOTING.md#database-issues) | +| Vault is sealed | [Troubleshooting - Vault](TROUBLESHOOTING.md#vault-issues) | +| Container keeps restarting | [Troubleshooting - Containers](TROUBLESHOOTING.md#container-issues) | +| Out of disk space | [Troubleshooting - Resources](TROUBLESHOOTING.md#resource-issues) | + +### Visual Troubleshooting Flow + +``` +Start Here + │ + ▼ +┌─────────────────────────────────┐ +│ Run: ./devstack health │ +└─────────────────────────────────┘ + │ + ├─── All healthy? ───▶ Problem is elsewhere + │ + ▼ +┌─────────────────────────────────┐ +│ Which service is unhealthy? │ +└─────────────────────────────────┘ + │ + ├─── Vault ───▶ ./devstack vault-status + │ └─── Sealed? ───▶ ./devstack vault-unseal + │ + ├─── Database ───▶ ./devstack logs + │ └─── Check credentials + │ + └─── Other ───▶ ./devstack logs + └─── Check dependencies +``` + +--- + +## Path 7: DevOps / Operations + +**Goal:** Manage DevStack in a team environment. + +**Time:** 1 hour + +### Step 1: Complete Setup Understanding (20 min) +📖 Follow: [Path 2: Understanding DevStack](#path-2-understanding-devstack) + +### Step 2: Backup and Recovery (15 min) +📖 Read: [Disaster Recovery](DISASTER_RECOVERY.md) + +**You'll learn:** +- Backup procedures +- Restore procedures +- RTO and RPO targets + +**Commands:** +```bash +# Backup all databases +./devstack backup + +# List backups +./devstack restore + +# Restore specific backup +./devstack restore 20250113_143022 +``` + +### Step 3: Certificate Management (10 min) +📖 Read: [TLS Certificate Management](TLS_CERTIFICATE_MANAGEMENT.md) + +**You'll learn:** +- Certificate lifecycle +- Renewal procedures +- Monitoring expiration + +### Step 4: Performance Tuning (10 min) +📖 Read: [Performance Tuning](PERFORMANCE_TUNING.md) + +**You'll learn:** +- Resource allocation +- Colima VM sizing +- Optimization strategies + +### Step 5: Upgrade Procedures (5 min) +📖 Read: [Upgrade Guide](UPGRADE_GUIDE.md) + +**You'll learn:** +- Version migration steps +- Breaking changes +- Rollback procedures + +--- + +## Path 8: Contributing + +**Goal:** Contribute to DevStack Core development. + +**Time:** 30 minutes + +### Step 1: Development Setup +📖 Follow: [Path 1: Just Start Developing](#path-1-just-start-developing) + +### Step 2: Contribution Guidelines +📖 Read: [Contributing](../CONTRIBUTING.md) + +**You'll learn:** +- Code style requirements +- PR process +- Testing requirements + +### Step 3: Architecture Understanding +📖 Read: [Architecture](ARCHITECTURE.md) (full document) + +### Step 4: Testing +📖 Read: [Testing Approach](TESTING_APPROACH.md) + +**Commands:** +```bash +# Run all tests +./tests/run-all-tests.sh + +# Run specific test +./tests/test-vault.sh + +# Run with parallel execution +./tests/run-all-tests.sh --parallel +``` + +--- + +## Documentation Index + +For complete documentation, see [docs/README.md](README.md). + +### By Topic + +| Topic | Documents | +|-------|-----------| +| **Getting Started** | [Getting Started](GETTING_STARTED.md), [Installation](INSTALLATION.md) | +| **CLI** | [CLI Reference](CLI_REFERENCE.md), [Quick Reference](QUICK_REFERENCE.md) | +| **Architecture** | [Architecture](ARCHITECTURE.md), [Network Segmentation](NETWORK_SEGMENTATION.md) | +| **Vault** | [Vault](VAULT.md), [Vault Security](VAULT_SECURITY.md) | +| **Databases** | [Services](SERVICES.md), [Redis](REDIS.md) | +| **Monitoring** | [Observability](OBSERVABILITY.md) | +| **Operations** | [Management](MANAGEMENT.md), [Disaster Recovery](DISASTER_RECOVERY.md) | +| **Troubleshooting** | [Troubleshooting](TROUBLESHOOTING.md), [FAQ](FAQ.md) | +| **Reference** | [Glossary](GLOSSARY.md), [Environment Variables](ENVIRONMENT_VARIABLES.md) | + +--- + +## Next Steps After Any Path + +1. **Bookmark frequently used docs:** + - [CLI Reference](CLI_REFERENCE.md) + - [Quick Reference](QUICK_REFERENCE.md) + - [Troubleshooting](TROUBLESHOOTING.md) + +2. **Join the community:** + - GitHub Issues for questions + - PRs for contributions + +3. **Keep learning:** + - Explore reference apps in `reference-apps/` + - Try different profiles + - Set up monitoring with full profile diff --git a/docs/README.md b/docs/README.md index f6fe548..a9144d8 100644 --- a/docs/README.md +++ b/docs/README.md @@ -33,6 +33,35 @@ This directory contains comprehensive documentation for the DevStack Core projec ## Documentation Index +### New to DevStack? Start Here + +- **[GETTING_STARTED.md](./GETTING_STARTED.md)** - Quick start guide (5-10 minutes) + - Install prerequisites and start DevStack + - Get database credentials from Vault + - Connect your application + - Daily workflow commands + +- **[GLOSSARY.md](./GLOSSARY.md)** - Terms and concepts explained + - Plain-language definitions for all key terms + - Vault, AppRole, PKI, Redis cluster concepts + - Service descriptions and common abbreviations + +- **[LEARNING_PATHS.md](./LEARNING_PATHS.md)** - Guided learning by goal + - Choose a path based on your role or objective + - Step-by-step reading order with time estimates + - Paths for beginners, developers, DevOps, and contributors + +- **[USE_CASES.md](./USE_CASES.md)** - Task-driven walkthroughs + - Connect Python/Go/Node.js apps to databases + - Use Vault secrets in your application + - Set up monitoring with Grafana + - Run integration tests against DevStack + +- **[CLI_REFERENCE.md](./CLI_REFERENCE.md)** - Complete CLI command reference + - All `./devstack` commands with examples + - Options, arguments, and exit codes + - Quick reference card for common tasks + ### Project Information - **[ACKNOWLEDGEMENTS.md](./ACKNOWLEDGEMENTS.md)** - Software acknowledgements and licenses diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index c59ca88..2991f51 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -22,6 +22,131 @@ Comprehensive troubleshooting guide for common issues in the DevStack Core infra --- +## Visual Troubleshooting Guides + +Use these focused flowcharts to quickly diagnose common issues. + +### Services Won't Start + +```mermaid +flowchart TD + A[Services won't start] --> B{Is Colima running?} + B -->|No| C[Run: ./devstack start] + B -->|Yes| D{Is Vault healthy?} + + D -->|No| E{Is Vault sealed?} + E -->|Yes| F[Run: ./devstack vault-unseal] + E -->|No| G[Check: ./devstack logs vault] + + D -->|Yes| H{Which service fails?} + H --> I[Check: ./devstack logs SERVICE] + + style A fill:#ffcdd2 + style C fill:#c8e6c9 + style F fill:#c8e6c9 +``` + +### Database Connection Failed + +```mermaid +flowchart TD + A[Can't connect to database] --> B{Is container running?} + B -->|No| C[Run: ./devstack start] + B -->|Yes| D{Is it healthy?} + + D -->|No| E[Check logs: ./devstack logs postgres] + D -->|Yes| F{Using correct password?} + + F -->|Not sure| G[Get password: ./devstack vault-show-password postgres] + F -->|Yes| H{Correct port?} + + H -->|Not sure| I["PostgreSQL: 5432\nMySQL: 3306\nMongoDB: 27017"] + + style A fill:#ffcdd2 + style C fill:#c8e6c9 + style G fill:#c8e6c9 +``` + +### Vault is Sealed + +```mermaid +flowchart TD + A[Vault is sealed] --> B{Do keys exist?} + B -->|Yes| C[Run: ./devstack vault-unseal] + B -->|No| D{Fresh install?} + + D -->|Yes| E[Run: ./devstack vault-init] + D -->|No| F[Keys lost - must reset] + + F --> G[./devstack reset] + G --> H[./devstack start] + H --> I[./devstack vault-init] + I --> J[./devstack vault-bootstrap] + + style A fill:#ffcdd2 + style C fill:#c8e6c9 + style E fill:#c8e6c9 +``` + +### Redis Cluster Not Working + +```mermaid +flowchart TD + A[Redis cluster issues] --> B{Which profile?} + B -->|minimal| C[Minimal uses single Redis - no cluster] + B -->|standard/full| D{All 3 nodes running?} + + D -->|No| E[./devstack start --profile standard] + D -->|Yes| F{Cluster initialized?} + + F -->|No| G[Run: ./devstack redis-cluster-init] + F -->|Yes| H[Check: redis-cli cluster info] + + style A fill:#ffcdd2 + style C fill:#fff9c4 + style G fill:#c8e6c9 +``` + +### Container Keeps Restarting + +```mermaid +flowchart TD + A[Container restarting] --> B[Check logs first] + B --> C[./devstack logs SERVICE] + + C --> D{Error type?} + D -->|Permission denied| E[Volume permission issue] + D -->|Connection refused| F[Dependency not ready] + D -->|Out of memory| G[Increase Colima memory] + + E --> H[./devstack reset and restart] + F --> I[Wait or restart dependencies] + G --> J["COLIMA_MEMORY=12 ./devstack start"] + + style A fill:#ffcdd2 + style H fill:#c8e6c9 + style I fill:#c8e6c9 + style J fill:#c8e6c9 +``` + +### Quick Diagnostic Commands + +``` +┌─────────────────────────────────────────────────────────────┐ +│ QUICK DIAGNOSTICS │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ Overall status: ./devstack health │ +│ Service logs: ./devstack logs SERVICE │ +│ Vault status: ./devstack vault-status │ +│ Container status: docker ps -a │ +│ Resource usage: docker stats │ +│ │ +└─────────────────────────────────────────────────────────────┘ +``` + +--- + ## Service Profile Issues (NEW v1.3) ### Issue: Wrong Services Starting diff --git a/docs/USE_CASES.md b/docs/USE_CASES.md new file mode 100644 index 0000000..8ea6b87 --- /dev/null +++ b/docs/USE_CASES.md @@ -0,0 +1,723 @@ +# Use Cases + +Step-by-step walkthroughs for common development tasks. Each use case shows exactly what you need to accomplish a specific goal. + +--- + +## Quick Links + +| Task | Time | +|------|------| +| [Connect a Python app to PostgreSQL](#use-case-1-connect-python-app-to-postgresql) | 5 min | +| [Connect a Go app to Redis](#use-case-2-connect-go-app-to-redis) | 5 min | +| [Connect a Node.js app to RabbitMQ](#use-case-3-connect-nodejs-app-to-rabbitmq) | 5 min | +| [Set up local development database](#use-case-4-set-up-local-development-database) | 10 min | +| [Use Vault secrets in your app](#use-case-5-use-vault-secrets-in-your-app) | 10 min | +| [Monitor your app with Grafana](#use-case-6-monitor-your-app-with-grafana) | 15 min | + +--- + +## Use Case 1: Connect Python App to PostgreSQL + +**Goal:** Connect a Python (FastAPI/Flask) application to the PostgreSQL database. + +### What You're Building + +``` +┌──────────────┐ ┌──────────────┐ ┌──────────────┐ +│ Your App │────▶│ DevStack │────▶│ PostgreSQL │ +│ (Python) │ │ localhost │ │ :5432 │ +└──────────────┘ └──────────────┘ └──────────────┘ +``` + +### Step 1: Start DevStack + +```bash +cd ~/devstack-core +./devstack start +./devstack health # Verify postgres is healthy +``` + +### Step 2: Get Credentials + +```bash +./devstack vault-show-password postgres +``` + +**Output:** +``` +Username: devuser +Password: Hx7kL9mNpQr2sTuVwXyZ12345 +Database: devdb +Host: localhost +Port: 5432 +``` + +### Step 3: Install Python Dependencies + +```bash +pip install psycopg2-binary sqlalchemy +``` + +### Step 4: Connect in Your Code + +**Option A: Direct Connection (psycopg2)** +```python +import psycopg2 + +conn = psycopg2.connect( + host="localhost", + port=5432, + database="devdb", + user="devuser", + password="Hx7kL9mNpQr2sTuVwXyZ12345" # From vault-show-password +) + +cursor = conn.cursor() +cursor.execute("SELECT version();") +print(cursor.fetchone()) +conn.close() +``` + +**Option B: SQLAlchemy** +```python +from sqlalchemy import create_engine + +DATABASE_URL = "postgresql://devuser:Hx7kL9mNpQr2sTuVwXyZ12345@localhost:5432/devdb" +engine = create_engine(DATABASE_URL) + +with engine.connect() as conn: + result = conn.execute("SELECT 1") + print(result.fetchone()) +``` + +**Option C: FastAPI with Environment Variables** +```python +import os +from sqlalchemy import create_engine + +# Load from environment (set via: source scripts/load-vault-env.sh) +DATABASE_URL = os.environ.get( + "DATABASE_URL", + f"postgresql://{os.environ['POSTGRES_USER']}:{os.environ['POSTGRES_PASSWORD']}@localhost:5432/{os.environ['POSTGRES_DB']}" +) + +engine = create_engine(DATABASE_URL) +``` + +### Step 5: Test Connection + +```bash +python -c " +import psycopg2 +conn = psycopg2.connect(host='localhost', port=5432, database='devdb', user='devuser', password='YOUR_PASSWORD') +print('Connected successfully!') +conn.close() +" +``` + +--- + +## Use Case 2: Connect Go App to Redis + +**Goal:** Connect a Go application to the Redis cluster. + +### What You're Building + +``` +┌──────────────┐ ┌──────────────┐ +│ Your App │────▶│ Redis Cluster│ +│ (Go) │ │ :6379-6381 │ +└──────────────┘ └──────────────┘ + 3 nodes, auto-sharding +``` + +### Step 1: Ensure Redis Cluster is Running + +```bash +./devstack start --profile standard +./devstack redis-cluster-init # Only needed once +./devstack health | grep redis +``` + +### Step 2: Get Redis Password + +```bash +./devstack vault-show-password redis +``` + +### Step 3: Install Go Redis Client + +```bash +go get github.com/redis/go-redis/v9 +``` + +### Step 4: Connect in Your Code + +**Single Node Connection** +```go +package main + +import ( + "context" + "fmt" + "github.com/redis/go-redis/v9" +) + +func main() { + ctx := context.Background() + + rdb := redis.NewClient(&redis.Options{ + Addr: "localhost:6379", + Password: "YOUR_REDIS_PASSWORD", // From vault-show-password + }) + + // Test connection + pong, err := rdb.Ping(ctx).Result() + if err != nil { + panic(err) + } + fmt.Println("Connected:", pong) + + // Set and get + rdb.Set(ctx, "key", "value", 0) + val, _ := rdb.Get(ctx, "key").Result() + fmt.Println("key:", val) +} +``` + +**Cluster Connection** +```go +package main + +import ( + "context" + "github.com/redis/go-redis/v9" +) + +func main() { + ctx := context.Background() + + rdb := redis.NewClusterClient(&redis.ClusterOptions{ + Addrs: []string{ + "localhost:6379", + "localhost:6380", + "localhost:6381", + }, + Password: "YOUR_REDIS_PASSWORD", + }) + + // Cluster automatically routes keys to correct node + rdb.Set(ctx, "user:1", "Alice", 0) + rdb.Set(ctx, "user:2", "Bob", 0) // May go to different node +} +``` + +--- + +## Use Case 3: Connect Node.js App to RabbitMQ + +**Goal:** Connect a Node.js application to RabbitMQ for message queuing. + +### What You're Building + +``` +┌──────────────┐ ┌──────────────┐ ┌──────────────┐ +│ Producer │──── send ───▶│ RabbitMQ │◀── receive ──│ Consumer │ +│ (Your App) │ │ :5672 │ │ (Your App) │ +└──────────────┘ └──────────────┘ └──────────────┘ +``` + +### Step 1: Get RabbitMQ Credentials + +```bash +./devstack vault-show-password rabbitmq +``` + +### Step 2: Install amqplib + +```bash +npm install amqplib +``` + +### Step 3: Producer Code + +```javascript +const amqp = require('amqplib'); + +async function sendMessage() { + // Connect to RabbitMQ + const connection = await amqp.connect({ + hostname: 'localhost', + port: 5672, + username: 'devuser', + password: 'YOUR_RABBITMQ_PASSWORD', // From vault-show-password + vhost: 'dev_vhost' + }); + + const channel = await connection.createChannel(); + const queue = 'my_queue'; + + // Create queue if it doesn't exist + await channel.assertQueue(queue, { durable: true }); + + // Send message + const message = { task: 'process_order', orderId: 123 }; + channel.sendToQueue(queue, Buffer.from(JSON.stringify(message))); + console.log('Sent:', message); + + await channel.close(); + await connection.close(); +} + +sendMessage(); +``` + +### Step 4: Consumer Code + +```javascript +const amqp = require('amqplib'); + +async function consumeMessages() { + const connection = await amqp.connect({ + hostname: 'localhost', + port: 5672, + username: 'devuser', + password: 'YOUR_RABBITMQ_PASSWORD', + vhost: 'dev_vhost' + }); + + const channel = await connection.createChannel(); + const queue = 'my_queue'; + + await channel.assertQueue(queue, { durable: true }); + console.log('Waiting for messages...'); + + channel.consume(queue, (msg) => { + const content = JSON.parse(msg.content.toString()); + console.log('Received:', content); + channel.ack(msg); // Acknowledge message + }); +} + +consumeMessages(); +``` + +### Step 5: Monitor in RabbitMQ UI + +Open http://localhost:15672 and login with the credentials from `vault-show-password rabbitmq`. + +--- + +## Use Case 4: Set Up Local Development Database + +**Goal:** Create a fresh database for your project with proper isolation. + +### The Setup + +``` +┌─────────────────────────────────────────────────┐ +│ DevStack PostgreSQL │ +│ │ +│ ┌─────────────┐ ┌─────────────┐ │ +│ │ devdb │ │ myproject │ ◀── New DB │ +│ │ (default) │ │ (yours) │ │ +│ └─────────────┘ └─────────────┘ │ +└─────────────────────────────────────────────────┘ +``` + +### Step 1: Connect to PostgreSQL + +```bash +./devstack shell postgres +``` + +### Step 2: Create Your Database + +```sql +-- Inside postgres container +psql -U devuser devdb + +-- Create new database +CREATE DATABASE myproject; + +-- Create project-specific user (optional) +CREATE USER myproject_user WITH PASSWORD 'mypassword'; +GRANT ALL PRIVILEGES ON DATABASE myproject TO myproject_user; + +-- Verify +\l -- List databases +``` + +### Step 3: Connect from Your App + +``` +postgresql://myproject_user:mypassword@localhost:5432/myproject +``` + +### Alternative: Use the Default Database + +For simpler setups, just use the default `devdb` database: + +```bash +# Get default credentials +./devstack vault-show-password postgres + +# Connection string +postgresql://devuser:PASSWORD@localhost:5432/devdb +``` + +--- + +## Use Case 5: Use Vault Secrets in Your App + +**Goal:** Fetch secrets from Vault at runtime instead of hardcoding. + +### The Pattern + +``` +┌──────────────┐ ┌──────────────┐ ┌──────────────┐ +│ Your App │────▶│ Vault │────▶│ Secrets │ +│ starts │ │ :8200 │ │ (passwords) │ +└──────────────┘ └──────────────┘ └──────────────┘ + │ │ + └─────────────── uses ──────────────────┘ +``` + +### Option A: Environment Variables (Simplest) + +```bash +# Load all secrets into shell environment +source scripts/load-vault-env.sh + +# Now use in your app +echo $POSTGRES_PASSWORD +echo $MYSQL_PASSWORD +echo $REDIS_PASSWORD +``` + +### Option B: Fetch at Runtime (Python) + +```python +import hvac # pip install hvac + +# Initialize Vault client +client = hvac.Client(url='http://localhost:8200') + +# Read token (for development) +with open('/Users/YOU/.config/vault/root-token') as f: + client.token = f.read().strip() + +# Fetch PostgreSQL credentials +secret = client.secrets.kv.v2.read_secret_version(path='postgres') +credentials = secret['data']['data'] + +print(f"Username: {credentials['user']}") +print(f"Password: {credentials['password']}") +print(f"Database: {credentials['database']}") +``` + +### Option C: Fetch at Runtime (Go) + +```go +package main + +import ( + "fmt" + vault "github.com/hashicorp/vault/api" + "io/ioutil" + "strings" +) + +func main() { + // Create Vault client + client, _ := vault.NewClient(&vault.Config{ + Address: "http://localhost:8200", + }) + + // Read token + token, _ := ioutil.ReadFile("/Users/YOU/.config/vault/root-token") + client.SetToken(strings.TrimSpace(string(token))) + + // Fetch secret + secret, _ := client.KVv2("secret").Get(nil, "postgres") + data := secret.Data + + fmt.Printf("User: %s\n", data["user"]) + fmt.Printf("Password: %s\n", data["password"]) +} +``` + +### Option D: Docker Container with Vault + +For apps running in Docker alongside DevStack: + +```yaml +# docker-compose.yml for your app +services: + myapp: + build: . + environment: + VAULT_ADDR: http://vault:8200 + VAULT_TOKEN: ${VAULT_TOKEN} + networks: + - devstack-core_vault-network + depends_on: + vault: + condition: service_healthy + +networks: + devstack-core_vault-network: + external: true +``` + +--- + +## Use Case 6: Monitor Your App with Grafana + +**Goal:** Add custom metrics to your app and visualize them in Grafana. + +### The Monitoring Stack + +``` +┌──────────────┐ ┌──────────────┐ ┌──────────────┐ +│ Your App │────▶│ Prometheus │────▶│ Grafana │ +│ /metrics │ │ :9090 │ │ :3001 │ +└──────────────┘ └──────────────┘ └──────────────┘ + exposes scrapes visualizes +``` + +### Step 1: Start Full Profile + +```bash +./devstack start --profile full +./devstack health | grep -E "(prometheus|grafana)" +``` + +### Step 2: Add Metrics to Your Python App + +```python +from prometheus_client import Counter, Histogram, generate_latest +from flask import Flask, Response + +app = Flask(__name__) + +# Define metrics +REQUEST_COUNT = Counter( + 'myapp_requests_total', + 'Total requests', + ['method', 'endpoint'] +) + +REQUEST_LATENCY = Histogram( + 'myapp_request_latency_seconds', + 'Request latency', + ['endpoint'] +) + +@app.route('/api/users') +def get_users(): + REQUEST_COUNT.labels(method='GET', endpoint='/api/users').inc() + with REQUEST_LATENCY.labels(endpoint='/api/users').time(): + # Your logic here + return {"users": []} + +@app.route('/metrics') +def metrics(): + return Response(generate_latest(), mimetype='text/plain') +``` + +### Step 3: Configure Prometheus to Scrape Your App + +Add to `configs/prometheus/prometheus.yml`: + +```yaml +scrape_configs: + - job_name: 'myapp' + static_configs: + - targets: ['host.docker.internal:5000'] # Your app port +``` + +Restart Prometheus: +```bash +docker compose restart prometheus +``` + +### Step 4: Create Grafana Dashboard + +1. Open http://localhost:3001 (admin/admin) +2. Go to Dashboards → New Dashboard +3. Add a panel +4. Use these PromQL queries: + +**Request Rate:** +```promql +rate(myapp_requests_total[5m]) +``` + +**Latency (p95):** +```promql +histogram_quantile(0.95, rate(myapp_request_latency_seconds_bucket[5m])) +``` + +**Request Count by Endpoint:** +```promql +sum by (endpoint) (rate(myapp_requests_total[5m])) +``` + +### Step 5: View in Grafana + +``` +┌─────────────────────────────────────────────────────────┐ +│ My Application Metrics │ +├─────────────────────────────────────────────────────────┤ +│ ┌─────────────────┐ ┌─────────────────┐ │ +│ │ Request Rate │ │ Latency (p95) │ │ +│ │ ▄▄▄▄ │ │ ▁▂▃▄ │ │ +│ │ ▄█████▄ │ │ ▁▂███▃▁ │ │ +│ │ 50 req/s │ │ 120ms │ │ +│ └─────────────────┘ └─────────────────┘ │ +└─────────────────────────────────────────────────────────┘ +``` + +--- + +## Use Case 7: Run Integration Tests Against DevStack + +**Goal:** Use DevStack as the backend for your integration tests. + +### Test Setup Pattern + +``` +┌──────────────────────────────────────────────────────────┐ +│ Test Execution │ +│ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ +│ │ Setup │────▶│ Test │────▶│ Teardown │ │ +│ │ (DevStack) │ │ (Your │ │ (cleanup) │ │ +│ │ │ │ tests) │ │ │ │ +│ └────────────┘ └────────────┘ └────────────┘ │ +└──────────────────────────────────────────────────────────┘ +``` + +### Step 1: Test Setup Script + +```bash +#!/bin/bash +# test-setup.sh + +# Start DevStack if not running +./devstack start --profile standard + +# Wait for healthy +./devstack health || exit 1 + +# Get credentials +source scripts/load-vault-env.sh + +# Export for tests +export TEST_POSTGRES_URL="postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@localhost:5432/${POSTGRES_DB}" +export TEST_REDIS_URL="redis://:${REDIS_PASSWORD}@localhost:6379" +``` + +### Step 2: Python pytest Example + +```python +# conftest.py +import pytest +import os +import psycopg2 + +@pytest.fixture(scope="session") +def db_connection(): + """Provide database connection for tests.""" + conn = psycopg2.connect(os.environ['TEST_POSTGRES_URL']) + yield conn + conn.close() + +@pytest.fixture(scope="function") +def clean_db(db_connection): + """Clean database before each test.""" + cursor = db_connection.cursor() + cursor.execute("TRUNCATE TABLE users CASCADE") + db_connection.commit() + yield + db_connection.rollback() + +# test_users.py +def test_create_user(db_connection, clean_db): + cursor = db_connection.cursor() + cursor.execute("INSERT INTO users (name) VALUES ('Alice')") + db_connection.commit() + + cursor.execute("SELECT name FROM users") + assert cursor.fetchone()[0] == 'Alice' +``` + +### Step 3: Run Tests + +```bash +# Setup and run +source test-setup.sh +pytest tests/ -v + +# Or in CI +./devstack start --profile standard +./devstack health +source scripts/load-vault-env.sh +pytest tests/ -v +./devstack stop +``` + +--- + +## Common Patterns + +### Loading Credentials Pattern + +```bash +# Best practice: Use environment variables +source scripts/load-vault-env.sh + +# Access in any language: +# Python: os.environ['POSTGRES_PASSWORD'] +# Go: os.Getenv("POSTGRES_PASSWORD") +# Node: process.env.POSTGRES_PASSWORD +# Rust: std::env::var("POSTGRES_PASSWORD") +``` + +### Connection String Pattern + +| Database | Connection String | +|----------|-------------------| +| PostgreSQL | `postgresql://USER:PASS@localhost:5432/DB` | +| MySQL | `mysql://USER:PASS@localhost:3306/DB` | +| MongoDB | `mongodb://USER:PASS@localhost:27017/DB` | +| Redis | `redis://:PASS@localhost:6379` | +| RabbitMQ | `amqp://USER:PASS@localhost:5672/vhost` | + +### Health Check Pattern + +```python +# Check all services before starting app +import requests + +services = { + 'vault': 'http://localhost:8200/v1/sys/health', + 'postgres': None, # Use connection test + 'redis': None, # Use ping +} + +for name, url in services.items(): + if url: + resp = requests.get(url) + assert resp.status_code == 200, f"{name} unhealthy" +``` + +--- + +## See Also + +- [Getting Started](GETTING_STARTED.md) - Initial setup +- [CLI Reference](CLI_REFERENCE.md) - All commands +- [Glossary](GLOSSARY.md) - Term definitions +- [Learning Paths](LEARNING_PATHS.md) - Guided learning