Skip to content

MIGRATIONS-MAINTENANCE.md

Latest commit

 

History

History
286 lines (196 loc) · 6.87 KB

MIGRATIONS-MAINTENANCE.md

File metadata and controls

286 lines (196 loc) · 6.87 KB

Migrations Directory Maintenance Guide

Overview

This document describes strategies for maintaining the core/migrations/ directory as your application evolves.

Directory Structure

core/migrations/
├── atlas.sum                          # Checksum file (auto-generated by Atlas)
├── 20250101120000_initial_schema.sql  # Initial migration
├── 20250102093000_add_users_table.sql # Add new table
├── 20250103153000_add_email_column.sql # Add column
└── ...                                 # More migrations over time

Core Principles

1. Never Modify Applied Migrations ⚠️

Once a migration has been applied to any environment (dev, staging, production):

  • ✅ DO: Create a new migration to make changes
  • ❌ DON'T: Edit the existing migration file
  • ❌ DON'T: Delete applied migration files

Why: Atlas uses atlas.sum to verify migration integrity. Modifying files will break checksum validation.

2. Linear History

Migrations must form a linear sequence:

Migration 1 → Migration 2 → Migration 3 → Current State

Avoid:

  • Branching (parallel migrations)
  • Out-of-order timestamps
  • Skipping sequences

Maintenance Strategies

Strategy 1: Keep All Migrations (Recommended for Production)

Approach: Never delete migrations, accumulate them over time.

Pros:

  • Complete audit trail
  • Easy rollback to any point
  • Safe for distributed teams

Cons:

  • Directory grows indefinitely
  • Longer atlas migrate status checks

When to use: Production applications, compliance requirements

Strategy 2: Squashing (Baseline Reset)

Approach: Periodically consolidate multiple migrations into a single baseline.

Process:

  1. Choose a baseline point (e.g., v1.0.0 release)
  2. Backup production database
  3. Create baseline migration: 
    # Generate full schema from current state
./core/bin/apprun migrate diff baseline_v1.0.0
  4. Archive old migrations: 
    mkdir -p core/migrations/archive/v0.x/
mv core/migrations/202501*.sql core/migrations/archive/v0.x/
  5. Update atlas.sum: 
    cd core && atlas migrate hash --dir file://migrations/
  6. Document in CHANGELOG

⚠️ Critical: All environments must be on the same version before squashing.

Strategy 3: Development Squashing

Approach: Squash migrations that only exist in development (not in production).

When: After completing a feature branch:

# Example: You created 5 migrations while developing "user-auth" feature
# Before merging to main, consolidate them:

# 1. Reset local database
./core/bin/apprun migrate reset

# 2. Delete development migrations
rm core/migrations/*user_auth*.sql

# 3. Regenerate as single migration
./core/bin/apprun migrate diff add_user_auth_feature

# 4. Apply
./core/bin/apprun migrate apply

⚠️ Warning: Only do this for migrations not yet in production/staging.

Recommended Workflow by Environment

Development Environment

Goal: Fast iteration, easy experimentation

# Modify Ent schema
vim core/ent/schema/user.go

# Generate migration
./core/bin/apprun migrate diff add_user_field

# Test
./core/bin/apprun migrate apply

# If something goes wrong, reset and try again
./core/bin/apprun migrate reset
./core/bin/apprun migrate sync   # Now handles empty DB correctly!

Squashing: OK to squash before pushing to shared branch

Staging/Production

Goal: Stability, auditability

# Pull latest migrations
git pull

# Check status
./core/bin/apprun migrate status

# Apply pending
./core/bin/apprun migrate apply

# Never reset in production!

Squashing: Only during major version releases with coordination

Migration Naming Conventions

Use descriptive names:

# ✅ Good
./core/bin/apprun migrate diff add_user_email_column
./core/bin/apprun migrate diff create_audit_log_table
./core/bin/apprun migrate diff add_index_user_email

# ❌ Bad
./core/bin/apprun migrate diff update
./core/bin/apprun migrate diff fix
./core/bin/apprun migrate diff change

Dealing with Large Migration Counts

Symptoms

  • migrate status takes >5 seconds
  • Migration directory has >100 files

Solutions

Option A: Archive Old Migrations

Move to subdirectory (maintain in VCS):

mkdir -p core/migrations/archive/2025-q1/
mv core/migrations/202501*.sql core/migrations/archive/2025-q1/

Update atlas.sum:

cd core && atlas migrate hash --dir file://migrations/

Option B: Use Atlas Cloud (Future)

Atlas Pro provides:

  • Remote migration storage
  • Faster status checks
  • Team collaboration features

Rollback Considerations

Before Squashing

Test rollback scenarios:

# Test rolling back last 3 migrations
./core/bin/apprun migrate rollback --count 3

# Verify database state
./core/bin/apprun migrate status

# Re-apply
./core/bin/apprun migrate apply

After Squashing

Rollback only possible to the baseline:

Baseline v1.0.0 → Migration 1 → Migration 2 → Current
                  ↑
                  Can rollback here

Cannot rollback before baseline without restoring database backup.

File Deletion Safety Checklist

Before deleting any migration file, ensure:

  • File has never been applied to production
  • File has never been applied to staging
  • No other team members have applied it
  • You have a database backup (if in doubt)
  • You've communicated with the team

Automation Scripts

Check Migration Count

#!/bin/bash
# scripts/check-migration-count.sh

COUNT=$(ls -1 core/migrations/*.sql 2>/dev/null | wc -l)
echo "📊 Migration count: $COUNT"

if [ "$COUNT" -gt 100 ]; then
    echo "⚠️  Consider squashing old migrations"
fi

Archive Old Migrations

#!/bin/bash
# scripts/archive-old-migrations.sh

ARCHIVE_DIR="core/migrations/archive/$(date +%Y-q%q)/"
mkdir -p "$ARCHIVE_DIR"

# Move migrations older than 6 months
find core/migrations/ -name "*.sql" -mtime +180 -exec mv {} "$ARCHIVE_DIR" \;

echo "✅ Archived old migrations to $ARCHIVE_DIR"

Best Practices Summary

  1. Never modify applied migrations - Always create new ones
  2. Use descriptive names - Future you will thank you
  3. Test rollbacks - Before deploying to production
  4. Squash cautiously - Only for unreleased migrations or with team coordination
  5. Keep atlas.sum in sync - Run atlas migrate hash after manual changes
  6. Document major changes - Update CHANGELOG when squashing
  7. Backup before squashing - Safety first

References

Questions?

See ARCHITECTURE.md for system design decisions or ask the team in Slack.