diff --git a/CHANGELOG.md b/CHANGELOG.md index bb536a1..783b31e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,18 @@ Versioning: [Semantic Versioning](https://semver.org/spec/v2.0.0.html) --- +## [1.6.0] — 2026-06-17 + +### Added +- **Fine-grained migration step control** — `MRTFixture` gains `upgrade_to(revision)`, `upgrade_one()`, `downgrade_one()`, `downgrade_to(revision)`, and `current_revision()` for testing data migration logic at any intermediate point in the chain. Achieves full API parity with pytest-alembic. (#86, #92) + +### Fixed +- **Django mode step methods raised `AttributeError` instead of `RuntimeError`** — `upgrade()`, `upgrade_to()`, `upgrade_one()`, `downgrade()`, `downgrade_one()`, `downgrade_to()`, and `current_revision()` all called `self._runner.*` without checking `_django_mode`. In Django mode `_runner` is `None`, so these methods crashed with an uninformative `AttributeError`. They now raise `RuntimeError` with a clear message, consistent with `check_revision()` and `assert_reversible()`. +- **`check_static()` raised `AttributeError` in Django mode** — `check_static()` called `self._runner.get_versions_dir()` without a Django mode guard. Now raises `RuntimeError` with a message directing users to `check_migration()` / `check_all()`. +- **Timeout path did not attempt DB state recovery** — when a migration exceeded `MRTConfig.migration_timeout`, the `FuturesTimeout` was caught but no recovery was attempted. The DB could be left in an upgraded state, corrupting all subsequent revisions in `check_all()`. Recovery now mirrors the `except Exception` path added in v0.7. + +--- + ## [1.5.0] — 2026-06-11 ### Removed diff --git a/README.md b/README.md index 4f83997..25979a3 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ CI Coverage Production/Stable - Python 3.10-3.13 + Python 3.10-3.14 MIT License Contributors

@@ -167,14 +167,14 @@ Add to `.pre-commit-config.yaml` to run `mrt check` automatically before every p ```yaml # Alembic - repo: https://github.com/croc100/pytest-mrt - rev: v1.4.0 + rev: v1.5.0 hooks: - id: mrt-check args: [alembic/versions/] # Django - repo: https://github.com/croc100/pytest-mrt - rev: v1.4.0 + rev: v1.5.0 hooks: - id: mrt-check args: [myapp/migrations/] @@ -263,13 +263,25 @@ Legacy syntax `# mrt: ignore` is still supported for backward compatibility. The key difference from pytest-alembic: pytest-mrt seeds actual rows before each rollback and verifies they survive. A migration that reverses the schema cleanly but silently destroys data will pass pytest-alembic and fail pytest-mrt. -## What's new in v1.4.0 +## What's new in v1.6.0 -- **`mrt check --format json/html`** — structured JSON output for CI tooling; self-contained HTML safety report -- **`mrt check --watch`** — re-runs automatically whenever a migration file changes -- **`mrt check --min-revision`** — skip revisions older than a configured floor (mirrors `MRTConfig.minimum_downgrade_revision`) -- **Django squashmigrations detection** — MRT601/MRT602 catch unsafe `RunPython` in squashed migrations -- **`minimum_downgrade_revision` in dynamic tests** — floor now respected by the `mrt` fixture `check_all()`, not just static analysis +- **Fine-grained migration step control** — `upgrade_to()`, `upgrade_one()`, `downgrade_one()`, `downgrade_to()`, `current_revision()` let you test data migration logic at any point in the chain: + +```python +def test_data_migration(mrt): + mrt.upgrade_to("abc123") # upgrade to a specific revision + mrt.seed("users", [...]) # seed data at that checkpoint + mrt.upgrade_one() # apply exactly one more step + assert mrt.current_revision() == "def456" + mrt.downgrade_one() # roll back one step + mrt.downgrade_to("base") # roll all the way back +``` + +## Migrating from v1.4.x + +**v1.5.0 removed `mrt fix` and `mrt clean-backups`** — migration code generation is a *transform* operation, not a *verify* operation, and was out of scope. Projects relying on these commands should pin `pytest-mrt<1.5.0`. + +The `fixable` field in `mrt check --format json` output was also removed in v1.5.0. ## Changelog diff --git a/ROADMAP.md b/ROADMAP.md index f384d5c..c396ae8 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,6 +1,6 @@ # Roadmap -## Current status: Production/Stable (v1.3.0) +## Current status: Production/Stable (v1.5.0 on PyPI — v1.6.0 on main) pytest-mrt is production-ready. The core API (`MRTConfig`, `mrt` fixture, `mrt check`) is stable and breaking changes will be versioned. See [`docs/api.md`](docs/api.md) for the stability guarantee. @@ -64,45 +64,57 @@ breaking changes will be versioned. See [`docs/api.md`](docs/api.md) for the sta - Test coverage: `default_tests.py` and `drift.py` at 100% - Documentation fully updated (pattern counts, version refs, suppression docs) -## v1.3.0 — Incremental CI + pre-commit + Django fix (shipped) +## v1.3.0 — Incremental CI + pre-commit (shipped) - **`mrt check --since `** — check only migrations added since a given revision; eliminates re-scanning full history on every PR - **pre-commit hook** — `.pre-commit-hooks.yaml` ships with the package; two-line setup -- **Django-aware `mrt fix`** — auto-generates reverse operations for `RunSQL`, `RunPython`, `RemoveField`, `DeleteModel` with transactional DB backup/restore scaffolding -- **`mrt clean-backups`** — CLI command to remove `_mrt_backups` data after deployment --- -## v1.4.0 — Under consideration +## v1.4.0 — CI integration + rolling deploy (shipped) -- **`mrt check --watch`** — re-run on file change during development -- **Django: `squashmigrations` detection** — squashed migrations with unresolved refs -- **Per-pattern confidence scores** in JSON output -- **HTML report: source line links** — click a finding to jump to the migration file -- **Sentry integration** — report migration failures as Sentry events -- **GitHub App** — automated PR comments with migration risk summary -- **VS Code extension** — inline warnings in migration files +- **`mrt check --format json/html`** — structured JSON output for CI tooling; self-contained HTML safety report (`--output` to write file) +- **`mrt check --watch`** — re-run automatically when migration files change; Ctrl-C to stop +- **`mrt check --min-revision`** — skip revisions at or before a floor; mirrors `MRTConfig.minimum_downgrade_revision` +- **`mrt check --check-compat`** — rolling-deploy compatibility checks (MRT701–MRT705): DROP COLUMN, RENAME COLUMN, DROP TABLE, ADD NOT NULL without default, incompatible type changes +- **Django squashmigrations detection** — MRT601/MRT602: catches `RunPython` without `reverse_code` in squashed migrations and suspicious squash filenames +- **`MRTConfig.minimum_downgrade_revision`** — permanent rollback floor respected by `check_all()` in the `mrt` fixture +- **`croc100/pytest-mrt-action` v1.0.0** — GitHub Actions action that posts findings as a job summary + +--- + +## v1.5.0 — Scope reduction (breaking, shipped) + +- **Removed `mrt fix` and `mrt clean-backups`** — migration code generation is a *transform* operation, not a *verify* operation. Out of scope. Projects that rely on these must pin `pytest-mrt<1.5.0`. +- **Removed `fixable` field from `mrt check --format json`** — advertised the removed auto-fix capability. Breaking for downstream tooling that read this field. +- **Fixed** `RollbackVerifier` false positive on failed custom seeds — rows whose `INSERT` fails are no longer tracked, eliminating "lost after rollback" false positives. + +--- + +## v1.6.0 — Fine-grained step control (shipped on main, pending release) + +- **`upgrade_to(revision)`**, **`upgrade_one()`**, **`downgrade_one()`**, **`downgrade_to(revision)`**, **`current_revision()`** — call any migration step from a test, enabling mid-chain data seeding and assertion --- -## Next / Under consideration +## Under consideration -These are not committed to a specific version yet: +These are not committed to a version yet: -- **`mrt check --watch`** — re-run on file change during development -- **Django: `squashmigrations` detection** — squashed migrations with unresolved refs +- **Django squashmigrations: full rollback plan** — detect and test rollback paths through squashed migration graphs dynamically (static detection is already in v1.4.0) - **Per-pattern confidence scores** in JSON output - **HTML report: source line links** — click a finding to jump to the migration file - **Sentry integration** — report migration failures as Sentry events - **GitHub App** — automated PR comments with migration risk summary - **VS Code extension** — inline warnings in migration files +- **`mrt check --check-compat` Django support** — currently Alembic only --- ## What won't be in scope - Executing migrations in production (this is a *testing* tool only) -- **Migration code generation / auto-fix** — pytest-mrt verifies migrations; it does not rewrite them. The former `mrt fix` / `mrt clean-backups` commands were removed in v1.5.0 (out of scope: "transform", not "verify") +- **Migration code generation / auto-fix** — removed in v1.5.0 (out of scope: "transform", not "verify") - Schema diff tools (use `alembic check` or `django-migration-linter`) - ORM-agnostic support (focused on Alembic and Django) diff --git a/docs/api.md b/docs/api.md index 1ceba6b..f802548 100644 --- a/docs/api.md +++ b/docs/api.md @@ -16,16 +16,22 @@ Configuration object passed to `pytest_configure` to set up migration rollback t | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `alembic_ini` | `str` | `"alembic.ini"` | Path to your `alembic.ini` file | +| `alembic_ini` | `str` | `"alembic.ini"` | Path to your `alembic.ini` file (ignored in Django mode) | | `db_url` | `str` | `""` | SQLAlchemy database URL for the test database | | `seed_rows` | `int` | `3` | Number of rows to seed per table during rollback verification | | `skip` | `dict[str, str]` | `{}` | Revisions to skip, with documented reasons | | `severity_overrides` | `dict[str, str]` | `{}` | Override severity of specific risk patterns | | `custom_seeds` | `dict[str, Callable]` | `{}` | Custom seed functions per table | -| `custom_checks` | `list[Callable]` | `[]` | Additional static analysis check functions | -| `migration_timeout` | `int \| None` | `None` | Per-migration timeout in seconds (`None` = no limit) | +| `custom_checks` | `list[Callable]` | `[]` | Additional static analysis check functions (Alembic only) | +| `migration_timeout` | `int \| None` | `60` | Per-migration timeout in seconds (`None` = no limit) | +| `minimum_downgrade_revision` | `str \| None` | `None` | Skip revisions at or before this floor in `check_all()` | +| `target_metadata` | `str \| None` | `None` | Import path for SQLAlchemy `Base`/`MetaData` used by `assert_schema_matches()` | +| `django_settings` | `str \| None` | `None` | Django settings module — enables Django mode | +| `django_apps` | `list[str] \| None` | `None` | Restrict dynamic testing to specific Django app labels | +| `django_project_dir` | `str \| None` | `None` | Path added to `sys.path` before Django import | +| `explain_model` | `str` | `"claude-opus-4-5"` | Claude model used by `mrt explain` | -### Example +### Example (Alembic) ```python # conftest.py @@ -46,6 +52,23 @@ def pytest_configure(config): custom_seeds={ "users": lambda: [{"id": 1, "name": "Alice", "email": "alice@example.com"}], }, + minimum_downgrade_revision="a1b2c3d4", + target_metadata="myapp.models:Base", + ) +``` + +### Example (Django) + +```python +# conftest.py +import os +from pytest_mrt import MRTConfig + +def pytest_configure(config): + config._mrt_config = MRTConfig( + db_url=os.environ["TEST_DATABASE_URL"], + django_settings="myproject.settings_test", + django_apps=["users", "orders"], ) ``` @@ -62,18 +85,18 @@ skip={ ### `severity_overrides` -Promotes warnings to errors (or demotes errors to warnings) for specific risk pattern names. Pattern names match the `pattern` field in the JSON output of `mrt check`. +Promotes warnings to errors (or demotes errors to warnings) for specific risk pattern names. Pattern names match the `pattern` field in `mrt check --format json` output. ```python severity_overrides={ - "INDEX without CONCURRENTLY": "error", # treat as error in your org - "noop downgrade": "warning", # already handled by your deploy process + "INDEX without CONCURRENTLY": "error", + "noop downgrade": "warning", } ``` ### `custom_checks` -Each function receives a `MigrationAST` and returns a list of `RiskWarning` objects. Custom checks run in addition to the built-in checks. +Each function receives a `MigrationAST` and returns a list of `RiskWarning` objects. Custom checks run in addition to the built-in checks. Alembic mode only. ```python from pytest_mrt.core.ast_analyzer import MigrationAST @@ -106,28 +129,74 @@ def test_migrations(mrt): ### Migration control +> **Django mode**: `upgrade_*`, `downgrade_*`, and `current_revision()` are Alembic-only and raise `RuntimeError` in Django mode. Use `check_migration()` / `check_all()` for Django projects. + #### `mrt.upgrade(revision="head")` Run `alembic upgrade` to the given revision. ```python -mrt.upgrade("head") # upgrade to latest -mrt.upgrade("001abc") # upgrade to a specific revision +mrt.upgrade("head") +mrt.upgrade("001abc") +``` + +#### `mrt.upgrade_to(revision)` + +Upgrade to a specific revision. Equivalent to `upgrade(revision)`. + +```python +mrt.upgrade_to("abc123") +``` + +#### `mrt.upgrade_one()` + +Upgrade exactly one step from the current revision. + +```python +mrt.upgrade_one() ``` #### `mrt.downgrade(revision="-1")` -Run `alembic downgrade` by one step (or to a specific revision). +Run `alembic downgrade` by one step or to a specific revision. ```python mrt.downgrade() # roll back one step mrt.downgrade("base") # roll back to empty schema ``` +#### `mrt.downgrade_one()` + +Downgrade exactly one step from the current revision. + +```python +mrt.downgrade_one() +``` + +#### `mrt.downgrade_to(revision)` + +Downgrade to a specific revision. + +```python +mrt.downgrade_to("abc123") +mrt.downgrade_to("base") +``` + +#### `mrt.current_revision() → str | None` + +Return the current Alembic revision ID, or `None` if at base. + +```python +rev = mrt.current_revision() +assert rev == "abc123" +``` + --- ### Static analysis +> Not available in Django mode — raises `RuntimeError`. Use `check_migration()` / `check_all()` for Django rollback testing. + #### `mrt.check_static(versions_dir=None) → list[RiskWarning]` Run static analysis on migration files. Returns all detected risk warnings. @@ -156,9 +225,7 @@ def test_no_unsafe_migrations(mrt): #### `mrt.check_revision(revision) → RevisionResult` -Test a single revision for safe reversibility. The database must be at the state just _before_ this revision when called. - -Returns a `RevisionResult` (see below). +Test a single Alembic revision for safe reversibility. The database must be at the state just _before_ this revision when called. Alembic mode only. ```python mrt.upgrade("001abc") @@ -166,9 +233,18 @@ result = mrt.check_revision("002def") assert result.passed ``` -#### `mrt.check_all() → list[RevisionResult]` +#### `mrt.check_migration(app_label, migration_name) → RevisionResult` -Test every migration in the chain. Internally runs in O(n) upgrade operations. +Test a single Django migration by app label and migration name. Django mode only. + +```python +result = mrt.check_migration("users", "0003_add_phone") +assert result.passed +``` + +#### `mrt.check_all(apps=None) → list[RevisionResult]` + +Test every migration in the chain. Runs in O(n) upgrade operations. In Django mode, pass `apps` to restrict to specific app labels (overrides `MRTConfig.django_apps` for this call). ```python results = mrt.check_all() @@ -177,16 +253,16 @@ failed = [r for r in results if not r.passed] #### `mrt.assert_reversible(revision="head")` -Assert that a single revision is safely reversible. Fails the test if not. +Assert that a single Alembic revision is safely reversible. Fails the test if not. Alembic mode only. ```python def test_latest_migration(mrt): mrt.assert_reversible() ``` -#### `mrt.assert_all_reversible()` +#### `mrt.assert_all_reversible(apps=None)` -Assert every migration in the chain is safely reversible. Prints a summary table and fails if any migration fails. +Assert every migration in the chain is safely reversible. Prints a summary table and fails if any migration fails. Works for both Alembic and Django. ```python def test_all_migrations(mrt): @@ -199,36 +275,52 @@ def test_all_migrations(mrt): #### `mrt.seed(table, rows, pk_col="id")` -Manually seed rows into a table (currently open schema). +Manually seed rows into a table at the current schema state. Combine with step control methods for mid-chain data assertions. ```python -mrt.upgrade("001abc") -mrt.seed("users", [{"id": 1, "name": "Alice"}]) +def test_data_migration(mrt): + mrt.upgrade_to("abc123") + mrt.seed("users", [{"id": 1, "name": "Alice"}]) + mrt.upgrade_one() + mrt.downgrade_one() + mrt.assert_data_intact() ``` #### `mrt.assert_data_intact()` Assert that all previously seeded rows still exist and have their original values. -```python -mrt.upgrade("head") -mrt.downgrade() -mrt.assert_data_intact() -``` - #### `mrt.reset()` Clear the internal seed state. Called automatically at fixture teardown. --- +### Schema drift + +#### `mrt.assert_schema_matches(target_metadata=None, metadata_path=None)` + +Fail if the DB schema does not match the SQLAlchemy model definitions. In Django mode, delegates to `manage.py makemigrations --check`. + +```python +from myapp.models import Base + +def test_no_drift(mrt): + mrt.upgrade("head") + mrt.assert_schema_matches(Base) +``` + +Or configure once via `MRTConfig(target_metadata="myapp.models:Base")` and rely on the built-in `test_mrt_schema_matches_models`. + +--- + ## `RevisionResult` -Return type of `mrt.check_revision()` and elements of `mrt.check_all()`. +Return type of `mrt.check_revision()`, `mrt.check_migration()`, and elements of `mrt.check_all()`. | Attribute | Type | Description | |-----------|------|-------------| -| `revision` | `str` | The Alembic revision ID | +| `revision` | `str` | The revision ID (Alembic) or `app/name` (Django) | | `passed` | `bool` | `True` if rollback was safe | | `skipped` | `bool` | `True` if this revision is in `MRTConfig.skip` | | `skip_reason` | `str` | The documented reason for skipping | @@ -255,10 +347,11 @@ Returned by `mrt.check_static()` and by `custom_checks` functions. |-----------|------|-------------| | `revision` | `str` | Revision ID or filename stem | | `file` | `str` | Migration filename | -| `pattern` | `str` | Short pattern name (e.g. `"DROP COLUMN"`) | +| `pattern` | `str` | Short pattern name (e.g. `"DROP COLUMN in upgrade"`) | | `message` | `str` | Human-readable explanation | | `severity` | `str` | `"error"` or `"warning"` | | `line` | `int \| None` | Line number in the migration file | +| `code` | `str` | Rule code (e.g. `"MRT201"`) | --- @@ -266,50 +359,61 @@ Returned by `mrt.check_static()` and by `custom_checks` functions. ### `mrt check ` -Statically analyze migration files for rollback risks. +Statically analyze migration files for rollback risk patterns. Auto-detects Django migrations. ``` mrt check alembic/versions/ -mrt check alembic/versions/ --strict -mrt check alembic/versions/ --format json +mrt check myapp/migrations/ --strict +mrt check alembic/versions/ --format json --output report.json +mrt check alembic/versions/ --format html --output report.html +mrt check alembic/versions/ --watch +mrt check alembic/versions/ --since a1b2c3d4 +mrt check alembic/versions/ --check-compat ``` | Option | Default | Description | |--------|---------|-------------| -| `--strict` | `False` | Exit 1 on warnings as well as errors | -| `--format` / `-f` | `table` | Output format: `table` or `json` | +| `--strict` | `False` | Treat warnings as errors (exit 2) | +| `--format` / `-f` | `table` | Output format: `table`, `json`, or `html` | +| `--output` / `-o` | `None` | Write output to file. For `--format html` defaults to `mrt-report.html`. | +| `--since` | `None` | Only check migrations added after this revision. Alembic: revision ID. Django: `app_label.migration_name`. Graph checks (orphan, data-hole detection) are skipped when `--since` is active. | +| `--min-revision` | `None` | Skip revisions at or older than this point. Alembic: revision ID. Django: `app_label.migration_name`. Mirrors `MRTConfig.minimum_downgrade_revision`. | +| `--watch` / `-w` | `False` | Re-run automatically when migration files change. `--format table` only. Ctrl-C to stop. | +| `--check-compat` | `False` | Also run rolling-deploy compatibility checks (MRT701–MRT705). Alembic only. | -**Exit codes:** `0` = safe, `1` = errors found (or warnings with `--strict`) +**Exit codes:** `0` = no findings, `1` = warnings only, `2` = one or more errors (or warnings with `--strict`) **JSON output schema:** ```json -[ - { - "revision": "001abc", - "file": "001_create_users.py", - "pattern": "DROP COLUMN", - "severity": "error", - "message": "op.drop_column('users', 'email') — data permanently lost...", - "line": 12 - } -] +{ + "version": "1.6.0", + "checked_at": "2026-06-17T12:00:00Z", + "summary": { "total_issues": 1, "errors": 1, "warnings": 0 }, + "findings": [ + { + "file": "001_create_users.py", + "line": 12, + "rule": "MRT201", + "severity": "error", + "pattern": "DROP COLUMN in upgrade", + "message": "op.drop_column('users', 'email') — data permanently lost on rollback" + } + ] +} ``` --- -### `mrt report ` +### `mrt drift ` -Generate an HTML safety report for all migrations. +Compare the live DB schema against SQLAlchemy model definitions and print a diff. ``` -mrt report alembic/versions/ -mrt report alembic/versions/ --output report.html +mrt drift myapp.models:Base --config alembic.ini --db-url sqlite:///test.db ``` -| Option | Default | Description | -|--------|---------|-------------| -| `--output` / `-o` | `migration_report.html` | Output file path | +Exits `1` if drift is detected, `0` if schema matches. --- @@ -333,6 +437,8 @@ mrt explain alembic/versions/001_create_users.py Requires: `pip install pytest-mrt[ai]` and `ANTHROPIC_API_KEY` environment variable. +Override the model with `MRTConfig(explain_model="claude-haiku-4-5-20251001")`. + --- ### `mrt version` diff --git a/docs/cli.md b/docs/cli.md index cb16cf2..0832c63 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -145,7 +145,7 @@ Output schema: ```json { - "version": "1.4.0", + "version": "1.6.0", "checked_at": "2026-06-10T11:00:00Z", "summary": { "total_issues": 2, "errors": 1, "warnings": 1 }, "findings": [ @@ -232,7 +232,7 @@ When there are no problems: ```bash mrt version -# pytest-mrt 1.4.0 +# pytest-mrt 1.6.0 ``` --- diff --git a/docs/index.md b/docs/index.md index 97f0a81..8fc9c44 100644 --- a/docs/index.md +++ b/docs/index.md @@ -112,7 +112,9 @@ pytest tests/test_migrations.py -s |---|---|---| | PostgreSQL | Yes | Yes | | SQLite | Yes | Yes | -| MySQL / MariaDB | Yes | planned | +| MySQL / MariaDB | Yes | Yes | +| Oracle | Yes | Yes | +| SQL Server | Yes | Yes | --- diff --git a/docs/migration-guide.md b/docs/migration-guide.md index 8583c29..f1e9f45 100644 --- a/docs/migration-guide.md +++ b/docs/migration-guide.md @@ -4,29 +4,79 @@ This document covers breaking changes and required actions when upgrading pytest --- -## v1.x → v1.2 +## v1.5.x → v1.6 -No breaking changes. Drop-in upgrade. +No breaking changes. Drop-in upgrade. -**New in v1.2:** +**New in v1.6:** -- Every pattern now has a unique rule code (`MRT101`–`MRT902`). -- `# noqa: MRTxxx` suppression syntax (ruff/flake8-compatible). -- Legacy `# mrt: ignore` syntax is retained and still works. +- `upgrade_to(revision)`, `upgrade_one()`, `downgrade_one()`, `downgrade_to(revision)`, `current_revision()` — fine-grained step control in the `mrt` fixture. + +**Action required:** None. + +--- + +## v1.4.x → v1.5 + +**Breaking changes:** + +| Area | Change | +|------|--------| +| `mrt fix` command | Removed entirely | +| `mrt clean-backups` command | Removed entirely | +| `mrt check --format json` | `fixable` field removed from each finding | + +**Migration steps:** + +1. Remove any `mrt fix` or `mrt clean-backups` calls from your scripts and CI pipelines. +2. If downstream tooling reads the `fixable` field from `mrt check --format json`, update it to ignore or omit that field. +3. If you have `_mrt_backups` tables left over from a previous `mrt fix` run, drop them manually. + +Pin to `pytest-mrt<1.5.0` if you need to keep using `mrt fix`. + +--- + +## v1.3.x → v1.4 -**Action required:** None. Existing suppression comments continue to work. +No breaking changes. Drop-in upgrade. + +**New in v1.4:** + +- `mrt check --format json/html` — structured output and self-contained HTML reports. +- `mrt check --watch` — re-run on file change during development. +- `mrt check --min-revision` / `MRTConfig.minimum_downgrade_revision` — rollback testing floor. +- `mrt check --check-compat` — rolling-deploy compatibility checks (MRT701–MRT705). +- Django squashmigrations detection (MRT601/MRT602). + +**Action required:** None. --- -## v1.x → v1.3 (upcoming) +## v1.x → v1.3 -No breaking changes planned. +No breaking changes. Drop-in upgrade. **New in v1.3:** - `mrt check --since ` — incremental analysis for CI. - pre-commit hook support via `.pre-commit-hooks.yaml`. +**Action required:** None. + +--- + +## v1.x → v1.2 + +No breaking changes. Drop-in upgrade. + +**New in v1.2:** + +- Every pattern now has a unique rule code (`MRT101`–`MRT902`). +- `# noqa: MRTxxx` suppression syntax (ruff/flake8-compatible). +- Legacy `# mrt: ignore` syntax is retained and still works. + +**Action required:** None. Existing suppression comments continue to work. + --- ## v0.x → v1.0 @@ -36,8 +86,8 @@ No breaking changes planned. | Area | Change | |------|--------| | `MRTConfig` | `engine_url` renamed to `db_url` | -| Default tests | 6 built-in tests are now auto-injected when `mrt` fixture is configured. Opt out per test with `skip_default_tests={"test_mrt_upgrade"}`. | -| Exit codes | `mrt check` now exits `0` (clean) / `1` (warnings) / `2` (errors). Previously always `0` or `1`. | +| Default tests | 6 built-in tests are now auto-injected when `mrt` fixture is configured. Opt out per test with `skip_default_tests={...}`. | +| Exit codes | `mrt check` now exits `0` (clean) / `1` (warnings) / `2` (errors). Previously always `0` or `1`. | **Migration steps:** @@ -47,14 +97,6 @@ No breaking changes planned. --- -## v0.8 → v0.9 - -No breaking changes. - -**New:** Django dynamic rollback support via `DjangoMigrationRunner` and `DjangoRollbackVerifier`. No action needed for Alembic-only projects. - ---- - ## Checking your installed version ```bash diff --git a/docs/roadmap.md b/docs/roadmap.md index 151f4e8..83d6617 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -4,49 +4,79 @@ Items are tracked as GitHub issues. This page is a high-level overview. --- -## Near-term +## Shipped -### Single-head revision check ([#72](https://github.com/croc100/pytest-mrt/issues/72)) +### v1.6.0 — Fine-grained step control (main, pending release) -Add a built-in default test that fails when the migration chain has more than one head. A diverged head is almost always an unresolved merge conflict and will break deployments silently. pytest-alembic has this; pytest-mrt should too. +- `upgrade_to()`, `upgrade_one()`, `downgrade_one()`, `downgrade_to()`, `current_revision()` — test data migration logic at any intermediate point in the migration chain -### GitHub Actions action ([#74](https://github.com/croc100/pytest-mrt/issues/74)) +### v1.5.0 — Scope reduction -A dedicated action (`croc100/pytest-mrt-action`) that wraps `mrt check`, posts findings as a job summary, and optionally annotates changed migration files inline. The `--format json` output already exists — this is the CI integration layer on top. +- Removed `mrt fix` and `mrt clean-backups` — out of scope for a *testing* tool +- Fixed `RollbackVerifier` false positive on failed custom seeds -```yaml -- uses: croc100/pytest-mrt-action@v1 - with: - migrations-dir: alembic/versions/ -``` +### v1.4.0 — CI integration + rolling deploy safety -### r/Python showcase post ([#75](https://github.com/croc100/pytest-mrt/issues/75)) +- `mrt check --format json/html` — structured output for CI pipelines and HTML safety reports +- `mrt check --watch` — re-run on file change during development +- `mrt check --min-revision` — skip revisions older than a configured floor +- `mrt check --check-compat` — rolling-deploy compatibility checks (MRT701–MRT705) +- Django squashmigrations detection — MRT601/MRT602 +- `croc100/pytest-mrt-action` v1.0.0 — GitHub Actions integration -After rolling deploy compat checks ship, write a showcase covering what pytest-mrt does, how it differs from pytest-alembic and django-migration-linter, and a concrete example. Goal: grow star count toward 50+ for awesome-django eligibility. +### v1.3.0 — Incremental CI + pre-commit + +- `mrt check --since` — scan only new migrations in PRs +- pre-commit hook (`.pre-commit-hooks.yaml`) + +### v1.1.0–v1.2.0 — Default tests + rule codes + +- Six built-in default tests auto-injected on fixture configuration +- Schema drift detection (`mrt drift`, `assert_schema_matches()`) +- MRT rule codes (MRT101–MRT902) and `# noqa: MRTxxx` suppression + +### v1.0.0 — Production/Stable + +- Full database coverage: PostgreSQL, SQLite, MySQL/MariaDB, Oracle, SQL Server +- Full migration framework coverage: Alembic (static + dynamic) and Django (static + dynamic) +- 44+ static analysis patterns, public accuracy report --- -## Medium-term +## Under consideration + +### Django squashmigrations: dynamic rollback testing -### Rolling deploy compatibility checks ([#73](https://github.com/croc100/pytest-mrt/issues/73)) +Static detection of squashmigrations is already in v1.4.0 (MRT601/MRT602). The next step is dynamic verification: run the rollback plan through the squashed migration graph and verify it succeeds. Currently skipped in `check_all()`. -Static analysis pass that checks whether a migration is safe during a rolling deploy — i.e., whether the old app version can still run against the new schema while the deploy is in progress. +### `--check-compat` Django support -This is a different axis from rollback safety: +Rolling-deploy compatibility checks (`--check-compat`, MRT7xx) are currently Alembic-only. Extending to Django migrations requires mapping Django operation types to the same compat patterns. -- **Rollback safety** (what pytest-mrt tests today): can the migration be undone? -- **Rolling deploy safety** (new): can the old app survive the new schema long enough to be replaced? +### Per-pattern confidence scores -Patterns to detect: DROP COLUMN, RENAME COLUMN, ADD NOT NULL without default, DROP TABLE, incompatible type changes. +Add a `confidence` field to `mrt check --format json` output. Lets downstream tooling suppress known false-positive-prone patterns without using `# noqa`. -New flag: `mrt check --check-compat`. Alembic and Django both. +### HTML report: source line links + +Click a finding in the HTML report to jump to the exact line in the migration file. Requires either embedding file contents or linking to the file on disk/GitHub. + +### Sentry integration + +Report dynamic rollback failures as Sentry events, enabling production-side alerting when a migration that was not tested fails in the field. + +--- -### pytest-dev contribution ([#76](https://github.com/croc100/pytest-mrt/issues/76)) +## What won't be in scope -PR [#14576](https://github.com/pytest-dev/pytest/pull/14576) to pytest-dev: show which items are missing in `dict.items()` / `dict.keys()` set comparisons (`>=`, `<=`, `>`, `<`), the same way set comparisons already work. Waiting for maintainer review. +- Executing migrations in production (this is a *testing* tool only) +- Migration code generation / auto-fix (removed in v1.5.0 — out of scope) +- Schema diff tools (use `alembic check` or `django-migration-linter`) +- ORM-agnostic support (focused on Alembic and Django) --- -## Why this order +## How to influence the roadmap -The single-head check and GitHub Action are both small, self-contained, and directly address gaps vs. competing tools. Rolling deploy compat is bigger and becomes the centerpiece of the showcase post — do it second so the post has something new to announce. +Open an issue tagged `roadmap` with your use case. +Sponsorship fast-tracks specific items — see [GitHub Sponsors](https://github.com/sponsors/croc100). diff --git a/pyproject.toml b/pyproject.toml index 095ccf0..45d328f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "pytest-mrt" -version = "1.5.0" +version = "1.6.0" description = "Catch database migration rollback failures before they reach production" readme = "README.md" license = { text = "MIT" } diff --git a/pytest_mrt/core/verifier.py b/pytest_mrt/core/verifier.py index dd1baea..2ff7e61 100644 --- a/pytest_mrt/core/verifier.py +++ b/pytest_mrt/core/verifier.py @@ -150,6 +150,19 @@ def check_revision(self, revision: str) -> RevisionResult: "The migration may be deadlocked or running a long data operation. " "Increase MRTConfig.migration_timeout or split the migration." ) + # Best-effort state recovery: the timed-out thread may have + # left the DB in a partial upgrade/downgrade state. + try: + current = self.runner.current_revision() + if current != start_revision: + self.runner.downgrade_base() + if start_revision is not None: + self.runner.upgrade(start_revision) + except Exception as recovery_exc: + failures.append( + f"State recovery failed after timeout — DB may be in unknown state: " + f"{recovery_exc}" + ) else: failures = self._run_migration_check(revision, schema_before, seeder) diff --git a/pytest_mrt/plugin.py b/pytest_mrt/plugin.py index c0127c8..38ee99c 100644 --- a/pytest_mrt/plugin.py +++ b/pytest_mrt/plugin.py @@ -103,32 +103,44 @@ def __init__(self, config: MRTConfig): # ── migration control ────────────────────────────────────────────── + def _require_alembic(self, method: str) -> None: + if self._django_mode: + raise RuntimeError( + f"{method}() is not available in Django mode. " + "Use check_migration() / check_all() for Django projects." + ) + def upgrade(self, revision: str = "head") -> None: + self._require_alembic("upgrade") self._runner.upgrade(revision) def upgrade_to(self, revision: str) -> None: """Upgrade to a specific revision. Equivalent to upgrade(revision).""" + self._require_alembic("upgrade_to") self._runner.upgrade(revision) def upgrade_one(self) -> None: """Upgrade exactly one step from the current revision.""" + self._require_alembic("upgrade_one") self._runner.upgrade("+1") def downgrade(self, revision: str = "-1") -> None: + self._require_alembic("downgrade") self._runner.downgrade(revision) def downgrade_one(self) -> None: """Downgrade exactly one step from the current revision.""" + self._require_alembic("downgrade_one") self._runner.downgrade("-1") def downgrade_to(self, revision: str) -> None: """Downgrade to a specific revision.""" + self._require_alembic("downgrade_to") self._runner.downgrade(revision) def current_revision(self) -> str | None: """Return the current Alembic revision, or None if at base.""" - if self._django_mode: - raise RuntimeError("current_revision() is not available in Django mode.") + self._require_alembic("current_revision") return self._runner.current_revision() # ── manual seeding ──────────────────────────────────────────────── @@ -147,6 +159,12 @@ def check_static(self, versions_dir: str | None = None) -> list[RiskWarning]: Includes built-in checks + any custom_checks registered in MRTConfig. severity_overrides from config are applied to the results. """ + if self._django_mode: + raise RuntimeError( + "check_static() is not available in Django mode — " + "static analysis targets Alembic migration files. " + "Use mrt check_migration() / check_all() for Django rollback testing." + ) if versions_dir is None: versions_dir = self._runner.get_versions_dir() diff --git a/tests/test_plugin.py b/tests/test_plugin.py index 48d1363..42de034 100644 --- a/tests/test_plugin.py +++ b/tests/test_plugin.py @@ -970,6 +970,45 @@ def test_assert_reversible_raises_in_django_mode(alembic_env): fixture.assert_reversible("001") +@pytest.mark.parametrize("method,args", [ + ("upgrade", ("head",)), + ("upgrade_to", ("abc123",)), + ("upgrade_one", ()), + ("downgrade", ()), + ("downgrade_one", ()), + ("downgrade_to", ("abc123",)), + ("current_revision", ()), +]) +def test_alembic_step_methods_raise_in_django_mode(alembic_env, method, args): + """upgrade/downgrade step methods raise RuntimeError in Django mode, not AttributeError.""" + cfg = MRTConfig(alembic_ini=alembic_env["ini"], db_url=alembic_env["db_url"]) + fixture = MRTFixture.__new__(MRTFixture) + fixture._config = cfg + fixture._django_mode = True + fixture._runner = None + fixture._verifier = None + fixture._django_verifier = None + fixture._seeder = None + + with pytest.raises(RuntimeError, match="not available in Django mode"): + getattr(fixture, method)(*args) + + +def test_check_static_raises_in_django_mode(alembic_env): + """check_static() raises RuntimeError in Django mode, not AttributeError.""" + cfg = MRTConfig(alembic_ini=alembic_env["ini"], db_url=alembic_env["db_url"]) + fixture = MRTFixture.__new__(MRTFixture) + fixture._config = cfg + fixture._django_mode = True + fixture._runner = None + fixture._verifier = None + fixture._django_verifier = None + fixture._seeder = None + + with pytest.raises(RuntimeError, match="not available in Django mode"): + fixture.check_static() + + # ── _auto_detect_django ImportError branch ──────────────────────────────────── diff --git a/tests/test_verifier_unit.py b/tests/test_verifier_unit.py index 7a21dee..e376b48 100644 --- a/tests/test_verifier_unit.py +++ b/tests/test_verifier_unit.py @@ -72,6 +72,28 @@ def slow_check(*args, **kwargs): assert any("timed out" in f for f in result.failures) +def test_check_revision_timeout_triggers_recovery(): + """After a timeout, check_revision attempts DB state recovery.""" + runner = _make_runner_mock() + # After timeout: current_revision returns "rev1" (DB is in upgraded state) + runner.current_revision.side_effect = [None, "rev1"] + + def slow_check(*args, **kwargs): + import time + + time.sleep(0.5) + + verifier = _make_verifier(runner, timeout=0.05) + + with patch.object(verifier, "_run_migration_check", side_effect=slow_check): + result = verifier.check_revision("rev1") + + assert not result.passed + assert any("timed out" in f for f in result.failures) + # current != start_revision → recovery should have been attempted + runner.downgrade_base.assert_called() + + # ── error recovery ────────────────────────────────────────────────────────────