Add CI check that the previous release works against the new schema#3506
Add CI check that the previous release works against the new schema#3506jonathangreen wants to merge 2 commits into
Conversation
Add a backwards-compatibility CI job that runs the previous release's database tests (pytest -m db) against the schema produced by the current code, exercising the online-migration requirement that a release's database must keep working with version N-1 of the code while its webservers are still serving traffic. - docker/ci/test_backwards_compatibility.yml: compose overlay defining a webapp-prev service (the previous release's image) in external-schema mode. - docker/ci/test_backwards_compatibility.sh: resolves the previous release from the latest GitHub release, builds the new schema with the current image, then runs the previous release's db tests against it. Skips when the previous release predates the external-schema seam. - build.yml: new backwards-compatibility-test job, added to the push job's needs. This is the second of two PRs and depends on the test seam from the first. Until a release containing the seam becomes the previous release, the check skips.
|
Claude finished @jonathangreen's task in 5m 53s —— View job Review
SummaryNice addition — this closes a real gap by enforcing the online-migration N-1 invariant in CI before images are pushed. The digest/artifact handling and the Details
|
…bility job The previous release now ships the external-schema test seam, so the job no longer needs to detect older images that lack it and skip.
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| - uses: actions/checkout@v6 | ||
| with: | ||
| persist-credentials: false |
There was a problem hiding this comment.
actions/checkout step in this workflow uses @v7 (lines 45, 264, 335, 384), but the new job pins @v6. If @v6 resolves to a different major-version tag (or doesn't exist), the checkout step will either fail outright or silently use an older version with different semantics.
| - uses: actions/checkout@v6 | |
| with: | |
| persist-credentials: false | |
| - uses: actions/checkout@v7 | |
| with: | |
| persist-credentials: false |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3506 +/- ##
=======================================
Coverage 93.44% 93.44%
=======================================
Files 511 511
Lines 46465 46465
Branches 6339 6339
=======================================
Hits 43419 43419
- Misses 1969 1970 +1
+ Partials 1077 1076 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…3507) ## Description Makes four tests robust to running serially with pytest-xdist disabled (`-n0`). They pass under the normal xdist test run but fail under `-n0`, which is how the backwards-compatibility CI check (#3506) runs a previous release's database suite against the current schema. - **Three script tests** (`test_equivalents`, `test_novelist`, `test_search`) constructed a `Script` with no `cmd_args` and called `do_run()`, so the script's argparse fell back to `sys.argv[1:]`. Under xdist each worker has a clean execnet `sys.argv`, but under `-n0` `sys.argv` is the real pytest command line. That caused argparse to error on unrecognized arguments (`equivalents`, `novelist`), and the leaked `-m db` marker matched `RebuildSearchIndexScript`'s `-m/--migration` flag, sending it down the migration path so `search_reindex.delay` was never called (`search`). Each now passes an explicit `cmd_args=[]` to express "no arguments" independent of `sys.argv`. - **The OIDC logout test** (`test_oidc_logout_initiate_no_stored_id_token`) asserted on `caplog` without setting a capture level. Under serial execution the effective log level is left raised by an earlier test, so the warning was not captured and `caplog.text` was empty. It now sets the level explicitly, matching its passing sibling `test_oidc_logout_initiate_revocation_only`. No production code changes — scripts intentionally default `cmd_args` to `sys.argv` for real CLI use via `Script.run()`. ## Motivation and Context The backwards-compatibility check added in #3506 runs the previous release's `-m db` suite under `-n0` (external-schema mode requires serial execution — the single shared database can't be used by parallel xdist workers). Its first real run surfaced these four latent test-isolation bugs, which xdist had been masking. None are schema-compatibility failures; all four fail identically against the previous release's own schema. Fixing them here means that once a release containing these fixes becomes the "previous release", the check will pass against it. ## How Has This Been Tested? Reproduced the failures on the unfixed tests by running them under `-n0` in the docker test environment (`tox -e py312-docker -- -n0 ...`), observing the argparse `unrecognized arguments` errors. With the fixes applied, all four pass under `-n0` (including with `-m db` present, which exercises the `search` flag-collision case): `60 passed, 19 deselected`. ## Checklist - [x] I have updated the documentation accordingly. - [x] All new and existing tests passed.
|
This is going to fail until #3507 gets into a release, so I'll leave it in draft for now. |
1 participant
Description
Adds a CI job,
backwards-compatibility-test, that verifies the previous released version of the application still works against the database schema produced by the current code. This enforces our online-migration requirement: a release's database must keep working with version N-1 of the code, because new migrations run while the previous version's webservers are still serving traffic during a deploy.The check works by:
circ-webappimage.pytest -m db) against that schema in external-schema mode (PALACE_TEST_DATABASE_EXTERNAL_SCHEMA), so the older code exercises the new schema. If those tests fail, the migration is not backwards compatible.New files:
docker/ci/test_backwards_compatibility.sh— orchestrates the check.docker/ci/test_backwards_compatibility.yml— compose overlay adding awebapp-prevservice that runs the previous release's image in external-schema mode.The
pushjob now depends onbackwards-compatibility-testso images are not tagged and pushed unless the check passes.Motivation and Context
Our migrations are run online, while the previous version of the application is still serving traffic, so a release's database must work with version N-1 of the code. Until now nothing in CI verified that invariant — a migration that broke backwards compatibility would only be caught during a deploy. This job catches such regressions before images are pushed.
How Has This Been Tested?
The check is itself a CI job and runs as part of this PR's build against the latest published release image. The script also supports a
PREV_RELEASE_IMAGEoverride for running it locally against a chosen previous image.Checklist