Make catalog refresh atomic

[simonw]

Refs:

• Update transactions code for sqlite-utils 4.0 #2831

Summary

• collect all schema metadata before mutating the internal catalog
• update catalog_databases and replace the five child catalog tables in one execute_write_fn() transaction
• delete child rows before parent rows so the operation remains valid if a prepare_connection plugin enables foreign-key enforcement

Why

The previous refresh committed the catalog_databases update, catalog deletion, and five insertion batches separately. Readers could observe an updated schema version with empty or partially rebuilt catalog tables, and each refresh paid for seven transactions.

The new flow prepares the replacement rows first and then applies the complete per-database catalog update atomically. A failure restores the previous catalog through the outer transaction.

This implements the atomic catalog refresh recommendation from the Datasette transaction report.

Performance

A local microbenchmark used Python 3.14.3, a persistent internal database, repeated schema changes, and generated schemas; setup and schema mutation were excluded from timing.

Tables	Baseline mean	New mean	Improvement
150	7.96 ms	5.68 ms	28.7%
300	12.80 ms	10.02 ms	21.7%

For 300 tables, median improved 15.3% and p95 improved 41.9%. Measured transactions per refresh fell from seven to one.

Validation

• uv run pytest -q — 2,294 passed, 40 skipped, 6 xfailed, 15 xpassed, 141 subtests passed
• uv run pytest -q tests/test_internal_db.py — 11 passed
• uv run black --check datasette/app.py datasette/utils/internal_db.py tests/test_internal_db.py
• uv run ruff check datasette/app.py datasette/utils/internal_db.py tests/test_internal_db.py
• git diff --check

[codecov]

Codecov Report

❌ Patch coverage is 0% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (10088df) to head (de7023c).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
datasette/utils/internal_db.py	0.00%	11 Missing ⚠️
datasette/app.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##            main   #2847   +/-   ##
=====================================
  Coverage   0.00%   0.00%           
=====================================
  Files         73      73           
  Lines      12215   12207    -8     
=====================================
+ Misses     12215   12207    -8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.