Serialize local sqlite initialization under a file lock

With WEB_CONCURRENCY=2, both gunicorn workers import the app
concurrently on a fresh instance; both could pass the sqlite
exists-check and race initialize() — the losing worker died at boot on
'UNIQUE constraint failed: policy.id' (observed as 503 bursts during
the first real Cloud Run scale-out, Stage 5 LB validation), and a
worker could also observe a created-but-unseeded database and skip
initialization entirely, serving with empty reference tables. An
exclusive fcntl lock around the exists-check + initialize means exactly
one process initializes while the others wait, then see the complete
file and skip.

Empirical verification (6 concurrent fresh boots x 3 rounds): unfixed
code failed every round (boot crashes and row counts of 0/2/5 across
workers); fixed code passes with all workers seeing exactly 5 seed
rows.

Fixes #3746

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: anth-volk

changelog.d/local-db-init-race.fixed.md

@@ -0,0 +1 @@
+Serialize local sqlite initialization under a file lock so concurrent gunicorn worker boots cannot race the seed inserts or observe a partially initialized database.

policyengine_api/data/data.py

@@ -1,3 +1,4 @@
+import fcntl
 import sqlite3
 from policyengine_api.constants import REPO, COUNTRY_PACKAGE_VERSIONS
 from policyengine_api.utils import hash_object
@@ -106,8 +107,20 @@ def __init__(
         if local:
             # Local development uses a sqlite database.
             self.db_url = REPO / "policyengine_api" / "data" / "policyengine.db"
-            if initialize or not Path(self.db_url).exists():
-                self.initialize()
+            # Serialize the exists-check + initialize under an exclusive file
+            # lock: with multiple gunicorn workers importing concurrently on a
+            # fresh instance, both can otherwise pass the exists() check and
+            # race initialize() (seed INSERTs collide -> worker dies at boot),
+            # or one can observe a created-but-unseeded file and skip
+            # initialization entirely.
+            lock_path = str(self.db_url) + ".init.lock"
+            with open(lock_path, "w") as lock_file:
+                fcntl.flock(lock_file, fcntl.LOCK_EX)
+                try:
+                    if initialize or not Path(self.db_url).exists():
+                        self.initialize()
+                finally:
+                    fcntl.flock(lock_file, fcntl.LOCK_UN)
         else:
             self._create_pool()
             if initialize: