Fixed racy first-start database initialization

[zachmu]

Dolt's CREATE DATABASE is non-transactional and non-atomic, which means that a different session might see a partially initialized database. This is a much bigger problem for Doltgres because it creates a postgres database on first run if no database exists, and clients polling for readiness might see this database before it's fully initialized.

To fix this, we create the initial database as a startup service, before we begin accepting connections.

Additionally, we add more defensive readiness checks in go-sql-driver, where this race was most commonly observed. They should be redundant at this point.

Depends on dolthub/dolt#11286

[github-actions]

	Main	PR
covering_index_scan_postgres	1901.04/s	1902.31/s	0.0%
groupby_scan_postgres	131.42/s	125.76/s	-4.4%
index_join_postgres	666.45/s	657.92/s	-1.3%
index_join_scan_postgres	850.36/s	844.30/s	-0.8%
index_scan_postgres	27.27/s	27.00/s	-1.0%
oltp_delete_insert_postgres	829.99/s	807.69/s	-2.7%
oltp_insert	697.38/s	710.87/s	+1.9%
oltp_point_select	3055.21/s	3053.59/s	-0.1%
oltp_read_only	3030.27/s	2982.99/s	-1.6%
oltp_read_write	2329.55/s	2309.57/s	-0.9%
oltp_update_index	754.26/s	749.65/s	-0.7%
oltp_update_non_index	788.90/s	791.78/s	+0.3%
oltp_write_only	1794.49/s	1785.92/s	-0.5%
select_random_points	1969.48/s	1943.32/s	-1.4%
select_random_ranges	1597.64/s	1586.95/s	-0.7%
table_scan_postgres	25.69/s	25.85/s	+0.6%
types_delete_insert_postgres	794.48/s	793.72/s	-0.1%
types_table_scan_postgres	11.31/s	11.29/s	-0.2%

[itoqa]

Commit: c1ed181: 10 test cases ran, 1 failed ❌, 7 passed ✅, 2 additional findings ⚠️.

Summary

This run exercised first-run startup and bootstrap behavior end to end, including connection gating, default database creation, readiness timing, retry handling, and restart recovery under failure conditions. Overall core startup and driver initialization flows look healthy, with issues concentrated in edge-case failure windows rather than normal initialization.

Merge with caution — one medium-severity failure is attributable to this PR and indicates a startup-ordering gap where the service can appear ready and accept real queries before full initialization viability is confirmed. Other failures are non-attributable caveats in existing bootstrap error-handling paths and are not the primary merge blocker for this change.

Tests run by Ito

Result	Severity	Type	Description
❌		Startup	The gate is released before replication startup is validated, so the process accepts external traffic during a transient window where overall startup is not yet viable.
✅	—	Bootstrap	With a fresh data directory and DOLTGRES_DB=teamdb, the server started successfully and only teamdb existed in pg_database while connections to postgres failed because that default database was not created.
✅	—	Driver	RepoStore.MakeRepo completed successfully, a fresh server immediately observed the CREATE DATABASE entry in dolt_log, and an immediate CREATE TABLE succeeded, confirming initDatabase waited for commit visibility before returning.
✅	—	Driver	Two consecutive MakeRepo/initDatabase calls completed quickly (0.12s each), showing no retry-cap edge artifact and matching expected readiness behavior.
✅	—	Driver	waitForDatabaseInit exhausted the configured retry budget and returned an explicit readiness error when the CREATE DATABASE commit signal never appeared, which matches the expected behavior.
✅	—	Startup	On first-run startup with an empty data directory, external TCP connections were only accepted after the internal bootstrap connection completed and the startup gate was released.
✅	—	Startup	Re-execution confirms startup fails closed as designed: invalid first-run database name causes CREATE DATABASE to error, the controller stops, and external connections are refused.
✅	—	Startup	Three overlapping restart attempts during first-run bootstrap converged to one stable server state with no accept/refuse oscillation.
⚠️		Bootstrap	Expected: retry reaches a consistent running state without manual intervention after partial bootstrap state. Actual: startup repeatedly exits with create-database errors (database exists / incomplete database directory ... remove the directory and try again) and only succeeds after manual directory deletion.
⚠️		Bootstrap	Safe failure was not atomic: startup failed, but CREATE DATABASE db side effects persisted on disk before the later statement error terminated startup.

Additional Findings Details

These findings are unrelated to the current changes but were observed during testing.

🟡 Retry fails after interrupted database bootstrap

• Severity: Medium
• Description: Expected: retry reaches a consistent running state without manual intervention after partial bootstrap state. Actual: startup repeatedly exits with create-database errors (database exists / incomplete database directory ... remove the directory and try again) and only succeeds after manual directory deletion.
• Impact: Under interrupted bootstrap conditions, the service can fail to create or start the target database until manual directory cleanup is performed. This can prolong downtime for the affected environment during recovery.
• Steps to Reproduce:
  1. Start the server with DOLTGRES_DB=teamdb and a data directory that already contains teamdb/ in a partial state.
  2. Allow startup to run its default bootstrap path that executes CREATE DATABASE teamdb.
  3. Restart again without removing the partial directory and observe startup fail with the same create-database errors until manual rm -rf cleanup is done.
• Stub / mock content: No stubs, mocks, or bypasses were applied for this test in the recorded run.
• Code Analysis: In server/server.go, runServer immediately returns startup failure when createDefaultDatabase returns an error, with no repair path for stale bootstrap state. In createDefaultDatabase, the code executes CREATE DATABASE <name> and directly returns the resulting error; there is no branch that detects an interrupted prior create and self-heals before retry. The PR adds startup gate wiring and internal dialing, but those changed lines do not introduce this failure mode or provide recovery logic for pre-existing partial database directories.

Evidence Package

• text/BOOTSTRAP-3/BOOTSTRAP-3-retry-failure-evidence.txt — 3 KB
• text/BOOTSTRAP-3/BOOTSTRAP-3-reproduction-script.txt — 2 KB

⚪ Malformed DOLTGRES_DB input leaves partial database side effects

• Severity: Minor
• Description: Safe failure was not atomic: startup failed, but CREATE DATABASE db side effects persisted on disk before the later statement error terminated startup.
• Impact: The bootstrap check can stall and leave teams without a reliable readiness signal, which can delay releases and obscure real product regressions.
• Steps to Reproduce:
  1. Set DOLTGRES_DB to db;DROP TABLE x;-- and start the server on a fresh empty data directory.
  2. Observe startup exits with an error and external connections are refused.
  3. Inspect the data directory and confirm a new db/ directory was created despite the failed startup.
• Stub / mock content: No stubs, mocks, or bypasses were applied for this test in the recorded run.
• Code Analysis: getDefaultDatabaseName returns DOLTGRES_DB directly, and createDefaultDatabase interpolates that value into CREATE DATABASE %s; via fmt.Sprintf before conn.Exec. With semicolon-delimited input, the first statement can execute and persist state before a subsequent statement fails, matching the observed partial db/ directory. A targeted fix is to validate database names as strict identifiers (or safely quote/escape identifiers) and reject multi-statement characters before issuing SQL.

Evidence Package

• text/BOOTSTRAP-2/BOOTSTRAP-2-malformed-db-side-effect.txt — 3 KB

Tip

Reply with @itoqa to send us feedback on this test run.

[itoqa]

Startup gate opens before replication viability is known

What failed: The gate is released before replication startup is validated, so the process accepts external traffic during a transient window where overall startup is not yet viable.

Impact · Steps · Stub / mock · Analysis · Why this is likely a bug

• Severity: Medium
• Impact: When replication startup fails, the server can briefly accept real client queries before shutting down, causing interrupted sessions and unreliable startup behavior for database users.
• Steps to Reproduce:
  1. Start the server with an empty data directory so first-run default database initialization runs under the startup gate.
  2. Configure replication to a non-routable upstream (for example 10.255.255.1) so startReplication() cannot establish viability promptly.
  3. Race external psql connections immediately after startup and observe successful queries before replication startup later fails/stalls.
• Stub / mock content: No stubs, mocks, or bypasses were applied for this test in the recorded run.
• Code Analysis: In server/server.go, the PR-modified startup path calls createDefaultDatabase(ssCfg, gate), then gate.Release(), and only afterward runs startReplication(cfg, ssCfg). If replication initialization blocks or fails, external connections have already been allowed. Runtime evidence matches this ordering: logs show Starting replication followed by external NewConnection IDs 2-8, and captured psql queries (SELECT 1, SELECT version()) succeed during that window.
• Why this is likely a bug: The expected behavior for this test is that external acceptance is deferred until replication viability is known, or that temporary accepts are safely isolated. The current code does neither: it intentionally opens the gate first and then performs replication startup, which allows real client queries in an inconsistent startup state.

Relevant code

server/server.go:194-207

if initializeDefaultDatabase {
	err = createDefaultDatabase(ssCfg, gate)
	if err != nil {
		controller.Stop()
		return nil, err
	}
	// Release the startup gate to accept external connections now that init is finished
	gate.Release()
}

// TODO: shutdown replication cleanly when we stop the server
_, err = startReplication(cfg, ssCfg)
if err != nil {
	controller.Stop()
	return nil, err
}

Evidence Package

• text/STARTUP-3/STARTUP-3-gate-release-replication-window.txt — 4 KB

Copy prompt for an agent

Ito QA identified the following failure during automated PR testing. Please investigate and propose a fix.

**Medium severity — Startup gate opens before replication viability is known**

**What failed:** The gate is released before replication startup is validated, so the process accepts external traffic during a transient window where overall startup is not yet viable.

- **Impact:** When replication startup fails, the server can briefly accept real client queries before shutting down, causing interrupted sessions and unreliable startup behavior for database users.
- **Steps to reproduce:**
  1. Start the server with an empty data directory so first-run default database initialization runs under the startup gate.
  2. Configure replication to a non-routable upstream (for example `10.255.255.1`) so `startReplication()` cannot establish viability promptly.
  3. Race external `psql` connections immediately after startup and observe successful queries before replication startup later fails/stalls.
- **Stub / mock content:** No stubs, mocks, or bypasses were applied for this test in the recorded run.
- **Code analysis:** In `server/server.go`, the PR-modified startup path calls `createDefaultDatabase(ssCfg, gate)`, then `gate.Release()`, and only afterward runs `startReplication(cfg, ssCfg)`. If replication initialization blocks or fails, external connections have already been allowed. Runtime evidence matches this ordering: logs show `Starting replication` followed by external `NewConnection` IDs 2-8, and captured `psql` queries (`SELECT 1`, `SELECT version()`) succeed during that window.
- **Why this is likely a bug:** The expected behavior for this test is that external acceptance is deferred until replication viability is known, or that temporary accepts are safely isolated. The current code does neither: it intentionally opens the gate first and then performs replication startup, which allows real client queries in an inconsistent startup state.

**Relevant code:**

`server/server.go:194-207`

~~~go
if initializeDefaultDatabase {
	err = createDefaultDatabase(ssCfg, gate)
	if err != nil {
		controller.Stop()
		return nil, err
	}
	// Release the startup gate to accept external connections now that init is finished
	gate.Release()
}

// TODO: shutdown replication cleanly when we stop the server
_, err = startReplication(cfg, ssCfg)
if err != nil {
	controller.Stop()
	return nil, err
}
~~~

[github-actions]

	Main	PR
Total	42090	42090
Successful	18275	18274
Failures	23815	23816
Partial Successes1	5335	5335

	Main	PR
Successful	43.4189%	43.4165%
Failures	56.5811%	56.5835%

${\color{red}Regressions (1)}$

random

QUERY:          (SELECT unique1 AS random
  FROM onek ORDER BY random() LIMIT 1)
INTERSECT
(SELECT unique1 AS random
  FROM onek ORDER BY random() LIMIT 1)
INTERSECT
(SELECT unique1 AS random
  FROM onek ORDER BY random() LIMIT 1);
RECEIVED ERROR: expected row count 0 but received 1

Footnotes

1. These are tests that we're marking as Successful, however they do not match the expected output in some way. This is due to small differences, such as different wording on the error messages, or the column names being incorrect while the data itself is correct. ↩