Problem
When running init against SerenDB/Neon targets where the database already exists, the job hangs indefinitely at the database_is_empty() check. This has been reproduced on multiple jobs:
- Job
438c3593-057c-45f3-9d23-6ddce6cea1bb
- Job
314eacf8-5fcc-48d5-822a-3350f3b5a0e3
Both jobs hang for 6+ minutes at the exact same point:
2025-12-03T20:53:37Z INFO: Database 'serendb' already exists on target
[HANG - no further output]
Root Cause Analysis
The issue occurs in src/commands/init.rs at line 403:
- Line 377:
target_client connects to the target database
- Line 388:
CREATE DATABASE fails with DUPLICATE_DATABASE error (expected)
- Line 403:
database_is_empty() is called, which creates a new connection at line 677
The problem is that database_is_empty() creates a second connection while target_client is still holding the first one. SerenDB/Neon has connection pooling limits, and the second connection blocks waiting for a slot.
Additionally, tokio_postgres::connect has no connection timeout, so it hangs indefinitely rather than failing.
Proposed Fix
Change database_is_empty() to accept an existing &Client reference instead of creating a new connection:
// Before (creates new connection)
async fn database_is_empty(target_url: &str, db_name: &str) -> Result<bool> {
let db_url = replace_database_in_url(target_url, db_name)?;
let client = postgres::connect_with_retry(&db_url).await?; // HANG HERE
// ...
}
// After (reuses existing connection)
async fn database_is_empty(client: &Client) -> Result<bool> {
let query = "
SELECT COUNT(*)
FROM information_schema.tables
WHERE table_schema NOT IN ('pg_catalog', 'information_schema')
";
let row = client.query_one(query, &[]).await?;
let count: i64 = row.get(0);
Ok(count == 0)
}
Then update the call site at line 403 to pass the existing target_client.
Additional Improvement
Consider adding a connection timeout to connect_with_retry() to prevent indefinite hangs in similar scenarios.
Environment
- SerenDB/Neon targets with
channel_binding=require
- Connection pooling with limited concurrent connections
- Remote execution via AWS EC2 workers
Problem
When running
initagainst SerenDB/Neon targets where the database already exists, the job hangs indefinitely at thedatabase_is_empty()check. This has been reproduced on multiple jobs:438c3593-057c-45f3-9d23-6ddce6cea1bb314eacf8-5fcc-48d5-822a-3350f3b5a0e3Both jobs hang for 6+ minutes at the exact same point:
Root Cause Analysis
The issue occurs in
src/commands/init.rsat line 403:target_clientconnects to the target databaseCREATE DATABASEfails withDUPLICATE_DATABASEerror (expected)database_is_empty()is called, which creates a new connection at line 677The problem is that
database_is_empty()creates a second connection whiletarget_clientis still holding the first one. SerenDB/Neon has connection pooling limits, and the second connection blocks waiting for a slot.Additionally,
tokio_postgres::connecthas no connection timeout, so it hangs indefinitely rather than failing.Proposed Fix
Change
database_is_empty()to accept an existing&Clientreference instead of creating a new connection:Then update the call site at line 403 to pass the existing
target_client.Additional Improvement
Consider adding a connection timeout to
connect_with_retry()to prevent indefinite hangs in similar scenarios.Environment
channel_binding=require