fix(rest-postgres): idempotent DB init so a partial init can't silently drop databases (#2623)

## Problem
`rest-api/deploy/kustomize/base/postgres/init-configmap.yaml` runs once
via `docker-entrypoint-initdb.d` under `psql -v ON_ERROR_STOP=1`. If any
statement errors, psql aborts and the remaining `CREATE
DATABASE`/`CREATE USER` statements are skipped, leaving a **partial**
state — e.g. `nico`+`keycloak` created but
`temporal`/`temporal_visibility` missing.

Because PostgreSQL only runs init scripts on an **empty** data dir, the
script **never re-runs**, so the missing databases are silent and
permanent. The Temporal install then fails its pre-install hook:
```
Error: failed pre-install: timed out waiting for the condition
FATAL: password authentication failed for user "temporal"
DETAIL: Role "temporal" does not exist.
```
Reproduced during a clean v0.10.3 bring-up.

## Fix
Make every database/role creation **idempotent**:
- `SELECT … WHERE NOT EXISTS (…)\gexec` guard for `CREATE DATABASE`
- `DO $$ … IF NOT EXISTS … $$` for roles

The script is now safe to re-run, and a single benign "already exists"
error no longer aborts it and skips later databases.

## Test
Re-applied against a partially-initialized postgres (had only
nico+keycloak): all four DBs (`nico`, `keycloak`, `temporal`,
`temporal_visibility`) present; Temporal install proceeds.

Signed-off-by: ekirson <ekirson@nvidia.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Hasan Khan <hasank@nvidia.com>

Author: 3 people

rest-api/deploy/kustomize/base/postgres/init-configmap.yaml

@@ -13,41 +13,40 @@ data:
     -- Enable pg_trgm extension (required for text search)
     CREATE EXTENSION IF NOT EXISTS pg_trgm;
 
-    -- Create nico database and user (used by API, Workflow)
-    CREATE DATABASE nico WITH ENCODING 'UTF8';
-    CREATE USER nico WITH PASSWORD 'nico';
+    -- nico database + user (idempotent: safe on re-run / partial init)
+    SELECT 'CREATE DATABASE nico WITH ENCODING ''UTF8'''
+      WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'nico')\gexec
+    DO $$ BEGIN IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname='nico') THEN CREATE ROLE nico LOGIN PASSWORD 'nico'; END IF; END $$;
     GRANT ALL PRIVILEGES ON DATABASE nico TO nico;
-
-    -- Connect to nico database to set up permissions
     \c nico
     GRANT ALL ON SCHEMA public TO nico;
     CREATE EXTENSION IF NOT EXISTS pg_trgm;
-
-    -- Create keycloak database
     \c postgres
-    CREATE DATABASE keycloak WITH ENCODING 'UTF8';
-    CREATE USER keycloak WITH PASSWORD 'keycloak';
-    GRANT ALL PRIVILEGES ON DATABASE keycloak TO keycloak;
 
+    -- keycloak database + user
+    SELECT 'CREATE DATABASE keycloak WITH ENCODING ''UTF8'''
+      WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'keycloak')\gexec
+    DO $$ BEGIN IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname='keycloak') THEN CREATE ROLE keycloak LOGIN PASSWORD 'keycloak'; END IF; END $$;
+    GRANT ALL PRIVILEGES ON DATABASE keycloak TO keycloak;
     \c keycloak
     GRANT ALL ON SCHEMA public TO keycloak;
-
-    -- Create temporal databases (for Temporal workflow engine)
     \c postgres
-    CREATE DATABASE temporal WITH ENCODING 'UTF8';
-    CREATE USER temporal WITH PASSWORD 'temporal' CREATEDB;
+
+    -- temporal databases (for Temporal workflow engine)
+    SELECT 'CREATE DATABASE temporal WITH ENCODING ''UTF8'''
+      WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'temporal')\gexec
+    DO $$ BEGIN IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname='temporal') THEN CREATE ROLE temporal LOGIN PASSWORD 'temporal' CREATEDB; END IF; END $$;
     GRANT ALL PRIVILEGES ON DATABASE temporal TO temporal;
     ALTER DATABASE temporal OWNER TO temporal;
-
     \c temporal
     GRANT ALL ON SCHEMA public TO temporal;
     ALTER SCHEMA public OWNER TO temporal;
-
     \c postgres
-    CREATE DATABASE temporal_visibility WITH ENCODING 'UTF8';
+
+    SELECT 'CREATE DATABASE temporal_visibility WITH ENCODING ''UTF8'''
+      WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'temporal_visibility')\gexec
     GRANT ALL PRIVILEGES ON DATABASE temporal_visibility TO temporal;
     ALTER DATABASE temporal_visibility OWNER TO temporal;
-
     \c temporal_visibility
     GRANT ALL ON SCHEMA public TO temporal;
     ALTER SCHEMA public OWNER TO temporal;