Self-host: share one libSQL connection with Better Auth (fix WAL data loss) (#1008)

Better Auth opened its own LibsqlDialect connection (@libsql/core 0.8) to the
same file as executor-core's @libsql/client (0.17). Opening second, it unlinked
executor-core's -wal/-shm and created new ones, orphaning executor-core onto a
deleted WAL inode. Every executor-core write (integrations, connections, tools)
then vanished on the next restart while Better Auth's own tables survived — the
'reconnected account, zero tools' data-loss bug. The earlier 'single Better Auth
instance' change removed only the throwaway bootstrap connection; the long-lived
one unlinked the WAL just the same.

Build Better Auth's LibsqlDialect on SelfHostDb's existing client ({ client },
closeClient=false) so there is one connection and one WAL. Also resolves the
documented cross-connection read lag.

Author: RhysSullivan

.changeset/selfhost-wal-shared-connection.md

@@ -0,0 +1,12 @@
+---
+"executor": patch
+---
+
+Self-hosted instances no longer lose data on restart. Better Auth now shares
+the same libSQL connection as the rest of the instance instead of opening its
+own. Previously the two connections each managed their own write-ahead log on
+the shared database file, and the second one to open could orphan the first —
+so integrations, connections, and tools written after startup landed in a
+discarded log and disappeared on the next restart, while sign-in data survived.
+This is the "reconnected my account but it has zero tools" failure; a single
+shared connection removes the split entirely.

apps/host-selfhost/src/auth/better-auth.ts

@@ -3,7 +3,7 @@ import { APIError } from "better-auth/api";
 import { admin, bearer, mcp, organization } from "better-auth/plugins";
 import { apiKey } from "@better-auth/api-key";
 import { type Client } from "@libsql/client";
-import { LibsqlDialect } from "@libsql/kysely-libsql";
+import { LibsqlDialect, type LibsqlDialectConfig } from "@libsql/kysely-libsql";
 import { Context } from "effect";
 
 import { loadConfig } from "../config";
@@ -26,42 +26,42 @@ interface SignupGate {
 const SIGNUP_PATH = "/sign-up/email";
 
 // ---------------------------------------------------------------------------
-// Better Auth instance over the SAME libSQL `file:` URL as the FumaDB executor
-// tables ("one file, two schema regions").
+// Better Auth instance over the SAME libSQL CONNECTION as the FumaDB executor
+// tables ("one connection, two schema regions").
 //
-// Schema-at-boot: passing `{ dialect: new LibsqlDialect({ url }), type: "sqlite" }`
-// makes Better Auth's createKyselyAdapter take its `"dialect" in db` branch (no
-// native dep, no bun:sqlite); `runMigrations()` creates the auth tables
-// idempotently in that file. `makeAuthOptions` is the single source of truth so
-// the migrator and runtime instance never drift.
+// Schema-at-boot: passing `{ dialect: new LibsqlDialect({ client }), type:
+// "sqlite" }` makes Better Auth's createKyselyAdapter take its `"dialect" in db`
+// branch (no native dep, no bun:sqlite); `runMigrations()` creates the auth
+// tables idempotently. `makeAuthOptions` is the single source of truth so the
+// migrator and runtime instance never drift.
 //
-// CRITICAL: LibsqlDialect opens its OWN libSQL connection to the file — it does
-// NOT share SelfHostDb's drizzle connection. Both target one file, and a row
-// Better Auth writes via this dialect is immediately readable through the
-// drizzle/FumaDB client (proven by seed.ts's reads + better-auth.test.ts). The
-// per-connection foreign_keys/WAL PRAGMAs SelfHostDb set on its own connection
-// do NOT carry to this one; for the auth tables that is fine (Kysely issues no
-// FK-dependent reads at boot and WAL is already a file-level mode), and the
-// shared file stays consistent because writes go through SQLite's file lock.
+// CRITICAL: LibsqlDialect is handed SelfHostDb's EXISTING `@libsql/client` (the
+// `{ client }` config branch), NOT a fresh `{ url }` connection. This is the
+// crux of the self-host data-loss fix: libSQL connections each manage their own
+// `-wal`/`-shm`, and when Better Auth opened a SECOND connection to the same
+// file (`{ url }`), its open unlinked SelfHostDb's `-wal`/`-shm` and created new
+// ones — orphaning SelfHostDb onto a now-deleted WAL inode. Every executor-core
+// write (integrations, connections, tools) then landed in that deleted inode
+// and vanished on the next restart, while Better Auth's own writes (on the live
+// WAL) survived — the "reconnected account, zero tools" bug, reproducing even
+// after the throwaway-bootstrap-instance fix because the LONG-LIVED auth
+// connection unlinked it just the same. Sharing one client means one WAL: no
+// unlink, and SelfHostDb's foreign_keys/WAL/busy_timeout PRAGMAs now cover auth
+// queries too (same connection). `{ client }` sets closeClient=false, so the
+// dialect never closes the handle — SelfHostDb owns the file lifecycle and
+// closes its client at shutdown. NEVER call .destroy() during normal operation.
 //
 // We build exactly ONE auth instance, held for the process lifetime. An earlier
-// design built a throwaway "bootstrap" instance to run migrations + seed before
-// the org id was known, then discarded it — but its LibsqlDialect connection
-// (a DIFFERENT native libSQL build than SelfHostDb's) was GC-closed mid-boot,
-// and that close unlinked the shared `-wal` out from under SelfHostDb's
-// still-open connection. Every executor write then landed in a deleted WAL
-// inode and vanished on the next restart (the "reconnected account, zero tools"
-// data-loss bug). Keeping one long-lived auth connection — with the org id
-// late-bound the same way the signup gate's `getAuth` already is — removes the
-// discarded connection entirely. NEVER call .destroy() during normal operation;
-// SelfHostDb owns the file lifecycle and closes its client at shutdown.
+// design also built a throwaway "bootstrap" instance (discarded mid-boot); that
+// is gone too — the org id is late-bound the same way the signup gate's
+// `getAuth` already is, so no second instance is ever needed.
 //
 // `satisfies BetterAuthOptions` (not a return annotation) keeps the literal
 // plugin tuple so `betterAuth` infers the plugin-augmented `auth.api` and
 // session/user shapes (activeOrganizationId, role, createUser, ...).
 // ---------------------------------------------------------------------------
 
-const makeAuthOptions = (url: string, getOrganizationId: () => string, gate?: SignupGate) => {
+const makeAuthOptions = (client: Client, getOrganizationId: () => string, gate?: SignupGate) => {
   const config = loadConfig();
   // Always resolved (generated + persisted when no env is set); this guards only
   // an explicitly-set env secret that is too weak.
@@ -71,7 +71,22 @@ const makeAuthOptions = (url: string, getOrganizationId: () => string, gate?: Si
     throw new Error("BETTER_AUTH_SECRET (or AUTH_SECRET), if set, must be at least 32 characters");
   }
   return {
-    database: { dialect: new LibsqlDialect({ url }), type: "sqlite" as const },
+    // Hand Better Auth the SAME libSQL client SelfHostDb already opened — NOT a
+    // fresh `{ url }` connection. `{ client }` makes LibsqlDialect adopt the
+    // existing handle (closeClient=false, so SelfHostDb keeps ownership). One
+    // connection means one WAL: see the header comment for why a second
+    // connection is the self-host data-loss bug.
+    //
+    // The cast bridges a dependency skew: @libsql/kysely-libsql pins an older
+    // @libsql/core (0.8) than @libsql/client (0.17), so the two `Client` types
+    // differ — only in `.sync()` (embedded-replica replication, unused here).
+    // The dialect calls execute/batch/transaction/close, which are identical
+    // across both versions, so sharing the 0.17 client is sound at runtime.
+    database: {
+      // oxlint-disable-next-line executor/no-double-cast -- boundary: the two @libsql/core versions' Client types are structurally identical for the calls the dialect makes (see above); no schema/decode applies to a native client handle.
+      dialect: new LibsqlDialect({ client } as unknown as LibsqlDialectConfig),
+      type: "sqlite" as const,
+    },
     secret,
     baseURL: config.webBaseUrl,
     // The browser Origin must match this exactly; CLI/MCP bearer requests carry
@@ -191,11 +206,12 @@ const inviteCodeFrom = (context: { body?: unknown }): string | undefined => {
   return undefined;
 };
 
-// Count org members via Better Auth's OWN adapter — the SAME connection that
-// `addMember` writes through. SelfHostDb opens a SEPARATE libSQL connection
-// whose snapshot can lag Better Auth's writes (observed under Bun: a just-added
-// member is invisible to that connection for a while), so any membership read
-// that gates behaviour MUST go through here to stay consistent with the writes.
+// Count org members via Better Auth's OWN adapter. Now that auth shares
+// SelfHostDb's libSQL client (one connection), this no longer guards against a
+// cross-connection snapshot lag — that lag is gone with the second connection.
+// It stays the canonical read because the adapter already models the `member`
+// table and the count gates the first-run claim; reading through it keeps the
+// gate logic next to the writes.
 export const countOrgMembers = (auth: Auth, organizationId: string): Promise<number> =>
   auth.$context.then(({ adapter }) =>
     adapter.count({ model: "member", where: [{ field: "organizationId", value: organizationId }] }),
@@ -208,8 +224,8 @@ const orgHasNoMembers = async (gate: SignupGate): Promise<boolean> => {
   return (await countOrgMembers(auth, gate.organizationId)) === 0;
 };
 
-const createAuthInstance = (url: string, getOrganizationId: () => string, gate?: SignupGate) =>
-  betterAuth(makeAuthOptions(url, getOrganizationId, gate));
+const createAuthInstance = (client: Client, getOrganizationId: () => string, gate?: SignupGate) =>
+  betterAuth(makeAuthOptions(client, getOrganizationId, gate));
 
 export type Auth = ReturnType<typeof createAuthInstance>;
 
@@ -241,13 +257,13 @@ export class BetterAuth extends Context.Service<BetterAuth, BetterAuthHandle>()(
  * `/sign-up/email` path — the seed's admin `createUser`/`createOrganization`
  * pass straight through, exactly as the old gate-free bootstrap instance did.
  *
- * `url` is the SAME libSQL `file:` URL SelfHostDb opened; `client` is
- * SelfHostDb's drizzle connection to that file, used by the seed for its two
- * idempotency reads against the auth tables Better Auth just migrated (proving
- * the cross-connection invariant: Better Auth writes via LibsqlDialect are
- * visible through SelfHostDb's client on the same file).
+ * `client` is SelfHostDb's libSQL connection. Better Auth's LibsqlDialect is
+ * built on this SAME client (not a fresh `{ url }` one — see the header
+ * comment's data-loss note), so auth tables and executor tables share one
+ * connection and one WAL. The seed also uses it directly for its two
+ * idempotency reads against the auth tables Better Auth just migrated.
  */
-export const buildBetterAuth = async (url: string, client: Client): Promise<BetterAuthHandle> => {
+export const buildBetterAuth = async (client: Client): Promise<BetterAuthHandle> => {
   const config = loadConfig();
 
   // The org id is resolved by the seed below, AFTER this instance is built; the
@@ -265,7 +281,7 @@ export const buildBetterAuth = async (url: string, client: Client): Promise<Bett
     getAuth: () => auth,
   };
 
-  auth = createAuthInstance(url, () => orgRef.id, gate);
+  auth = createAuthInstance(client, () => orgRef.id, gate);
   // `runMigrations()` flows through the LibsqlDialect and is idempotent.
   await (await auth.$context).runMigrations();
   await ensureInviteCodeTable(client);

apps/host-selfhost/src/auth/index.ts

@@ -36,7 +36,7 @@ export interface ResolvedAuthProviders {
 export const resolveAuthProviders = async (
   dbHandle: SelfHostDbHandle,
 ): Promise<ResolvedAuthProviders> => {
-  const betterAuth = await buildBetterAuth(dbHandle.url, dbHandle.client);
+  const betterAuth = await buildBetterAuth(dbHandle.client);
   const betterAuthLayer = Layer.succeed(BetterAuth)(betterAuth);
 
   // The consent redirect from Better Auth's authorize only carries the opaque

apps/host-selfhost/src/db/self-host-db.ts

@@ -32,11 +32,13 @@ import { SELF_HOST_NAMESPACE, SELF_HOST_SCHEMA_VERSION } from "../config";
 //
 // Driver: libSQL (@libsql/client + drizzle-orm/libsql), not bun:sqlite, so the
 // self-host server runs on Node AND Bun (and the same code path serves edge by
-// swapping the `file:` URL for an https Turso URL). Better Auth opens its OWN
-// libSQL connection (LibsqlDialect) to the SAME file: URL — see better-auth.ts.
-// Because libSQL connections are NOT a single shared in-process handle the way
-// bun:sqlite's was, the WAL/busy_timeout/synchronous/foreign_keys PRAGMAs are
-// re-applied PER connection (here, and again in the Better Auth dialect path).
+// swapping the `file:` URL for an https Turso URL). Better Auth SHARES this very
+// `@libsql/client` (its LibsqlDialect is built with `{ client }`, not a fresh
+// `{ url }` connection) — so the PRAGMAs set here cover auth queries too, and
+// there is exactly ONE connection and ONE WAL. That sharing is load-bearing: a
+// second libSQL connection to the same file unlinks this one's `-wal`/`-shm` on
+// open, orphaning executor-core writes onto a deleted inode that vanishes on
+// restart (the self-host data-loss bug — see better-auth.ts's header).
 // ---------------------------------------------------------------------------
 
 /**
@@ -55,10 +57,10 @@ export interface SelfHostDbHandle<TTables extends FumaTables = FumaTables> {
   readonly fuma: FumaDB<SelfHostFumaSchema<TTables>[]>;
   readonly drizzle: LibSQLDatabase<Record<string, unknown>>;
   /**
-   * The libSQL client for this handle's `file:` URL. Better Auth opens its own
-   * separate connection to the same file via LibsqlDialect; the seed reads
-   * Better Auth's tables through this client (async), so the URL is carried
-   * alongside so callers can hand it to the dialect.
+   * The libSQL client for this handle's `file:` URL. Better Auth's LibsqlDialect
+   * is built on THIS client (one shared connection — see better-auth.ts), and
+   * the seed reads Better Auth's tables through it too. `url` is retained for
+   * callers that still need the `file:` string (diagnostics, edge swap).
    */
   readonly client: Client;
   readonly url: string;
@@ -82,11 +84,11 @@ export const createSqliteExecutorDb = async <const TTables extends FumaTables>(
 
   const url = toLibsqlFileUrl(options.path);
   const client = createClient({ url });
-  // PER-CONNECTION PRAGMAs: libSQL gives drizzle and Better Auth SEPARATE
-  // connections to this file (no single shared handle), so these must be set on
-  // this connection here and again on Better Auth's dialect connection. WAL is a
-  // file-level mode once any connection enables it; foreign_keys is strictly
-  // per-connection and MUST be re-set on each.
+  // Connection PRAGMAs. This is the ONE libSQL connection for the process —
+  // drizzle (executor tables) and Better Auth's LibsqlDialect both run on this
+  // same client (see better-auth.ts), so these apply to every query, auth
+  // included. WAL is a file-level mode; foreign_keys/busy_timeout/synchronous
+  // are connection-level and set once here.
   await client.execute("PRAGMA foreign_keys = ON");
   await client.execute("PRAGMA journal_mode = WAL");
   // Survive concurrent writes from the multi-user HTTP server, and trade