fix(analytics,driver-sql): coerce datetime filter comparands to storage form

Dashboard time-series charts and "last N months" KPIs that filter/group by a
`Field.datetime` column silently returned "No rows" even though data existed;
`Field.date` columns worked.

Root cause: `NativeSQLStrategy` expands dashboard relative-date tokens
(`{12_months_ago}`, `{today}`, …) to ISO date strings and binds them directly
into raw SQL, bypassing the driver's CRUD filter coercion. Under better-sqlite3
a `Field.datetime` column is stored as an INTEGER epoch (ms), so
`assessed_at >= '2025-06-18'` is a TEXT-vs-INTEGER affinity compare that is
always false → empty result. `Field.date` stores ISO TEXT and compared fine.

Fix: make the comparand coercion type- and storage-aware via a new optional
`StrategyContext.coerceTemporalFilterValue` hook, wired by the analytics plugin
to the driver's public `SqlDriver.temporalFilterValue` — the single source of
truth for the storage convention (reuses the existing `coerceFilterValue`).
Coercion is dialect-correct: SQLite `Field.datetime` → epoch ms; `Field.date`
text and native-timestamp dialects (Postgres/MySQL) are left unchanged, so
Postgres is never handed an epoch integer. Also gated the driver's existing
datetime→epoch coercion on `isSqlite` to make the native-timestamp path correct.
Applied to gte/lte/gt/lt/equals, in/notIn, and the dateRange/timeDimension path.

Tests:
- service-analytics: NativeSQLStrategy binds epoch for a datetime `gte`/range/in,
  leaves ISO text unchanged when the hook reports no coercion (Postgres/date),
  and is backward-compatible with no hook.
- driver-sql: E2E SQLite repro proving 0→4 rows once coerced; dialect-gating unit
  test asserting Postgres/MySQL are NOT epoch-coerced (no regression).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: os-zhuang

.changeset/analytics-datetime-epoch-filter.md

@@ -0,0 +1,11 @@
+---
+"@objectstack/service-analytics": patch
+"@objectstack/driver-sql": patch
+"@objectstack/spec": patch
+---
+
+Fix dashboard time-series charts / "last N months" KPIs that filter or group by a `Field.datetime` column silently returning "No rows".
+
+The analytics `NativeSQLStrategy` compiles dashboard relative-date tokens (`{12_months_ago}`, `{today}`, …) to ISO date strings and binds them directly into raw SQL, bypassing the driver's own filter coercion. Under better-sqlite3 a `Field.datetime` column is stored as an INTEGER epoch (ms), so `assessed_at >= '2025-06-18'` became a TEXT-vs-INTEGER affinity compare that is always false — an empty result even though the rows exist. `Field.date` columns store ISO TEXT and were unaffected.
+
+The strategy now coerces a temporal comparand to the column's on-disk storage form via a new optional `StrategyContext.coerceTemporalFilterValue` hook, wired to the driver's public `SqlDriver.temporalFilterValue` (the single source of truth for the storage convention). Coercion is dialect-correct: SQLite `Field.datetime` → epoch ms; `Field.date` text and native-timestamp dialects (Postgres/MySQL) are left unchanged, so Postgres is never handed an epoch integer. Applied to `gte`/`lte`/`gt`/`lt`/`equals`, `in`/`notIn`, and the `dateRange`/timeDimension `BETWEEN` path.

packages/plugins/driver-sql/src/sql-driver-analytics-datetime.test.ts

@@ -0,0 +1,102 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * End-to-end repro of the dashboard time-series "No rows" bug at the storage
+ * level, and proof of the fix.
+ *
+ * The analytics `NativeSQLStrategy` compiles dashboard relative-date tokens
+ * (e.g. `{12_months_ago}`) to ISO date strings and binds them into a raw
+ * `SELECT … WHERE col >= ?` that it runs through the driver's `execute()` —
+ * bypassing the normal `find()` filter coercion. Under better-sqlite3 a
+ * `Field.datetime` column is stored as an INTEGER epoch (ms), so the ISO TEXT
+ * comparand never matches (TEXT sorts after every INTEGER) → 0 rows, even though
+ * the rows exist. A `Field.date` column stores ISO TEXT and matches fine.
+ *
+ * This test reproduces both the broken (raw ISO bind → 0) and fixed (epoch bind
+ * via the driver's public `temporalFilterValue` → N) behaviour against a real
+ * SQLite database, mirroring exactly what the analytics strategy now does.
+ */
+
+import { describe, it, expect, beforeEach, afterEach } from 'vitest';
+import { SqlDriver } from '../src/index.js';
+
+describe('Analytics datetime filter — SQLite epoch storage (E2E repro)', () => {
+  let driver: SqlDriver;
+  const TABLE = 'compliance_assessment';
+  const CUTOFF = '2025-06-18'; // ISO date token the dashboard expands to
+
+  beforeEach(async () => {
+    driver = new SqlDriver({
+      client: 'better-sqlite3',
+      connection: { filename: ':memory:' },
+      useNullAsDefault: true,
+    });
+
+    await driver.initObjects([
+      {
+        name: TABLE,
+        fields: {
+          title: { type: 'string' },
+          assessed_at: { type: 'datetime' }, // stored as INTEGER epoch ms
+          assessed_on: { type: 'date' },      // stored as YYYY-MM-DD text
+        },
+      },
+    ]);
+
+    // Four assessments AFTER the cutoff, one well before — inserted with real
+    // Date objects so better-sqlite3 stores `assessed_at` as INTEGER epoch ms,
+    // exactly the path the seed loader takes.
+    const rows = [
+      ['a1', new Date('2024-01-01T00:00:00Z'), '2024-01-01'], // before cutoff
+      ['a2', new Date('2025-06-18T09:00:00Z'), '2025-06-18'], // on/after
+      ['a3', new Date('2025-09-01T09:00:00Z'), '2025-09-01'],
+      ['a4', new Date('2026-01-15T09:00:00Z'), '2026-01-15'],
+      ['a5', new Date('2026-05-20T09:00:00Z'), '2026-05-20'],
+    ] as const;
+    for (const [id, at, on] of rows) {
+      await driver.create(
+        TABLE,
+        { id, title: id, assessed_at: at, assessed_on: on },
+        { bypassTenantAudit: true },
+      );
+    }
+  });
+
+  afterEach(async () => {
+    await driver.disconnect();
+  });
+
+  const countWhere = async (col: string, bind: unknown): Promise<number> => {
+    const res: any = await driver.execute(
+      `SELECT count(*) AS n FROM "${TABLE}" WHERE "${col}" >= ?`,
+      [bind],
+    );
+    const row = Array.isArray(res) ? res[0] : res?.rows?.[0] ?? res;
+    return Number(row.n);
+  };
+
+  it('BUG: a raw ISO comparand against the epoch datetime column returns 0 rows', async () => {
+    // This is what the type-blind strategy used to bind — the silent failure.
+    expect(await countWhere('assessed_at', CUTOFF)).toBe(0);
+  });
+
+  it('FIX: the driver-coerced epoch comparand returns the 4 matching rows', async () => {
+    // `temporalFilterValue` is exactly the hook NativeSQLStrategy now calls.
+    const coerced = driver.temporalFilterValue(TABLE, 'assessed_at', CUTOFF);
+    expect(typeof coerced).toBe('number'); // epoch ms, not the ISO string
+    expect(await countWhere('assessed_at', coerced)).toBe(4);
+  });
+
+  it('CONTROL: the `Field.date` text column already matched the raw ISO comparand', async () => {
+    // Proves the date/text path was never broken and is left untouched.
+    const coerced = driver.temporalFilterValue(TABLE, 'assessed_on', CUTOFF);
+    expect(typeof coerced).toBe('string'); // YYYY-MM-DD, NOT coerced to epoch
+    expect(await countWhere('assessed_on', coerced)).toBe(4);
+    // and the raw ISO bind matches identically (no coercion needed for text)
+    expect(await countWhere('assessed_on', CUTOFF)).toBe(4);
+  });
+
+  it('does not touch a non-temporal column', () => {
+    expect(driver.temporalFilterValue(TABLE, 'title', 'hello')).toBe('hello');
+  });
+});

packages/plugins/driver-sql/src/sql-driver-temporal-dialect.test.ts

@@ -0,0 +1,69 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * Dialect-correctness of `temporalFilterValue` (the hook the analytics layer
+ * uses). The datetime → epoch-ms coercion is SQLite-ONLY: SQLite stores
+ * `Field.datetime` as an INTEGER epoch, but Postgres/MySQL map it to a native
+ * TIMESTAMP where an ISO string / Date binds correctly. Coercing to an epoch
+ * integer on a native-timestamp dialect would compare INTEGER vs TIMESTAMP and
+ * break the query — the exact Postgres regression we must NOT introduce.
+ *
+ * No DB connection is needed: we seed the field-type maps the way `initObjects`
+ * would and exercise the pure coercion logic across dialects.
+ */
+
+import { describe, it, expect } from 'vitest';
+import { SqlDriver } from '../src/index.js';
+
+/** Test double that injects the field-type metadata without a live connection. */
+class ProbeDriver extends SqlDriver {
+  seedDatetime(table: string, field: string): void {
+    (this.datetimeFields[table] ??= new Set()).add(field);
+  }
+  seedDate(table: string, field: string): void {
+    (this.dateFields[table] ??= new Set()).add(field);
+  }
+}
+
+function makeDriver(client: string): ProbeDriver {
+  // Connection is never opened — we only call the synchronous coercion path.
+  return new ProbeDriver({ client, connection: { filename: ':memory:' }, useNullAsDefault: true } as any);
+}
+
+const ISO = '2025-06-18';
+const EPOCH = Date.parse('2025-06-18T00:00:00.000Z');
+
+describe('temporalFilterValue dialect gating', () => {
+  it('SQLite: datetime ISO comparand → epoch ms', () => {
+    const d = makeDriver('better-sqlite3');
+    d.seedDatetime('t', 'at');
+    expect(d.temporalFilterValue('t', 'at', ISO)).toBe(EPOCH);
+  });
+
+  it('Postgres: datetime ISO comparand is LEFT UNCHANGED (no epoch coercion → no regression)', () => {
+    const d = makeDriver('pg');
+    d.seedDatetime('t', 'at');
+    expect(d.temporalFilterValue('t', 'at', ISO)).toBe(ISO);
+  });
+
+  it('MySQL: datetime ISO comparand is left unchanged', () => {
+    const d = makeDriver('mysql2');
+    d.seedDatetime('t', 'at');
+    expect(d.temporalFilterValue('t', 'at', ISO)).toBe(ISO);
+  });
+
+  it('Field.date normalises to YYYY-MM-DD text on every dialect', () => {
+    for (const client of ['better-sqlite3', 'pg', 'mysql2']) {
+      const d = makeDriver(client);
+      d.seedDate('t', 'on');
+      expect(d.temporalFilterValue('t', 'on', '2025-06-18T12:00:00Z')).toBe('2025-06-18');
+    }
+  });
+
+  it('non-temporal fields pass through unchanged on every dialect', () => {
+    for (const client of ['better-sqlite3', 'pg', 'mysql2']) {
+      const d = makeDriver(client);
+      expect(d.temporalFilterValue('t', 'name', 'hello')).toBe('hello');
+    }
+  });
+});

packages/plugins/driver-sql/src/sql-driver.ts

@@ -1614,6 +1614,14 @@ export class SqlDriver implements IDataDriver {
     };
 
     if (isDatetime) {
+      // Only SQLite stores `Field.datetime` as an INTEGER epoch (better-sqlite3
+      // binds a JS `Date` as `.getTime()`); there the ISO/text comparand MUST be
+      // coerced to epoch ms or it collapses to a TEXT-vs-INTEGER affinity compare
+      // that never matches. Postgres/MySQL map datetime to a native TIMESTAMP
+      // (see `defineColumn` → `table.timestamp`), where Knex binds an ISO string
+      // or `Date` correctly — coercing to an epoch integer there would compare an
+      // INTEGER against a TIMESTAMP and break the query. So gate on dialect.
+      if (!this.isSqlite) return value;
       const ms = toMs(value);
       return ms == null ? value : ms;
     }
@@ -1622,6 +1630,30 @@ export class SqlDriver implements IDataDriver {
     return this.toDateOnly(value);
   }
 
+  /**
+   * Public, dialect-correct temporal filter-value coercion for callers that
+   * build SQL *outside* the normal `find()`/`applyFilters()` path — chiefly the
+   * analytics native-SQL strategy, which compiles a raw `SELECT … WHERE col >= $N`
+   * and binds the value directly, bypassing `coerceFilterValue`.
+   *
+   * Given a logical object (table) name, a field name and a filter value
+   * (typically an ISO date/datetime string from a dashboard relative-date
+   * token like `{12_months_ago}`), this returns the value in the column's
+   * on-disk storage form:
+   *   - SQLite `Field.datetime` → epoch milliseconds (INTEGER), so the
+   *     comparison matches the stored integer rather than failing a
+   *     TEXT-vs-INTEGER affinity compare.
+   *   - `Field.date` (any dialect)   → `YYYY-MM-DD` text.
+   *   - Native-timestamp dialects / non-temporal fields → value unchanged.
+   *
+   * This is a thin, intentionally narrow wrapper over the same `coerceFilterValue`
+   * the driver already uses, so there is exactly one source of truth for the
+   * storage convention and the analytics path can never drift from CRUD.
+   */
+  public temporalFilterValue(objectName: string, field: string, value: any): any {
+    return this.coerceFilterValue(objectName, field, value);
+  }
+
   protected applyFilters(builder: Knex.QueryBuilder, filters: any) {
     if (!filters) return;
     const table = this.tableNameForBuilder(builder);