fix(driver-sql,service-analytics): bare-day upper bounds cover the whole day on Field.datetime (#3777) (#4041)

* fix(driver-sql,service-analytics): bare-day upper bounds cover the whole day on Field.datetime (#3777)

A bare YYYY-MM-DD comparand anchors to midnight UTC — right for a lower
bound, silently wrong for an upper one: the dashboard date-range filter
compiles { $gte: from, $lte: to } with bare-day bounds, so on a datetime
column every row created after 00:00 of the final day vanished. The
default configuration hit it: the filter's default field is created_at
(a system-injected Field.datetime) and 7 of 13 presets end "today".

The translation is operator-sensitive and half-open ([gte, lt) — the
drill ranges' shape, never an inclusive 23:59:59.999), applied at the
emitters that own an operator, while temporalFilterValue stays form-only:

- SqlDriver (SqliteWasmDriver inherits): $lte/<= bare-day on datetime ->
  < next-day-midnight in storage form; $between [min, max] with a
  bare-day max decomposes to >= min AND < next-day(max). Covers the
  plain and the legacy-repair (mixed-storage) column paths and both
  where spellings.
- NativeSQLStrategy: dateRange windows and lte filters bind < next-day.
- ObjectQLStrategy: execution keeps bare bounds (the driver rewrite is
  the single authority); /analytics/sql renders half-open so the echoed
  SQL reproduces execution.
- Preview evaluator: the same rule in memory, replacing the '~' hack.

One shared primitive: nextUtcCalendarDay (@objectstack/core) — rejects
instants, Dates and impossible days rather than inventing a bound.
Unchanged per the #3777 semantics table: date/time columns,
full-ISO/Date comparands, $gte/$gt/$lt.

Tests: the issue repro verbatim as the acceptance gate (row results),
operator x field-type x storage-form matrix incl. un-backfilled mixed
storage, dialect physical forms, boundary rollovers, a wasm inheritance
pin, strategy SQL shapes, preview parity. ADR-0053 gains the D-D
addendum; D-A3's matrix gains the bound-semantics axis.

Closes #3777

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EkL3RGJrzjLEGURsLxfS2f

* docs(objectql): state the calendar-day bound semantics where date filters are documented (#3777)

The Date Filters section demos the exact shape whose upper-bound semantics
#3777 fixed, without saying what the bounds mean. Now it does: bare-day
lower bound = start of day, bare-day upper bound = the whole day (compiled
half-open on datetime columns), full ISO timestamp = exact instant.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EkL3RGJrzjLEGURsLxfS2f

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: os-zhuang

.changeset/calendar-day-upper-bound.md

@@ -0,0 +1,39 @@
+---
+"@objectstack/core": minor
+"@objectstack/driver-sql": patch
+"@objectstack/driver-sqlite-wasm": patch
+"@objectstack/service-analytics": patch
+---
+
+fix(driver-sql,service-analytics): a bare-day upper bound covers the whole day on `Field.datetime` (#3777)
+
+A bare `YYYY-MM-DD` comparand anchors to midnight UTC. That is right for a
+lower bound and was silently wrong for an upper one: the dashboard date-range
+filter compiles `{ $gte: from, $lte: to }` with bare-day bounds, so on a
+`datetime` column every row created after 00:00 of the `to` day vanished from
+the result — no error, the chart renders, the numbers are just smaller. The
+default configuration hit it: the filter's default field is `created_at`
+(a system-injected `Field.datetime`) and 7 of the 13 presets end "today".
+
+The translation is operator-sensitive and half-open, applied at every
+comparison emitter:
+
+- `SqlDriver` (and `SqliteWasmDriver` by inheritance): `$lte`/`<=` with a
+  bare-day comparand on a `datetime` column compiles to `< next-day-midnight`
+  in the column's storage form; `$between [min, max]` with a bare-day max
+  decomposes to `>= min AND < next-day(max)`. Both the plain and the
+  legacy-repair (mixed-storage) column paths, both `where` spellings.
+- `NativeSQLStrategy`: `dateRange` windows and `lte` filters bind `< next-day`
+  instead of an inclusive `BETWEEN`/`<=` when the bound is a bare day.
+- The `/analytics/sql` rendering and the dataset preview evaluator apply the
+  same rule, so the echoed SQL and drafted numbers reproduce execution.
+
+`@objectstack/core` gains the shared primitive `nextUtcCalendarDay(value)`:
+the next calendar day of a valid bare `YYYY-MM-DD` (else `null` — instants,
+`Date`s and impossible days are never widened).
+
+Unchanged on purpose, per the semantics table on #3777: `date`/`time` columns
+(`<= day` is already whole-day-correct there), full-ISO/`Date` comparands
+(instant semantics), and `$gte`/`$gt`/`$lt` (midnight anchoring is correct for
+those). No authored metadata changes: a dashboard's existing
+`{ $gte, $lte }` window now simply includes its final day.

content/docs/protocol/objectql/query-syntax.mdx

@@ -306,6 +306,13 @@ where: { due_date: { $lt: '2024-06-01' } }
 where: { created_at: { $gte: '2024-01-01' } }
 ```
 
+A bare `YYYY-MM-DD` bound is a **calendar day**. As a lower bound (`$gte`) it
+means the start of that day (midnight UTC); as an upper bound (`$lte`, or the
+max of a `$between`) it covers the **whole** day — on a `datetime` column the
+driver compiles it half-open (`< next day`), so the `$between` above includes
+everything that happened on Dec 31. A full ISO timestamp keeps exact-instant
+semantics on every operator.
+
 ### Null Checks
 
 {/* os:check */}

docs/adr/0053-date-and-datetime-semantics.md

@@ -658,3 +658,62 @@ host's zone) is not recoverable. Regression cover:
 `sql-driver-time-canonical-storage.test.ts`, `sql-driver-time-of-day.test.ts`
 (SQLite), `sql-driver-time-live-dialects.test.ts` (live PG + MySQL, in the CI
 temporal-conformance job's non-UTC matrix).
+
+---
+
+## Addendum (2026-07-30) — a bare-day UPPER bound means the whole day (#3777)
+
+> **Status:** landed. Extends Phase 1's calendar-day semantics with the
+> operator-sensitive half the original decision left implicit — and the default
+> dashboard configuration hit it: the date-range filter's default field is
+> `created_at`, a system-injected `Field.datetime`, and 7 of 13 presets end
+> "today", so `{ $lte: <today> }` anchored to midnight silently dropped every
+> row created after 00:00 of the final day.
+
+### D-D1 — Operator-sensitive translation lives at the comparison EMITTERS
+
+`YYYY-MM-DD` anchors to midnight UTC (D-B1). That instant is the correct
+comparand for `$gte`/`$gt`/`$lt` — and the wrong one for `$lte`, whose author
+means "through the whole of that day". The translation is therefore a property
+of the *comparison*, not of the value: `temporalFilterValue` stays
+operator-blind (form only), and each emitter that owns an operator compiles a
+bare-day upper bound half-open:
+
+- **`SqlDriver` filter compiler** (`calendarDayUpperBoundRewrite` /
+  `calendarDayBetweenRewrite`): `$lte`/`<=` → `< next-day-midnight` in storage
+  form; `$between [min, max]` with a bare-day max decomposes to
+  `>= min AND < next-day(max)`. Applies on both the plain and the
+  CASE-normalised (D-B2) column paths, and to the Mongo-style and array
+  `where` spellings. `driver-sqlite-wasm` inherits.
+- **`NativeSQLStrategy`** windows and `lte` filters bind `< next-day` instead
+  of `BETWEEN`/`<=` when the bound is a bare day.
+- **`ObjectQLStrategy`** leaves its lowered `{$gte, $lte}` bounds bare — the
+  driver rewrite is the single execution-path authority — and renders
+  `/analytics/sql` half-open so the echoed SQL reproduces execution.
+- **The dataset preview evaluator** applies the same rule in memory, replacing
+  its `'~'`-suffix string hack, so draft numbers match published numbers.
+
+One primitive backs all of them: `nextUtcCalendarDay` (`@objectstack/core`),
+which rejects instants, `Date`s and impossible days (`2026-02-30`) rather than
+inventing a bound. Half-open — never an inclusive `23:59:59.999`, which
+re-opens the gap at whatever precision the dialect stores beyond milliseconds
+(Postgres keeps microseconds), and is the same `[gte, lt)` shape the drill
+ranges (#1752) already emit. `< next-day` is also order-equivalent to `<= day`
+for `Field.date` text, which is what lets the type-blind emitters (raw SQL,
+preview) apply it unconditionally; the driver, which knows the column type,
+scopes the rewrite to `datetime` so `date`/`time` columns compile byte-identical
+to before.
+
+The filter-token resolver (`filter-tokens.ts`) keeps its documented refusal to
+widen: a resolver-side fix would change what a token *is*; the emitter-side fix
+changes what a comparison *does* with it, per column type — which is the layer
+that owns that knowledge.
+
+### D-D2 — Consequences for D-A3
+
+The conformance matrix gains a **bound-semantics** axis (`point`, `whole-day`):
+row-result coverage for the `$lte`/`$between` upper-bound cells now lives in
+`sql-driver-calendar-day-upper-bound.test.ts` (canonical + legacy-mixed
+storage, dialect physical forms, boundary rollovers) and the strategy/preview
+suites; the full matrix program (relative-token × live-driver × timezone)
+remains open under D-A3.

packages/core/src/utils/datetime.test.ts

@@ -6,7 +6,7 @@
 // database, and midnight must land exactly on the day boundary in that zone.
 
 import { describe, it, expect } from 'vitest';
-import { zonedDateStartToUtcMs, calendarPartsInTz } from './datetime.js';
+import { zonedDateStartToUtcMs, calendarPartsInTz, nextUtcCalendarDay } from './datetime.js';
 
 const iso = (s: string) => Date.parse(s);
 
@@ -60,3 +60,29 @@ describe('zonedDateStartToUtcMs — round-trips to the day boundary in the zone'
     });
   }
 });
+
+describe('nextUtcCalendarDay — the exclusive upper bound of a bare calendar day (#3777)', () => {
+  it('advances one day, rolling month, year and leap boundaries', () => {
+    expect(nextUtcCalendarDay('2026-07-28')).toBe('2026-07-29');
+    expect(nextUtcCalendarDay('2026-07-31')).toBe('2026-08-01');
+    expect(nextUtcCalendarDay('2026-12-31')).toBe('2027-01-01');
+    expect(nextUtcCalendarDay('2024-02-28')).toBe('2024-02-29'); // leap year
+    expect(nextUtcCalendarDay('2025-02-28')).toBe('2025-03-01');
+    expect(nextUtcCalendarDay(' 2026-07-28 ')).toBe('2026-07-29'); // trimmed
+  });
+
+  it('returns null for anything that is not a valid bare calendar day', () => {
+    // Instants keep instant semantics — never widened.
+    expect(nextUtcCalendarDay('2026-07-28T12:00:00Z')).toBeNull();
+    expect(nextUtcCalendarDay(new Date('2026-07-28T00:00:00Z'))).toBeNull();
+    // Impossible days are rejected, not rolled into an invented bound.
+    expect(nextUtcCalendarDay('2026-02-30')).toBeNull();
+    expect(nextUtcCalendarDay('2026-13-01')).toBeNull();
+    // Non-strings / junk.
+    expect(nextUtcCalendarDay(1753660800000)).toBeNull();
+    expect(nextUtcCalendarDay(null)).toBeNull();
+    expect(nextUtcCalendarDay(undefined)).toBeNull();
+    expect(nextUtcCalendarDay('7/28/2026')).toBeNull();
+    expect(nextUtcCalendarDay('2026-7-28')).toBeNull(); // not zero-padded → not the canonical shape
+  });
+});

packages/core/src/utils/datetime.ts

@@ -102,6 +102,39 @@ export function zonedDateStartToUtcMs(ymd: string, tz?: string): number {
   }
 }
 
+/**
+ * The calendar day after a bare `YYYY-MM-DD` string — the exclusive upper
+ * bound of that day, for compiling "through day X" into the half-open
+ * `[X, X+1)` a `datetime` column needs (#3777).
+ *
+ * A bare calendar day used as an upper bound (`$lte`, a `dateRange` end, a
+ * `{current_month_end}` token) always carries whole-day intent — the author
+ * means "including everything that happened on X", never "up to the stroke of
+ * midnight that begins X". On a `date` column `<= X` already says that; on a
+ * `datetime` column it silently drops every instant after 00:00. The correct
+ * translation is the half-open pair `>= start AND < nextUtcCalendarDay(end)`
+ * — the same shape the analytics drill ranges already emit — rather than an
+ * inclusive `23:59:59.999` constant, which re-opens the gap at whatever
+ * precision the dialect stores beyond milliseconds.
+ *
+ * `< nextUtcCalendarDay(X)` is also *equivalent* to `<= X` for a `date` column
+ * (plain `YYYY-MM-DD` text ordering), so emitters that cannot see the column
+ * type (raw-SQL strategies, the dataset preview evaluator) can apply it
+ * unconditionally to a bare-day bound and be right on both column types.
+ *
+ * Returns `null` for anything that is not a valid bare calendar day — full
+ * ISO timestamps keep instant semantics and must NOT be widened, and an
+ * impossible day (`2026-02-30`) is rejected the same way
+ * {@link bucketKeyToCalendarRange} rejects it, so a caller falls back to the
+ * untranslated comparand instead of inventing a bound.
+ */
+export function nextUtcCalendarDay(value: unknown): string | null {
+  if (typeof value !== 'string') return null;
+  const day = value.trim();
+  if (!/^\d{4}-\d{2}-\d{2}$/.test(day)) return null;
+  return bucketKeyToCalendarRange(day, 'day')?.end ?? null;
+}
+
 /**
  * Granularity of a canonical date-bucket key. Mirrors `@objectstack/spec`'s
  * `DateGranularity` enum but kept as a local literal union so this low-level