Skip to content

Commit 98a1535

Browse files
os-zhuangclaude
andauthored
fix(driver-sql): canonical ISO-8601-Z audit timestamps on SQLite (ADR-0074) (#2342)
The SQLite write paths disagreed on the created_at/updated_at format: INSERT fell back to the CURRENT_TIMESTAMP column default ('YYYY-MM-DD HH:MM:SS') while UPDATE stamped 'YYYY-MM-DD HH:MM:SS.mmm' — both timezone-naive, space-separated strings that Date.parse reads as LOCAL time. On a non-UTC runtime a stored UTC wall-clock silently shifted by the host offset (the objectos kernel freshness-probe miss: a shifted updated_at landed before builtAtMs, so the per-env kernel never evicted). create/bulkCreate/upsert/update now stamp one canonical ISO-8601 instant with an explicit Z (new Date().toISOString()), matching the caller-stamped paths (sys_metadata, service outboxes) and Postgres/MySQL native now(). Applied app-side (not via the column default) so EXISTING tenant DBs are fixed immediately. formatOutput repairs legacy/raw zone-naive audit timestamps to the same format on read (idempotent) — no data migration. upsert treats created_at as insert-only (never clobbered on merge). Postgres/MySQL unaffected. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent aa33b02 commit 98a1535

4 files changed

Lines changed: 374 additions & 7 deletions

File tree

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
---
2+
'@objectstack/driver-sql': patch
3+
---
4+
5+
Fix: store SQLite `created_at`/`updated_at` in one canonical, timezone-explicit format (ADR-0074)
6+
7+
The two SQLite write paths disagreed on the audit-timestamp format. INSERT fell
8+
back to the column default `CURRENT_TIMESTAMP` (`'YYYY-MM-DD HH:MM:SS'`) while
9+
UPDATE stamped `toISOString().replace('T',' ').replace('Z','')`
10+
(`'YYYY-MM-DD HH:MM:SS.mmm'`) — both **timezone-naive**, space-separated strings
11+
that `Date.parse` reads as *local* time. On a non-UTC runtime a stored UTC
12+
wall-clock silently shifted by the host offset; e.g. the objectos kernel
13+
freshness probe compared a shifted `updated_at` against an absolute `builtAtMs`
14+
and never evicted (publishes/installs/config toggles didn't take effect until the
15+
LRU TTL expired).
16+
17+
`create` / `bulkCreate` / `upsert` / `update` now stamp a single canonical
18+
ISO-8601 instant with an explicit `Z` (`new Date().toISOString()`) — matching the
19+
caller-stamped paths (`sys_metadata`, the service outboxes) and Postgres/MySQL's
20+
native `now()`. Because the stamp is applied app-side (not via the column
21+
default), **existing** tenant databases are fixed immediately, not just freshly
22+
created tables. `formatOutput` additionally repairs any legacy/raw zone-naive
23+
audit timestamp to the same format on read (idempotent), so old rows read back
24+
unambiguously without a data migration. `upsert` now treats `created_at` as
25+
insert-only — a conflicting merge never overwrites it.
26+
27+
Postgres/MySQL are unaffected (they store a real zone-aware `TIMESTAMP`).
Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# ADR-0074: Audit timestamps are stored in one canonical, timezone-explicit format on SQLite
2+
3+
**Status**: Accepted (2026-06-26)
4+
**Deciders**: ObjectStack Protocol Architects
5+
**Builds on**: [ADR-0053](./0053-date-and-datetime-semantics.md) (`datetime` is an instant stored as UTC)
6+
**Consumers**: `@objectstack/driver-sql` (`create`/`bulkCreate`/`upsert`/`update`, `formatOutput`), `@objectstack/objectql` (optimistic locking via `updated_at`, `sys_metadata` writes), report/analytics date bucketing, and any out-of-tree consumer of `created_at`/`updated_at` (notably the objectos kernel freshness probe).
7+
**Surfaced by**: a self-hosted objectos-ee on a UTC+8 host where the per-environment kernel never evicted after a publish/install/config toggle — `updated_at` parsed as local time landed *before* the kernel's `builtAtMs`, so the freshness probe always reported "fresh".
8+
9+
---
10+
11+
## TL;DR
12+
13+
SQLite has no native timestamp type. The driver's two write paths disagreed on
14+
how they wrote the builtin `created_at`/`updated_at` audit columns:
15+
16+
- **INSERT** relied on the column default `defaultTo(knex.fn.now())`, which on
17+
SQLite is `CURRENT_TIMESTAMP``'2026-06-26 10:23:40'` (space-separated, no
18+
millis, **no zone**).
19+
- **UPDATE** stamped `new Date().toISOString().replace('T',' ').replace('Z','')`
20+
`'2026-06-26 10:23:40.246'` (space-separated, with millis, **no zone**).
21+
22+
Both are **timezone-naive**. A zone-less, space-separated string is not
23+
ISO-8601, so `Date.parse` (V8) interprets it as **local** time. On any non-UTC
24+
runtime a stored UTC wall-clock therefore shifts by the host offset when read
25+
back as an instant — silently corrupting every `new Date(updated_at)` comparison,
26+
not just the freshness probe that surfaced it.
27+
28+
**Decision.** On SQLite, every driver write path stamps audit timestamps in a
29+
single canonical format — **full ISO-8601 with an explicit `Z`**
30+
(`new Date().toISOString()`, e.g. `'2026-06-26T10:23:40.246Z'`) — and the read
31+
path repairs any legacy/raw zone-naive value to that same format. INSERT and
32+
UPDATE now agree, and the stored instant is unambiguous. Postgres/MySQL are
33+
unchanged: they store a real zone-aware `TIMESTAMP` via native `now()` and never
34+
had the ambiguity.
35+
36+
This extends ADR-0053's "`datetime` is an instant stored as UTC" to the builtin
37+
audit columns: an instant must be stored zone-explicitly, never as a bare
38+
wall-clock string.
39+
40+
## Decision detail
41+
42+
1. **Write — app-side stamp, not the column default.** `create`, `bulkCreate`,
43+
`upsert` and `update` stamp `created_at`/`updated_at` to
44+
`new Date().toISOString()` (SQLite only; gated on `tablesWithTimestamps`).
45+
A caller-provided value (a seed fixture, the `sys_metadata` writer, a service
46+
outbox) is preserved on insert — only an empty slot is filled.
47+
48+
Stamping **app-side** (rather than changing the column DDL default to a
49+
`strftime('%Y-%m-%dT%H:%M:%fZ','now')` expression) is deliberate: a DDL
50+
default only applies to **newly created** tables, so existing tenant databases
51+
— exactly the ones exhibiting the bug — would keep emitting the naive
52+
`CURRENT_TIMESTAMP` on insert and continue to mix formats with the now-fixed
53+
UPDATE. App-side stamping fixes every database immediately and needs no
54+
schema migration. The legacy `CURRENT_TIMESTAMP` column default is left in
55+
place as a harmless fallback for raw inserts that bypass the driver.
56+
57+
2. **`created_at` is insert-only.** `upsert`'s conflict `merge()` now excludes
58+
`created_at`, so a merge that updates an existing row advances `updated_at`
59+
but never rewrites the original `created_at`.
60+
61+
3. **Read — tolerant reader for legacy/mixed rows.** `formatOutput` repairs a
62+
zone-naive `created_at`/`updated_at` string to canonical ISO-8601-`Z`,
63+
interpreting the stored wall-clock as UTC (what `CURRENT_TIMESTAMP` and the
64+
old UPDATE stamp both wrote). The repair is **idempotent** (an
65+
already-zone-explicit value is returned unchanged) and **total** (a
66+
`Field.datetime`-typed audit column stored as epoch-ms INTEGER, and any
67+
unrecognised shape, pass through untouched). This mirrors the existing
68+
read-repair the `Field.date`/numeric-scalar paths already perform, and means
69+
existing rows read back unambiguously **without a data migration**.
70+
71+
## Consequences
72+
73+
- **Unambiguous instants.** `new Date(created_at|updated_at)` yields the correct
74+
UTC instant on every host timezone. The objectos freshness probe (already
75+
hardened defensively on the consumer side) now also receives a correct value
76+
at the source.
77+
- **No on-disk format mixing for new writes.** Because INSERT and UPDATE share
78+
one format, a SQL-level `ORDER BY updated_at` (e.g. objectql metadata loads)
79+
sorts chronologically. Lexicographic order of ISO-8601-`Z` equals chronological
80+
order.
81+
- **Optimistic locking stays stable.** objectql `assertVersionMatch` compares the
82+
`updated_at` token app-side, on both sides through `formatOutput`; the
83+
idempotent reader keeps the token deterministic across reads, including for
84+
legacy rows.
85+
- **Legacy rows at rest stay naive until rewritten.** Like ADR-0053's date-only
86+
repair, the on-disk value is only normalized when a row is next written through
87+
the driver; reads are repaired transparently. A raw SQL `ORDER BY` over rows
88+
that straddle the fix (some naive, some `Z`) can mis-order until those rows are
89+
rewritten — an accepted, self-healing residual, not a regression of the common
90+
driver-mediated path.
91+
- **Scope.** Only the builtin `created_at`/`updated_at` audit columns are
92+
covered. A user-declared `datetime` field with a `defaultValue: 'NOW()'` still
93+
takes the naive `CURRENT_TIMESTAMP` default on SQLite; aligning those is a
94+
possible follow-up but was out of scope for this fix.
Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* Canonical audit-timestamp format on SQLite.
5+
*
6+
* SQLite has no native timestamp type. The two write paths used to disagree:
7+
* INSERT fell back to the column default `CURRENT_TIMESTAMP`
8+
* (`'YYYY-MM-DD HH:MM:SS'`) while UPDATE stamped
9+
* `toISOString().replace('T',' ').replace('Z','')`
10+
* (`'YYYY-MM-DD HH:MM:SS.mmm'`). BOTH were timezone-NAIVE: `Date.parse` reads a
11+
* zone-less, space-separated string as LOCAL time, so a UTC wall-clock value
12+
* silently shifted by the host offset on a non-UTC runtime — the bug that made
13+
* the objectos kernel freshness probe never evict (it compared a shifted
14+
* `updated_at` against an absolute `builtAtMs`).
15+
*
16+
* These tests pin the fix: every driver write path stamps a single canonical
17+
* ISO-8601-with-`Z` instant, INSERT and UPDATE agree on that one format, and
18+
* legacy/raw zone-naive rows are repaired to the same format on read.
19+
*/
20+
21+
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
22+
import { SqlDriver } from '../src/index.js';
23+
24+
const ISO_Z = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;
25+
26+
describe('SqlDriver canonical audit-timestamp format (SQLite)', () => {
27+
let driver: SqlDriver;
28+
let raw: any;
29+
30+
beforeEach(async () => {
31+
driver = new SqlDriver({
32+
client: 'better-sqlite3',
33+
connection: { filename: ':memory:' },
34+
useNullAsDefault: true,
35+
});
36+
raw = (driver as any).knex;
37+
await driver.initObjects([
38+
{ name: 'thing', fields: { name: { type: 'string' } } },
39+
]);
40+
});
41+
42+
afterEach(async () => {
43+
await driver.disconnect();
44+
});
45+
46+
it('create() stamps created_at AND updated_at as canonical ISO-8601 Z (raw on disk)', async () => {
47+
await driver.create('thing', { id: 't1', name: 'A' }, { bypassTenantAudit: true });
48+
const row = await raw('thing').where('id', 't1').first();
49+
expect(row.created_at).toMatch(ISO_Z);
50+
expect(row.updated_at).toMatch(ISO_Z);
51+
// both stamped from one instant on insert
52+
expect(row.created_at).toBe(row.updated_at);
53+
});
54+
55+
it('update() stamps updated_at canonical, INSERT and UPDATE agree on the format, created_at is preserved', async () => {
56+
await driver.create('thing', { id: 't2', name: 'A' }, { bypassTenantAudit: true });
57+
const inserted = await raw('thing').where('id', 't2').first();
58+
await new Promise((r) => setTimeout(r, 5));
59+
await driver.update('thing', 't2', { name: 'B' }, { bypassTenantAudit: true });
60+
const updated = await raw('thing').where('id', 't2').first();
61+
62+
expect(updated.updated_at).toMatch(ISO_Z); // same canonical shape as insert
63+
expect(inserted.updated_at).toMatch(ISO_Z);
64+
expect(updated.created_at).toBe(inserted.created_at); // created_at immutable
65+
// Lexicographic == chronological for ISO-8601-Z, so a SQL ORDER BY is correct.
66+
expect(updated.updated_at >= updated.created_at).toBe(true);
67+
});
68+
69+
it('no on-disk format mixing: an inserted-only row and an updated row share one format (SQL ORDER BY safe)', async () => {
70+
await driver.create('thing', { id: 'never', name: 'x' }, { bypassTenantAudit: true });
71+
await driver.create('thing', { id: 'edited', name: 'y' }, { bypassTenantAudit: true });
72+
await driver.update('thing', 'edited', { name: 'y2' }, { bypassTenantAudit: true });
73+
const rows = await raw('thing').select('updated_at');
74+
for (const r of rows) expect(r.updated_at).toMatch(ISO_Z);
75+
});
76+
77+
it('preserves a caller-provided created_at, still stamps a missing updated_at', async () => {
78+
const provided = '2025-03-01T12:00:00.000Z';
79+
await driver.create('thing', { id: 't3', name: 'A', created_at: provided }, { bypassTenantAudit: true });
80+
const row = await raw('thing').where('id', 't3').first();
81+
expect(row.created_at).toBe(provided);
82+
expect(row.updated_at).toMatch(ISO_Z);
83+
});
84+
85+
it('bulkCreate stamps canonical timestamps', async () => {
86+
await driver.bulkCreate('thing', [
87+
{ id: 'b1', name: '1' },
88+
{ id: 'b2', name: '2' },
89+
], { bypassTenantAudit: true });
90+
const rows = await raw('thing').whereIn('id', ['b1', 'b2']).select('created_at', 'updated_at');
91+
for (const r of rows) {
92+
expect(r.created_at).toMatch(ISO_Z);
93+
expect(r.updated_at).toMatch(ISO_Z);
94+
}
95+
});
96+
97+
it('upsert: insert stamps canonical; a conflicting merge preserves created_at and advances updated_at', async () => {
98+
await driver.upsert('thing', { id: 'u1', name: 'first' }, ['id'], { bypassTenantAudit: true });
99+
const afterInsert = await raw('thing').where('id', 'u1').first();
100+
expect(afterInsert.created_at).toMatch(ISO_Z);
101+
expect(afterInsert.updated_at).toMatch(ISO_Z);
102+
103+
await new Promise((r) => setTimeout(r, 5));
104+
await driver.upsert('thing', { id: 'u1', name: 'second' }, ['id'], { bypassTenantAudit: true });
105+
const afterMerge = await raw('thing').where('id', 'u1').first();
106+
expect(afterMerge.name).toBe('second');
107+
expect(afterMerge.created_at).toBe(afterInsert.created_at); // created_at immutable on merge
108+
expect(afterMerge.updated_at).toMatch(ISO_Z);
109+
expect(afterMerge.updated_at >= afterInsert.updated_at).toBe(true);
110+
});
111+
112+
// ── Read-side tolerant reader (legacy / raw zone-naive rows) ────────────────
113+
114+
it('repairs a legacy space-separated row to canonical ISO-Z on read, interpreting it as UTC', async () => {
115+
// Simulate a row written by the OLD update stamp / CURRENT_TIMESTAMP default,
116+
// bypassing the driver write path entirely.
117+
await raw('thing').insert({ id: 'legacy', name: 'L', created_at: '2026-01-15 08:30:00', updated_at: '2026-01-15 08:30:00.246' });
118+
const row: any = await driver.findOne('thing', 'legacy', { bypassTenantAudit: true });
119+
expect(row.created_at).toBe('2026-01-15T08:30:00.000Z');
120+
expect(row.updated_at).toBe('2026-01-15T08:30:00.246Z');
121+
});
122+
123+
it('REGRESSION (freshness probe): the repaired instant equals the UTC wall-clock, host-timezone-independent', async () => {
124+
// The zone-naive '2026-01-15 08:30:00' must mean 08:30 UTC, NOT 08:30 local.
125+
await raw('thing').insert({ id: 'fr', name: 'F', created_at: '2026-01-15 08:30:00', updated_at: '2026-01-15 08:30:00' });
126+
const row: any = await driver.findOne('thing', 'fr', { bypassTenantAudit: true });
127+
expect(new Date(row.updated_at as string).getTime()).toBe(Date.parse('2026-01-15T08:30:00.000Z'));
128+
});
129+
130+
it('read-repair is idempotent: an already-canonical value is returned unchanged', async () => {
131+
const canonical = '2026-02-02T02:02:02.222Z';
132+
await raw('thing').insert({ id: 'canon', name: 'C', created_at: canonical, updated_at: canonical });
133+
const row: any = await driver.findOne('thing', 'canon', { bypassTenantAudit: true });
134+
expect(row.created_at).toBe(canonical);
135+
expect(row.updated_at).toBe(canonical);
136+
});
137+
138+
it('does not mangle a Field.datetime-typed audit column stored as epoch ms', async () => {
139+
// When created_at is declared `datetime`, better-sqlite3 stores a JS Date as
140+
// INTEGER ms; the repair must leave that number alone (Field.datetime owns it).
141+
const dd = new SqlDriver({
142+
client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true,
143+
});
144+
try {
145+
await dd.initObjects([{ name: 'evt', fields: { created_at: { type: 'datetime' }, label: { type: 'string' } } }]);
146+
await dd.create('evt', { id: 'e1', label: 'x', created_at: new Date('2026-04-04T04:04:04.004Z') }, { bypassTenantAudit: true });
147+
const row: any = await dd.findOne('evt', 'e1', { bypassTenantAudit: true });
148+
expect(typeof row.created_at).toBe('number');
149+
expect(row.created_at).toBe(Date.parse('2026-04-04T04:04:04.004Z'));
150+
} finally {
151+
await dd.disconnect();
152+
}
153+
});
154+
});

0 commit comments

Comments
 (0)