Skip to content

Commit 601cc11

Browse files
os-zhuangclaude
andauthored
feat(analytics): timezone-aware date bucketing (ADR-0053 Phase 2, #1982) (#2001)
Day/week/month/quarter/year buckets resolve on a reference timezone's calendar days, so a row near a tz day-boundary lands in the bucket a user in that zone expects — identically on SQLite and Postgres. Per decision D2, non-UTC bucketing runs in-memory uniformly rather than emitting dialect-specific `date_trunc … AT TIME ZONE` (SQLite/MySQL lack loaded tz data → cross-driver boundary skew). `engine.aggregate({ timezone })` forces the in-memory path for a non-UTC zone; the date-range `where` still goes to the driver. UTC / unset keeps the native fast path unchanged. - New shared DST-safe `calendarPartsInTz`/`calendarPartsInTzOrUtc` in @objectstack/core (Intl-based; falls back to UTC for unset/UTC/invalid). - Thread the reference tz: EngineAggregateOptions → analytics executeAggregate bridge / ObjectQLStrategy → applyInMemoryAggregation → bucketDateValue, plus the draft-preview evaluator's bucketDate. - formatDateBucket stays UTC by design (re-labels already-bucketed values). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent cb352a0 commit 601cc11

14 files changed

Lines changed: 330 additions & 22 deletions

File tree

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
---
2+
"@objectstack/core": minor
3+
"@objectstack/spec": minor
4+
"@objectstack/objectql": minor
5+
"@objectstack/service-analytics": minor
6+
---
7+
8+
feat(analytics): timezone-aware date bucketing (ADR-0053 Phase 2)
9+
10+
Analytics day/week/month/quarter/year buckets now resolve on a **reference timezone's** calendar days, so a row near a tz day-boundary lands in the bucket a user in that zone would expect — identically on SQLite and Postgres.
11+
12+
Per ADR-0053 decision **D2**, bucketing is done **in-memory, uniformly** for non-UTC zones rather than emitting dialect-specific `date_trunc … AT TIME ZONE` (SQLite has no tz database and MySQL needs tz tables loaded, so splitting by dialect would shift bucket boundaries for the same data). `engine.aggregate({ timezone })` therefore forces the in-memory aggregation path when a non-UTC reference tz is set — the date-range `where` still goes to the driver, so only matching rows are fetched. **UTC / unset keeps the native driver fast path unchanged.**
13+
14+
- New shared `calendarPartsInTz` / `calendarPartsInTzOrUtc` util in `@objectstack/core` (DST-safe via `Intl.DateTimeFormat`, never hand-rolled offset math; falls back to UTC for an unset/`'UTC'`/invalid zone).
15+
- `EngineAggregateOptions` and the analytics `executeAggregate` bridge / `ObjectQLStrategy` thread the reference timezone (sourced from the dataset selection / `ExecutionContext`) through to `applyInMemoryAggregation``bucketDateValue`, and the draft-preview evaluator's `bucketDate`.
16+
- `formatDateBucket` (dimension labels) stays UTC-only by design: it re-labels values that were *already* bucketed upstream, so re-applying a timezone there would shift a correct bucket by a day.

packages/core/src/index.ts

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,9 @@ export * from './security/index.js';
2323
// Export environment utilities
2424
export * from './utils/env.js';
2525

26+
// Export timezone-aware calendar utilities (ADR-0053 Phase 2)
27+
export * from './utils/datetime.js';
28+
2629
// Export in-memory fallbacks for core-criticality services
2730
export * from './fallbacks/index.js';
2831

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* Timezone-aware calendar utilities (ADR-0053 Phase 2).
5+
*
6+
* The one primitive everything else builds on is {@link calendarPartsInTz}:
7+
* the year/month/day an instant falls on *as seen in a reference timezone*.
8+
* It uses `Intl.DateTimeFormat().formatToParts()` so DST transitions are
9+
* handled by the platform's tz database — never hand-rolled offset math, which
10+
* is the classic source of off-by-one-hour bucket errors.
11+
*
12+
* This lives in `@objectstack/core` (not `@objectstack/formula`) because both
13+
* the ObjectQL aggregation engine and the analytics service need it and both
14+
* already depend on core, whereas neither depends on formula's public surface.
15+
* (`@objectstack/formula` keeps its own private copy for `today()`/`daysFromNow`
16+
* to avoid a layering dependency on core.)
17+
*/
18+
19+
/** Calendar-day parts in a reference timezone. `month` is 1-12. */
20+
export interface CalendarParts {
21+
year: number;
22+
month: number;
23+
day: number;
24+
}
25+
26+
/**
27+
* The year/month/day an instant falls on in `tz`. Throws if `tz` is not a
28+
* valid IANA zone (callers treat that as a fall-through to UTC).
29+
*/
30+
export function calendarPartsInTz(d: Date, tz: string): CalendarParts {
31+
const parts = new Intl.DateTimeFormat('en-US', {
32+
timeZone: tz,
33+
year: 'numeric',
34+
month: '2-digit',
35+
day: '2-digit',
36+
}).formatToParts(d);
37+
const get = (t: string) => Number(parts.find((p) => p.type === t)?.value);
38+
return { year: get('year'), month: get('month'), day: get('day') };
39+
}
40+
41+
/**
42+
* The calendar-day parts of an instant, in `tz` when it's a real non-UTC zone,
43+
* otherwise in UTC. Never throws: an unset, `'UTC'`, or invalid zone falls back
44+
* to the UTC calendar day. This is the safe entry point for bucketing code that
45+
* must degrade to the historical UTC behavior rather than error.
46+
*/
47+
export function calendarPartsInTzOrUtc(d: Date, tz?: string): CalendarParts {
48+
if (tz && tz !== 'UTC') {
49+
try {
50+
return calendarPartsInTz(d, tz);
51+
} catch {
52+
// unknown zone → fall through to UTC
53+
}
54+
}
55+
return {
56+
year: d.getUTCFullYear(),
57+
month: d.getUTCMonth() + 1,
58+
day: d.getUTCDate(),
59+
};
60+
}
Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license.
2+
//
3+
// ADR-0053 Phase 2 (D2): `engine.aggregate({ timezone })` routing.
4+
//
5+
// Native driver date bucketing (`date_trunc`) is UTC-only, so a non-UTC
6+
// reference timezone must force the in-memory path (uniform across drivers)
7+
// while UTC / unset keeps the native fast path. These tests pin both halves
8+
// using a driver that advertises native day bucketing and records whether its
9+
// native `aggregate()` was taken.
10+
11+
import { describe, it, expect } from 'vitest';
12+
import { ObjectQL } from './engine.js';
13+
14+
/**
15+
* A driver advertising native `day` granularity. Its native `aggregate()`
16+
* "buckets" in UTC (the only thing a real SQL `date_trunc` can do here) and
17+
* flags that it ran, so the test can assert which path the engine chose.
18+
*/
19+
function makeBucketingDriver(rows: any[]) {
20+
let nativeAggregateCalls = 0;
21+
const driver: any = {
22+
name: 'bucketing-mock',
23+
version: '0.0.0',
24+
supports: { queryDateGranularity: { day: true, week: true, month: true, quarter: true, year: true } },
25+
async connect() {}, async disconnect() {}, async checkHealth() { return true; }, async execute() { return null; },
26+
async find() { return rows.slice(); },
27+
findStream() { throw new Error('ni'); },
28+
async findOne() { return rows[0] ?? null; },
29+
async create(_o: string, d: any) { return d; },
30+
async update(_o: string, _id: string, d: any) { return d; },
31+
async delete() { return true; },
32+
async count() { return rows.length; },
33+
async bulkCreate(_o: string, r: any[]) { return r; },
34+
async bulkUpdate() { return []; }, async bulkDelete() {},
35+
async beginTransaction() { return { __trx: true, commit: async () => {}, rollback: async () => {} }; },
36+
async commit() {}, async rollback() {},
37+
async aggregate(_object: string, ast: any) {
38+
nativeAggregateCalls += 1;
39+
// Simulate UTC `date_trunc` day bucketing + sum.
40+
const buckets = new Map<string, number>();
41+
const gran = ast.groupBy?.[0];
42+
const field = typeof gran === 'string' ? gran : gran.field;
43+
for (const r of rows) {
44+
const day = new Date(String(r[field])).toISOString().slice(0, 10);
45+
buckets.set(day, (buckets.get(day) ?? 0) + Number(r.amount));
46+
}
47+
return Array.from(buckets, ([closed_at, total]) => ({ closed_at, total }));
48+
},
49+
};
50+
return { driver, nativeCalls: () => nativeAggregateCalls };
51+
}
52+
53+
// Two events 4h apart straddling the NY midnight: same UTC day (03-01),
54+
// different NY days (02-29 / 03-01).
55+
const ROWS = [
56+
{ closed_at: '2024-03-01T03:00:00.000Z', amount: 10 }, // NY 02-29
57+
{ closed_at: '2024-03-01T07:00:00.000Z', amount: 5 }, // NY 03-01
58+
];
59+
60+
async function makeEngine(rows: any[]) {
61+
const { driver, nativeCalls } = makeBucketingDriver(rows);
62+
const engine = new ObjectQL();
63+
engine.registerDriver(driver, true);
64+
await engine.init();
65+
engine.registry.registerObject({
66+
name: 'deal',
67+
fields: { closed_at: { type: 'date' }, amount: { type: 'number' } },
68+
} as any);
69+
return { engine, nativeCalls };
70+
}
71+
72+
const dayBucket = {
73+
groupBy: [{ field: 'closed_at', dateGranularity: 'day' }],
74+
aggregations: [{ function: 'sum', field: 'amount', alias: 'total' }],
75+
};
76+
77+
describe('engine.aggregate timezone routing (ADR-0053 Phase 2)', () => {
78+
it('UTC / unset takes the native driver fast path', async () => {
79+
const { engine, nativeCalls } = await makeEngine(ROWS);
80+
const utc = await engine.aggregate('deal', { ...dayBucket, timezone: 'UTC' } as any);
81+
expect(nativeCalls()).toBe(1);
82+
expect(utc).toEqual([{ closed_at: '2024-03-01', total: 15 }]);
83+
84+
const unset = await engine.aggregate('deal', { ...dayBucket } as any);
85+
expect(nativeCalls()).toBe(2); // native again
86+
expect(unset).toEqual([{ closed_at: '2024-03-01', total: 15 }]);
87+
});
88+
89+
it('non-UTC timezone forces in-memory bucketing on the reference day', async () => {
90+
const { engine, nativeCalls } = await makeEngine(ROWS);
91+
const ny = (await engine.aggregate('deal', {
92+
...dayBucket,
93+
timezone: 'America/New_York',
94+
} as any)).sort((a: any, b: any) => String(a.closed_at).localeCompare(String(b.closed_at)));
95+
96+
expect(nativeCalls()).toBe(0); // native path skipped
97+
expect(ny).toEqual([
98+
{ closed_at: '2024-02-29', total: 10 },
99+
{ closed_at: '2024-03-01', total: 5 },
100+
]);
101+
});
102+
});

packages/objectql/src/engine.ts

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2484,16 +2484,27 @@ export class ObjectQL implements IDataEngine {
24842484
if (!g?.dateGranularity) return true; // plain {field} object is fine
24852485
return granularityCaps?.[g.dateGranularity] === true;
24862486
});
2487-
if (typeof drv.aggregate === 'function' && allStructuredSupported) {
2487+
// ADR-0053 Phase 2 (D2): native driver date bucketing (`date_trunc`) is
2488+
// UTC-only — SQLite has no tz database and MySQL needs tz tables loaded,
2489+
// so pushing tz-aware bucketing down splits boundaries per dialect. When
2490+
// a non-UTC reference timezone is in play we therefore force the
2491+
// in-memory path: the date-range `where` still goes to the driver (only
2492+
// matching rows are fetched), but bucketing runs uniformly in JS so a
2493+
// row near a tz day-boundary lands identically on every driver.
2494+
const tz = query.timezone;
2495+
const hasDateBucket = structuredItems.some((g: any) => !!g?.dateGranularity);
2496+
const tzRequiresInMemory = !!tz && tz !== 'UTC' && hasDateBucket;
2497+
if (typeof drv.aggregate === 'function' && allStructuredSupported && !tzRequiresInMemory) {
24882498
return drv.aggregate(object, ast, this.buildDriverOptions(opCtx.context));
24892499
}
24902500
// In-memory fallback path: ask the driver for raw rows, then bucket +
24912501
// aggregate here. This guarantees `groupBy` (incl. structured items
24922502
// carrying `dateGranularity`) and `aggregations` always work even on
24932503
// drivers that have no native aggregation support (driver-rest,
2494-
// driver-memory, partial SQL drivers).
2504+
// driver-memory, partial SQL drivers), and is the path that honours a
2505+
// non-UTC reference timezone.
24952506
const raw = await driver.find(object, ast, this.buildDriverOptions(opCtx.context));
2496-
return applyInMemoryAggregation(raw, ast);
2507+
return applyInMemoryAggregation(raw, ast, tz);
24972508
});
24982509

24992510
return opCtx.result as any[];

packages/objectql/src/in-memory-aggregation.test.ts

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,4 +105,56 @@ describe('bucketDateValue', () => {
105105
expect(bucketDateValue(null, 'month')).toBe('(null)');
106106
expect(bucketDateValue('not-a-date', 'month')).toBe('(null)');
107107
});
108+
109+
// ADR-0053 Phase 2 (D2): a non-UTC reference timezone shifts the calendar day.
110+
describe('timezone-aware bucketing', () => {
111+
// 2024-03-01T03:00Z is still 2024-02-29 (22:00) in America/New_York.
112+
const nearMidnight = '2024-03-01T03:00:00.000Z';
113+
114+
it('buckets on the reference zone calendar day (day/month/quarter)', () => {
115+
expect(bucketDateValue(nearMidnight, 'day', 'America/New_York')).toBe('2024-02-29');
116+
expect(bucketDateValue(nearMidnight, 'month', 'America/New_York')).toBe('2024-02');
117+
expect(bucketDateValue(nearMidnight, 'quarter', 'America/New_York')).toBe('2024-Q1');
118+
// ...while UTC sees the next day/month.
119+
expect(bucketDateValue(nearMidnight, 'day', 'UTC')).toBe('2024-03-01');
120+
expect(bucketDateValue(nearMidnight, 'month', 'UTC')).toBe('2024-03');
121+
});
122+
123+
it('shifts the ISO week when the zone moves the day across a Monday', () => {
124+
// 2024-03-04T02:00Z is a Monday in UTC (ISO week 10) but still
125+
// 2024-03-03 Sunday (ISO week 9) in America/New_York.
126+
const mondayUtc = '2024-03-04T02:00:00.000Z';
127+
expect(bucketDateValue(mondayUtc, 'week', 'UTC')).toBe('2024-W10');
128+
expect(bucketDateValue(mondayUtc, 'week', 'America/New_York')).toBe('2024-W09');
129+
});
130+
131+
it('falls back to UTC for unset / UTC / invalid zones', () => {
132+
expect(bucketDateValue(nearMidnight, 'day')).toBe('2024-03-01');
133+
expect(bucketDateValue(nearMidnight, 'day', 'UTC')).toBe('2024-03-01');
134+
expect(bucketDateValue(nearMidnight, 'day', 'Not/AZone')).toBe('2024-03-01');
135+
});
136+
137+
it('groups rows into the right tz bucket via applyInMemoryAggregation', () => {
138+
// Two events 4h apart that straddle the NY midnight: in UTC they share
139+
// the 2024-03-01 day; in NY they split across 02-29 and 03-01.
140+
const rows = [
141+
{ closed_at: '2024-03-01T03:00:00.000Z', amount: 10 }, // NY: 02-29
142+
{ closed_at: '2024-03-01T07:00:00.000Z', amount: 5 }, // NY: 03-01
143+
];
144+
const ast = {
145+
groupBy: [{ field: 'closed_at', dateGranularity: 'day' as const }],
146+
aggregations: [{ function: 'sum', field: 'amount', alias: 'total' }],
147+
};
148+
const utc = applyInMemoryAggregation(rows, ast, 'UTC');
149+
expect(utc).toEqual([{ closed_at: '2024-03-01', total: 15 }]);
150+
151+
const ny = applyInMemoryAggregation(rows, ast, 'America/New_York').sort(
152+
(a, b) => String(a.closed_at).localeCompare(String(b.closed_at)),
153+
);
154+
expect(ny).toEqual([
155+
{ closed_at: '2024-02-29', total: 10 },
156+
{ closed_at: '2024-03-01', total: 5 },
157+
]);
158+
});
159+
});
108160
});

packages/objectql/src/in-memory-aggregation.ts

Lines changed: 25 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -25,15 +25,22 @@
2525
// invalid values bucket as the literal string `'(null)'` to remain
2626
// consistent with the client `useReportData` hook.
2727

28+
import { calendarPartsInTzOrUtc } from '@objectstack/core';
2829
import type { QueryAST, GroupByNode, AggregationNode, DateGranularityValue } from '@objectstack/spec/data';
2930

3031
/**
3132
* Group + aggregate raw rows according to the AST's `groupBy` /
3233
* `aggregations`. When neither is present, returns the rows unchanged.
34+
*
35+
* `timezone` (ADR-0053 Phase 2) shifts date bucketing to a reference timezone
36+
* so a row near a tz day-boundary lands in the right day/week/month/quarter.
37+
* It is only consulted by `groupBy` items carrying a `dateGranularity`; an
38+
* unset or `'UTC'` value keeps the historical UTC bucketing.
3339
*/
3440
export function applyInMemoryAggregation(
3541
rows: any[],
3642
ast: Pick<QueryAST, 'groupBy' | 'aggregations'>,
43+
timezone?: string,
3744
): any[] {
3845
const groupBy = (ast.groupBy ?? []) as GroupByNode[];
3946
const aggregations = (ast.aggregations ?? []) as AggregationNode[];
@@ -50,7 +57,7 @@ export function applyInMemoryAggregation(
5057
const parts: string[] = [];
5158
for (const g of groupBy) {
5259
const fieldName = typeof g === 'string' ? g : (g.alias ?? g.field);
53-
const value = projectGroupValue(row, g);
60+
const value = projectGroupValue(row, g, timezone);
5461
key[fieldName] = value;
5562
parts.push(`${fieldName}=${value}`);
5663
}
@@ -71,11 +78,11 @@ export function applyInMemoryAggregation(
7178
return out;
7279
}
7380

74-
function projectGroupValue(row: any, g: GroupByNode): string {
81+
function projectGroupValue(row: any, g: GroupByNode, timezone?: string): string {
7582
const field = typeof g === 'string' ? g : g.field;
7683
const v = row?.[field];
7784
if (typeof g !== 'string' && g.dateGranularity) {
78-
return bucketDateValue(v, g.dateGranularity);
85+
return bucketDateValue(v, g.dateGranularity, timezone);
7986
}
8087
return v == null ? '(null)' : String(v);
8188
}
@@ -161,13 +168,23 @@ function toNumber(v: any): number {
161168
/**
162169
* Bucket a date-like value into an ISO-formatted period label. Weeks start
163170
* Monday and use ISO week numbering.
171+
*
172+
* `timezone` (ADR-0053 Phase 2) resolves the calendar day in a reference zone
173+
* so an instant near a tz day-boundary buckets where a user in that zone would
174+
* expect. An unset / `'UTC'` / invalid zone keeps the historical UTC bucketing.
175+
* The y/m/d are taken in the reference zone and the ISO-week math then runs on
176+
* a UTC date built from those parts — the parts already carry the zone shift,
177+
* so the week boundary lands correctly without re-applying any offset.
164178
*/
165-
export function bucketDateValue(value: unknown, granularity: DateGranularityValue): string {
179+
export function bucketDateValue(
180+
value: unknown,
181+
granularity: DateGranularityValue,
182+
timezone?: string,
183+
): string {
166184
if (value == null) return '(null)';
167185
const d = value instanceof Date ? value : new Date(String(value));
168186
if (Number.isNaN(d.getTime())) return '(null)';
169-
const y = d.getUTCFullYear();
170-
const m = d.getUTCMonth() + 1;
187+
const { year: y, month: m, day } = calendarPartsInTzOrUtc(d, timezone);
171188
switch (granularity) {
172189
case 'year':
173190
return String(y);
@@ -176,10 +193,10 @@ export function bucketDateValue(value: unknown, granularity: DateGranularityValu
176193
case 'month':
177194
return `${y}-${String(m).padStart(2, '0')}`;
178195
case 'day':
179-
return `${y}-${String(m).padStart(2, '0')}-${String(d.getUTCDate()).padStart(2, '0')}`;
196+
return `${y}-${String(m).padStart(2, '0')}-${String(day).padStart(2, '0')}`;
180197
case 'week': {
181198
// ISO-8601 week date: week 1 contains the first Thursday of the year.
182-
const target = new Date(Date.UTC(y, d.getUTCMonth(), d.getUTCDate()));
199+
const target = new Date(Date.UTC(y, m - 1, day));
183200
const dayNum = (target.getUTCDay() + 6) % 7; // Mon=0..Sun=6
184201
target.setUTCDate(target.getUTCDate() - dayNum + 3);
185202
const firstThursday = new Date(Date.UTC(target.getUTCFullYear(), 0, 4));

packages/services/service-analytics/src/__tests__/preview-evaluator.test.ts

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,4 +150,14 @@ describe('evaluateAnalyticsQueryOverRows', () => {
150150
expect(matchesWhere({ a: 5 }, { a: { $in: [1, 5] } })).toBe(true);
151151
expect(matchesWhere({ a: 'Hello World' }, { a: { $contains: 'world' } })).toBe(true);
152152
});
153+
154+
it('helpers: bucketDate resolves the calendar day in a reference timezone', () => {
155+
// 2024-03-01T03:00Z is still 2024-02-29 in America/New_York.
156+
const near = '2024-03-01T03:00:00.000Z';
157+
expect(bucketDate(near, 'day', 'America/New_York')).toBe('2024-02-29');
158+
expect(bucketDate(near, 'month', 'America/New_York')).toBe('2024-02');
159+
// Unset / UTC keep the historical UTC bucketing.
160+
expect(bucketDate(near, 'day')).toBe('2024-03-01');
161+
expect(bucketDate(near, 'day', 'UTC')).toBe('2024-03-01');
162+
});
153163
});

packages/services/service-analytics/src/dimension-labels.ts

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,12 @@ const pad = (n: number) => String(n).padStart(2, '0');
6060
* month → "2026-04"
6161
* week → "2026-04-13" (ISO date of the bucket)
6262
* day → "2026-04-15"
63+
*
64+
* Intentionally UTC-only (ADR-0053 Phase 2): timezone bucketing happens
65+
* upstream in `bucketDate` / `bucketDateValue`, so by the time a value reaches
66+
* here it is *already* the reference-zone bucket (often a label string like
67+
* "2026-Q2"). Re-applying a timezone here would shift an already-correct
68+
* `YYYY-MM-DD` day bucket by a day — this is a pure, idempotent re-labeler.
6369
*/
6470
export function formatDateBucket(value: unknown, granularity?: DateGranularity | string): unknown {
6571
if (value == null || value instanceof Date === false) {

0 commit comments

Comments
 (0)