Skip to content

Commit ebc7ecb

Browse files
os-zhuangclaude
andauthored
feat(dogfood): add in-process runtime regression gate (#2020)
Static gates (build, unit tests, spec-liveness, CodeQL) verify each layer in isolation, usually against mocks, and cannot catch a break that only appears when the real engine + strategies + services + HTTP context run together. The #2018 tz-bucketing regression is the canonical case: green on every static gate (900+ unit tests included) because each of its three broken seams was individually correct and individually mocked. This adds `@objectstack/dogfood` (private), a harness that boots real example apps in-process against in-memory SQLite — wired with the same service plugins `objectstack dev` loads — and exercises them through the real HTTP surface via Hono request-injection (no ports, no sockets, CI-stable). Tests act as a browser client: sign in, hit /api/v1/..., assert on real responses. - src/harness.ts — bootDogfoodStack(config) → { kernel, api, raw, signIn, apiAs, stop } - test/analytics-timezone.dogfood.test.ts — golden regression for #1982/#2018: org timezone (set via the real settings route) must shift a boundary-crossing instant's date bucket (UTC 2024-03-01 → America/Los_Angeles 2024-02-29) with the correct count. Verified to FAIL when any #2018 seam is reverted. - CI: new "Dogfood Regression Gate" job (also runs under turbo run test). Boot-to-assert ~2s. Private package — no changeset. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent fd07027 commit ebc7ecb

7 files changed

Lines changed: 440 additions & 0 deletions

File tree

.github/workflows/ci.yml

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,61 @@ jobs:
106106
path: packages/spec/coverage/
107107
retention-days: 30
108108

109+
dogfood:
110+
name: Dogfood Regression Gate
111+
needs: filter
112+
if: needs.filter.outputs.core == 'true'
113+
runs-on: ubuntu-latest
114+
permissions:
115+
contents: read
116+
117+
steps:
118+
- name: Checkout repository
119+
uses: actions/checkout@v6
120+
121+
- name: Setup Node.js
122+
uses: actions/setup-node@v6
123+
with:
124+
node-version: '20'
125+
126+
- name: Enable Corepack
127+
run: corepack enable
128+
129+
- name: Verify pnpm version
130+
run: pnpm --version
131+
132+
- name: Get pnpm store directory
133+
shell: bash
134+
run: |
135+
echo "STORE_PATH=$(pnpm store path --silent)" >> $GITHUB_ENV
136+
137+
- name: Setup pnpm cache
138+
uses: actions/cache@v5
139+
with:
140+
path: ${{ env.STORE_PATH }}
141+
key: ${{ runner.os }}-pnpm-store-v3-${{ hashFiles('**/pnpm-lock.yaml') }}
142+
restore-keys: |
143+
${{ runner.os }}-pnpm-store-v3-
144+
145+
- name: Setup Turbo cache
146+
uses: actions/cache@v5
147+
with:
148+
path: .turbo/cache
149+
key: ${{ runner.os }}-turbo-${{ github.job }}-${{ github.ref_name }}-${{ github.sha }}
150+
restore-keys: |
151+
${{ runner.os }}-turbo-${{ github.job }}-${{ github.ref_name }}-
152+
${{ runner.os }}-turbo-${{ github.job }}-
153+
154+
- name: Install dependencies
155+
run: pnpm install --frozen-lockfile
156+
157+
# Boots real example apps in-process (in-memory SQLite) and exercises them
158+
# through the real HTTP + service stack — catches runtime regressions that
159+
# build / unit tests / spec-liveness pass over (e.g. the #2018 tz-bucketing
160+
# break, which was green on every static gate).
161+
- name: Boot example apps and exercise real user flows
162+
run: pnpm turbo run test --filter=@objectstack/dogfood
163+
109164
build-core:
110165
name: Build Core
111166
needs: filter

packages/dogfood/README.md

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# @objectstack/dogfood
2+
3+
Runtime regression gate. Private (never published).
4+
5+
## Why
6+
7+
Static gates — `build`, unit tests, spec-liveness, CodeQL — verify each layer in
8+
isolation, usually against mocks. They cannot catch a break that only appears
9+
when the **real engine + strategies + services + HTTP context run together**.
10+
11+
The canonical example is [#2018](https://github.com/objectstack-ai/framework/pull/2018):
12+
"organization timezone drives analytics date bucketing" was broken across three
13+
seams (analytics strategy routing, in-memory count, REST execution-context).
14+
Every static gate was green — 900+ unit tests included — because each layer was
15+
individually correct and individually mocked. The bug was only visible by
16+
booting the app and comparing a date bucket under UTC vs a non-UTC org timezone.
17+
18+
This package boots **real example apps in-process** (in-memory SQLite), wired
19+
with the same service plugins `objectstack dev` loads, and exercises them
20+
through the **real HTTP surface** (Hono request-injection — no ports, no
21+
sockets, CI-stable). Tests act as a browser client would: sign in, hit
22+
`/api/v1/...`, assert on real responses.
23+
24+
## Layout
25+
26+
- `src/harness.ts``bootDogfoodStack(config)``{ kernel, api, raw, signIn, apiAs, stop }`.
27+
- `test/*.dogfood.test.ts` — golden flows. Each should assert on **observable
28+
output** (a number, a bucket label, a row count), not just "no error".
29+
30+
## Adding a golden test
31+
32+
1. Pick a real user flow that a static test can't cover (it spans engine +
33+
service + HTTP, or depends on seeded/written data).
34+
2. `bootDogfoodStack(<appConfig>)`, `signIn()`, drive it via `api()/apiAs()`.
35+
3. Assert on the concrete result.
36+
4. **Prove it catches the bug**: temporarily revert the relevant fix and confirm
37+
the test goes red. A green-on-the-bug test is not a gate.
38+
39+
Runs in CI as the `Dogfood Regression Gate` job (and under `turbo run test`).

packages/dogfood/package.json

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
{
2+
"name": "@objectstack/dogfood",
3+
"version": "0.0.0",
4+
"private": true,
5+
"license": "Apache-2.0",
6+
"description": "Dogfood regression gate — boots real example apps in-process and exercises them through the real HTTP/service stack, catching runtime regressions that static checks (build, unit tests, spec liveness) miss.",
7+
"type": "module",
8+
"scripts": {
9+
"test": "vitest run"
10+
},
11+
"dependencies": {
12+
"@objectstack/core": "workspace:*",
13+
"@objectstack/runtime": "workspace:*",
14+
"@objectstack/objectql": "workspace:*",
15+
"@objectstack/spec": "workspace:*",
16+
"@objectstack/driver-sqlite-wasm": "workspace:*",
17+
"@objectstack/plugin-hono-server": "workspace:*",
18+
"@objectstack/rest": "workspace:*",
19+
"@objectstack/plugin-auth": "workspace:*",
20+
"@objectstack/plugin-security": "workspace:*",
21+
"@objectstack/service-settings": "workspace:*",
22+
"@objectstack/service-analytics": "workspace:*",
23+
"@objectstack/example-crm": "workspace:*"
24+
},
25+
"devDependencies": {
26+
"@types/node": "^25.9.3",
27+
"typescript": "^6.0.3",
28+
"vitest": "^4.1.9"
29+
}
30+
}

packages/dogfood/src/harness.ts

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
//
3+
// Dogfood boot harness.
4+
//
5+
// Boots a real ObjectStack app **in-process** against an in-memory SQLite
6+
// database, wired with the same service plugins `objectstack dev` loads, and
7+
// exposes the live HTTP surface via Hono's request-injection (no port, no
8+
// sockets — CI-stable). Tests then exercise the app exactly as a browser
9+
// client would: sign in, hit `/api/v1/...`, assert on real responses.
10+
//
11+
// Why this exists: the bucketing regression fixed in #2018 passed every static
12+
// gate (build, 900+ unit tests, spec-liveness, CodeQL) because each layer was
13+
// individually correct and individually mocked — the break only appeared when
14+
// the real engine + strategy + settings + REST context ran together. Unit
15+
// tests that mock the protocol/server (e.g. rest.test.ts) cannot catch that.
16+
// This harness runs the integrated stack so they can.
17+
18+
import { ObjectKernel, AppPlugin, DriverPlugin, createDispatcherPlugin } from '@objectstack/runtime';
19+
import { ObjectQLPlugin } from '@objectstack/objectql';
20+
import { SqliteWasmDriver } from '@objectstack/driver-sqlite-wasm';
21+
import { HonoServerPlugin } from '@objectstack/plugin-hono-server';
22+
import { createRestApiPlugin } from '@objectstack/rest';
23+
import { AuthPlugin } from '@objectstack/plugin-auth';
24+
import { SecurityPlugin } from '@objectstack/plugin-security';
25+
import { SettingsServicePlugin } from '@objectstack/service-settings';
26+
import { AnalyticsServicePlugin } from '@objectstack/service-analytics';
27+
28+
/** A Hono app exposes `.request(path, init)` returning a standard `Response`. */
29+
interface InjectableApp {
30+
request(input: string, init?: RequestInit): Promise<Response>;
31+
}
32+
33+
const API_PREFIX = '/api/v1';
34+
const DEFAULT_ADMIN_EMAIL = 'admin@objectos.ai';
35+
const DEFAULT_ADMIN_PASSWORD = 'admin123';
36+
37+
export interface DogfoodStack {
38+
/** The booted kernel — for direct service calls when bypassing HTTP is intentional. */
39+
kernel: ObjectKernel;
40+
/** Inject an HTTP request through the real Hono app (no socket). Path is relative to `/api/v1`. */
41+
api(path: string, init?: RequestInit): Promise<Response>;
42+
/** Inject a request at an absolute path (e.g. `/api/settings/...`). */
43+
raw(path: string, init?: RequestInit): Promise<Response>;
44+
/** Sign in through the real auth route; returns a bearer token. Defaults to the dev admin. */
45+
signIn(email?: string, password?: string): Promise<string>;
46+
/** Convenience: an authed JSON request relative to `/api/v1`. */
47+
apiAs(token: string, method: string, path: string, body?: unknown): Promise<Response>;
48+
/** Tear down the kernel (close DB / HTTP handles). */
49+
stop(): Promise<void>;
50+
}
51+
52+
export interface BootOptions {
53+
/** Override the dev admin credentials the harness signs in with. */
54+
admin?: { email: string; password: string };
55+
}
56+
57+
/**
58+
* Boot an app config in-process and return a live dogfood stack.
59+
*
60+
* `NODE_ENV` is forced to `development` so the auth plugin's dev-admin
61+
* bootstrap provisions a known, loginable admin (mirrors `objectstack dev`).
62+
*/
63+
export async function bootDogfoodStack(
64+
// eslint-disable-next-line @typescript-eslint/no-explicit-any
65+
config: any,
66+
opts: BootOptions = {},
67+
): Promise<DogfoodStack> {
68+
process.env.NODE_ENV = 'development';
69+
70+
const kernel = new ObjectKernel();
71+
72+
// Data engine + in-memory SQLite (pure-JS WASM driver — no native build, CI-safe).
73+
await kernel.use(new ObjectQLPlugin());
74+
await kernel.use(new DriverPlugin(new SqliteWasmDriver({ filename: ':memory:' })));
75+
76+
// HTTP server (registers the `http-server` IHttpServer service the REST +
77+
// dispatcher plugins mount their routes onto). Port 0 = ephemeral; we never
78+
// hit the socket — requests are injected through the Hono app directly.
79+
await kernel.use(new HonoServerPlugin({ port: 0 }));
80+
81+
// The app under test (objects, datasets, cubes, flows, seed data).
82+
await kernel.use(new AppPlugin(config));
83+
84+
// Service plugins `objectstack dev` auto-loads for an app of this shape.
85+
await kernel.use(new SettingsServicePlugin());
86+
await kernel.use(new AnalyticsServicePlugin());
87+
await kernel.use(new AuthPlugin({ secret: 'dogfood-regression-secret' }));
88+
await kernel.use(new SecurityPlugin());
89+
90+
// REST + dispatcher route surfaces (mount onto the http-server service).
91+
await kernel.use(createRestApiPlugin({ api: { api: { requireAuth: true } } as never }));
92+
await kernel.use(createDispatcherPlugin({}));
93+
94+
// Fire the ready lifecycle: seed data, dev-admin bootstrap, route registration.
95+
await kernel.bootstrap();
96+
97+
const httpServer = await kernel.getServiceAsync<{ getRawApp(): InjectableApp; close?(): Promise<void> }>(
98+
'http-server',
99+
);
100+
const app = httpServer.getRawApp();
101+
102+
const raw = (path: string, init?: RequestInit) => app.request(path, init);
103+
const api = (path: string, init?: RequestInit) => raw(`${API_PREFIX}${path}`, init);
104+
105+
const admin = opts.admin ?? { email: DEFAULT_ADMIN_EMAIL, password: DEFAULT_ADMIN_PASSWORD };
106+
107+
const signIn = async (
108+
email: string = admin.email,
109+
password: string = admin.password,
110+
): Promise<string> => {
111+
const res = await api('/auth/sign-in/email', {
112+
method: 'POST',
113+
headers: { 'Content-Type': 'application/json' },
114+
body: JSON.stringify({ email, password }),
115+
});
116+
if (!res.ok) {
117+
throw new Error(`dogfood signIn failed: ${res.status} ${await res.text()}`);
118+
}
119+
const data = (await res.json()) as { token?: string };
120+
if (!data.token) throw new Error('dogfood signIn: no token in response');
121+
return data.token;
122+
};
123+
124+
const apiAs = (token: string, method: string, path: string, body?: unknown) =>
125+
api(path, {
126+
method,
127+
headers: {
128+
'Content-Type': 'application/json',
129+
Authorization: `Bearer ${token}`,
130+
},
131+
...(body !== undefined ? { body: JSON.stringify(body) } : {}),
132+
});
133+
134+
const stop = async () => {
135+
try {
136+
await httpServer.close?.();
137+
} catch {
138+
/* best-effort */
139+
}
140+
try {
141+
// eslint-disable-next-line @typescript-eslint/no-explicit-any
142+
await (kernel as any).shutdown?.();
143+
} catch {
144+
/* best-effort */
145+
}
146+
};
147+
148+
return { kernel, api, raw, signIn, apiAs, stop };
149+
}
Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
//
3+
// GOLDEN REGRESSION for #1982 / #2018 — "organization timezone drives analytics
4+
// date bucketing", exercised end-to-end through the real HTTP + service stack.
5+
//
6+
// This is the test that would have caught #2018 before merge. That bug passed
7+
// every static gate: it lived in the *integration* of NativeSQLStrategy routing
8+
// + in-memory count + REST execution-context resolution. Only booting the app
9+
// and comparing UTC vs non-UTC buckets surfaces it.
10+
11+
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
12+
import crmStack from '@objectstack/example-crm';
13+
import { bootDogfoodStack, type DogfoodStack } from '../src/harness.js';
14+
15+
// 03:00 UTC on 2024-03-01 is still 2024-02-29 (19:00) in America/Los_Angeles
16+
// (PST = UTC-8, before DST). So a *day* bucket labels this instant 2024-03-01
17+
// under UTC but 2024-02-29 under LA — the exact boundary behavior #2018 fixed.
18+
const BOUNDARY = '2024-03-01T03:00:00.000Z';
19+
const N_LEADS = 3;
20+
21+
const leadByDay = {
22+
name: 'lead_by_day',
23+
label: 'Leads by day',
24+
object: 'crm_lead',
25+
dimensions: [
26+
{ name: 'created', label: 'Created', field: 'created_at', type: 'date', dateGranularity: 'day' },
27+
],
28+
measures: [{ name: 'cnt', label: 'Count', aggregate: 'count' }],
29+
};
30+
31+
describe('dogfood: org timezone drives analytics date bucketing (#1982/#2018)', () => {
32+
let stack: DogfoodStack;
33+
let token: string;
34+
35+
beforeAll(async () => {
36+
stack = await bootDogfoodStack(crmStack);
37+
38+
// Deterministic fixture: N leads pinned to the tz-boundary instant, inserted
39+
// as system so the write path's defaults/validation don't fight the setup.
40+
// eslint-disable-next-line @typescript-eslint/no-explicit-any
41+
const ql = await stack.kernel.getServiceAsync<any>('objectql');
42+
for (let i = 0; i < N_LEADS; i++) {
43+
await ql.insert(
44+
'crm_lead',
45+
{ name: `tz-lead-${i}`, status: 'new', created_at: BOUNDARY },
46+
{ context: { isSystem: true } },
47+
);
48+
}
49+
// Sanity: confirm created_at actually persisted as the boundary instant
50+
// (if a stamp hook overrode it the whole premise is void — fail loudly).
51+
const mine = (await ql.find('crm_lead', {
52+
where: { name: { $in: ['tz-lead-0', 'tz-lead-1', 'tz-lead-2'] } },
53+
context: { isSystem: true },
54+
})) as Array<Record<string, unknown>>;
55+
expect(mine).toHaveLength(N_LEADS);
56+
for (const r of mine) {
57+
expect(new Date(r.created_at as string).toISOString()).toBe(BOUNDARY);
58+
}
59+
60+
token = await stack.signIn();
61+
}, 60_000);
62+
63+
afterAll(async () => {
64+
await stack?.stop();
65+
});
66+
67+
// Set the org-wide timezone via the real settings route (NO explicit per-query
68+
// tz), then bucket — exercising settings → REST exec-context → analytics.
69+
async function bucketsForOrgTz(tz: string): Promise<Record<string, number>> {
70+
const put = await stack.raw('/api/settings/localization', {
71+
method: 'PUT',
72+
headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${token}` },
73+
body: JSON.stringify({ timezone: tz }),
74+
});
75+
expect(put.status).toBe(200);
76+
77+
const res = await stack.apiAs(token, 'POST', '/analytics/dataset/query', {
78+
dataset: leadByDay,
79+
selection: { dimensions: ['created'], measures: ['cnt'] },
80+
});
81+
expect(res.status).toBe(200);
82+
const body = (await res.json()) as { rows?: Array<{ created: string; cnt: number }> };
83+
const out: Record<string, number> = {};
84+
for (const row of body.rows ?? []) out[row.created] = row.cnt;
85+
return out;
86+
}
87+
88+
it('buckets the boundary instant on the UTC calendar day with the correct count', async () => {
89+
const utc = await bucketsForOrgTz('UTC');
90+
// The 3 boundary leads land on 2024-03-01; cnt must be 3 (not 0 — the
91+
// count-all `*` in-memory bug), and not split across raw timestamps.
92+
expect(utc['2024-03-01']).toBe(N_LEADS);
93+
expect(utc['2024-02-29']).toBeUndefined();
94+
});
95+
96+
it('shifts the bucket to the previous day under America/Los_Angeles', async () => {
97+
const la = await bucketsForOrgTz('America/Los_Angeles');
98+
// The regression: before #2018 this stayed on 2024-03-01 (tz dropped) or
99+
// returned cnt 0. The org timezone must move it to 2024-02-29.
100+
expect(la['2024-02-29']).toBe(N_LEADS);
101+
expect(la['2024-03-01']).toBeUndefined();
102+
});
103+
});

0 commit comments

Comments
 (0)