Skip to content

Commit 9a7b3cc

Browse files
claudeclaude
committed
feat(spec,objectql,client,plugin-webhooks): honest bulk event contract for predicate writes (#4639)
A `multi: true` update/delete reaches `IDataDriver.updateMany`/`deleteMany`, contracted to resolve an affected row COUNT and nothing else. That satisfies neither `DataEvent.recordId` (required) nor `before`/`after`/`changes`, so before #4626 the engine fabricated `recordId: ''` with `after: <count>` — an event every schema-compliant consumer must reject, which the webhook enqueuer's `?? 'unknown'` fallback turned into a real delivery naming an unidentifiable record. #4626 removed the fabrication and published nothing instead: honest, but webhooks, knowledge sync and `subscribeData` all went silent for predicate writes. Bulk writes now get their own contract rather than impersonating a per-record one or going dark. - spec: new `BulkDataEventType` / `BulkDataEventSchema` — `data.records.updated` / `data.records.deleted` carrying `object` and `matched`. A separate schema, not a widened `DataEvent`: the type alone tells a consumer no `recordId` is coming, instead of it discovering an empty string at runtime. No `where` — the only predicate in hand at publish time is the middleware-composed AST, whose filter embeds the security layer's injected row scoping (RLS, sharing), and publishing it would ship tenant internals to whatever external URL a webhook points at. - objectql: `publishBulkDataEvent` from the two `multi` branches, validated before publish. A predicate matching zero rows publishes nothing (no data changed), and a driver resolving a non-count publishes nothing and warns rather than asserting an unverified number. Per-record writes are untouched, including a scalar `where.id` with `multi: true`, which stays a single-record target. - plugin-webhooks: opt-in `bulk_update` / `bulk_delete` triggers. Not extra sources for `create`/`update`/`delete` — the body has no `recordId` and no record, so routing it to per-record subscribers would hand them a payload missing every field they read. Dedups on the producer's event uuid, since two sweeps in the same millisecond are distinct events a timestamp key would collapse. Self-heal now also refreshes on a predicate write to `sys_webhook`. - client: `subscribeBulkData`, with the same loud boundary validation. Separate from `subscribeData` so a `BulkDataEvent` never reaches a `(event: DataEvent) => void` callback. - service-knowledge: a knowledge index is a per-record projection and a count names no record, so bulk events cannot drive it. Says so rather than no-opping silently; reconciliation tracked in #4672. Also pays off the measurement debt from #4655, which claimed the write-path cost of event publishing had been measured but never published it: `engine-data-events.bench.ts` puts it at ~7-9us per event against an in-memory driver, paid once per bulk write regardless of match-set size. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LYnZrTwXbrctB8E8HpJAPT
1 parent 5966c2a commit 9a7b3cc

22 files changed

Lines changed: 1383 additions & 92 deletions

‎.changeset/bulk-data-event-contract.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
---
2+
"@objectstack/spec": minor
3+
"@objectstack/objectql": minor
4+
"@objectstack/client": minor
5+
"@objectstack/plugin-webhooks": minor
6+
"@objectstack/service-knowledge": patch
7+
---
8+
9+
feat(spec,objectql,client,plugin-webhooks): predicate writes get an honest bulk event contract (#4639)
10+
11+
A `multi: true` update/delete reaches `IDataDriver.updateMany` / `deleteMany`,
12+
which are contracted to resolve an affected row COUNT and nothing else. That
13+
satisfies neither `DataEvent.recordId` (required) nor `before` / `after` /
14+
`changes`, so before #4626 the engine fabricated a per-record event with
15+
`recordId: ''` and `after: <count>` — an event every schema-compliant consumer
16+
must reject, and one the webhook enqueuer's `?? 'unknown'` fallback turned into
17+
a real delivery naming an unidentifiable record. #4626 removed the fabrication
18+
and published nothing instead: honest, but it left webhooks, knowledge sync and
19+
`subscribeData` silent for every predicate write.
20+
21+
Bulk writes now get their **own** contract rather than impersonating a
22+
per-record one or going dark:
23+
24+
- **New `BulkDataEvent`** (`@objectstack/spec/api`): `data.records.updated` /
25+
`data.records.deleted` — note the plural — carrying `id`, `type`, `object`,
26+
`matched`, `userId?`, `timestamp`. Deliberately a separate schema from
27+
`DataEvent`, not a widened one: a consumer that receives
28+
`data.records.updated` knows from the type alone that no `recordId` is
29+
coming, instead of discovering an empty string at runtime.
30+
- **Engine** publishes it from the `multi: true` branches of `update()` /
31+
`delete()`, validated with `BulkDataEventSchema.parse` before publish. A
32+
predicate that matched **zero** rows publishes nothing (no data changed — this
33+
is what keeps an idle background sweep from becoming an hourly "0 records"
34+
delivery), and a driver that resolves a non-count publishes nothing and warns
35+
rather than asserting a number it cannot verify. Per-record writes are
36+
untouched, including a scalar `where.id` with `multi: true`, which is still a
37+
single-record target and still emits `data.record.deleted`.
38+
- **Webhooks**: two new opt-in triggers, `bulk_update` and `bulk_delete`
39+
(`WebhookTriggerType`, and the `sys_webhook.triggers` multi-select). They are
40+
**not** extra sources for `create` / `update` / `delete`: the delivered body
41+
has no `recordId` and no record, so routing it to existing per-record
42+
subscribers would hand them a payload missing every field they read — the
43+
same class of breakage as the old `recordId: ''`, from the other direction. A
44+
webhook that wants both subscribes to both. Bulk deliveries dedup on the
45+
producer's event uuid, since two sweeps in the same millisecond are genuinely
46+
different events that a timestamp-based key would collapse.
47+
- **Client SDK**: new `client.events.subscribeBulkData(object, cb)`, with the
48+
same loud boundary validation as `subscribeData`. Kept a separate method for
49+
the same reason — delivering a `BulkDataEvent` to a `(event: DataEvent) =>
50+
void` callback would recreate exactly the "typed field, `undefined` at
51+
runtime" defect #4626 removed. `subscribeData`'s own guard was also tightened
52+
from `data.` to `data.record.`, so an aggregate event is ignored rather than
53+
rejected as off-contract.
54+
- **Knowledge sync** now says out loud that a predicate write leaves its index
55+
stale. A knowledge index is a per-record projection and `matched: 40` names no
56+
record, so no event shape could drive it — the durable fix is reconciliation,
57+
tracked in #4672.
58+
59+
The event carries no `where` predicate. The only one available at publish time
60+
is the middleware-composed AST, whose filter embeds the security layer's
61+
injected row scoping (RLS, sharing) — publishing it would ship tenant scoping
62+
internals to whatever external URL a webhook points at.
63+
64+
Also pays off a measurement debt from #4655, which claimed the write-path cost
65+
of event publishing had been measured but never published the numbers:
66+
`packages/objectql/src/engine-data-events.bench.ts` measures it. Against an
67+
in-memory driver, publishing costs ~7–9µs per event (insert 0.021ms vs 0.012ms,
68+
single-id update 0.013ms vs 0.007ms). A bulk write pays that **once** regardless
69+
of how many rows matched (0.040ms vs 0.034ms over a 100-row match set), so its
70+
relative cost shrinks as the match set grows.

‎content/docs/automation/webhooks.mdx‎

Lines changed: 54 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -94,7 +94,7 @@ in `definition_json`, a serialised `Webhook` JSON (canonical schema:
9494
| `name` | text | Unique snake_case name — referenced in logs and audit. |
9595
| `label` | text | Optional display label. |
9696
| `object_name` | text | Short object name whose record events fire this webhook. |
97-
| `triggers` | select | Multi-select of `create` / `update` / `delete`, stored as an array (the enqueuer also accepts a legacy comma-separated string). |
97+
| `triggers` | select | Multi-select of `create` / `update` / `delete` plus the opt-in bulk pair `bulk_update` / `bulk_delete` ([see below](#bulk-writes-bulk_update-and-bulk_delete)), stored as an array (the enqueuer also accepts a legacy comma-separated string). |
9898
| `url` | text | External endpoint that receives the POST. |
9999
| `method` | select | HTTP method — one of `GET` / `POST` / `PUT` / `PATCH` / `DELETE`. Default `POST`. |
100100
| `description` | textarea | Free-text description. |
@@ -214,11 +214,59 @@ is the spec's `DataEvent` (`@objectstack/spec/api`) — validated against
214214
}
215215
```
216216

217-
> **A multi-row write emits no record event.** `updateMany` / `deleteMany`
218-
> (`multi: true`) return only an affected count, so there is no record for a
219-
> `DataEvent` to name and the engine publishes nothing rather than an event
220-
> with an empty `recordId` — meaning webhooks do **not** fire for bulk writes
221-
> today. Tracked in [#4639](https://github.com/objectstack-ai/objectstack/issues/4639).
217+
### Bulk writes: `bulk_update` and `bulk_delete`
218+
219+
A predicate write — `updateMany` / `deleteMany` (`multi: true`) — reports only
220+
an affected count, so there is no record for a `DataEvent` to name. Rather than
221+
publish an event with an empty `recordId` (which every schema-compliant
222+
consumer must reject), the engine publishes a **separate** aggregate event
223+
(#4639):
224+
225+
```ts
226+
{
227+
type: 'data.records.updated', // note: recordS — plural
228+
object: 'account',
229+
timestamp: '<ISO 8601>',
230+
payload: {
231+
id: '<uuid>', // unique event id
232+
type: 'data.records.updated',
233+
object: 'account',
234+
matched: 40, // how many records the predicate affected
235+
userId: 'usr_1', // when the write names an actor
236+
timestamp: '<ISO 8601>',
237+
},
238+
}
239+
```
240+
241+
These dispatch under their own triggers, `bulk_update` and `bulk_delete`, and
242+
they are **opt-in**: a webhook declaring `update` does not receive them. That
243+
is deliberate — the body has no `recordId` and no record, so delivering it to a
244+
subscriber written against the per-record shape would hand it a payload missing
245+
everything it reads.
246+
247+
```ts
248+
webhooks: [{
249+
name: 'account_bulk_audit',
250+
object: 'account',
251+
triggers: ['update', 'bulk_update'], // subscribe to both if you want both
252+
url: 'https://example.com/hooks/accounts',
253+
}]
254+
```
255+
256+
Two properties worth knowing:
257+
258+
- **No event when nothing matched.** A predicate that affected zero rows
259+
changed no data, so it publishes nothing — an idle background sweep does not
260+
become an hourly "0 records" delivery.
261+
- **The predicate is not included.** The only filter available at publish time
262+
is the query after the security layer composed row scoping into it (RLS,
263+
sharing), so sending it would leak tenant scoping internals to the
264+
destination URL. The event states the count and nothing more.
265+
266+
Because a count names no rows, a bulk delivery cannot drive an incremental
267+
per-record projection (a cache, a search index, a mirror). Use it to invalidate,
268+
alert, or schedule a refetch; anything that must know *which* records changed
269+
has to reconcile against the source.
222270

223271
> **Not yet cluster-aware.** The only shipped `IRealtimeService` implementation
224272
> is `InMemoryRealtimeAdapter`, an in-process, single-node pub/sub with no

‎content/docs/references/api/events.mdx‎

Lines changed: 29 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,13 +26,39 @@ Examples:
2626
## TypeScript Usage
2727

2828
```typescript
29-
import { DataEventSchema, DataEventType, MetadataEventSchema, MetadataEventType } from '@objectstack/spec/api';
30-
import type { DataEvent, DataEventType, MetadataEvent, MetadataEventType } from '@objectstack/spec/api';
29+
import { BulkDataEventSchema, BulkDataEventType, DataEventSchema, DataEventType, MetadataEventSchema, MetadataEventType } from '@objectstack/spec/api';
30+
import type { BulkDataEvent, BulkDataEventType, DataEvent, DataEventType, MetadataEvent, MetadataEventType } from '@objectstack/spec/api';
3131

3232
// Validate data
33-
const result = DataEventSchema.parse(data);
33+
const result = BulkDataEventSchema.parse(data);
3434
```
3535

36+
---
37+
38+
## BulkDataEvent
39+
40+
### Properties
41+
42+
| Property | Type | Required | Description |
43+
| :--- | :--- | :--- | :--- |
44+
| **id** | `string` | ✅ | Unique event identifier |
45+
| **type** | `Enum<'data.records.updated' \| 'data.records.deleted'>` | ✅ | Event type |
46+
| **object** | `string` | ✅ | Object name |
47+
| **matched** | `integer` | ✅ | Number of records affected |
48+
| **userId** | `string` | optional | User who triggered the event |
49+
| **timestamp** | `string` | ✅ | Event timestamp |
50+
51+
52+
---
53+
54+
## BulkDataEventType
55+
56+
### Allowed Values
57+
58+
* `data.records.updated`
59+
* `data.records.deleted`
60+
61+
3662
---
3763

3864
## DataEvent

‎content/docs/references/automation/webhook.mdx‎

Lines changed: 25 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,26 @@ producer are declared here — an author can't subscribe to something that
1919

2020
never fires.
2121

22+
**Bulk triggers (#4639).** `bulk_update` / `bulk_delete` map to the engine's
23+
24+
aggregate `data.records.updated` / `data.records.deleted`, emitted when a
25+
26+
predicate write (`multi: true` → `IDataDriver.updateMany`/`deleteMany`)
27+
28+
affects a set of rows the driver reports only as a count. They are separate
29+
30+
trigger values, not extra sources for `update` / `delete`, because their
31+
32+
delivery has a different SHAPE: no `recordId`, no record body, just
33+
34+
`object` + `matched`. Folding them into the per-record triggers would send
35+
36+
every existing subscriber a body missing the fields it reads — the same
37+
38+
class of breakage as the pre-#4626 `recordId: ''` fabrication, arriving from
39+
40+
the other direction. A webhook that wants both subscribes to both.
41+
2242
Deliberately NOT triggers (#3196):
2343

2444
- `undelete` — there is no soft-delete / restore capability in the engine
@@ -47,7 +67,7 @@ value that silently never fires.
4767

4868
```typescript
4969
import { WebhookSchema, WebhookTriggerType } from '@objectstack/spec/automation';
50-
import type { Webhook } from '@objectstack/spec/automation';
70+
import type { Webhook, WebhookTriggerType } from '@objectstack/spec/automation';
5171

5272
// Validate data
5373
const result = WebhookSchema.parse(data);
@@ -63,8 +83,8 @@ const result = WebhookSchema.parse(data);
6383
| :--- | :--- | :--- | :--- |
6484
| **name** | `string` | ✅ | Webhook unique name (lowercase snake_case) |
6585
| **label** | `string` | optional | Human-readable webhook label |
66-
| **object** | `string` | optional | Object whose record events (create/update/delete) trigger this webhook |
67-
| **triggers** | `Enum<'create' \| 'update' \| 'delete'>[]` | optional | Events that trigger execution |
86+
| **object** | `string` | optional | Object whose record events (create/update/delete, bulk_update/bulk_delete) trigger this webhook |
87+
| **triggers** | `Enum<'create' \| 'update' \| 'delete' \| 'bulk_update' \| 'bulk_delete'>[]` | optional | Events that trigger execution |
6888
| **url** | `string` | ✅ | External webhook endpoint URL |
6989
| **method** | `Enum<'GET' \| 'POST' \| 'PUT' \| 'PATCH' \| 'DELETE'>` | ✅ | HTTP method |
7090
| **headers** | `Record<string, string>` | optional | Custom HTTP headers |
@@ -83,6 +103,8 @@ const result = WebhookSchema.parse(data);
83103
* `create`
84104
* `update`
85105
* `delete`
106+
* `bulk_update`
107+
* `bulk_delete`
86108

87109

88110
---

‎content/docs/references/integration/connector.mdx‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -178,7 +178,7 @@ Circuit breaker configuration
178178
| **triggers** | `{ key: string; label: string; description?: string; type: Enum<'polling' \| 'webhook'>; … }[]` | optional | Trigger definitions (not yet enforced — never read at registration; see #3197) |
179179
| **syncConfig** | `{ strategy?: Enum<'full' \| 'incremental' \| 'upsert' \| 'append_only'>; direction?: Enum<'import' \| 'export' \| 'bidirectional'>; schedule?: string \| { dialect: Enum<'cel' \| 'cron' \| 'template'>; source?: string; ast?: any; meta?: object }; realtimeSync?: boolean; … }` | optional | Data sync configuration |
180180
| **fieldMappings** | `{ source: string; target: string; transform?: { type: 'constant'; value: any } \| { type: 'cast'; targetType: Enum<'string' \| 'number' \| 'boolean' \| 'date'> } \| { type: 'lookup'; table: string; keyField: string; valueField: string } \| { type: 'javascript'; expression: string \| { dialect: Enum<'cel' \| 'cron' \| 'template'>; source?: string; ast?: any; meta?: object } } \| { type: 'map'; mappings: Record<string, any> }; defaultValue?: any; … }[]` | optional | Field mapping rules |
181-
| **webhooks** | `{ name: string; label?: string; object?: string; triggers?: Enum<'create' \| 'update' \| 'delete'>[]; … }[]` | optional | Webhook configurations (not yet enforced — never read at registration; see #3197) |
181+
| **webhooks** | `{ name: string; label?: string; object?: string; triggers?: Enum<'create' \| 'update' \| 'delete' \| 'bulk_update' \| 'bulk_delete'>[]; … }[]` | optional | Webhook configurations (not yet enforced — never read at registration; see #3197) |
182182
| **rateLimitConfig** | `{ strategy?: Enum<'fixed_window' \| 'sliding_window' \| 'token_bucket' \| 'leaky_bucket'>; maxRequests: number; windowSeconds: number; burstCapacity?: number; … }` | optional | Rate limiting configuration |
183183
| **retryConfig** | `{ strategy?: Enum<'exponential_backoff' \| 'linear_backoff' \| 'fixed_delay' \| 'no_retry'>; maxAttempts?: number; initialDelayMs?: number; maxDelayMs?: number; … }` | optional | Retry configuration |
184184
| **connectionTimeoutMs** | `number` | optional | Connection timeout in ms |
@@ -336,7 +336,7 @@ Connector type
336336
| **triggers** | `{ key: string; label: string; description?: string; type: Enum<'polling' \| 'webhook'>; … }[]` | optional | Trigger definitions (not yet enforced — never read at registration; see #3197) |
337337
| **syncConfig** | `{ strategy?: Enum<'full' \| 'incremental' \| 'upsert' \| 'append_only'>; direction?: Enum<'import' \| 'export' \| 'bidirectional'>; schedule?: string \| { dialect: Enum<'cel' \| 'cron' \| 'template'>; source?: string; ast?: any; meta?: object }; realtimeSync?: boolean; … }` | optional | Data sync configuration |
338338
| **fieldMappings** | `{ source: string; target: string; transform?: { type: 'constant'; value: any } \| { type: 'cast'; targetType: Enum<'string' \| 'number' \| 'boolean' \| 'date'> } \| { type: 'lookup'; table: string; keyField: string; valueField: string } \| { type: 'javascript'; expression: string \| { dialect: Enum<'cel' \| 'cron' \| 'template'>; source?: string; ast?: any; meta?: object } } \| { type: 'map'; mappings: Record<string, any> }; defaultValue?: any; … }[]` | optional | Field mapping rules |
339-
| **webhooks** | `{ name: string; label?: string; object?: string; triggers?: Enum<'create' \| 'update' \| 'delete'>[]; … }[]` | optional | Webhook configurations (not yet enforced — never read at registration; see #3197) |
339+
| **webhooks** | `{ name: string; label?: string; object?: string; triggers?: Enum<'create' \| 'update' \| 'delete' \| 'bulk_update' \| 'bulk_delete'>[]; … }[]` | optional | Webhook configurations (not yet enforced — never read at registration; see #3197) |
340340
| **rateLimitConfig** | `{ strategy?: Enum<'fixed_window' \| 'sliding_window' \| 'token_bucket' \| 'leaky_bucket'>; maxRequests: number; windowSeconds: number; burstCapacity?: number; … }` | optional | Rate limiting configuration |
341341
| **retryConfig** | `{ strategy?: Enum<'exponential_backoff' \| 'linear_backoff' \| 'fixed_delay' \| 'no_retry'>; maxAttempts?: number; initialDelayMs?: number; maxDelayMs?: number; … }` | optional | Retry configuration |
342342
| **connectionTimeoutMs** | `number` | optional | Connection timeout in ms |
@@ -459,8 +459,8 @@ Synchronization strategy
459459
| :--- | :--- | :--- | :--- |
460460
| **name** | `string` | ✅ | Webhook unique name (lowercase snake_case) |
461461
| **label** | `string` | optional | Human-readable webhook label |
462-
| **object** | `string` | optional | Object whose record events (create/update/delete) trigger this webhook |
463-
| **triggers** | `Enum<'create' \| 'update' \| 'delete'>[]` | optional | Events that trigger execution |
462+
| **object** | `string` | optional | Object whose record events (create/update/delete, bulk_update/bulk_delete) trigger this webhook |
463+
| **triggers** | `Enum<'create' \| 'update' \| 'delete' \| 'bulk_update' \| 'bulk_delete'>[]` | optional | Events that trigger execution |
464464
| **url** | `string` | ✅ | External webhook endpoint URL |
465465
| **method** | `Enum<'GET' \| 'POST' \| 'PUT' \| 'PATCH' \| 'DELETE'>` | ✅ | HTTP method |
466466
| **headers** | `Record<string, string>` | optional | Custom HTTP headers |

0 commit comments

Comments
 (0)