fix(objectql): honor expand where filter on lookup/master_detail expansion (#2140)

* fix(objectql): honor expand `where` filter on lookup/master_detail expansion

The expand post-processor batch-loads related records with an `$in` query but
never merged the nested QueryAST `where`, so a documented
`expand: { rel: { where: {...} } }` filter was silently ignored — every related
record came back regardless of the filter. The spec schema and
query-syntax.mdx both advertise the nested `where`.

Merge the nested filter via an explicit `$and` group
(`{ $and: [{ id: { $in } }, nestedAST.where] }`) rather than a shallow spread:
a spread would clobber the `id` constraint when the nested filter also keys
`id`, and is ambiguous for a top-level `$or`/`$and`. The `$and` wrapper is
correct for every FilterCondition shape and is executed by the memory matcher
and the SQL/mongo filter compilers.

`limit`/`offset` are intentionally still NOT propagated: this path batch-loads
every parent's related records in one `$in` query, so a per-parent limit can't
be expressed — forwarding it would globally cap the batch and drop records
other parents need. `orderBy` is likewise not observable here (records are
re-keyed by FK on injection). Docs + schema describe updated to match, with a
guard test asserting limit/offset are not pushed into the expand query.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore(changeset): add changeset for expand `where` propagation fix

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Jack Zhuang <277994282+os-zhuang@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: 3 people

.changeset/expand-where-propagation.md

@@ -0,0 +1,10 @@
+---
+"@objectstack/objectql": patch
+"@objectstack/spec": patch
+---
+
+Honor a nested `where` filter inside `expand` on lookup/master_detail expansion.
+
+The expand post-processor batch-loads related records with an `id $in [...]` query but never merged the nested QueryAST `where`, so a documented `expand: { rel: { where: {...} } }` filter was silently ignored and every related record came back. The nested filter is now AND-merged into the batch query via an explicit `$and` group (`{ $and: [{ id: { $in } }, nestedAST.where] }`) — robust against a nested filter that itself keys `id` or uses a top-level `$or`/`$and`, where a shallow spread would clobber or reorder the constraint.
+
+`limit`/`offset`/`orderBy` remain intentionally not honored on the expand path: it batch-loads every parent's related records in one `$in` query and re-keys them per parent by foreign key, so a per-parent page size or ordering can't be expressed there. Docs and the schema `describe()` are updated to match, with a guard test asserting `limit`/`offset` are not pushed into the expand query.

content/docs/protocol/objectql/query-syntax.mdx

@@ -442,23 +442,38 @@ The engine resolves `expand` via batch `$in` queries (driver-agnostic) with a de
 
 ### Filtered Expand
 
-You can filter, sort, and limit expanded relations:
+Expansion follows **`lookup`** and **`master_detail`** fields — i.e. the foreign key
+lives on the object you are querying. The nested `QueryAST` can **filter** (`where`) and
+**select** (`fields`) the related records:
 
 ```typescript
 const query: QueryAST = {
-  object: 'account',
-  fields: ['company_name'],
+  object: 'task',
+  fields: ['title', 'assignee'],
   expand: {
-    opportunities: {
-      object: 'opportunity',
-      where: { stage: { $ne: 'Closed Lost' } },
-      orderBy: [{ field: 'amount', order: 'desc' }],
-      limit: 5,
+    // assignee is a lookup → user; only resolve assignees that are still active.
+    assignee: {
+      object: 'user',
+      where: { active: { $eq: true } },
+      fields: ['name', 'email'],
     },
   },
 };
 ```
 
+The nested `where` is **AND-merged** with the batch `$in` the engine uses to load related
+records, so a related record is attached only when it also matches your filter. A foreign
+key whose target is filtered out is left as the raw id (unresolved) rather than dropped.
+
+<Callout type="warn">
+**Per-parent shaping (`limit` / `offset` / `orderBy`) is not honored on the expand path.**
+The engine batch-loads every parent's related records in a single `$in` query and then
+re-attaches them to each parent by foreign key, so it cannot express a per-parent page
+size, and the injected order follows each parent's own foreign-key value rather than the
+nested `orderBy`. To paginate or order related records, query the related object directly
+with its own `where` + `orderBy` + `limit`.
+</Callout>
+
 ---
 
 ## 5. Aggregations

packages/objectql/src/engine.test.ts

@@ -456,6 +456,127 @@ describe('ObjectQL Engine', () => {
             );
         });
 
+        it('should apply a nested expand where-filter to the related $in query', async () => {
+            // Regression: query-syntax.mdx documents `expand: { rel: { where: {...} } }`
+            // and the QueryAST schema accepts it, but the engine used to drop
+            // nestedAST.where — silently returning *all* related records.
+            vi.mocked(SchemaRegistry.getObject).mockImplementation((name) => {
+                if (name === 'task') return {
+                    name: 'task',
+                    fields: {
+                        assignee: { type: 'lookup', reference: 'user' },
+                        title: { type: 'text' },
+                    },
+                } as any;
+                if (name === 'user') return {
+                    name: 'user',
+                    fields: { name: { type: 'text' }, active: { type: 'boolean' } },
+                } as any;
+                return undefined;
+            });
+
+            vi.mocked(mockDriver.find)
+                .mockResolvedValueOnce([
+                    { id: 't1', title: 'Task 1', assignee: 'u1' },
+                    { id: 't2', title: 'Task 2', assignee: 'u2' },
+                ])
+                // Driver does the filtering; only the active user comes back.
+                .mockResolvedValueOnce([{ id: 'u1', name: 'Alice', active: true }]);
+
+            const result = await engine.find('task', {
+                expand: {
+                    assignee: { object: 'user', where: { active: { $eq: true } } },
+                },
+            });
+
+            // The nested filter is AND-merged with the batch $in (not clobbered).
+            expect(mockDriver.find).toHaveBeenCalledTimes(2);
+            const expandCall = vi.mocked(mockDriver.find).mock.calls[1];
+            expect(expandCall[1]).toEqual(
+                expect.objectContaining({
+                    object: 'user',
+                    where: {
+                        $and: [
+                            { id: { $in: ['u1', 'u2'] } },
+                            { active: { $eq: true } },
+                        ],
+                    },
+                }),
+            );
+
+            // Records the driver filtered out keep their raw FK id (unresolved).
+            expect(result[0].assignee).toEqual({ id: 'u1', name: 'Alice', active: true });
+            expect(result[1].assignee).toBe('u2');
+        });
+
+        it('should preserve $or logical groups in a nested expand where (no shallow clobber)', async () => {
+            // A shallow `{ ...nestedAST.where }` spread would be unsafe if the
+            // nested filter itself keyed `id` or used a top-level logical group.
+            vi.mocked(SchemaRegistry.getObject).mockImplementation((name) => {
+                if (name === 'task') return {
+                    name: 'task',
+                    fields: { assignee: { type: 'lookup', reference: 'user' } },
+                } as any;
+                if (name === 'user') return {
+                    name: 'user',
+                    fields: { role: { type: 'text' }, active: { type: 'boolean' } },
+                } as any;
+                return undefined;
+            });
+
+            vi.mocked(mockDriver.find)
+                .mockResolvedValueOnce([{ id: 't1', assignee: 'u1' }])
+                .mockResolvedValueOnce([{ id: 'u1', role: 'admin' }]);
+
+            const nestedWhere = {
+                $or: [{ role: { $eq: 'admin' } }, { active: { $eq: true } }],
+            };
+            await engine.find('task', {
+                expand: { assignee: { object: 'user', where: nestedWhere } },
+            });
+
+            const expandCall = vi.mocked(mockDriver.find).mock.calls[1];
+            expect(expandCall[1]).toEqual(
+                expect.objectContaining({
+                    where: { $and: [{ id: { $in: ['u1'] } }, nestedWhere] },
+                }),
+            );
+        });
+
+        it('should not push nested expand limit/offset into the batched $in query', async () => {
+            // The expand path batch-loads every parent's related records in one
+            // $in query, so a *per-parent* limit/offset can't be expressed here.
+            // Propagating them would globally cap the batch and silently drop
+            // records some parents need — so they are intentionally not forwarded.
+            vi.mocked(SchemaRegistry.getObject).mockImplementation((name) => {
+                if (name === 'task') return {
+                    name: 'task',
+                    fields: { assignee: { type: 'lookup', reference: 'user' } },
+                } as any;
+                if (name === 'user') return { name: 'user', fields: { name: { type: 'text' } } } as any;
+                return undefined;
+            });
+
+            vi.mocked(mockDriver.find)
+                .mockResolvedValueOnce([
+                    { id: 't1', assignee: 'u1' },
+                    { id: 't2', assignee: 'u2' },
+                ])
+                .mockResolvedValueOnce([
+                    { id: 'u1', name: 'Alice' },
+                    { id: 'u2', name: 'Bob' },
+                ]);
+
+            await engine.find('task', {
+                expand: { assignee: { object: 'user', limit: 1, offset: 2 } },
+            });
+
+            const expandCall = vi.mocked(mockDriver.find).mock.calls[1];
+            expect(expandCall[1]).not.toHaveProperty('limit');
+            expect(expandCall[1]).not.toHaveProperty('offset');
+            expect(expandCall[1]).not.toHaveProperty('top');
+        });
+
         it('should expand master_detail fields', async () => {
             vi.mocked(SchemaRegistry.getObject).mockImplementation((name) => {
                 if (name === 'order_item') return {

packages/objectql/src/engine.ts

@@ -1791,11 +1791,27 @@ export class ObjectQL implements IDataEngine {
 
       // Batch-load related records using $in query
       try {
+        // Constrain the batch to the collected FK ids. When the nested expand
+        // AST carries its own `where`, AND-merge it via an explicit `$and`
+        // group rather than spreading keys onto the id filter: a shallow spread
+        // would clobber the `id` constraint (if the nested filter also keys
+        // `id`) and is ambiguous when the nested filter is a top-level `$or` /
+        // `$and`. The `$and` wrapper is correct for every FilterCondition shape.
+        const idFilter = { id: { $in: uniqueIds } };
+        const where = nestedAST.where
+          ? { $and: [idFilter, nestedAST.where] }
+          : idFilter;
+
         const relatedQuery: QueryAST = {
           object: referenceObject,
-          where: { id: { $in: uniqueIds } },
+          where,
           ...(nestedAST.fields ? { fields: nestedAST.fields } : {}),
           ...(nestedAST.orderBy ? { orderBy: nestedAST.orderBy } : {}),
+          // NOTE: nestedAST.limit/offset are intentionally NOT forwarded here.
+          // This path batch-loads every parent's related records in a single
+          // $in query, so a *per-parent* limit/offset can't be expressed — a
+          // global cap on the batch would silently drop records other parents
+          // need. Paginate by querying the related object directly instead.
         };
 
         const driver = this.getDriver(referenceObject);

packages/spec/src/data/query.zod.ts

@@ -586,9 +586,11 @@ export type QueryInput = z.input<typeof BaseQuerySchema> & {
 export const QuerySchema: z.ZodType<QueryAST> = lazySchema(() => BaseQuerySchema.extend({
   expand: z.lazy(() => z.record(z.string(), QuerySchema)).optional().describe(
     'Recursive relation loading map. Keys are lookup/master_detail field names; '
-    + 'values are nested QueryAST objects that control select, filter, sort, and '
-    + 'further expansion on the related object. The engine resolves expand via '
-    + 'batch $in queries (driver-agnostic) with a default max depth of 3.'
+    + 'values are nested QueryAST objects that control select (`fields`) and filter '
+    + '(`where`, AND-merged with the batch $in), plus further expansion on the related '
+    + 'object. The engine resolves expand via batch $in queries (driver-agnostic) with a '
+    + 'default max depth of 3; per-parent `limit`/`offset`/`orderBy` are NOT applied on '
+    + 'this path.'
   ),
 }));