Skip to content

Commit bce47a0

Browse files
committed
feat(ai): HITL phase 3 — approval queue demos and UI polish
1 parent d48a0d6 commit bce47a0

12 files changed

Lines changed: 917 additions & 30 deletions

File tree

.changeset/v5-ai-hitl-e2e-demos.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
---
2+
'@objectstack/service-ai': patch
3+
---
4+
5+
**HITL Phase 3 — end-to-end demos + bug fix in handler-engine adapter.**
6+
7+
Two runnable integration demos for the action-approval queue ship under `examples/app-todo/test/`:
8+
9+
- `ai-hitl.test.ts` — drives the tool registry directly (no LLM). Asserts `variant:'danger'` actions register as tools, invocation returns `pending_approval`, row persists, `approvePendingAction(id, actor)` re-runs the handler, row flips to `executed`. Reject path covered too. Run with `pnpm --filter @example/app-todo test:hitl`.
10+
- `ai-hitl-llm.test.ts` — same scenario behind a real model on Vercel AI Gateway. The LLM autonomously picks `action_delete_completed`, the framework gates the call with `pending_approval`, the model summarises the wait without retrying, and the operator-side approve completes the deletion. Gated on `AI_GATEWAY_API_KEY`. Run with `AI_GATEWAY_API_KEY=... pnpm --filter @example/app-todo test:hitl:llm`.
11+
12+
While wiring the demos, two bugs surfaced in the bypass-approval dispatcher and the handler-engine adapter:
13+
14+
1. **Bulk delete from declarative handlers was silently failing.** The adapter built by `buildHandlerEngineAdapter()` wrapped multi-id deletes as `engine.delete(obj, { where: { id: { $in: ids } } })`, but `ObjectQLEngine.delete()` prefers the scalar `id` branch whenever `where.id` is set — so the `{ $in: [...] }` object was forwarded to `driver.delete(scalar)` and rejected as `"Wrong API use: tried to bind a value of an unknown type ([object Object])"`. The adapter now loops scalar deletes, which is correct and driver-agnostic.
15+
16+
2. **Approval pathway swallowed handler errors.** `createActionToolHandler` returns a `{ ok: false, error }` envelope on failure rather than throwing. The pre-registered bypass dispatcher just JSON-parsed and returned that envelope, so `approvePendingAction` thought the run succeeded and flipped the row to `executed`. The dispatcher now treats `ok === false` as a thrown error, so failed approvals are correctly persisted as `status: 'failed'` with the original message.
17+
18+
Also: added `delete`/`remove`/`purge`/`destroy`/`erase` to `MemoryLLMAdapter.ACTION_VERBS` so the in-memory adapter can route delete-style intents during tests that don't have a real LLM.
19+
20+
Docs: `content/docs/guides/ai-capabilities.mdx` now points at the two integration demos with copy-pasteable run commands.

.changeset/v5-ai-hitl-studio-ui.md

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
---
2+
'@objectstack/service-ai': minor
3+
---
4+
5+
Polish Studio HITL pending-action inbox UI
6+
7+
The `AiPendingActionView` shipped by `service-ai` is now an actual operator
8+
console rather than a flat grid:
9+
10+
- **Drawer detail panel** — clicking any row opens a side drawer
11+
(`navigation: { mode: 'drawer', view: 'detail' }`) with four sections:
12+
Proposal · Tool input · Conversation context · Decision.
13+
- **JSON widget** on `tool_input`, `result`, and `error` so structured tool
14+
arguments and responses are readable without copy-pasting into a formatter.
15+
- **Relative timestamps** (`type: 'datetime-relative'`) on `proposed_at` /
16+
`decided_at` columns and form fields.
17+
- **Conversation/message linkbacks** — the existing `Field.lookup` references
18+
to `ai_conversations` / `ai_messages` are surfaced in a collapsed
19+
"Conversation context" section, giving operators one-click access from a
20+
pending action back to the chat that proposed it.
21+
- **Status-conditional fields** via `visibleOn` predicates — `rejection_reason`
22+
only appears for rejected rows, `error` only for failed rows, etc.
23+
- **Per-row approve/reject buttons** on the Pending tab via `rowActions`
24+
pointing at the existing `approve_pending_action` / `reject_pending_action`
25+
object actions; the same actions also render in the drawer header.
26+
- **Status-coloured rows** to make pending vs failed vs executed scannable.
27+
28+
Snapshot-style tests in `__tests__/ai-pending-action.view.test.ts` lock the
29+
shape so future Studio contract changes (widget renames, navigation modes)
30+
fail loudly in one place.
31+
32+
This is a metadata-only change — Studio (`@object-ui/studio`) interprets the
33+
new view automatically. No backend, REST, or HITL semantics changed; the
34+
end-to-end demos in `examples/app-todo/test/ai-hitl*.test.ts` continue to
35+
pass unmodified.

content/docs/guides/ai-capabilities.mdx

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -433,6 +433,47 @@ The queue is exposed via four REST endpoints (`GET`, `GET/:id`, `POST/:id/approv
433433

434434
**Why a dedicated queue (not the multi-step `IApprovalService`)?** AI tool-call HITL is ephemeral: subject is the proposed *call*, not a stable record state; there's no predefined process; operators expect single-click yes/no. The pending-action queue is a thin write log + dispatcher map that delivers that UX without dragging in process-engine overhead.
435435

436+
#### End-to-end example
437+
438+
Two runnable demos live in `examples/app-todo/test/`:
439+
440+
- **`ai-hitl.test.ts`** — drives the tool registry directly (no LLM dependency). Asserts the full lifecycle: `variant:'danger'` action registered as a tool → invocation returns `pending_approval` without executing → row persisted → `approvePendingAction(id, actor)` re-runs the handler → row transitions to `executed`. Reject path is also covered.
441+
442+
```bash
443+
pnpm --filter @example/app-todo test:hitl
444+
```
445+
446+
- **`ai-hitl-llm.test.ts`** — same scenario but with a real model behind Vercel AI Gateway. The LLM is given the auto-generated tool description and asked to "delete all completed tasks"; the framework returns `pending_approval` so the model summarises the wait instead of retrying. After we call approve from the test, the deletion completes.
447+
448+
```bash
449+
AI_GATEWAY_API_KEY=vck_... pnpm --filter @example/app-todo test:hitl:llm
450+
```
451+
452+
Gated on `AI_GATEWAY_API_KEY` (or `OPENAI_API_KEY`); exits 0 with a notice if no key is provided, so it can be wired into CI without leaking spend.
453+
454+
A trimmed view of the integration path:
455+
456+
```typescript
457+
// 1. Plugin boots with approval gating enabled
458+
new AIServicePlugin({
459+
adapter: new VercelLLMAdapter({ model }),
460+
enableActionApproval: true, // ← opt-in
461+
})
462+
463+
// 2. LLM picks the gated tool — handler short-circuits to pending
464+
const result = await aiService.toolRegistry.execute({
465+
type: 'tool-call',
466+
toolName: 'action_delete_completed',
467+
input: {},
468+
} as never);
469+
const envelope = JSON.parse((result.output as { value: string }).value);
470+
// → { ok: true, status: 'pending_approval', pendingActionId: 'pa_...' }
471+
472+
// 3. Operator approves (REST or programmatic)
473+
const outcome = await ai.approvePendingAction(envelope.pendingActionId, 'alice@example.com');
474+
// → { status: 'executed', result: <handler return> }
475+
```
476+
436477
---
437478

438479
## RAG Pipelines

examples/app-todo/package.json

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,8 @@
1919
"test:ai": "tsx test/ai.test.ts",
2020
"test:agent": "tsx test/ai-agent.test.ts",
2121
"test:action": "tsx test/ai-action.test.ts",
22+
"test:hitl": "tsx test/ai-hitl.test.ts",
23+
"test:hitl:llm": "tsx test/ai-hitl-llm.test.ts",
2224
"test:llm": "tsx test/ai-llm.test.ts"
2325
},
2426
"dependencies": {

examples/app-todo/src/actions/task.actions.ts

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,11 @@ export const DeleteCompletedAction: Action = {
111111
type: 'script',
112112
target: 'deleteCompletedTasks',
113113
locations: ['list_toolbar'],
114+
// Destructive + irreversible — flag as danger so Studio paints it red
115+
// and the AI tool runtime routes through the HITL approval queue when
116+
// `enableActionApproval` is on.
117+
variant: 'danger',
118+
confirmText: 'Permanently delete all completed tasks? This cannot be undone.',
114119
successMessage: 'Completed tasks deleted!',
115120
refreshAfter: true,
116121
};
Lines changed: 239 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,239 @@
1+
// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license.
2+
//
3+
// **Real-LLM HITL smoke test** — confirms the Phase 3 approval queue
4+
// behaves correctly when invoked by an actual model (not just the
5+
// MemoryLLMAdapter heuristics).
6+
//
7+
// Flow:
8+
// 1. Boot with `enableActionApproval: true`.
9+
// 2. Ask the LLM to delete completed tasks.
10+
// 3. Model is expected to pick `action_delete_completed` (a
11+
// `variant: 'danger'` action) — the tool returns
12+
// `{status:'pending_approval'}` instead of executing.
13+
// 4. We assert: tasks NOT deleted, pending row persisted, LLM final
14+
// message acknowledges the pending approval.
15+
// 5. We call `IAIService.approvePendingAction()` (acting as the
16+
// operator) and confirm the underlying handler runs.
17+
//
18+
// Gated on `AI_GATEWAY_API_KEY` (or `OPENAI_API_KEY`). Without a key
19+
// the script exits 0 with a notice, so it can be chained in CI without
20+
// leaking spend.
21+
//
22+
// Run via: `AI_GATEWAY_API_KEY=... pnpm --filter @example/app-todo test:hitl:llm`
23+
24+
import { ObjectKernel, DriverPlugin, AppPlugin } from '@objectstack/runtime';
25+
import { SqliteWasmDriver } from '@objectstack/driver-sqlite-wasm';
26+
import { ObjectQLPlugin } from '@objectstack/objectql';
27+
import {
28+
AIServicePlugin,
29+
VercelLLMAdapter,
30+
registerQueryDataTool,
31+
registerActionsAsTools,
32+
} from '@objectstack/service-ai';
33+
import type { IAIService, IDataEngine } from '@objectstack/spec/contracts';
34+
import { createGateway } from '@ai-sdk/gateway';
35+
import TodoApp from '../objectstack.config';
36+
import { Task } from '../src/objects/task.object';
37+
import { registerTaskActionHandlers } from '../src/actions/register-handlers';
38+
39+
(async () => {
40+
const apiKey = process.env.AI_GATEWAY_API_KEY ?? process.env.OPENAI_API_KEY;
41+
if (!apiKey) {
42+
console.log('ℹ️ AI_GATEWAY_API_KEY not set — skipping real-LLM HITL smoke test.');
43+
console.log(' Provide a Vercel AI Gateway key (or OPENAI_API_KEY) to run.');
44+
process.exit(0);
45+
}
46+
47+
console.log('🛡️🤖 ObjectStack HITL × Real-LLM Smoke Test');
48+
console.log('──────────────────────────────────────────────');
49+
50+
const modelId = process.env.AI_MODEL ?? 'openai/gpt-4.1-mini';
51+
console.log(` Model: ${modelId}`);
52+
53+
process.env.OS_MULTI_TENANT = 'false';
54+
55+
const gateway = createGateway({ apiKey });
56+
const model = gateway.languageModel(modelId);
57+
58+
const kernel = new ObjectKernel();
59+
await kernel.use(new ObjectQLPlugin());
60+
await kernel.use(new DriverPlugin(new SqliteWasmDriver({ filename: ':memory:' })));
61+
await kernel.use(
62+
new AIServicePlugin({
63+
adapter: new VercelLLMAdapter({ model }),
64+
models: [
65+
{
66+
id: modelId,
67+
name: modelId,
68+
version: '1.0',
69+
provider: 'custom',
70+
capabilities: { textGeneration: true, toolCalling: true },
71+
limits: { maxTokens: 8192, contextWindow: 128_000 },
72+
},
73+
],
74+
defaultModelId: modelId,
75+
enableActionApproval: true,
76+
}),
77+
);
78+
await kernel.use(new AppPlugin(TodoApp));
79+
await kernel.bootstrap();
80+
81+
const ql = await (kernel as unknown as {
82+
getServiceAsync: <T>(name: string) => Promise<T>;
83+
}).getServiceAsync<{
84+
registerAction: (objectName: string, handlerName: string, fn: unknown) => void;
85+
}>('data');
86+
registerTaskActionHandlers(ql as never);
87+
88+
const ai = kernel.getService<IAIService>('ai');
89+
if (!ai?.chatWithTools) throw new Error('chatWithTools not available');
90+
const dataEngine = kernel.getService<IDataEngine>('data');
91+
if (!dataEngine) throw new Error('data engine not available');
92+
93+
const mergedTask = TodoApp.objects?.find(o => o.name === 'todo_task') ?? Task;
94+
const aiService = ai as IAIService & {
95+
toolRegistry: Parameters<typeof registerQueryDataTool>[0];
96+
};
97+
const fakeMetadata = { listObjects: async () => [mergedTask] } as never;
98+
registerQueryDataTool(aiService.toolRegistry, { ai, metadata: fakeMetadata, dataEngine });
99+
await registerActionsAsTools(aiService.toolRegistry, {
100+
metadata: fakeMetadata,
101+
dataEngine,
102+
enableActionApproval: true,
103+
aiService: ai,
104+
});
105+
106+
console.log('\n📊 Step 1 — seed snapshot');
107+
const before = (await dataEngine.find('todo_task', {})) as Array<Record<string, unknown>>;
108+
let completed = before.filter(r => r.status === 'completed');
109+
console.log(` ${before.length} tasks total, ${completed.length} completed`);
110+
if (completed.length === 0) {
111+
const first = before[0];
112+
if (!first) {
113+
console.error('❌ No tasks in seed data');
114+
process.exit(1);
115+
}
116+
console.warn(' ⚠️ Seeding: flipping first task to completed so the model has something to delete');
117+
await dataEngine.update(
118+
'todo_task',
119+
{ id: first.id, status: 'completed', completed_date: new Date().toISOString() },
120+
{ where: { id: first.id } },
121+
);
122+
completed = [{ ...first, status: 'completed' }];
123+
}
124+
console.log(` Completed tasks ready: ${completed.length}`);
125+
126+
console.log('\n🧠 Step 2 — ask the real LLM to delete completed tasks');
127+
const userQuestion = 'Please delete all completed tasks for me.';
128+
console.log(` User: "${userQuestion}"`);
129+
130+
const toolCallLog: Array<{ tool: string; args: unknown; output?: string; isError?: boolean }> = [];
131+
const origRegistry = aiService.toolRegistry;
132+
const origExecuteAll = (origRegistry as unknown as { executeAll: Function }).executeAll.bind(
133+
origRegistry,
134+
);
135+
(origRegistry as unknown as { executeAll: Function }).executeAll = async (
136+
calls: Array<Record<string, unknown>>,
137+
) => {
138+
const out = await origExecuteAll(calls);
139+
for (let i = 0; i < calls.length; i++) {
140+
const c = calls[i];
141+
const args = c.args ?? c.input ?? c.arguments ?? {};
142+
const tool = String(c.toolName ?? c.name);
143+
const r = (out as Array<{ output?: unknown; isError?: boolean }>)[i];
144+
const outText =
145+
r && typeof r.output === 'string' ? r.output : JSON.stringify(r?.output ?? r);
146+
toolCallLog.push({ tool, args, output: outText, isError: !!r?.isError });
147+
}
148+
return out;
149+
};
150+
151+
const t0 = Date.now();
152+
const result = await ai.chatWithTools!(
153+
[
154+
{
155+
role: 'system',
156+
content: [
157+
'You are the data_chat agent for an ObjectStack todo app.',
158+
'You can call action_* tools to perform business actions.',
159+
'When a tool returns {status:"pending_approval", pendingActionId, message}, the action',
160+
'has been queued for human review and has NOT executed. Do NOT retry the tool. Inform the',
161+
'user that approval was requested and briefly reference the pendingActionId.',
162+
].join('\n'),
163+
},
164+
{ role: 'user', content: userQuestion },
165+
],
166+
{
167+
model: modelId,
168+
toolChoice: 'auto',
169+
maxIterations: 4,
170+
},
171+
);
172+
const elapsed = Date.now() - t0;
173+
console.log(` Agent (${elapsed}ms): "${result.content}"`);
174+
console.log(` Tool invocations (${toolCallLog.length}):`);
175+
for (const c of toolCallLog) {
176+
const argStr = c.args == null ? '?' : JSON.stringify(c.args).slice(0, 200);
177+
console.log(` → ${c.tool}(${argStr}) ${c.isError ? '✗' : '✓'}`);
178+
if (c.output) console.log(` = ${c.output.slice(0, 400)}`);
179+
}
180+
181+
console.log('\n🛡️ Step 3 — verify HITL gate held');
182+
const deleteCalls = toolCallLog.filter(c => c.tool === 'action_delete_completed');
183+
if (deleteCalls.length === 0) {
184+
console.warn(
185+
' ⚠️ Model did not pick action_delete_completed — skipping HITL assertions (model behaviour, not framework).',
186+
);
187+
console.log('\n🎉 Smoke test complete (LLM chose a different path; framework not exercised).');
188+
process.exit(0);
189+
}
190+
const stillCompleted = (await dataEngine.find('todo_task', {
191+
where: { status: 'completed' },
192+
})) as Array<Record<string, unknown>>;
193+
console.log(` Completed tasks still present: ${stillCompleted.length}`);
194+
if (stillCompleted.length === 0) {
195+
console.error('❌ HITL gate failed — completed tasks were deleted without approval');
196+
process.exit(1);
197+
}
198+
const pending = await (ai as IAIService).listPendingActions!({ status: 'pending' });
199+
console.log(` Pending rows: ${pending.length}`);
200+
if (pending.length === 0) {
201+
console.error('❌ Expected at least one pending row');
202+
process.exit(1);
203+
}
204+
const myRow = pending.find(r => r.action_name === 'delete_completed');
205+
if (!myRow) {
206+
console.error('❌ Pending row for delete_completed not found');
207+
process.exit(1);
208+
}
209+
console.log(` ✓ Pending row: ${myRow.id} (proposed_by=${myRow.proposed_by})`);
210+
211+
console.log('\n✅ Step 4 — operator approves via REST contract');
212+
const outcome = await (ai as IAIService).approvePendingAction!(
213+
myRow.id,
214+
'operator@example.com',
215+
);
216+
console.log(` Outcome: ${outcome.status}`);
217+
if (outcome.status !== 'executed') {
218+
console.error(`❌ approve did not execute: ${outcome.error}`);
219+
process.exit(1);
220+
}
221+
const finalCompleted = (await dataEngine.find('todo_task', {
222+
where: { status: 'completed' },
223+
})) as Array<Record<string, unknown>>;
224+
console.log(` Completed after approval: ${finalCompleted.length}`);
225+
if (finalCompleted.length !== 0) {
226+
console.error('❌ Approval executed but completed tasks remain');
227+
process.exit(1);
228+
}
229+
230+
console.log('\n🎉 Real-LLM HITL Smoke Test Successful!');
231+
console.log(' • Real LLM picked action_delete_completed against an auto-generated tool description');
232+
console.log(' • Framework returned {status:"pending_approval"} — action did NOT run');
233+
console.log(' • Pending row persisted with the LLM-supplied input');
234+
console.log(' • Operator-side approve re-ran the handler and finished the work');
235+
process.exit(0);
236+
})().catch(err => {
237+
console.error('💥 HITL real-LLM smoke test failed:', err);
238+
process.exit(1);
239+
});

0 commit comments

Comments
 (0)