refactor(ai): skills-only AI surface — real Actions instead of 10 fictional tools, retire the two agents - #512
Merged
Conversation
…tional tools, retire the two agents (objectstack#3820, ADR-0109) HotCRM was the corpus that motivated objectstack#3820: 6 skills declaring 16 tool references, of which 10 named tools that exist nowhere. The copilot's instructions therefore claimed abilities it did not have — the runtime silently drops an unresolved tool, so the model improvised or failed when asked to use one. Every instance passed `objectstack validate` and `lint` cleanly. The fix is mostly deletion, because the references were a category error: "analyze the pipeline", "generate email copy", "score this lead" are REASONING, not tools. What actually needed a tool was the handful of steps that change state — and those already exist as HotCRM Actions, reachable through the `action_<name>` tools the runtime materialises from them (ADR-0109's default path: no tool records to author). - case_triage: `triage_case` → the priority rubric moves into instructions; `action_escalate_case` / `action_close_case` do the work. Also repoints the hand-off that named `response_drafting`, a skill that never existed (the audit's "hand-off to a nonexistent skill"), at the real `email_drafting`. - email_drafting: 4 fictional copy tools → the model writes the copy; `action_send_email` sends it, so recipient resolution, permissions and audit match the UI path. - lead_qualification: `analyze_lead` / `suggest_next_action` → BANT reasoning in instructions over `get_record`; `action_convert_lead` / `action_schedule_followup` act, and conversion requires confirmation. - revenue_forecasting: 3 fictional analytics tools → `aggregate_data` / `query_records` / `visualize_data` with the actual forecasting method spelled out in instructions. - live_data, customer_360: already correct, untouched but for prose. The two agents (`sales_copilot`, `service_copilot`) are DELETED. ADR-0063 §2 closed `*.agent.ts` to third parties and cloud#904 made the runtime refuse non-platform agent records, so they were unreachable metadata — carrying them is the ADR-0078 lie. The capability lives in the skills, which attach to the platform assistant by surface affinity. Verified with the merged reference-integrity rule (objectstack#3885 `validate-ai-tool-references`): 22 tool references, 0 findings — was 16 references with 10 dead. typecheck, `objectstack validate`, and the 67-test suite pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BHjroNkLkajskKbJaidko4
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…DR-0011) Follow-up to the skills rewrite in this PR. The first pass routed state changes through `action_escalate_case` / `action_close_case` / `action_send_email` — but the runtime materialises `action_<name>` ONLY for an Action that opts in (`ai.exposed` + `ai.description`) AND has a headless path. All three are `type: 'modal'`: they collect a reason / resolution / final copy from a person, so they are UI-only and no tool is ever generated. Referencing them would have re-created the bug this PR fixes, one layer down. - `convert_lead` and `schedule_followup` are flow-typed with targets, so they only needed the opt-in: both now carry `ai.exposed` + an LLM-facing description. `convert_lead` keeps its `confirmText`, which the runtime reads as approval-required — an agent invocation lands in the HITL queue instead of converting a lead unattended, which is what we want for an irreversible outcome. - `case_triage` and `email_drafting` no longer claim to act. They read, reason, and hand a ready-to-paste `reason` / `resolution` / draft to the user, pointing at the Escalate / Close / Send Email button. The instructions say plainly that the assistant cannot press it — a modal Action staying human-driven is the right shape for these, not a gap. Re-verified with the corrected rule (objectstack#3894, which now models AI exposure): 19 tool references, 0 findings. Both directions hold — the two opted-in flow Actions resolve, the three modal ones would not. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BHjroNkLkajskKbJaidko4
os-zhuang
marked this pull request as ready for review
July 28, 2026 15:35
This was referenced Jul 31, 2026
yinlianghui
pushed a commit
that referenced
this pull request
Jul 31, 2026
#512 deleted `src/agents/` when the AI surface went skills-only, but seven maintainer docs kept printing `src/agents/*.agent.ts` in their tree diagrams and registration tables. `code_examples.md` was the worst of it: after registering a skill it told the author to "add its name to an agent in `src/agents/*.agent.ts`" — an instruction that cannot be followed. `src/cubes/` had the same shape, dropped in favour of datasets (ADR-0021, noted in objectstack.config.ts) and still drawn in two trees. Nothing checked, because a path in prose is just prose. That is the same failure mode as #493 one layer out: metadata validation sees a well-formed string and stops, so the reference rots silently and the next reader — human or agent — is sent somewhere that does not exist. - README.md, AGENTS.md, docs/README.md, docs/STATUS.md, docs/ARCHITECTURE.md, docs/developers/code_examples.md: drop the agent and cube directories; README picks up `src/datasets/` (the layer that actually replaced cubes) and its flow count goes 20 → 23; AGENTS.md's `*.action.ts` becomes `*.actions.ts`, the real suffix. - code_examples.md: drop the skill `permissions: [...]` key — SkillSchema has no such field and strips it silently (#511) — and fill the gap the deletions leave. The example now says which two sources a `tools` name can resolve to (platform data tools, or `action_<name>` from an `ai.exposed` Action with a headless path), why `defineTool` is not a third one, that reasoning belongs in `instructions` (ADR-0109), and where the guard lives. - ARCHITECTURE.md: same note, plus the `case_triage` skill its table had been missing. Adds a repo-tree guard to test/docs-drift.test.ts — every `src/<dir>/` a maintainer doc names must resolve on disk, with `docs/archive/` excluded as a historical record. It earned its place immediately by failing on a stray `no src/agents/` parenthetical in this change's own first pass. 155 tests (+9), typecheck, validate and build pass; the 2 remaining validate warnings are the pre-existing campaign_enrollment ones. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y9STuduWbrAwcgviziaV4e
yinlianghui
pushed a commit
that referenced
this pull request
Jul 31, 2026
…real The allowlist added in #557 guessed the platform tool set from what the `@objectstack/mcp@16.1.0` bridge happens to register, and got it wrong in both directions: - It omitted `search_knowledge`, so the guard would have FAILED a legitimate reference. The tool is real — it is in `PLATFORM_PROVIDED_TOOL_NAMES`, and 16.1.0 documented it all along at `spec/src/ai/knowledge-source.zod.ts:114` ("Whether `search_knowledge` may expose this source to AI agents"). One grep for that name would have caught it; the same absent-from-node_modules inference was applied to `visualize_data` and came out the other way. - It reasoned about deliberately excluding `create_record` / `update_record` / `delete_record`. Those are MCP-bridge tools and are not in the platform registry at all, so there was nothing to exclude. The list is now transcribed verbatim from `PLATFORM_PROVIDED_TOOL_NAMES` in `@objectstack/spec@17.0.0-rc.0` — all 30 entries, diffed equal to the upstream set — carrying instructions to delete the literal and import it on the 17.0 upgrade. A hand-copied registry is the drift risk this file exists to catch, one level up; it is a stopgap for 16.1.0, which exposes no such export. Cross-checked against upstream's own `ai-skill-tool-unresolved` rule, which ships in `@objectstack/lint@17.0.0-rc.0` (objectstack#3820). Both give identical verdicts on seven probes: `search_knowledge`, `query_data`, `todo_write` and `action_convert_lead` accepted; `action_escalate_case` (not `ai.exposed`), `search_knowledgebase` and `triage_case` rejected. Also corrects the "nothing defines it" claim in three places — the `customer_360` docstring, its changeset, and `code_examples.md`. The skill still does not reference the tool, for a narrower and accurate reason: retrieval needs a declared knowledge source, `AIKnowledgeSchema` mounts only on `AgentSchema.knowledge`, and #512 deleted the agents — so a skills-only app has nowhere to declare one and the tool would resolve and return nothing. The skill's instruction line now says the model has no such tool *in this skill*, rather than that none exists. 168 tests pass, typecheck and build clean, validate holds at its 2 pre-existing warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y9STuduWbrAwcgviziaV4e
20 tasks
yinlianghui
pushed a commit
to yinlianghui/hotcrm
that referenced
this pull request
Jul 31, 2026
…in CI (objectstack-ai#493) `customer_360` declared `tools: ['search_knowledge']` — the survivor of the eleven fictional tools objectstack-ai#493 counted, missed by objectstack-ai#512 because that skill looked "already correct". The runtime silently drops an unresolved tool, so the skill shipped with its only declared capability resolving to nothing, while its instructions promised an account + cases + opportunities + knowledge roll-up it had no tool to read. Defining the missing tool was never the fix: `ToolSchema` is a read-only Studio projection — no `implementation` field, no executor loads it — so a hand-authored `search_knowledge` would validate, build, and still never run. And the knowledge base here is not a search service: it is `crm_knowledge_article`, a normal object. The skill now reads it with `query_records` alongside the contacts, cases and opportunities the description already promised, and quotes totals from `aggregate_data`. Adds `test/skills-integrity.test.ts` (its own file — metadata-references is a busy merge surface): - every skill tool resolves to a platform built-in or an `action_<name>` tool materialised from a real Action, with wildcard subscriptions resolved by prefix; - every referenced Action is `ai.exposed` with a ≥40-char LLM description AND a headless path (flow with a defined target, or a script body) — the regression objectstack-ai#512's follow-up commit had to fix by hand; - stack tool metadata cannot be used to satisfy a dangling reference, since it never executes; - every skill handed off to in instructions exists — the defect that pointed `case_triage` at a `response_drafting` skill that was never defined. Each guard verified against the real defect before landing: reinstating `search_knowledge`, `action_escalate_case` and the `response_drafting` hand-off each fails its own assertion. Also corrects two stale docstrings. `case_triage` and `email_drafting` justified their read-only posture by calling `escalate_case`, `close_case` and `send_email` `type: 'modal'` with no headless path; objectstack-ai#515 and the send_email rewrite retyped all three to flow/script. The posture is right, the reason was wrong — none of the three opts in via `ai.exposed` (ADR-0011 is default off), so no tool is materialised and the human review step stays a deliberate governance call. typecheck, validate, build and the full suite pass (146 tests, +6). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y9STuduWbrAwcgviziaV4e
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HotCRM is the corpus that motivated objectstack-ai/objectstack#3820: 6 skills declaring 16 tool references, 10 of which named tools that exist nowhere. The copilot's instructions claimed abilities it did not have — the runtime silently drops an unresolved tool, so the model improvised or failed when asked to use one. Every instance passed
objectstack validateandlintcleanly. This is the dogfood fix, and the first app to run on ADR-0109's authoring model.The fix is mostly deletion
The fictional references were a category error: "analyze the pipeline", "generate email copy", "score this lead" are reasoning, not tools. What genuinely needed a tool was the handful of steps that change state — and those already exist as HotCRM Actions, reachable through the
action_<name>tools the runtime materialises from them. That is ADR-0109's default path: no tool records to author.case_triagetriage_caseaction_escalate_case/action_close_casedo the workemail_draftinggenerate_email_copy,optimize_subject_line,personalize_content,generate_emailaction_send_emailsends itlead_qualificationanalyze_lead,suggest_next_actionget_record;action_convert_lead/action_schedule_followupactrevenue_forecastinganalyze_pipeline,identify_at_risk,forecast_revenueaggregate_data/query_records/visualize_data+ the forecasting method spelled outlive_data,customer_360Because every state change now goes through the same Action the UI button uses, permissions and audit are identical whether a human clicks or the assistant calls.
case_triagealso handed off toresponse_drafting— a skill that never existed (the original audit's "hand-off to a nonexistent skill", §7 non-goal because it lives in free-form prompt text). Repointed at the realemail_draftingwhile rewriting the instructions.The two agents are deleted
sales_copilotandservice_copilotwere unreachable metadata: ADR-0063 §2 closed*.agent.tsto third parties, and objectstack-ai/cloud#904 made the runtime refuse non-platform agent records. Carrying them is exactly the ADR-0078 lie the whole issue is about. The capability lives in the skills, which attach to the platform assistant by surface affinity — so this removes a fiction without removing a feature.Prose updated to match (
src/docs/crm_admin.md"AI copilots" → "AI skills", plus two stale comments naming the copilots).Verification
validate-ai-tool-references(the rule merged in feat(spec,lint): ADR-0109 revised + Phase 1 — skills-only default path, tool-name registry, advisory skill.tools lint (#3820 R7) objectstack#3885) run against the rewritten stack: 22 tool references, 0 findings — was 16 references with 10 dead.pnpm typecheckclean;objectstack validateclean (stack summary no longer lists agents); test suite 8 files / 67 tests pass.Refs objectstack-ai/objectstack#3820 · ADR-0109 · ADR-0063 · ADR-0078
🤖 Generated with Claude Code
https://claude.ai/code/session_01BHjroNkLkajskKbJaidko4
Generated by Claude Code