Upstream gap: `validate-ai-tool-references` does not ship in @objectstack/* 16.1.0 — skill tool references are unchecked platform-wide

[yinlianghui]

Why this is filed here

This is an upstream (objectstack-ai/objectstack) gap, filed in hotcrm as a tracker because the session that found it has no access to the upstream repo (add_repo → you need push access to objectstack-ai/objectstack). The ready-to-paste upstream body is at the bottom — someone with push access can move it over and close this.

The gap

#512's commit message states its fix was:

Verified with the merged reference-integrity rule (objectstack#3885 validate-ai-tool-references): 22 tool references, 0 findings

and the follow-up commit adds:

Re-verified with the corrected rule (objectstack#3894, which now models AI exposure): 19 tool references, 0 findings

That rule is not present in the @objectstack/*@16.1.0 packages this repo installs. Which is why search_knowledge in customer_360 survived #512 and had to be caught by hand in #493 / #557 — nothing in the shipped toolchain could see it.

Evidence

Checked three layers, not just one:

1. @objectstack/lint@16.1.0 — the strings skill, tool and agent do not appear anywhere in dist/index.js. Zero occurrences, so no rule of any kind inspects AI metadata.
2. objectstack validate (@objectstack/cli@16.1.0) — runs 13 checks; none touches AI metadata:
   JSX-source pages (ADR-0080), React-source page props (ADR-0081), React-source pages (ADR-0081), SDUI styling (ADR-0065), capability providers, capability references (ADR-0066), dashboard action references (ADR-0049), dashboard widget bindings (ADR-0021), flow trigger wiring, list-view navigation modes (ADR-0053), security posture (ADR-0090 D7), source-page styling (ADR-0065), view container shape.
3. lint-liveness-properties (@objectstack/cli) — this one does cover skills (TYPE_COLLECTIONS includes { type: 'skill', key: 'skills' }), but it is ledger-driven and only flags dead authored properties. It has no notion of resolving a tools[] name against a registry, so it cannot catch this class.

So the check either was never released, was dropped before 16.1.0, or lives only in the closed cloud distribution. Any of the three is worth knowing.

Why it matters beyond hotcrm

SkillSchema.tools is a list of bare strings. validate and build check only that each is well-formed snake_case; at runtime an unresolved name is silently dropped. The model is then handed instructions describing a capability it does not have, and improvises. Every instance of this ships green — hotcrm accumulated eleven of them across six skills before anyone noticed, and it is the reference app. Any app built on ObjectStack has the same blind spot.

Interim mitigation (already merged here)

#557 added test/skills-integrity.test.ts, a repo-local guard: every skill tool must resolve to a platform built-in or an action_<name> materialised from an Action that is ai.exposed with a headless path; AI tool metadata cannot satisfy a dangling reference (ToolSchema is a read-only projection with no executor); and skill-to-skill handoffs in instructions must resolve.

That is a workaround, not a fix — every ObjectStack app would have to reimplement it, and the built-in allowlist has to be hand-maintained because the AI runtime ships closed (ADR-0025 §2) and exposes no authoritative tool list to validate against.

Asks for upstream

1. Confirm whether validate-ai-tool-references (objectstack#3885 / #3894) shipped, and in which package/version. If it did not, that makes refactor(ai): skills-only AI surface — real Actions instead of 10 fictional tools, retire the two agents #512's "0 findings" verification unfounded — worth a note on that PR.
2. Ship it in the open-edition toolchain (@objectstack/lint or objectstack validate), not only in cloud. The metadata it validates is authored in the open edition, so the check belongs there.
3. Expose the platform built-in tool registry as data the open edition can read. Without it, any local guard hardcodes a list that silently drifts from the runtime — which is a smaller version of the same bug.

---

Ready-to-paste body for the upstream issue

Title: validate-ai-tool-references is not in the 16.1.0 open-edition toolchain — skill tools[] names are unvalidated

SkillSchema.tools holds bare strings. objectstack validate / build check only the snake_case shape; the runtime silently drops a name it cannot resolve. The result is a skill whose instructions describe tools the model does not have — it validates, builds, and passes CI in that state.

objectstack#3885 / #3894 describe a validate-ai-tool-references rule for exactly this, and downstream work cited it as the verification. It is not present in @objectstack/*@16.1.0:

• @objectstack/lint@16.1.0 dist/index.js contains zero occurrences of skill, tool, or agent.
• objectstack validate (@objectstack/cli@16.1.0) runs 13 checks, none over AI metadata (ADR-0080/0081 pages, SDUI/source styling ADR-0065, capability providers + references ADR-0066, dashboard action refs ADR-0049, widget bindings ADR-0021, flow trigger wiring, list-view nav ADR-0053, security posture ADR-0090 D7, view container shape).
• lint-liveness-properties does walk skills, but is ledger-driven over dead properties only — it has no reference-resolution step.

Impact: objectstack-ai/hotcrm, the reference app, accumulated eleven skill references to tools that exist nowhere (#493). Ten were removed by hand in #512, the last in #557, all found by human audit rather than by tooling.

Requests:

1. Confirm whether the rule shipped, and where.
2. Ship it in the open-edition toolchain — the metadata is authored there, so the check belongs there.
3. Expose the built-in tool registry as data the open edition can read. Any downstream guard currently has to hardcode the built-in list, which drifts from the runtime silently — a smaller instance of the same defect.

[yinlianghui]

Closing: fixed upstream in 17.0.0-rc.0, and duplicates an existing upstream issue

1. The rule ships in 17.0.0-rc.0

@objectstack/lint@17.0.0-rc.0 contains src/validate-ai-tool-references.ts, rule id ai-skill-tool-unresolved. Verified by installing the rc into a scratch tree and running it against this repo's compiled dist/objectstack.json:

input	result
current main	0 findings
pre-#557 state (customer_360.tools = ['search_knowledge'])	0 findings
control (search_knowledgebase, a real typo)	1 warning, with Did you mean "search_knowledge"?

It also imports PLATFORM_PROVIDED_TOOL_NAMES / PLATFORM_TOOL_FAMILY_PREFIXES from @objectstack/spec/system — the authoritative built-in registry this issue asked for (ask 3). Asks 2 and 3 are therefore both satisfied upstream.

2. Duplicate of an existing upstream issue

objectstack-ai/objectstack#3820 — "lint: no reference-integrity for AI metadata (agent skills, skill tools) — R7, blocked on D1/D2" — is open and covers exactly this. This tracker should not have been filed separately; upstream had it first. (This session's GitHub integration is allowlisted to objectstack-ai/hotcrm, so repo-scoped reads of the upstream issue are refused and only its title is visible from search — the body is not assessed here.)

3. Correction to this issue's premise, and to #557

The registry contradicts a claim made in #557 and repeated in this issue's body: search_knowledge is a real platform-provided tool. It is in PLATFORM_PROVIDED_TOOL_NAMES, and it was documented in 16.1.0 all along — @objectstack/spec/src/ai/knowledge-source.zod.ts:114 reads "Whether search_knowledge may expose this source to AI agents." The 17.0 rule reports zero findings against the pre-#557 state.

So #557 removed a reference to a tool that does exist. The removal's outcome still holds, for a different reason: search_knowledge needs a knowledge source, AIKnowledgeSchema mounts only on AgentSchema.knowledge, and agents were deleted in #512 — a skills-only app has nowhere to declare one, so the tool would resolve and search nothing. But the stated reason ("nothing defines it") is wrong in three places: the customer-360.skill.ts docstring, .changeset/skills-undefined-tool-references.md, and this issue's body.

4. What does not close with this issue

test/skills-integrity.test.ts's hand-maintained PLATFORM_TOOLS allowlist is wrong today, independent of any upgrade. It omits search_knowledge, query_data, validate_expression and the entire build-surface tool family, so it will fail a legitimate reference. Tracked separately — adopting 17.0 replaces the allowlist with an import of PLATFORM_PROVIDED_TOOL_NAMES, which is the durable fix; until then the list needs correcting by hand.

One divergence worth carrying to #3820 if anyone picks it up: the upstream rule's collectToolUniverse counts stack.tools metadata records as resolving, but ToolSchema in 17.0 still describes itself as a [READ-ONLY PROJECTION — not an execution entry point] with no executor. A metadata-authored tool would satisfy the linter and still never run.

Closing as a duplicate of objectstack-ai/objectstack#3820. Adopting the rule here is a dependency bump to 17.0, deferred for now while it is an rc.

---

Generated by Claude Code