Skip to content

docs: document 4th failure class — @utils.setVariables reserved-identifier blocklist#2

Open
acbb01 wants to merge 1 commit intoSalesforceAIResearch:mainfrom
acbb01:docs/fourth-failure-class-dispatch-validation
Open

docs: document 4th failure class — @utils.setVariables reserved-identifier blocklist#2
acbb01 wants to merge 1 commit intoSalesforceAIResearch:mainfrom
acbb01:docs/fourth-failure-class-dispatch-validation

Conversation

@acbb01
Copy link
Copy Markdown
Contributor

@acbb01 acbb01 commented Apr 20, 2026

Summary

Extends the "Pattern: Unexpected Error Responses" section in skills/developing-agentforce/references/agent-validation-and-debugging.md with a fourth documented cause for the platform's "I apologize, but I encountered an unexpected error" wrapper. The existing docs list three causes (two consecutive UNGROUNDED grounding failures, an action that returned an error, and a misconfigured topic transition); this PR adds:

4. Pre-Execution Tool Dispatch Validation Failure — the Atlas Reasoning Engine's dispatch validator silently rejects an entire @utils.setVariables invocation when one of the variable identifiers declared in the action's with <name> = ... bindings matches a reserved-identifier blocklist. The rejection is a property of the action's declared bindings, not of the LLM's runtime arguments. Renaming the offending variable to a non-blocklisted identifier is sufficient to make the same invocation succeed with no other changes.

What this PR used to claim (and why it was wrong)

Note for reviewers who saw the earlier revision: this PR was originally submitted with a narrower hypothesis — "the dispatch validator rejects a parallel-invoked @utils.setVariables call when any invoked tool declares ≥4 slots." That characterization was falsified during follow-up minimum-config bisection on the same org. A single 3-slot capture invoked alone, with only one LLM tool call and two arguments supplied, still drops whenever one of its declared variable names matches the blocklist. Conversely, a 4-slot capture whose declared identifiers are all non-blocklisted does NOT drop. Slot count and parallel invocation turned out not to be load-bearing; variable identifier matching is. The updated subsection calls out the earlier framing explicitly.

Matching rules observed on the blocklist

Two rules coexist, applied per-entry:

  1. Substring, case-insensitive. Triggers anywhere in the identifier, any prefix, any suffix, any case. Observed entries: sector, ciudadania.
  2. Capital-token, position-sensitive. Triggers when the identifier contains the term as a distinct token — bare or as a capital-leading token in a camelCase compound. Observed entries: Name, Industry. business_name (lowercase name inside snake_case compound) does NOT trigger.

Observed non-exhaustive blocklist: sector, ciudadania, Name, Industry. The list appears curated around Salesforce standard field names and Atlas NER entity labels (some of which are Spanish-localized).

Evidence

9-variant minimum-config bisection on a standard Agentforce org running Agent Script bundles. Bundle shape held constant: one start-agent router, one topic with one three-slot capture, identical instructions across all variants, Spanish locale, AgentforceServiceAgent type. Only the name of the third declared variable changed between variants. Reproduced in both sf agent preview --authoring-bundle (local compile) and sf agent preview --api-name (deployed) modes.

Third-slot variable name Outcome
slot_c, my_third_variable, social_security_number, cedula, company_nit, business_name Works
business_sector, foo_sector, businessSector, Sector, ciudadania, cedula_ciudadania, Industry, Name, slot_Name Drops silently

social_security_number passing rules out a generic PII/DLP filter — the most obviously-PII-named variable in the world writes fine, but ciudadania (citizenship, a related concept) drops.

The one-line repro: take any failing bundle, rename the offending variable to a non-blocklisted identifier, and the same utterance that produced the apology will now produce clean VariableUpdateStep rows with GROUNDED verdict — no other changes.

Why this belongs in the adlc docs

  • The debug playbook already documents the "I apologize" wrapper with three causes; this is a fourth empirically distinct cause that a developer following the current flow would not find from the trace alone.
  • The workaround is a simple variable rename with concrete guidance on the observed patterns — no architectural change required.
  • The detection signature uses the same LLMStep / VariableUpdateStep / ReasoningStep vocabulary the rest of the file uses, so it plugs into the existing trace-analysis workflow.

Test plan

  • Open the modified file on GitHub and confirm the new subsection renders correctly (heading depth, table, fenced code block)
  • Confirm the new #### heading sits one level below the parent ### "Unexpected Error" pattern and above the next ### pattern
  • Verify the rename-pattern guidance is actionable as-written (a reader can apply the business_sector -> business_category pattern to their own bundle without further context)

@salesforce-cla
Copy link
Copy Markdown

Thanks for the contribution! Before we can merge this, we need @acbb01 to sign the Salesforce Inc. Contributor License Agreement.

…-identifier blocklist in @utils.setVariables dispatch

Add a Pre-Execution Tool Dispatch Validation Failure subsection to the
known failure modes for the platform's generic error wrapper. The Atlas
Reasoning Engine's dispatch validator silently rejects an
@utils.setVariables invocation when one of the variables declared in
the action's `with <name> = ...` bindings matches a reserved-identifier
blocklist. The rejection is a property of the declared bindings, not
of the LLM's runtime arguments — renaming the offending variable is
sufficient to make the same invocation succeed with no other changes.

Two matching rules were observed on the blocklist, applied per-entry:
substring case-insensitive (e.g., `sector`, `ciudadania`) and
capital-token position-sensitive (e.g., `Name`, `Industry`). A single
3-slot capture invoked alone reproduces the drop whenever one of its
declared identifiers matches. Observed non-exhaustive blocklist:
sector, ciudadania, Name, Industry.

Reproduced via a 9-variant minimum-config bisection on a standard
Agentforce org using sf agent preview in both --authoring-bundle and
--api-name modes. Same bundle, single-variable rename, behavior
inverts from apology wrapper to clean VariableUpdateStep rows with
GROUNDED verdict.

Supersedes an earlier draft that hypothesized "parallel invocation of
two or more tools where one declares four or more slots" was the
trigger. That hypothesis was falsified during bisection — slot count
and parallel invocation are not load-bearing; variable identifier
matching is. The subsection calls out the earlier framing explicitly
so readers with prior context know what changed.

Workaround: scan every @utils.setVariables action's declared bindings
for identifiers that match the blocklist patterns and rename them to
domain-neutral alternatives (e.g., business_sector -> business_category).
@acbb01 acbb01 force-pushed the docs/fourth-failure-class-dispatch-validation branch from 2024e50 to 60229c8 Compare April 20, 2026 14:10
@acbb01 acbb01 changed the title docs: document 4th failure class — @utils.setVariables ≥4-slot parallel-dispatch silent drop docs: document 4th failure class — @utils.setVariables reserved-identifier blocklist Apr 20, 2026
@acbb01
Copy link
Copy Markdown
Contributor Author

acbb01 commented Apr 20, 2026

Heads up to anyone who saw the earlier revision of this PR: I force-pushed a corrected version. Same failure class (a 4th cause for the generic "I apologize" wrapper), but the root cause narrows to a different mechanism than I originally claimed.

What changed: The original draft said the trigger was "parallel invocation of two or more tools in one LLM turn, where at least one declares four or more slots." That was wrong. Follow-up minimum-config bisection showed slot count and parallel invocation are not load-bearing. A single 3-slot capture invoked alone drops whenever one of its declared variable names matches a reserved-identifier blocklist. Same bundle, rename one variable, behavior flips.

Observed blocklist so far (non-exhaustive): sector, ciudadania, Name, Industry. Two matching rules coexist on the list: substring-case-insensitive for some entries, capital-token-sensitive for others. social_security_number writes fine, which rules out a generic PII filter — the list looks curated around Salesforce standard field names and Atlas NER entity labels.

The updated subsection documents the new evidence matrix, the two matching rules, the detection signature, and the rename-based workaround with concrete patterns (e.g., business_sectorbusiness_category). Happy to provide the raw pass/fail session traces if useful.

Copy link
Copy Markdown
Contributor

@almandsky almandsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough write-up and bisection work. We attempted to reproduce the reserved-identifier blocklist behavior but were unable to on our end.

We created a minimal two-bundle test — identical except one uses business_sector and the other business_category — and ran both through sf agent preview in simulated and live-action modes. In all cases, business_sector produced clean VariableUpdateStep rows for all 3 variables with a GROUNDED verdict. No silent drop, no apology wrapper.

A couple of things we didn't match from your test conditions:

  • We tested with en_US locale (your bisection used Spanish)
  • We may be on a different platform version than when you originally tested

Could you:

  1. Re-reproduce today on the same org you originally tested on? If the behavior no longer triggers, it may have been patched server-side.
  2. Clarify whether locale is a factor — does this only trigger with Spanish locale?

The documentation quality is excellent — if the behavior is still reproducible under specific conditions, we'd want the docs to note those conditions clearly. If it turns out this was a platform-side fix, we should reframe the section accordingly.

@acbb01
Copy link
Copy Markdown
Contributor Author

acbb01 commented Apr 23, 2026

Update — unable to reproduce today.

I re-ran the bisection on the same org (PuntosDE) with identical conditions (business_sector variable, Spanish locale, same single-setter topology) via sf agent preview --use-live-actions. All 3 VariableUpdateStep rows fired cleanly; reasoning was GROUNDED; no apology wrapper. Trace evidence matches what you observed.

Trace excerpt:

[VariableUpdateStep] company_nit     ""  → "900.200.200"
[VariableUpdateStep] business_name   ""  → "Panaderia Luna SAS"
[VariableUpdateStep] business_sector ""  → "comercio"
[ReasoningStep] category = GROUNDED

Two hypotheses:

  • Platform patched between my original report (2026-04-20) and today (2026-04-23).
  • My original finding was UNDERFILL in disguise. The business_sectorbusiness_category rename may have nudged LLM compliance (causing it to fill all 3 declared slots) rather than escaping a reserved-identifier blocklist. Session 36 follow-up bisection separately confirmed a distinct UNDERFILL failure class: silent drop when the LLM underfills any declared slot of a multi-slot @utils.setVariables, which reliably reproduces via a clean 1/3-filled vs 3/3-filled test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants