docs: document 4th failure class — @utils.setVariables reserved-identifier blocklist#2
Conversation
|
Thanks for the contribution! Before we can merge this, we need @acbb01 to sign the Salesforce Inc. Contributor License Agreement. |
…-identifier blocklist in @utils.setVariables dispatch Add a Pre-Execution Tool Dispatch Validation Failure subsection to the known failure modes for the platform's generic error wrapper. The Atlas Reasoning Engine's dispatch validator silently rejects an @utils.setVariables invocation when one of the variables declared in the action's `with <name> = ...` bindings matches a reserved-identifier blocklist. The rejection is a property of the declared bindings, not of the LLM's runtime arguments — renaming the offending variable is sufficient to make the same invocation succeed with no other changes. Two matching rules were observed on the blocklist, applied per-entry: substring case-insensitive (e.g., `sector`, `ciudadania`) and capital-token position-sensitive (e.g., `Name`, `Industry`). A single 3-slot capture invoked alone reproduces the drop whenever one of its declared identifiers matches. Observed non-exhaustive blocklist: sector, ciudadania, Name, Industry. Reproduced via a 9-variant minimum-config bisection on a standard Agentforce org using sf agent preview in both --authoring-bundle and --api-name modes. Same bundle, single-variable rename, behavior inverts from apology wrapper to clean VariableUpdateStep rows with GROUNDED verdict. Supersedes an earlier draft that hypothesized "parallel invocation of two or more tools where one declares four or more slots" was the trigger. That hypothesis was falsified during bisection — slot count and parallel invocation are not load-bearing; variable identifier matching is. The subsection calls out the earlier framing explicitly so readers with prior context know what changed. Workaround: scan every @utils.setVariables action's declared bindings for identifiers that match the blocklist patterns and rename them to domain-neutral alternatives (e.g., business_sector -> business_category).
2024e50 to
60229c8
Compare
|
Heads up to anyone who saw the earlier revision of this PR: I force-pushed a corrected version. Same failure class (a 4th cause for the generic "I apologize" wrapper), but the root cause narrows to a different mechanism than I originally claimed. What changed: The original draft said the trigger was "parallel invocation of two or more tools in one LLM turn, where at least one declares four or more slots." That was wrong. Follow-up minimum-config bisection showed slot count and parallel invocation are not load-bearing. A single 3-slot capture invoked alone drops whenever one of its declared variable names matches a reserved-identifier blocklist. Same bundle, rename one variable, behavior flips. Observed blocklist so far (non-exhaustive): The updated subsection documents the new evidence matrix, the two matching rules, the detection signature, and the rename-based workaround with concrete patterns (e.g., |
almandsky
left a comment
There was a problem hiding this comment.
Thanks for the thorough write-up and bisection work. We attempted to reproduce the reserved-identifier blocklist behavior but were unable to on our end.
We created a minimal two-bundle test — identical except one uses business_sector and the other business_category — and ran both through sf agent preview in simulated and live-action modes. In all cases, business_sector produced clean VariableUpdateStep rows for all 3 variables with a GROUNDED verdict. No silent drop, no apology wrapper.
A couple of things we didn't match from your test conditions:
- We tested with en_US locale (your bisection used Spanish)
- We may be on a different platform version than when you originally tested
Could you:
- Re-reproduce today on the same org you originally tested on? If the behavior no longer triggers, it may have been patched server-side.
- Clarify whether locale is a factor — does this only trigger with Spanish locale?
The documentation quality is excellent — if the behavior is still reproducible under specific conditions, we'd want the docs to note those conditions clearly. If it turns out this was a platform-side fix, we should reframe the section accordingly.
|
Update — unable to reproduce today. I re-ran the bisection on the same org (PuntosDE) with identical conditions ( Trace excerpt: Two hypotheses:
|
Summary
Extends the "Pattern: Unexpected Error Responses" section in
skills/developing-agentforce/references/agent-validation-and-debugging.mdwith a fourth documented cause for the platform's "I apologize, but I encountered an unexpected error" wrapper. The existing docs list three causes (two consecutive UNGROUNDED grounding failures, an action that returned an error, and a misconfigured topic transition); this PR adds:4. Pre-Execution Tool Dispatch Validation Failure — the Atlas Reasoning Engine's dispatch validator silently rejects an entire
@utils.setVariablesinvocation when one of the variable identifiers declared in the action'swith <name> = ...bindings matches a reserved-identifier blocklist. The rejection is a property of the action's declared bindings, not of the LLM's runtime arguments. Renaming the offending variable to a non-blocklisted identifier is sufficient to make the same invocation succeed with no other changes.What this PR used to claim (and why it was wrong)
Matching rules observed on the blocklist
Two rules coexist, applied per-entry:
sector,ciudadania.Name,Industry.business_name(lowercasenameinside snake_case compound) does NOT trigger.Observed non-exhaustive blocklist:
sector,ciudadania,Name,Industry. The list appears curated around Salesforce standard field names and Atlas NER entity labels (some of which are Spanish-localized).Evidence
9-variant minimum-config bisection on a standard Agentforce org running Agent Script bundles. Bundle shape held constant: one start-agent router, one topic with one three-slot capture, identical instructions across all variants, Spanish locale, AgentforceServiceAgent type. Only the name of the third declared variable changed between variants. Reproduced in both
sf agent preview --authoring-bundle(local compile) andsf agent preview --api-name(deployed) modes.slot_c,my_third_variable,social_security_number,cedula,company_nit,business_namebusiness_sector,foo_sector,businessSector,Sector,ciudadania,cedula_ciudadania,Industry,Name,slot_Namesocial_security_numberpassing rules out a generic PII/DLP filter — the most obviously-PII-named variable in the world writes fine, butciudadania(citizenship, a related concept) drops.The one-line repro: take any failing bundle, rename the offending variable to a non-blocklisted identifier, and the same utterance that produced the apology will now produce clean
VariableUpdateSteprows withGROUNDEDverdict — no other changes.Why this belongs in the adlc docs
LLMStep/VariableUpdateStep/ReasoningStepvocabulary the rest of the file uses, so it plugs into the existing trace-analysis workflow.Test plan
####heading sits one level below the parent###"Unexpected Error" pattern and above the next###patternbusiness_sector->business_categorypattern to their own bundle without further context)