feat: Add `report/feedback` tool for agent feedback

[jirispilka]

Let's discuss specific purpose, name, and args first before implementing

Problem

When an agent gets stuck against our MCP server, we only see the failing tool call — never the higher-level "I was trying to do X but your tool's response didn't tell me how to proceed." We have no direct channel for agent feedback.

Related: #673 (Improve tool call accuracy).

Proposed solution

Ship a standalone helper tool (report-issue) that the agent calls when blocked.

• Add HelperTools.REPORT_ISSUE = 'report-issue' to src/const.ts:25.
• New tool file: src/tools/common/report_issue.ts (following the pattern of src/tools/common/search_apify_docs.ts — internal tool, Zod schema, AJV validate, buildMCPResponse return).
• Zod schema:
  • goal: z.string() — what the agent was trying to accomplish
  • attempted: z.string() — what it tried (tools, inputs)
  • blocker: z.string() — where it got stuck / what was confusing
  • tool_name: z.string().optional() — tool tied to the issue
• Register it in the appropriate category in src/tools/categories.ts (likely enabled by default like docs tools).
• Pipe submissions through telemetry: emit a new Segment event from src/telemetry.ts (e.g. 'MCP Agent Feedback') carrying the four fields plus mcp_session_id, mcp_client_name, etc.
• The Slack/GitHub fan-out lives in apify-mcp-server-internal (consumes the Segment event) — keep delivery out of the public repo.
• Mention the tool in the description footers of tools where agents commonly get stuck (call-actor, search-actors, fetch-actor-details) so the agent knows the channel exists. One sentence: "If you're blocked or our response is unclear, call report-issue so we can fix it."

Plan

• Add HelperTools.REPORT_ISSUE enum + tool implementation
• Add Segment event in src/telemetry.ts; extend types in src/types.ts
• Register in tool categories (default-enabled)
• Add tool-mention sentence to call-actor, search-actors, fetch-actor-details descriptions
• Unit test + tests/integration/suite.ts case
• Coordinate with apify-mcp-server-internal for Slack + GitHub fan-out

Alternatives considered

• Pure log-based feedback (softFail) — not visible to product, only to oncall.
• Bundling feedback into existing tools as an optional field — agents won't reach for an optional field; a dedicated tool framed as "get unstuck" has stronger pull.

[jancurn]

For context, see

https://apify.slack.com/archives/C04URRT3934/p1777047845832489
https://x.com/teddy_riker/status/2047312986696454584

[MQ37]

I like the approach in general but I would change the tool_name: z.string().optional() schema to use array of strings so it is parsable more easily - I imagine the Mixpanel where we will store that cannot easily parse comma separated list and multiple tools may be involved in a single report. Otherwise I agree and LGTM

[jancurn]

From duplicate #953:

To improve agentic experience, we need to get feedback from agents where they struggle, or which Actors are good.

Let's try adding a share_feedback MCP tool with following parameters:

• actorId?: string - Which Actor is this feedback related to?
• actorRunId?: string - Which Actor run is this feedback related to?
• npsRating?: numer - How likely you are to recommend this Actor to other agents? Enter number from 0 (very unlikely) to 10 (very likely).
• message: string - Short feedback explaining what went wrong and what went well.

[MQ37]

@jancurn great idea with the npsRating I like that - we should also add the relatedTools string array that I mentioned in the previous message as the feedback should be shared in general for any MCP tool and not just Actors. Also we must be very careful with the prompting and description and tell agents to never include anything personal or private from the conversation.

[RobertCrupa]

Before implementing I think it's important for everyone to align on a spec.

Firstly I have I have a few questions:

1. Do we want a dedicated event for this tool call? or we piggyback on the ToolCallTelemetryProperties and let the internal repo filter? I think dedicated
2. If we are setting it on by default but disabled on anthropic clients, I will match strings on client name, is this good enough?
3. Are we happy to keep the read only hint so it doesn't prompt approval? it isnt read only but it doesn't change any state, so it is readonly from the users point of view

Here is my proposal:

---

share-feedback — design spec

Tool

• Name / enum: HelperTools.FEEDBACK_SHARE = 'share-feedback'
• File: src/tools/feedback/share_feedback.ts (new feedback/ subdir — there's no common/ dir anymore; subdirs mirror categories). Follows the search_apify_docs.ts pattern: internal tool, Zod schema, AJV validate, buildMCPResponse.
• Schema:

Field	Type	Required
message	z.string().min(1)	yes
actorId	z.string()	no
actorRunId	z.string()	no
npsRating	z.number().int().min(0).max(10)	no
relatedTools	z.string().array()	no

• Strict on message/npsRating; permissive on the rest (no existence checks, no enum validation, no length cap). stripQuoteWrappers on actorId.
• Description: recruits for both moments — "report a problem you hit or rate an Actor" — plus a prominent privacy line: "Do NOT include personal data, credentials, secrets, or verbatim private conversation content. Describe the problem in your own words."
• Metadata: paymentRequired: false; no outputSchema; annotations { title: 'Share feedback', readOnlyHint: true, destructiveHint: false, idempotentHint: false, openWorldHint: false }. No widget; shared across default + apps modes.
• Return: always-acknowledge plain text — "Feedback submitted. Thank you — this helps improve Apify's tools." Same on success and when telemetry is disabled; never looks broken.

Registration & gating

• New category feedback: [shareFeedback] in registry.ts; add 'feedback' to toolCategoriesEnabledByDefault.
• Hard block for Anthropic clients: isAnthropicClient(initializeRequestData) in src/utils/mcp_clients.ts — case-insensitive substring match on "claude"/"anthropic" in clientInfo.name. Applied as a final filter that strips share-feedback even when explicitly selected.

Telemetry

• Dedicated Segment event 'MCP Agent Feedback' via new trackAgentFeedback(); new AgentFeedbackTelemetryProperties type in src/types.ts.
• Emitted server-side in server.ts, gated on tool.name === FEEDBACK_SHARE, reusing the already-assembled userId + client metadata.
• Payload: standard context (app, app_version, mcp_client_name, mcp_client_version, mcp_protocol_version, mcp_session_id, transport_type) + message (verbatim), actor_id, actor_run_id, nps_rating, related_tools.

Discoverability

• Append one uniform sentence to call-actor, search-actors, fetch-actor-details (via ${HelperTools.FEEDBACK_SHARE} interpolation): "If you got stuck, or this tool's response was confusing or wrong, use share-feedback to report it — you can also use it to rate an Actor."

[tangweigang-jpg]

One small safety detail I would add to the spec is feedback deduplication and client-visible sampling behavior.

If this tool is discoverable from normal agent turns, some clients will call it repeatedly after the same confusing response, especially if the tool description says "if you got stuck". I would include a stable local event key in telemetry, maybe derived from session id + tool name + actor/run id + coarse error class, and either coalesce duplicates server-side or return a short "already recorded for this run" acknowledgement.

That also makes the privacy line easier to enforce. The agent can report a compact failure category plus a short human-written message, while telemetry keeps enough context to join it back to server-side tool metadata without encouraging the model to paste private conversation text into message.

[MQ37]

Answers:

1. I agree and would use dedicated event
2. I would match for client id/name containing either "claude" or "antropic" and disabled that for these
3. Yes, I would keep it read only to not bother the user as it might confuse the user and hinder UX

In general I like that. I would include instructions regarding the message input field that it should be just a few sentences as we are not really interested in walls of text and huge sloppy reports.

@tangweigang-jpg has a great point with this possible spamming, but that would require storing all the reports in Redis for multi-node setup and somehow either rate limit that or match the texts. I would not complicate it and just prompt the agent to not spam the tool and if there is actual spam we can look into such solution.

I would be even more strict about the PII prompt that it should not include anything that could be personal or business intelligence like project names when for example someone might be using Claude code and working on something and it might leak project/company name or something like this.

[jirispilka]

Implementation review posted on #1050: #1050 (review)

@jancurn @MQ37 @RobertCrupa
Main proposals: rename share-feedback → report-problem, drop npsRating (failure-conditioned sampling makes it junk data), remove the MUST instruction and keep one gentle nudge, move client gating into the tool-compose step behind a generic blocklist, and confirm the design with Anthropic.

Please, let's continue the discussion on the PR.

---

Generated by Claude Code