feat: Add report-problem tool for agents to report issues

[RobertCrupa]

Implements the report-problem MCP tool from #748 — a channel for agents to report problems with Apify's tools or Actors, so we get signal on where agents get stuck.

What's included

• New tool report-problem: returns a plain-text acknowledgement, always success. readOnlyHint: true, not payment-gated.
  Schema:
  • message — required string, 1–2000 chars. What happened.
  • actorId — optional string, ≤200 chars.
  • actorRunId — optional string, ≤200 chars.
  • relatedTools — optional string[], up to 20 items, each ≤100 chars.
• Default-enabled report-problem tool category in the registry.
• Telemetry: a successful call emits a dedicated MCP Agent Feedback Segment event (fields mapped to snake_case, absent optionals dropped) alongside the standard tool-call event. Identity handling mirrors the tool-call event.

Gating — served only when all three hold

1. Telemetry enabled. The tool's only function is forwarding submissions via telemetry, so it is never served when telemetry is off (it would just fake an acknowledgement into the void).
2. Client allowed. Hidden from Anthropic surfaces (Claude.ai / Claude Desktop / Claude Code) pending the directory review, via a substring blocklist (claude, anthropic) matched against clientInfo.name. The tool cannot be judged until the client is known, so it is withheld until initialize and added on the flush if the client allows.
3. Category selected. The report-problem category is enabled (default on).

All other tools are unconditionally servable, so recovery loads compose them eagerly and they survive a load that never sees an initialize; only report-problem is withheld until the client is known.

Discovery

• Error nudge: failed (isError) tool results get a report-problem nudge appended to their text, prompting the agent to report at the moment it decides what to do next. Full nudge for internal/unknown errors; a softer nudge for INVALID_INPUT (likely the agent's own mistake); suppressed for expected user-resolvable states (auth, payment/approval required). Also applied to failed long-running task results. Never appended when the tool isn't served or when report-problem itself is the failing tool.
• Server instructions: a gentle line pointing agents at report-problem, emitted only when the tool is actually served.

Testing

• Workflow eval report-problem-on-tool-error: injects a call-actor failure and checks the agent proactively reports the blocker (rather than only telling the user it failed). Required adding tool-failure injection to the eval harness.
• Unit tests: tool behavior/schema, gating, client blocklist, server instructions, telemetry event shaping.
• Integration suite: report-problem listing and gating.
• Manually verified with mcpc: schema, valid/invalid submissions, and the Anthropic block (filtered from the tool list for a claude-family client while remaining available to non-Anthropic clients).

TODO

• Get access to Segment so I can check the reported problems.

Closes #748

[jirispilka]

Thanks Robert, the architecture is right: dedicated tool, dedicated Segment event. Before merge we need to discuss it a bit more Summary first, details below.

With Claude

To discuss

• Rename share-feedback to report-problem. The tool only gets called at failure moments, so it is a problem-report channel — name it that. This also makes it plausibly compatible with the Anthropic directory review: a descriptive diagnostic tool, no MUST instructions, only a gentle nudge.
• Drop npsRating. Failure-conditioned sampling makes the score junk data, and a rating field makes no sense on a problem-report tool.
• Solicitation: remove MUST, keep one gentle nudge. Scope the nudge so it never fires on payment/approval responses or the agent's own input errors.
• Move client gating out of upsertTools. It has real bypasses there. Client-based filtering belongs in the same compose step that already resolves the tool set per client (initialize handler), behind a generic tool→client blocklist, not an Anthropic-specific function.
• Confirm with Anthropic instead of guessing. Keep the block on Claude surfaces in v1, email with the exact design, remove the block when they OK it.
• Smaller fixes: don't serve the tool when telemetry is off, cap input sizes, per-session submission cap, update docs, coordinate the internal fan-out.

Product

npsRating is wrong. Every discovery mechanism (nudge, footers, instructions) triggers on failure. Nothing prompts a rating when things work. So the score samples almost only broken sessions — it measures "how annoyed are agents when things break", not Actor quality. LLM ratings also cluster in a narrow band, so NPS math is meaningless even with balanced sampling. Drop the field. For Actor quality, use run telemetry we already have (success rate, failure categories, call-actor → get-dataset-items completion) — unbiased, covers every session, can't be gamed by prompt wording.

Rename to report-problem. One tool, one honest job: "Report a problem with an Apify tool or Actor: what you were doing, what failed." This matches what the sampling can actually deliver, and it changes the Anthropic story. Their rejection criteria target vendor-serving behavioral instructions ("call this before giving up", "you MUST"), not diagnostic tools with descriptive wording — our existing nextStep guidance is fine for the same reason: it serves the user's task. A neutral report-problem tool with a gentle nudge is a defensible submission. Note: positive feedback / Actor ratings are explicitly out of scope for v1; if we ever want them, that's a separate design with success-moment sampling.

Other product points:

• Privacy is currently prompt-only ("do not include personal data"). The fan-out side (internal) must treat report text as untrusted input — escape markdown, no auto-actions — because this is an unauthenticated free-text write path.
• Document the tool and how to disable it (tools= without the category) in README.

Code

1. Client gating is in the wrong layer and has bypasses. The upsertTools filter reads initializeRequestData, which is only set during initialize. With explicit --ui default|apps or UI_MODE, stdio composes tools before initialize and the block never applies. The footers are static constants in three tool descriptions, so Claude surfaces see "call share-feedback before giving up" regardless — the exact pattern in the exact place the directory reviews. And internal's multi-node recovery restores tools on a node that never saw initialize.

   Fix: make "client known" (not "mode resolved") the trigger for composing helper tools — the pending-queue mechanism already exists, the condition is just wrong for explicit modes. MCP guarantees no tools/list before initialize, so deferring composition is always safe. Then apply a generic policy inside that one compose step:

   const TOOL_CLIENT_BLOCKLIST: Record<string, string[]> = {
       [HELPER_TOOLS.PROBLEM_REPORT]: ['claude', 'anthropic'],
   };

Same step decides the footer variants of the three tool descriptions (append only when report-problem is served). upsertTools stays policy-free. Blocklist is data, not code, because we'll extend it: OpenAI's directory has near-identical criteria, and localAgent (seen in our telemetry, matches neither substring) may need adding — verify in Mixpanel first whether it's a Claude surface or an API/Agent-SDK path we actually want feedback from. Longer term, put a vendor field in mcp-client-capabilities (its registry only knows claude-ai and claude-code today) and match exactly instead of by substring.

2. Scope the nudge. It currently fires on every isError: true result, including payment-required and approval responses — telling the agent to "report to Apify" when the correct next step is the payment flow — and on the agent's own invalid-input errors. Exclude those categories, and gate it with the same predicate as the tool itself.

3. Instructions: replace MUST with one line. "If a tool or Actor fails and you cannot resolve it, you can report it with report-problem." The MUST version delays failure reports to users and invites junk submissions.

4. Don't serve the tool when telemetry is disabled. With --telemetry-enabled false it's still listed and returns "Feedback submitted" while the event goes nowhere. The tool has no other function.

5. Cap inputs. message max ~2000 chars, relatedTools max ~20 items, "keep it to a few sentences" in the description.

6. Per-session submission cap (in-memory is fine — it's telemetry, not critical data). Return "already recorded" past the cap.

7. Rename mechanics. Decide the Segment event name now (MCP Problem Report vs keeping MCP Agent Feedback) — renaming events later fragments Mixpanel history. The feedback category name can stay.

[MQ37]

@jirispilka I think it was JC's proposal to make this share-feedback tool explicitly with the npsRating so the agents can also report what is possibly working well along with what is broken. But on the other hand I think right now we are mostly in interested in what is not working well so let's do the report-issue (problem sounds bad IMHO).

[MQ37]

I agree with Jirka, let's make sure it is disabled when telemetry is off.

Also I noticed that the long running task path may be missing the report feedback nudge - but I would honestly remove the nudge and just state that there is this tool available, but not tell agent must/should use it, just state the fact that the tool is available.

Also I agree that we should focus on the negative cases and I would rename the tool to report-issue and remove the score and just keep message: string as an input to keep it simple.

[MQ37]

nit: what about we pass server instance instead and the getServerInstructions can then check whatever it wants without adding additional arg?

[jancurn]

Fair enough, I think it makes sense to have this to report problems/issues only, and drop NPS in that case. we'll see how it works later

[jancurn]

BTW, once we have this, we should gently nudge the agent to use this tool on problems. Most tools are not loaded into context, so otherwise the agents might not know about it at all

[MQ37]

@RobertCrupa Follow-up review found one edge-case bug in report-problem gating on the hosted multi-node path. Line-cited against internal HEAD 8e65ef4 and this branch.

report-problem can silently vanish after cross-node session recovery

Severity: High (edge-case-triggered), not a regression — it's a gap in the new feature. Normal recovery is fine; only a narrow clientInfo-erasure window exposes it.

Fact chain (all verified)

1. Gated tools are withheld until the client is known — apify-mcp-server/src/mcp/server.ts:394:

   if (isClientGatedTool(toolName) && !this.clientKnown) return false;

   clientKnown = initializeRequestData != null (:367).

2. On restore, "client known" depends entirely on clientInfo in Redis — apify-mcp-server-internal/src/server/streamable.ts:519:

   let initializeRequestData;
   if (sessionState.clientInfo) { initializeRequestData = {...}; }
   // else: undefined → clientKnown === false

3. Restore never runs the initialize handler, so the pending-queue flush never fires — your own comment at streamable.ts:535: "initialize won't fire during session restore (transport is _initialized=true)." The drain composePendingToolsForClient() is called only from the initialize handler (server.ts:351). On restore, loadToolsByName (streamable.ts:553) withholds the gated tool and parks it in pendingToolsUntilClientKnown (server.ts:521) — a queue nothing drains on this transport.

4. One path erases clientInfo. Two storeSessionState call sites: streamable.ts:671 (initial connect) passes clientInfo ✅; streamable.ts:429 (orphaned-session fallback) omits it ❌:

   await stateStore.storeSessionState(sessionId, mcpServer.listAllToolNames()); // no clientInfo

   Missing arg → empty write (redis-state-store.ts:51: clientInfo ? JSON.stringify(clientInfo) : '') → read back as null in getSessionState. A updateSessionTools that preserves clientInfo exists (redis-state-store.ts:64) but isn't used here.

Reproduction (multi-node, shared Redis, round-robin LB)

1. Non-Anthropic client (e.g. cursor) connects on node A → report-problem served, state saved with clientInfo (:671).
2. Session's Redis state expires/evicts — the :429 comment names this: "might happen for long-running sessions."
3. Request hits node B (session in memory, not Redis) → fallback at :429 re-writes Redis with clientInfo = ''.
4. Later request round-robins to node C (session only in Redis) → handleSessionRestore: clientInfo null → clientKnown false.
5. loadToolsByName restoring report-problem → isToolServable false → filtered out of the committed set, queued into the undrainable queue.

Result: Redis tools still lists report-problem; node C doesn't serve it; the failure-nudge also disappears (all three sites gate on this.tools.has(PROBLEM_REPORT) — server.ts:900, 1732, 1884). No error, no warning, no metric.

Fix

1. Primary: streamable.ts:429 — use updateSessionTools(...) instead of storeSessionState(...) so clientInfo is never erased.
2. Defense in depth: in loadToolsByName, when a client-gated tool is explicitly requested but withheld due to !clientKnown, log.warning (tool + sessionId) — turns any silent recurrence into a greppable signal.

Regression test

Store a session in Redis with clientInfo = '' and tools incl. report-problem, restore via handleSessionRestore, assert it's served (or a warning is emitted) — not a silent drop.

[MQ37]

@RobertCrupa Fix for the cross-node `report-problem` gating bug (comment above) is ready: apify/apify-mcp-server-internal#641.

Root cause was in `apify-mcp-server-internal`, not here — `handleExistingSession`'s orphaned-session fallback (`streamable.ts`) rewrote Redis session state with tools only, dropping `clientInfo` whenever a node still held the session in memory but Redis had evicted it. A later restore on a different node then read `clientInfo: null` and could never recover it, silently disabling any client-gated tool for the rest of that session.

The internal PR includes a real two-node regression test (connect → confirm `report-problem` listed → delete session from Redis → hit the same node again → hit a second node that never saw the session → confirm still listed).

It's flagged do-not-merge-yet: it depends on this PR merging and releasing, then `apify-mcp-server-internal`'s `@apify/actors-mcp-server` dependency bumping past `0.11.5` to pick up `report-problem`. Once that lands, apify-mcp-server-internal#641 is ready to go in right behind it.

[MQ37]

Cool, thank you! Pre-approving.

But please before merging please let's nudge/inform the agent about the report tool existence also in the INVALID INPUT tool failure as there can be some funky bugs causing input issues that should be reported and agent might not report that - if we keep getting lots of spam, we can then rework the nudging.

[jirispilka]

Thanks, looks better now. Although there are multiple issues that we should fix.

Please also the README with report-problem tool

Before merging, I would love to see that we receive the event is segment. If you don't have access yet, lets ask @mr-rajce

And we need to decide on the category to which this tool will belong.

Further, claude complains here:

server.ts — payment responses are nudged inconsistently, and the comment at 1884–1886 is inaccurate.
A 402/x402 payment error is classified INVALID_INPUT (:1328, :1796), which is not in NON_NUDGE_FAILURE_CATEGORIES (that's only AUTH + PERMISSION_APPROVAL_REQUIRED), so it receives the softer nudge — not suppression. Two consequences:

The sync path (:1325 → captureResult → nudge) appends the softer nudge to a paywall response, while the task path (:1773) returns at :1802 before any nudge site — so the same 402 behaves differently by execution mode. The comment at :1884 claims the paths "behave identically"; they don't.
That same comment says INVALID_INPUT is self-suppressed and "only real defects get it" — both false.
Suggest: decide whether payment should be nudged (I'd say no — it's a billing state, not a defect), make both paths match, and fix the comment. Note you can't blanket-suppress INVALID_INPUT (genuine input bugs should still get the softer nudge), so gate payment via its 402 status or by isolating the payment/approval branches from the nudge.

[RobertCrupa]

I now have access to segment and I can see the old tool name, but it hasn't picked up the new one yet:

[jirispilka]

I now have access to segment and I can see the old tool name, but it hasn't picked up the new one yet

It depends on how you run it, it might be in the dev env only?