Skip to content

Enable async tool cancellation feature.#4241

Merged
filipi87 merged 8 commits into
mainfrom
filipi/async_tools_cancellable
Apr 10, 2026
Merged

Enable async tool cancellation feature.#4241
filipi87 merged 8 commits into
mainfrom
filipi/async_tools_cancellable

Conversation

@filipi87
Copy link
Copy Markdown
Contributor

@filipi87 filipi87 commented Apr 3, 2026

Enable async tool cancellation feature.

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 3, 2026

Codecov Report

❌ Patch coverage is 28.08989% with 64 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/pipecat/services/llm_service.py 20.27% 59 Missing ⚠️
src/pipecat/adapters/base_llm_adapter.py 54.54% 5 Missing ⚠️
Files with missing lines Coverage Δ
src/pipecat/utils/async_tool_cancellation.py 100.00% <100.00%> (ø)
src/pipecat/adapters/base_llm_adapter.py 86.11% <54.54%> (-5.83%) ⬇️
src/pipecat/services/llm_service.py 47.71% <20.27%> (-6.18%) ⬇️

... and 18 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@filipi87 filipi87 changed the title Trying to enable async tool cancellation feature. Enable async tool cancellation feature. Apr 3, 2026
Comment thread src/pipecat/processors/aggregators/llm_response_universal.py
@filipi87 filipi87 marked this pull request as ready for review April 3, 2026 20:23
@filipi87
Copy link
Copy Markdown
Contributor Author

filipi87 commented Apr 3, 2026

@aconchillo, @kompfner, @markbackman, should we invoke on_function_calls_started when a function call is used to cancel another function call? It feels like we shouldn’t.

Copy link
Copy Markdown

@JiwaniZakir JiwaniZakir left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In base_llm_adapter.py, the fallback branch in from_standard_tools (lines ~155–163) silently drops the user's legacy/provider-specific tools when _builtin_tools is non-empty—the original behavior was return tools which passed them through unchanged. Replacing them entirely with only built-in tools is a silent breaking change for anyone using a non-ToolsSchema format; a better approach would be to still pass the original tools through and emit the warning, rather than discarding them.

In _compose_system_instruction within llm_service.py, the old implementation unconditionally appended completion_instructions (i.e., even when _filter_incomplete_user_turns was False), whereas the new version gates this behind the flag. This is likely a bug fix, but it's worth verifying there aren't existing users or tests relying on the old unconditional behavior—a regression test covering the "turn completion off, async cancellation on" composition path would help here.

In llm_response_universal.py, tool_call_id is now duplicated—once inside the JSON content payload and once as a top-level message key. If this is intentional (e.g., for a provider that reads it from the content body), a brief comment explaining why would prevent future readers from removing it as redundant.

@filipi87 filipi87 force-pushed the filipi/async_tools_stream branch from 283aa29 to 7b45a56 Compare April 9, 2026 12:04
Base automatically changed from filipi/async_tools_stream to main April 9, 2026 13:26
@filipi87 filipi87 force-pushed the filipi/async_tools_cancellable branch from 58882cc to 772fb57 Compare April 9, 2026 13:29
Comment thread src/pipecat/services/llm_service.py Outdated
await super().start(frame)
if not self._run_in_parallel:
await self._create_sequential_runner_task()
if self._has_async_functions():
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably also add a property to the class so users can disable this behavior if they wish.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wondering...would it ever make sense to make async tool cancellation a per-tool configuration as opposed to an all-or-nothing configuration?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can start with an all-or-nothing approach and make it a per-tool cancellation if needed.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the thinking that we'll add this param in a follow-up PR?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was planning to add this parameter in this PR so we can keep it disabled by default, preserving the current behavior unless the user specifies otherwise.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

disabled by default

Oh, huh. I think I was imagining that it would be enabled by default (but honestly, hadn't thought about it too much). The nice thing is we haven't shipped async tool support yet, so there's no "shipped behavior" to consider and we can make this decision before we ship any of it.

What was your reasoning for having it disabled by default? Are LLMs already pretty good at simply ignoring tool results that come back after they're no longer relevant, i.e. after the conversation has already moved onto another topic and it's clear that the user is no longer interested in the result?

I guess a simple test would be "get me the weather in san francisco...oh wait, sorry, i meant san diego".

Copy link
Copy Markdown
Contributor

@kompfner kompfner Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

at simply ignoring tool results that come back after they're no longer relevant

Hmm, actually, thinking about this a bit more, our architecture doesn't really always let the LLM make that judgment call...when the result comes back it will trigger an LLM response.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea was to avoid injecting anything for the user by default, unless they explicitly ask for it.

However, LLMs are not yet good at ignoring function call results, at least in my tests. 🤕

Copy link
Copy Markdown
Contributor

@kompfner kompfner Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LLMs are not yet good at ignoring function call results, at least in my tests

We've actually set it up so they can't, I think—we ask them to run inference with those results as the latest message(s).

I wonder if the way to get them to reliably ignore irrelevant function call results is with some prompting akin to what @markbackman did with USER_TURN_COMPLETION_INSTRUCTIONS: asking the LLM to disregard "async_tool" results that are no longer relevant to the conversation, including in the case where the result is the only message after the last assistant message (in which case we could ask the LLM to output a symbol that will be ignored, essentially making it "skip" responding).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added a new property for now: enable_async_tool_cancellation

Comment thread src/pipecat/adapters/base_llm_adapter.py Outdated
Comment thread src/pipecat/adapters/base_llm_adapter.py Outdated
Comment thread src/pipecat/adapters/base_llm_adapter.py Outdated
Comment thread src/pipecat/services/llm_service.py Outdated
Comment thread src/pipecat/services/llm_service.py Outdated
Comment thread src/pipecat/services/llm_service.py
Comment thread src/pipecat/services/llm_service.py Outdated
Comment thread src/pipecat/utils/async_tool_cancellation.py
Comment thread src/pipecat/services/llm_service.py Outdated
if name != CANCEL_ASYNC_TOOL_NAME
)

def _setup_async_tool_cancellation(self):
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we ever want to "tear down" async tool cancellation, say when the LLM receives a context with an updated set of tools that no longer contain async tools?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the right way to do this would be to validate, when unregister_function is invoked, whether any async tools remain, and if not, then we should remove that.

What do you think ?

Copy link
Copy Markdown
Contributor

@kompfner kompfner Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, hmm...not sure. unregister_function is pretty optional, in terms of telling the LLM which set of tools is available to it...the user could choose to keep all tools "registered" with the Pipecat service but at runtime set LLMSetToolsFrames to change which tools are available for each inference.

Kind of makes me think we probably want to reconsider when/where we run the logic that determines if we have async tools and sets up the tool cancellation mechanism.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we probably want to reconsider when/where we run the logic that determines if we have async tools and sets up the tool cancellation mechanism

Exploring what that would look like, in case it would be too gnarly...

We would have the "hook in" point for setting up the async tool cancellation mechanism be right before running inference, once we've resolved the actual set of tools available to the LLM for that inference run (not the set of tools register_functioned with the Pipecat LLM service). So...completely internal to the adapter get_llm_invocation_params function.

Advantages of this approach:

  • It only adds the async tool cancellation mechanism if we really need it (in the case of more tools being registered than are being used)
  • Correctly omits the tool cancellation mechanism if the user doesn't unregister_function

Disadvantage of this approach:

  • There are some LLMs where tools can also be specified directly in the init, and the resolution of the "active set" of tools happens after get_llm_invocation_params.
    • Though...I'm not sure async tools even makes sense for realtime LLMs, since we don't really have a good way for all of them to report async result updates mid-conversation...maybe we should
  • It would require an update to all the get_llm_invocation_params implementations in the adapters
  • It might be beneficial to the LLM to have the async-tool-cancellation-related system instructions in place continually throughout the conversation...

Taking a step back: the reason to change your approach would be to make the cancellation mechanism's lifespan be "tighter" around when async tools are actually available to the LLM. But it doesn't hurt to have the mechanism in place for longer than that. So...maybe let's do nothing.

Comment thread src/pipecat/utils/async_tool_cancellation.py Outdated
Comment thread src/pipecat/utils/async_tool_cancellation.py Outdated
Comment thread src/pipecat/utils/async_tool_cancellation.py Outdated
@kompfner
Copy link
Copy Markdown
Contributor

kompfner commented Apr 9, 2026

I think we'll really want an example file to demonstrate where this behavior would be useful, with some helpful commenting explaining how the user might exercise it.

@kompfner
Copy link
Copy Markdown
Contributor

kompfner commented Apr 9, 2026

Thoughts after talking with @filipi87...

It seems like async tool cancellation is sort of being designed to be two things:

  1. A way to disregard stale results that come in after they're no longer needed in the convo
  2. As a task control function, stopping long-running "subagent-like" or "process-like" async tools that might have some external side effects and may be sending a stream of results into the conversation

Re 1: I'm not sure the cancellation mechanism as it exists in this PR will excel at this without some modification, as tool calls are "expensive"—requiring a back and forth—and the LLM will respond to cancellation tool call results coming in (the latter point which maybe we can address with additional prompt engineering in the cancellation tool call result message). It's possible an entirely different approach like this one or something we're not yet considering, would be better suited to the task.

Re 2: I feel a smidge uneasy using a hidden tool cancellation mechanism as a control interface for long-running subagent-like tasks - if the developer is adding something like "start_location_tracking", it would be more natural for them to provide a "stop_location_tracking" (and possibly even an "update_location_tracking", if they so desire). This then makes me wonder if these sorts of tasks are better suited for the WIP subagents and their corresponding control systems...which then makes me wonder whether we're making a mistake by bundling up long-running, streamed-response agent-like things into our conception of "tools" in the first place.

Maybe I'm overthinking things and spiraling out...but I think it's worth a bit of more discussion, as we really want to get this right since it's a pretty foundational pattern. Thanks for bearing with me!

cc @markbackman @aconchillo

@kompfner
Copy link
Copy Markdown
Contributor

kompfner commented Apr 9, 2026

Re 1: talked to @filipi87 and he'll try addressing the issue of

the LLM will respond to cancellation tool call results coming in

🤞 hopefully that'll be enough to make things work fairly well for both the disregard-result case as well as the cancel-long-running-task case (we talked about the validation scenarios that make sense for this).

Re 2: after talking to @aconchillo and @filipi87, my existential unease about longer-running streamed-result tools have been assuaged. I'm back on board.

Maybe I'm overthinking things and spiraling out...

Yes, probably! For those who know me: very much classic me.

but I think it's worth a bit of more discussion

I think it was worth it 👍.

@aconchillo
Copy link
Copy Markdown
Contributor

aconchillo commented Apr 10, 2026

This looks great to me @filipi87 ! My only concern would be the cancellation prompt. Worst case, if the prompt doesn't work for some use case/LLM, users are able to modify the module variables.

@filipi87
Copy link
Copy Markdown
Contributor Author

filipi87 commented Apr 10, 2026

Worst case, if the prompt doesn't work for some use case/LLM, users are able to modify the module variables.

Or they can simply leave the mechanism disabled and handle it on their own.

@filipi87
Copy link
Copy Markdown
Contributor Author

@kompfner , it looks like we were already respecting when the run_llm is False:

@filipi87
Copy link
Copy Markdown
Contributor Author

@kompfner, it looks like we need to keep run_llm set to True, otherwise, the LLM doesn’t reply. With it set to True, it seems to be working well.

For example:

[
   {
      "role":"assistant",
      "content":"How can I help you today?"
   },
   {
      "role":"user",
      "content":"Can you tell me the current weather in Florianópolis?"
   },
   {
      "role":"assistant",
      "tool_calls":[
         {
            "id":"call_RsG3xBDlDIgxhNz57aNxvAbY",
            "function":{
               "name":"get_current_weather",
               "arguments":"{\"location\": \"Florianópolis, SC\", \"format\": \"celsius\"}"
            },
            "type":"function"
         }
      ]
   },
   {
      "role":"tool",
      "content":"{\"type\": \"async_tool\", \"status\": \"running\", \"tool_call_id\": \"call_RsG3xBDlDIgxhNz57aNxvAbY\", \"description\": \"An asynchronous task associated with this tool_call_id has started running. Expect results to arrive later as developer messages that look roughly like this one (with \\'type=async_tool\\' and a matching tool_call_id) but with a \\'result\\' field. Note that there *may* be more than one result (i.e., a stream of results), but there doesn\\'t have to be (there may be only one). The last result will come in a message with \\'status=finished\\'.\"}",
      "tool_call_id":"call_RsG3xBDlDIgxhNz57aNxvAbY"
   },
   {
      "role":"assistant",
      "content":"Let me check on that."
   },
   {
      "role":"user",
      "content":"Actually, would you mind? Tell me the weather in Curitiba instead."
   },
   {
      "role":"assistant",
      "tool_calls":[
         {
            "id":"call_zduKgNDnbjxqbk9pBWeVPJom",
            "function":{
               "name":"cancel_async_tool_call",
               "arguments":"{\"tool_call_id\": \"call_RsG3xBDlDIgxhNz57aNxvAbY\"}"
            },
            "type":"function"
         }
      ]
   },
   {
      "role":"tool",
      "content":"{\"cancelled\": \"call_RsG3xBDlDIgxhNz57aNxvAbY\"}",
      "tool_call_id":"call_zduKgNDnbjxqbk9pBWeVPJom"
   },
   {
      "role":"assistant",
      "tool_calls":[
         {
            "id":"call_x5TCeVP8aUgKADUvabYvQXFq",
            "function":{
               "name":"get_current_weather",
               "arguments":"{\"location\": \"Curitiba, PR\", \"format\": \"celsius\"}"
            },
            "type":"function"
         }
      ]
   },
   {
      "role":"tool",
      "content":"{\"type\": \"async_tool\", \"status\": \"running\", \"tool_call_id\": \"call_x5TCeVP8aUgKADUvabYvQXFq\", \"description\": \"An asynchronous task associated with this tool_call_id has started running. Expect results to arrive later as developer messages that look roughly like this one (with \\'type=async_tool\\' and a matching tool_call_id) but with a \\'result\\' field. Note that there *may* be more than one result (i.e., a stream of results), but there doesn\\'t have to be (there may be only one). The last result will come in a message with \\'status=finished\\'.\"}",
      "tool_call_id":"call_x5TCeVP8aUgKADUvabYvQXFq"
   },
   {
      "role":"developer",
      "content":"{\"type\": \"async_tool\", \"tool_call_id\": \"call_x5TCeVP8aUgKADUvabYvQXFq\", \"status\": \"finished\", \"description\": \"This is the final result for the asynchronous task associated with this tool_call_id. The task has completed. No further results will arrive for this tool_call_id.\", \"result\": \"{\\\"conditions\\\": \\\"nice\\\", \\\"temperature\\\": \\\"75\\\"}\"}"
   }
]
Screenshot 2026-04-10 at 08 15 00

@filipi87 filipi87 requested review from aconchillo and kompfner April 10, 2026 11:20
@filipi87
Copy link
Copy Markdown
Contributor Author

And this is an example with multiple stream responses:

[
   {
      "role":"developer",
      "content":"Please introduce yourself to the user."
   },
   {
      "role":"assistant",
      "content":"Hello! I’m your helpful assistant. How can I assist you today?"
   },
   {
      "role":"user",
      "content":"Could you track my current location?"
   },
   {
      "role":"assistant",
      "content":"Sure, tracking your location now."
   },
   {
      "role":"assistant",
      "tool_calls":[
         {
            "id":"call_J4LO1WCW2Re1ow99rxb9ya3H",
            "function":{
               "name":"track_current_location",
               "arguments":"{}"
            },
            "type":"function"
         }
      ]
   },
   {
      "role":"tool",
      "content":"{\"type\": \"async_tool\", \"status\": \"running\", \"tool_call_id\": \"call_J4LO1WCW2Re1ow99rxb9ya3H\", \"description\": \"An asynchronous task associated with this tool_call_id has started running. Expect results to arrive later as developer messages that look roughly like this one (with \\'type=async_tool\\' and a matching tool_call_id) but with a \\'result\\' field. Note that there *may* be more than one result (i.e., a stream of results), but there doesn\\'t have to be (there may be only one). The last result will come in a message with \\'status=finished\\'.\"}",
      "tool_call_id":"call_J4LO1WCW2Re1ow99rxb9ya3H"
   },
   {
      "role":"developer",
      "content":"{\"type\": \"async_tool\", \"tool_call_id\": \"call_J4LO1WCW2Re1ow99rxb9ya3H\", \"status\": \"running\", \"description\": \"This is an intermediate result for the asynchronous task associated with this tool_call_id. The task is still running. More intermediate results may follow, or the next result may be the final one with \\'status=finished\\'.\", \"result\": \"{\\\"gps\\\": {\\\"lat\\\": 37.731, \\\"lng\\\": -122.4527}, \\\"city\\\": \\\"San Francisco\\\"}\"}"
   },
   {
      "role":"assistant",
      "content":"I’ve started tracking your current location. You’re currently in San Francisco. Let me know if you"
   },
   {
      "role":"developer",
      "content":"{\"type\": \"async_tool\", \"tool_call_id\": \"call_J4LO1WCW2Re1ow99rxb9ya3H\", \"status\": \"running\", \"description\": \"This is an intermediate result for the asynchronous task associated with this tool_call_id. The task is still running. More intermediate results may follow, or the next result may be the final one with \\'status=finished\\'.\", \"result\": \"{\\\"gps\\\": {\\\"lat\\\": 33.96003, \\\"lng\\\": -118.40639}, \\\"city\\\": \\\"Los Angeles\\\"}\"}"
   },
   {
      "role":"user",
      "content":"Actually, we can cancel that now."
   },
   {
      "role":"assistant",
      "tool_calls":[
         {
            "id":"call_ARVAj8QVyoSIhtdugrgtKcHA",
            "function":{
               "name":"cancel_async_tool_call",
               "arguments":"{\"tool_call_id\": \"call_J4LO1WCW2Re1ow99rxb9ya3H\"}"
            },
            "type":"function"
         }
      ]
   },
   {
      "role":"tool",
      "content":"{\"cancelled\": \"call_J4LO1WCW2Re1ow99rxb9ya3H\"}",
      "tool_call_id":"call_ARVAj8QVyoSIhtdugrgtKcHA"
   },
   {
      "role":"assistant",
      "content":"Location tracking has been cancelled. If you need help with anything else, just let me know!"
   },
   {
      "role":"user",
      "content":"Okay. Thank you."
   }
]
Screenshot 2026-04-10 at 08 28 29

@filipi87
Copy link
Copy Markdown
Contributor Author

And this one, is one test with multiple function calls:

[
   {
      "role":"developer",
      "content":"Please introduce yourself to the user."
   },
   {
      "role":"assistant",
      "content":"Hello! I'm your helpful voice assistant. How can I assist you today?"
   },
   {
      "role":"user",
      "content":"Could you tell me how is the weather in"
   },
   {
      "role":"user",
      "content":"San Diego?"
   },
   {
      "role":"user",
      "content":"And track my current location?"
   },
   {
      "role":"assistant",
      "content":"Sure, tracking your location now."
   },
   {
      "role":"assistant",
      "tool_calls":[
         {
            "id":"call_MWF6W3GyUb7rqEq0rTehNsEQ",
            "function":{
               "name":"get_current_weather",
               "arguments":"{\"location\": \"San Diego, CA\", \"format\": \"fahrenheit\"}"
            },
            "type":"function"
         }
      ]
   },
   {
      "role":"tool",
      "content":"{\"type\": \"async_tool\", \"status\": \"running\", \"tool_call_id\": \"call_MWF6W3GyUb7rqEq0rTehNsEQ\", \"description\": \"An asynchronous task associated with this tool_call_id has started running. Expect results to arrive later as developer messages that look roughly like this one (with \\'type=async_tool\\' and a matching tool_call_id) but with a \\'result\\' field. Note that there *may* be more than one result (i.e., a stream of results), but there doesn\\'t have to be (there may be only one). The last result will come in a message with \\'status=finished\\'.\"}",
      "tool_call_id":"call_MWF6W3GyUb7rqEq0rTehNsEQ"
   },
   {
      "role":"assistant",
      "tool_calls":[
         {
            "id":"call_kqGHoNfdvpJBnY6cUu0rFMgH",
            "function":{
               "name":"track_current_location",
               "arguments":"{}"
            },
            "type":"function"
         }
      ]
   },
   {
      "role":"tool",
      "content":"{\"type\": \"async_tool\", \"status\": \"running\", \"tool_call_id\": \"call_kqGHoNfdvpJBnY6cUu0rFMgH\", \"description\": \"An asynchronous task associated with this tool_call_id has started running. Expect results to arrive later as developer messages that look roughly like this one (with \\'type=async_tool\\' and a matching tool_call_id) but with a \\'result\\' field. Note that there *may* be more than one result (i.e., a stream of results), but there doesn\\'t have to be (there may be only one). The last result will come in a message with \\'status=finished\\'.\"}",
      "tool_call_id":"call_kqGHoNfdvpJBnY6cUu0rFMgH"
   },
   {
      "role":"developer",
      "content":"{\"type\": \"async_tool\", \"tool_call_id\": \"call_kqGHoNfdvpJBnY6cUu0rFMgH\", \"status\": \"running\", \"description\": \"This is an intermediate result for the asynchronous task associated with this tool_call_id. The task is still running. More intermediate results may follow, or the next result may be the final one with \\'status=finished\\'.\", \"result\": \"{\\\"gps\\\": {\\\"lat\\\": 37.731, \\\"lng\\\": -122.4527}, \\\"city\\\": \\\"San Francisco\\\"}\"}"
   },
   {
      "role":"assistant",
      "content":"I'm checking the weather in San Diego for you. Meanwhile, your current GPS coordinates place you in San Francisco."
   },
   {
      "role":"developer",
      "content":"{\"type\": \"async_tool\", \"tool_call_id\": \"call_kqGHoNfdvpJBnY6cUu0rFMgH\", \"status\": \"running\", \"description\": \"This is an intermediate result for the asynchronous task associated with this tool_call_id. The task is still running. More intermediate results may follow, or the next result may be the final one with \\'status=finished\\'.\", \"result\": \"{\\\"gps\\\": {\\\"lat\\\": 33.96003, \\\"lng\\\": -118.40639}, \\\"city\\\": \\\"Los Angeles\\\"}\"}"
   },
   {
      "role":"user",
      "content":"Okay. We can cancel tracking my location."
   },
   {
      "role":"assistant",
      "tool_calls":[
         {
            "id":"call_ezcvw1BvgSrK2bxspZV0EIwc",
            "function":{
               "name":"cancel_async_tool_call",
               "arguments":"{\"tool_call_id\": \"call_kqGHoNfdvpJBnY6cUu0rFMgH\"}"
            },
            "type":"function"
         }
      ]
   },
   {
      "role":"tool",
      "content":"{\"cancelled\": \"call_kqGHoNfdvpJBnY6cUu0rFMgH\"}",
      "tool_call_id":"call_ezcvw1BvgSrK2bxspZV0EIwc"
   },
   {
      "role":"assistant",
      "content":"Location tracking has been cancelled. If you need to start it again or have any other requests, just let me know!"
   },
   {
      "role":"developer",
      "content":"{\"type\": \"async_tool\", \"tool_call_id\": \"call_MWF6W3GyUb7rqEq0rTehNsEQ\", \"status\": \"finished\", \"description\": \"This is the final result for the asynchronous task associated with this tool_call_id. The task has completed. No further results will arrive for this tool_call_id.\", \"result\": \"{\\\"conditions\\\": \\\"nice\\\", \\\"temperature\\\": \\\"75\\\"}\"}"
   },
   {
      "role":"assistant",
      "content":"The weather in San Diego is nice, with a temperature around 75 degrees Fahrenheit. If you need anything else, feel free to ask!"
   },
   {
      "role":"user",
      "content":"Thank you."
   }
]
Screenshot 2026-04-10 at 08 47 53

self._function_call_task_finished(task)

if cancelled_items:
await self._call_event_handler("on_function_calls_cancelled", cancelled_items)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't there be a single tool call that matches tool_call_id? So, this should be on_function_call_cancelled (singular) and we don't need to keep track of a list, I think.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I created on_function_calls_cancelled to match the behavior of on_function_calls_started, and this callback is invoked even when there are no async function calls.

So, for example, if you have two function calls in progress (non-async) and you interrupt the bot, both will be cancelled.

@kompfner
Copy link
Copy Markdown
Contributor

Thanks for sharing the outcome of running the examples, @filipi87!

it looks like we need to keep run_llm set to True, otherwise, the LLM doesn’t reply

I see: if we hadn't set run_llm=True, the assistant wouldn't have fired off the next tool call to get the weather for the right place.

Our of curiosity: what was the assistant's response at the end of the ignore-my-first-weather-request example? I assume the assistant did the right thing and didn't feel the need to mention canceling the first request.

Copy link
Copy Markdown
Contributor

@kompfner kompfner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for bearing with all my comments! This is a tricky feature! Nice work.

@filipi87
Copy link
Copy Markdown
Contributor Author

filipi87 commented Apr 10, 2026

Our of curiosity: what was the assistant's response at the end of the ignore-my-first-weather-request example? I assume the assistant did the right thing and didn't feel the need to mention canceling the first request.

Exactly. In that case, after we provided the result of the cancelled function, the assistant simply invoked get_weather again, but this time for Curitiba. It didn’t mention it had cancelled the get_weather for Florianopolis.

@filipi87 filipi87 requested a review from aconchillo April 10, 2026 17:17
Comment thread src/pipecat/services/llm_service.py Outdated
logger.info(f"Starting {len(function_calls)} function calls")

@task.event_handler("on_function_calls_cancelled")
async def on_function_calls_cancelled(service, cancelled):
Copy link
Copy Markdown
Contributor

@aconchillo aconchillo Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update to:

 async def on_function_calls_cancelled(service, function_calls: List[FunctionCallFromLLM]):

same in the started case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@aconchillo
Copy link
Copy Markdown
Contributor

LGTM!

@filipi87 filipi87 merged commit b1204cc into main Apr 10, 2026
6 checks passed
@filipi87 filipi87 deleted the filipi/async_tools_cancellable branch April 10, 2026 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants