anthropic prompt caching for tool result and tool use#1141
Conversation
|
DescriptionSummary By MatterAI
🔄 What ChangedThis pull request introduces support for Anthropic's prompt caching for tool results and tool use. Key changes include:
🔍 Impact of the ChangeThese changes enable the system to leverage Anthropic's prompt caching capabilities for tool interactions, potentially improving performance and reducing token usage for repeated tool calls or results. It also enhances type safety and flexibility for tool result content. 📁 Total Files Changed
🧪 Test AddedThe PR description includes an example JSON payload demonstrating the new 🔒Security VulnerabilitiesNo new security vulnerabilities were detected in the provided code changes. MotivationThis PR is motivated by the need to implement Anthropic prompt caching for tool results and tool use, as referenced by #1140. Type of Change
How Has This Been Tested?
Screenshots (if applicable)N/A Checklist
Related IssuesTip Quality Recommendations
Sequence DiagramsequenceDiagram
participant User
participant AnthropicChatCompleteConfig
participant transformAssistantMessage
participant transformToolMessage
participant AnthropicAPI
User->>AnthropicChatCompleteConfig: Sends Chat Completion Request (messages)
AnthropicChatCompleteConfig->>AnthropicChatCompleteConfig: Iterates through messages
alt Message Role is 'assistant'
AnthropicChatCompleteConfig->>transformAssistantMessage: Call transformAssistantMessage(msg: Message)
transformAssistantMessage-->>AnthropicChatCompleteConfig: Returns AnthropicMessage (with tool_code and optional cache_control)
end
alt Message Role is 'tool'
AnthropicChatCompleteConfig->>transformToolMessage: Call transformToolMessage(msg: Message)
transformToolMessage-->>AnthropicChatCompleteConfig: Returns AnthropicMessage (with tool_result and potentially complex content)
end
AnthropicChatCompleteConfig->>AnthropicAPI: Sends Transformed Messages to /chat/completions
AnthropicAPI-->>AnthropicChatCompleteConfig: Returns API Response
AnthropicChatCompleteConfig-->>User: Returns Chat Completion Response
|
![matterai-app[bot]](https://avatars.githubusercontent.com/in/474549?s=60&v=4){kind=small}
| type: 'tool_result'; | ||
| tool_use_id: string; | ||
| content?: string; | ||
| content?: |
There was a problem hiding this comment.
💡 Optional Recommendation
Issue: The cache_control property is added without validation or documentation.
Fix: Add JSDoc comments explaining the purpose and structure of the cache_control property.
Impact: Improves code maintainability and helps other developers understand the feature's purpose and usage.
| type: 'tool_result'; | |
| tool_use_id: string; | |
| content?: string; | |
| content?: | |
| type: 'tool_result'; | |
| tool_use_id: string; | |
| /** | |
| * Content for the tool result, which can include cache control directives | |
| * to optimize API usage by caching responses when appropriate. | |
| * | |
| * @property {Object} cache_control - Controls caching behavior | |
| * @property {string} cache_control.type - Type of caching directive | |
| * @property {number} cache_control.ttl - Time to live in seconds | |
| */ | |
| content?: |
| ...(toolCall.cache_control && { | ||
| cache_control: toolCall.cache_control, | ||
| }), |
There was a problem hiding this comment.
💡 Optional Recommendation
Issue: The cache_control property is passed directly without validation.
Fix: Add validation to ensure cache_control has the expected structure before passing it to the API.
Impact: Prevents potential runtime errors if cache_control has an unexpected format.
| ...(toolCall.cache_control && { | |
| cache_control: toolCall.cache_control, | |
| }), | |
| ...(toolCall.cache_control && { | |
| cache_control: validateCacheControl(toolCall.cache_control), | |
| }), |
| } else if (msg.role === 'tool') { | ||
| // even though anthropic supports images in tool results, openai doesn't support it yet | ||
| messages.push(transformToolMessage(msg)); |
There was a problem hiding this comment.
🛠️ Code Refactor
Issue: The comment about Anthropic supporting images in tool results is incomplete and could be more informative.
Fix: Expand the comment to provide more context about the current limitations and future compatibility.
Impact: Improves code documentation and helps other developers understand the implementation decisions.
| } else if (msg.role === 'tool') { | |
| // even though anthropic supports images in tool results, openai doesn't support it yet | |
| messages.push(transformToolMessage(msg)); | |
| } else if (msg.role === 'tool') { | |
| // Anthropic supports images in tool results, but OpenAI doesn't support it yet. | |
| // We're transforming the tool message to ensure cross-provider compatibility. | |
| messages.push(transformToolMessage(msg)); |
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
|
@VisargD the example posted is returning a 500 for me: |
3 participants
#1140
documentation for prompt caching from anthropic: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
https://docs.anthropic.com/en/api/messages#body-messages-content-cache-control
example payload
{ "model": "claude-3-5-sonnet-20240620", "max_tokens": 200, "stream": false, "messages": [ { "role": "user", "content": "Based on the temperature and time in California what do you suggest I wear?" }, { "role": "assistant", "content": "Certainly! I'd be happy to provide you with the current temperature and time in San Francisco. To get this information, I'll need to use two separate tools. Let me fetch that data for you.", "tool_calls": [ { "id": "toolu_01BspjPLY4dtHkKGurn8q9A6", "type": "function", "function": { "name": "get_current_temperature", "arguments": "" }, "cache_control": { "type": "ephemeral" } }, { "id": "toolu_014jEfKqGbfFvRaKfiauxgPv", "type": "function", "function": { "name": "get_current_time", "arguments": "{\"location\":\"San Francisco, CA\"}" } } ] }, { "role": "tool", "content": [ { "type": "text", "text": "15 degrees", "cache_control": { "type": "ephemeral" } } ], "tool_call_id": "toolu_01BspjPLY4dtHkKGurn8q9A6" }, { "role": "tool", "content": "10 PM", "tool_call_id": "toolu_014jEfKqGbfFvRaKfiauxgPv" } ], "tools": [ { "type": "function", "function": { "name": "get_current_temperature", "description": "Get the current temperature for a specific location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g., San Francisco, CA" }, "unit": { "type": "string", "enum": [ "Celsius", "Fahrenheit" ], "description": "The temperature unit to use. Infer this from the user's location." } }, "required": [ "location", "unit" ] } } }, { "type": "function", "function": { "name": "get_current_time", "description": "Get the current time for a specific location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g., San Francisco, CA" } }, "required": [ "location" ] } } } ] }