Skip to content

Commit 611230c

Browse files
authored
Python: improve misc-integration test robustness (#5295)
* Python: use local MCP server for hosted tools test and broaden image assertion The hosted tools integration test was hitting rate limits on the external learn.microsoft.com MCP server, causing persistent failures that retries couldn't recover from. Switch to the local MCP server already spun up in CI via LOCAL_MCP_URL, skipping when the env var isn't set. Also broaden the image description assertion to accept common synonyms (cottage, mansion, villa, etc.) instead of just "house", since the model legitimately uses varied vocabulary for the same image. * Address review feedback: validate LOCAL_MCP_URL scheme and use word boundaries - Skip hosted tools test when LOCAL_MCP_URL lacks http/https scheme, matching the pattern used in test_mcp.py. - Use regex word boundaries for image assertion to avoid false matches like "villain" matching "villa".
1 parent f112150 commit 611230c

1 file changed

Lines changed: 9 additions & 3 deletions

File tree

python/packages/anthropic/tests/test_anthropic_client.py

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
# Copyright (c) Microsoft. All rights reserved.
22
import os
3+
import re
34
from pathlib import Path
45
from typing import Annotated, Any
56
from unittest.mock import MagicMock, patch
@@ -1503,15 +1504,19 @@ async def test_anthropic_client_integration_function_calling() -> None:
15031504
@skip_if_anthropic_integration_tests_disabled
15041505
async def test_anthropic_client_integration_hosted_tools() -> None:
15051506
"""Integration test for hosted tools."""
1507+
local_mcp_url = os.environ.get("LOCAL_MCP_URL", "")
1508+
if not local_mcp_url or not local_mcp_url.startswith(("http://", "https://")):
1509+
pytest.skip("LOCAL_MCP_URL not set or not an HTTP URL; skipping hosted tools test")
1510+
15061511
client = AnthropicClient()
15071512

15081513
messages = [Message(role="user", contents=["What tools do you have available?"])]
15091514
tools = [
15101515
AnthropicClient.get_web_search_tool(),
15111516
AnthropicClient.get_code_interpreter_tool(),
15121517
AnthropicClient.get_mcp_tool(
1513-
name="example-mcp",
1514-
url="https://learn.microsoft.com/api/mcp",
1518+
name="local-mcp",
1519+
url=local_mcp_url,
15151520
),
15161521
]
15171522

@@ -1607,7 +1612,8 @@ async def test_anthropic_client_integration_images() -> None:
16071612

16081613
assert response is not None
16091614
assert response.messages[0].text is not None
1610-
assert "house" in response.messages[0].text.lower()
1615+
text = response.messages[0].text.lower()
1616+
assert re.search(r"\b(house|home|building|cottage|mansion|villa)\b", text)
16111617

16121618

16131619
# Response Format Tests

0 commit comments

Comments
 (0)