fix(fetch): handle 5xx responses and add timeout in robots.txt check

[isheng-eqi]

Summary

Two fixes in check_may_autonomously_fetch_url in the fetch server.

Problem 1: 5xx responses silently parsed as robots.txt

When a robots.txt endpoint returns 5xx, the code fell through to the robots.txt parsing block. The error page HTML was fed to Protego.parse(), which would silently produce a default-allow parser. The 4xx check on line 92 only covers 400–499; 500+ had no branch.

Problem 2: No timeout on robots.txt fetch

The check_may_autonomously_fetch_url function's client.get() had no timeout parameter. If the robots.txt endpoint hangs (network partition, slow server), the client hangs indefinitely. The main fetch_url function already uses timeout=30 — this is a consistency fix.

Changes

• src/fetch/src/mcp_server_fetch/server.py (+6 lines):
  • Added elif response.status_code >= 500 branch that raises McpError instead of parsing the error page as robots.txt
  • Added timeout=30 to the client.get() call for robots.txt

Testing

from mcp_server_fetch.server import check_may_autonomously_fetch_url
# Import verified OK, no syntax or type errors