Skip to content

Commit 29d301b

Browse files
authored
fix(playwright): dispose of APIResponse body for send_request (#1771)
### Description Dispose of Playwright's APIResponse body, otherwise it stays in memory until the context closes. If not done, this can lead to excessive memory usage during crawling when using `send_request` within a handler. ### Testing `send_request` is already covered by tests, so if they pass I think it should be OK.
1 parent d4c51f8 commit 29d301b

File tree

1 file changed

+6
-1
lines changed

1 file changed

+6
-1
lines changed

src/crawlee/crawlers/_playwright/_types.py

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,15 @@
33
from dataclasses import dataclass
44
from typing import TYPE_CHECKING, Literal, Protocol, TypedDict
55

6+
from playwright.async_api import APIResponse
7+
68
from crawlee import HttpHeaders
79
from crawlee._utils.docs import docs_group
810

911
if TYPE_CHECKING:
1012
from collections.abc import AsyncGenerator
1113

12-
from playwright.async_api import APIResponse, Response
14+
from playwright.async_api import Response
1315
from typing_extensions import NotRequired, Self
1416

1517

@@ -56,6 +58,9 @@ async def from_playwright_response(cls, response: Response | APIResponse, protoc
5658
# Used http protocol version cannot be obtained from `Response` and has to be passed as additional argument.
5759
http_version = protocol
5860
_content = await response.body()
61+
# If not called then the body will stay in memory until the context closes.
62+
if isinstance(response, APIResponse):
63+
await response.dispose()
5964

6065
return cls(http_version=http_version, status_code=status_code, headers=headers, _content=_content)
6166

0 commit comments

Comments
 (0)