Skip to content

Memory is critically overloaded WARNING #550

@stephenleo

Description

@stephenleo

I'm running the quickstart code in WSL2 on Windows 11. Crawlee is not using the full available memory in WSL2. I have 15Gi total memory in WSL2

$  free -mh
               total        used        free      shared  buff/cache   available
Mem:            15Gi       2.7Gi       8.9Gi       3.0Mi       3.9Gi        12Gi
Swap:          4.0Gi          0B       4.0Gi

Crawlee is only using 3.88GB for some reason.

[crawlee._autoscaling.snapshotter] WARN  Memory is critically overloaded. Using 5.24 GB of 3.88 GB (135%). Consider increasing available memory.

How can I make Crawlee use a larger percentage of available memory?

Code (same code as the quickstart):

import asyncio

from crawlee.playwright_crawler import PlaywrightCrawler, PlaywrightCrawlingContext


async def main() -> None:
    # PlaywrightCrawler crawls the web using a headless browser controlled by the Playwright library.
    crawler = PlaywrightCrawler()

    # Define a request handler to process each crawled page and attach it to the crawler using a decorator.
    @crawler.router.default_handler
    async def request_handler(context: PlaywrightCrawlingContext) -> None:
        # Extract relevant data from the page context.
        data = {
            'url': context.request.url,
            'title': await context.page.title(),
        }
        # Store the extracted data.
        await context.push_data(data)
        # Extract links from the current page and add them to the crawling queue.
        await context.enqueue_links()

    # Add first URL to the queue and start the crawl.
    await crawler.run(['https://crawlee.dev'])


if __name__ == '__main__':
    asyncio.run(main())

Metadata

Metadata

Assignees

No one assigned

    Labels

    t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions