0.2.0

0.2.0 (2024-08-05)

🚀 Features

• Add new curl impersonate HTTP client (#387) (9c06260)
• (playwright) infinite_scroll helper (#393) (34f74bd)

0.1.2

0.1.2 (2024-07-30)

🚀 Features

• Add URL validation (#343) (1514538)

🐛 Bug Fixes

• Minor log fix (#341) (0688bf1)
• Also use error_handler for context pipeline errors (#331) (7a66445)
• Strip whitespace from href in enqueue_links (#346) (8a3174a)
• Warn instead of crashing when an empty dataset is being exported (#342) (22b95d1)
• Avoid Github rate limiting in project bootstrapping test (#364) (992f07f)
• Pass crawler configuration to storages (#375) (b2d3a52)
• Purge request queue on repeated crawler runs (#377) (7ad3d69)

0.1.1

Features

• Support for proxy configuration in PlaywrightCrawler.
• Blocking detection in PlaywrightCrawler.
• Expose crawler.log to public.

Bug fixes

• Fix Pylance reportPrivateImportUsage errors by defining __all__ in modules __init__.py.
• Set HTTPX logging level to WARNING by default.
• Fix CLI behavior with existing project folders

0.1.0

• Crawlee is a web scraping and browser automation library.
• Launching Crawlee for Python blog post

Features

Why is Crawlee the preferred choice for web scraping and crawling?

Why use Crawlee instead of just a random HTTP library with an HTML parser?

• Unified interface for HTTP & headless browser crawling.
• Automatic parallel crawling based on available system resources.
• Written in Python with type hints - enhances DX (IDE autocompletion) and reduces bugs (static type checking).
• Automatic retries on errors or when you’re getting blocked.
• Integrated proxy rotation and session management.
• Configurable request routing - direct URLs to the appropriate handlers.
• Persistent queue for URLs to crawl.
• Pluggable storage of both tabular data and files.
• Robust error handling.

Why to use Crawlee rather than Scrapy?

• Crawlee has out-of-the-box support for headless browser crawling (Playwright).
• Crawlee has a minimalistic & elegant interface - Set up your scraper with fewer than 10 lines of code.
• Complete type hint coverage.
• Based on standard Asyncio.

0.0.7

Fixes

• selector handling for RETRY_CSS_SELECTORS in _handle_blocked_request in BeautifulSoupCrawler
• selector handling in enqueue_links in BeautifulSoupCrawler
• improve AutoscaledPool state management

0.0.6

Adds

• BREAKING: BasicCrawler.export_data helper method which replaces BasicCrawler.export_to
• Configuration.get_global_configuration method
• Automatic logging setup
• Context helper for logging (context.log)

Fixes

• Handling of relative URLs in add_requests
• Graceful exit in BasicCrawler.run

0.0.5

Adds

• Add explicit error messages for missing package extras during import
• Better browser abstraction:
  • BrowserController - Wraps a single browser instance and maintains its state.
  • BrowserPlugin - Manages the browser automation framework, and basically acts as a factory for controllers.
• Browser rotation with a maximum number of pages opened per browser.
• Add emit persist state event to event manager
• Add batched request addition in RequestQueue
• Add start requests option to BasicCrawler
• Add storage-related helpers get_data, push_data and export_to to BasicCrawler and BasicContext
• Add PlaywrightCrawler's enqueue links helper

Fixes

• Fix type error in persist state of statistics

0.0.4

Another internal release, adding statistics capturing, proxy configuration,
and the initial version of browser management and PlaywrightCrawler.

Adds

• Statistics
• ProxyConfiguration
• BrowserPool
• PlaywrightCrawler

0.0.3

Another internal release, adding mainly session management and BeautifulSoupCrawler.

Adds

• HttpxClient
• SessionPool
• BeautifulSoupCrawler
• BaseStorageClient
• Storages and MemoryStorageClient were refactored

Was added in 0.0.2

• EventManager & LocalEventManager
• Snapshotter
• AutoscaledPool
• MemoryStorageClient
• Storages
• BasicCrawler & HttpCrawler