feat: Add FirecrawlCrawler

[bogdankostic]

Related Issues

• part of https://github.com/deepset-ai/haystack-private/issues/89

Proposed Changes:

Added new FirecrawlFetcher component under haystack_integrations.components.fetchers.firecrawl.
implementing both sync (run) and async (run_async) crawling flows with Firecrawl clients.

How did you test it?

I added unit tests and integration tests.

Notes for the reviewer

Firecrawl also provides websearch functionality. We can add that in a separate PR by introducing a FirecrawlWebSearch component, analogous to the already exisitng SearchApiWebSearch and SerperDevWebSearch components.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.

[anakin87]

Nice work.

I left some initial comments and also added a task list (copied from our new integration template) in https://github.com/deepset-ai/haystack-private/issues/89#issuecomment-3932711305.

[anakin87]

Please add the integration to the Inventory section of this repo README

[anakin87]

Looks good!