This new component should use the Tavily Extract API endpoint, that extracts web page content from one or more specified URLs.

If you want to contribute with this feature, make sure you have a Tavily API key to test the new component yourself locally. Please do not submit a PR with code you haven't tried yourself.

Brief design idea

Should live in the Tavily integrations, under a new fetchers folder.

The run signature might look like the following:

@component.output_types(documents=list[Document], meta=dict[str, Any])
def run(self, urls: list[str]) -> dict[str, Any]:

I expect meta to contain info related to the request like response_time, usage and request_id. See API docs above.

We should also understand if only webpages are supported or also other content like PDFs.

Checklist

• The code is documented with docstrings and was merged in the main branch
• Docs are published at https://docs.haystack.deepset.ai/
• There is a Github workflow running the tests for the integration nightly and at every PR
• The package has been released on PyPI
• An integration tile with a usage example has been added to https://github.com/deepset-ai/haystack-integrations
• The integration has been listed in the Inventory section of this repo README