diff --git a/docs-website/docs/concepts/data-classes.mdx b/docs-website/docs/concepts/data-classes.mdx index 636fa5fd34c..141d080d318 100644 --- a/docs-website/docs/concepts/data-classes.mdx +++ b/docs-website/docs/concepts/data-classes.mdx @@ -116,7 +116,7 @@ image = ByteStream.from_file_path("dog.jpg") ### ChatMessage -`ChatMessage` is the central abstraction to represent a message for a LLM. It contains role, metadata and several types of content, including text, tool calls and tool calls results. +`ChatMessage` is the central abstraction to represent a message for a LLM. It contains role, metadata and several types of content, including text, tool calls and tool calls results. Read the detailed documentation for the `ChatMessage` data class on a dedicated [ChatMessage](data-classes/chatmessage.mdx) page. @@ -301,7 +301,7 @@ print(info.name) # Name assigned in the pipeline, if any #### Overview -The `SparseEmbedding` class represents a sparse embedding: a vector where most values are zeros. +The `SparseEmbedding` class represents a sparse embedding: a vector where most values are zeros. #### Attributes diff --git a/docs-website/docs/concepts/data-classes/chatmessage.mdx b/docs-website/docs/concepts/data-classes/chatmessage.mdx index ab15e9caa72..ab4e1777bcc 100644 --- a/docs-website/docs/concepts/data-classes/chatmessage.mdx +++ b/docs-website/docs/concepts/data-classes/chatmessage.mdx @@ -9,7 +9,7 @@ description: "`ChatMessage` is the central abstraction to represent a message fo `ChatMessage` is the central abstraction to represent a message for a LLM. It contains role, metadata and several types of content, including text, images, tool calls, tool call results, and reasoning content. -To create a `ChatMessage` instance, use `from_user`, `from_system`, `from_assistant`, and `from_tool` class methods. +To create a `ChatMessage` instance, use `from_user`, `from_system`, `from_assistant`, and `from_tool` class methods. The [content](#types-of-content) of the `ChatMessage` can then be inspected using the `text`, `texts`, `image`, `images`, `file`, `files`, `tool_call`, `tool_calls`, `tool_call_result`, `tool_call_results`, `reasoning`, and `reasonings` properties. diff --git a/docs-website/docs/concepts/document-store.mdx b/docs-website/docs/concepts/document-store.mdx index 7c22becab7d..ef9c93771c3 100644 --- a/docs-website/docs/concepts/document-store.mdx +++ b/docs-website/docs/concepts/document-store.mdx @@ -38,7 +38,7 @@ See the installation and initialization details for each Document Store in the " ### Work with Documents -Convert your data into `Document` objects before writing them into a Document Store along with its metadata and document ID. +Convert your data into `Document` objects before writing them into a Document Store along with its metadata and document ID. The ID field is mandatory, so if you don’t choose a specific ID yourself, Haystack will do its best to come up with a unique ID based on the document’s information and assign it automatically. However, since Haystack uses the document’s contents to create an ID, two identical documents might have identical IDs. Keep it in mind as you update your documents, as the ID will not be updated automatically. diff --git a/docs-website/docs/concepts/document-store/creating-custom-document-stores.mdx b/docs-website/docs/concepts/document-store/creating-custom-document-stores.mdx index d00579b6203..c8f01f91088 100644 --- a/docs-website/docs/concepts/document-store/creating-custom-document-stores.mdx +++ b/docs-website/docs/concepts/document-store/creating-custom-document-stores.mdx @@ -152,9 +152,9 @@ Haystack comes with some testing functionalities you can use in a custom Documen ### Implementation Tips - The best way to learn how to write a custom Document Store is to look at the existing ones: the `InMemoryDocumentStore`, which is part of Haystack, or the [`ElasticsearchDocumentStore`](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/elasticsearch), which is a Core Integration, are good places to start. -- When starting from scratch, it might be easier to create the four CRUD methods of the `DocumentStore` protocol one at a time and test them one at a time as well. For example: +- When starting from scratch, it might be easier to create the four CRUD methods of the `DocumentStore` protocol one at a time and test them one at a time as well. For example: 1. Implement the logic for `count_documents`. - 2. In your `test_document_store.py` module, define the test class `TestDocumentStore(CountDocumentsTest)`. Note how we only inherit from the specific testing mix-in `CountDocumentsTest`. + 2. In your `test_document_store.py` module, define the test class `TestDocumentStore(CountDocumentsTest)`. Note how we only inherit from the specific testing mix-in `CountDocumentsTest`. 3. Make the tests pass. 4. Implement the logic for `write_documents`. 5. Change `test_document_store.py` so that your class now also derives from the `WriteDocumentsTest` mix-in: `TestDocumentStore(CountDocumentsTest, WriteDocumentsTest)`. diff --git a/docs-website/docs/concepts/experimental-package.mdx b/docs-website/docs/concepts/experimental-package.mdx index 5852f72c1c8..5f0a4cdfbef 100644 --- a/docs-website/docs/concepts/experimental-package.mdx +++ b/docs-website/docs/concepts/experimental-package.mdx @@ -9,13 +9,13 @@ description: "Try out new experimental features with Haystack." Try out new experimental features with Haystack. -The `haystack-experimental` package allows you to test new experimental features without committing to their official release. Its main goal is to gather user feedback and iterate on new features quickly. +The `haystack-experimental` package allows you to test new experimental features without committing to their official release. Its main goal is to gather user feedback and iterate on new features quickly. Check out the `haystack-experimental` [GitHub repository](https://github.com/deepset-ai/haystack-experimental) for the latest catalog of available features, or take a look at our [Experiments API Reference](/reference). ### Installation -For simplicity, every release of `haystack-experimental` includes all the available experiments at that time. To install the latest features, run: +For simplicity, every release of `haystack-experimental` includes all the available experiments at that time. To install the latest features, run: ```shell pip install -U haystack-experimental @@ -45,7 +45,7 @@ c = FoobarGenerator() c.run([ChatMessage.from_user("What's an experiment? Be brief.")]) ``` -Experiments can also override existing Haystack features. For example, you can opt into an experimental type of `Pipeline` by changing the usual import: +Experiments can also override existing Haystack features. For example, you can opt into an experimental type of `Pipeline` by changing the usual import: ```python # from haystack import Pipeline diff --git a/docs-website/docs/concepts/integrations.mdx b/docs-website/docs/concepts/integrations.mdx index ee6af756d04..19754a67669 100644 --- a/docs-website/docs/concepts/integrations.mdx +++ b/docs-website/docs/concepts/integrations.mdx @@ -9,7 +9,7 @@ description: "The Haystack ecosystem integrates with many other technologies, su The Haystack ecosystem integrates with many other technologies, such as vector databases, model providers and even custom components made by the community. Here you can explore our integrations, which may be maintined by deepset, or submitted by others. -Haystack integrates with a number of other technologies and tools. For example, you can use a number of different model providers or databases with Haystack. +Haystack integrates with a number of other technologies and tools. For example, you can use a number of different model providers or databases with Haystack. There are two main types of integrations: @@ -20,7 +20,7 @@ There are two main types of integrations: An integration is any type of external technology that can be used to extend the capabilities of the Haystack framework. Some integration examples are those providing access to model providers like OpenAI or Cohere, to databases like Weaviate and Qdrant, or even to monitoring tools such as Traceloop. They can be components, Document Stores, or any other feature that can be used with Haystack. -We maintain a list of available integrations on the [Haystack Integrations](https://haystack.deepset.ai/integrations) page, where you can see which integrations we maintain or which have been contributed by the community. +We maintain a list of available integrations on the [Haystack Integrations](https://haystack.deepset.ai/integrations) page, where you can see which integrations we maintain or which have been contributed by the community. An integrations page focuses on explaining how Haystack integrates with that technology. For example, the OpenAI integration page will provide a summary of the various ways Haystack and OpenAI can work together. @@ -48,13 +48,13 @@ Check out the [video walkthrough](https://www.youtube.com/watch?v=SWC0QecAMcI) f ## How do I showcase my integration? -To make your integration visible to the Haystack community, contribute it to our [haystack-integrations](https://github.com/deepset-ai/haystack-integrations) GitHub repository. There are several requirements you have to follow: +To make your integration visible to the Haystack community, contribute it to our [haystack-integrations](https://github.com/deepset-ai/haystack-integrations) GitHub repository. There are several requirements you have to follow: - Make sure your contribution is [packaged](https://packaging.python.org/en/latest/), installable, and runnable. We suggest using [hatch](https://hatch.pypa.io/latest/) for this purpose. - Provide the GitHub repo and issue link. -- Create a Pull Request in the [haystack-integrations](https://github.com/deepset-ai/haystack-integrations) repo by following the [draft-integration.md](https://github.com/deepset-ai/haystack-integrations/blob/main/draft-integration.md) and include a clear explanation of what your integration is. This page should include: +- Create a Pull Request in the [haystack-integrations](https://github.com/deepset-ai/haystack-integrations) repo by following the [draft-integration.md](https://github.com/deepset-ai/haystack-integrations/blob/main/draft-integration.md) and include a clear explanation of what your integration is. This page should include: - Installation instructions - A list of the components the integration includes - Examples of how to use it with clear/runnable code - Licensing information - - (Optionally) Documentation and/or API docs that you’ve generated for your repository \ No newline at end of file + - (Optionally) Documentation and/or API docs that you’ve generated for your repository diff --git a/docs-website/docs/concepts/jinja-templates.mdx b/docs-website/docs/concepts/jinja-templates.mdx index 769abbfd831..f651f56b7da 100644 --- a/docs-website/docs/concepts/jinja-templates.mdx +++ b/docs-website/docs/concepts/jinja-templates.mdx @@ -55,4 +55,4 @@ Due to how we use Jinja in some Components, there are some security consideratio To mitigate this risk, Jinja templates are executed and rendered in a [sandbox environment](https://jinja.palletsprojects.com/en/3.1.x/sandbox/). While this approach is safer, it's also less flexible and limits the expressiveness of the template. If you need the more advanced functionality of Jinja templates, components that use them provide an `unsafe` init parameter - setting it to `False` will disable the sandbox environment and enable unsafe template rendering. -With unsafe template rendering, the [OutputAdapter](../pipeline-components/converters/outputadapter.mdx) and [ConditionalRouter](../pipeline-components/routers/conditionalrouter.mdx) components allow their `output_type` to be set to one of the [Haystack data classes](data-classes.mdx) such as `ChatMessage`, `Document`, or `Answer`. +With unsafe template rendering, the [OutputAdapter](../pipeline-components/converters/outputadapter.mdx) and [ConditionalRouter](../pipeline-components/routers/conditionalrouter.mdx) components allow their `output_type` to be set to one of the [Haystack data classes](data-classes.mdx) such as `ChatMessage`, `Document`, or `Answer`. diff --git a/docs-website/docs/concepts/metadata-filtering.mdx b/docs-website/docs/concepts/metadata-filtering.mdx index 583c5b54ebd..1ca7e246c68 100644 --- a/docs-website/docs/concepts/metadata-filtering.mdx +++ b/docs-website/docs/concepts/metadata-filtering.mdx @@ -76,10 +76,10 @@ The available logic operators may vary depending on the specific Document Store Here is a more complex filter that uses both Comparison and Logic to find documents where: -- Meta field `type` is "article", -- Meta field `date` is between 1420066800 and 1609455600 (a specific date range), -- Meta field `rating` is greater than or equal to 3, -- Documents are either classified as `genre`  ["economy", "politics"] `OR` the meta field `publisher` is "nytimes". +- Meta field `type` is "article", +- Meta field `date` is between 1420066800 and 1609455600 (a specific date range), +- Meta field `rating` is greater than or equal to 3, +- Documents are either classified as `genre` ["economy", "politics"] `OR` the meta field `publisher` is "nytimes". ```python filters = { @@ -133,7 +133,7 @@ pipeline.run( ) ``` -In Document Stores, the `filter_documents` method is used to apply filters to stored documents, if the specific integration supports filtering. +In Document Stores, the `filter_documents` method is used to apply filters to stored documents, if the specific integration supports filtering. The example below shows how filters can be passed to the `QdrantDocumentStore`: diff --git a/docs-website/docs/concepts/pipelines/pipeline-breakpoints.mdx b/docs-website/docs/concepts/pipelines/pipeline-breakpoints.mdx index 0660f68bd33..8a7295ddc37 100644 --- a/docs-website/docs/concepts/pipelines/pipeline-breakpoints.mdx +++ b/docs-website/docs/concepts/pipelines/pipeline-breakpoints.mdx @@ -11,7 +11,7 @@ Learn how to pause and resume Haystack pipeline or Agent execution using breakpo ## Introduction -Haystack pipelines support breakpoints for debugging complex execution flows. A `Breakpoint` allows you to pause the execution at specific components, inspect the pipeline state, and resume execution from saved snapshots. This feature works for any regular component as well as an `Agent` component. +Haystack pipelines support breakpoints for debugging complex execution flows. A `Breakpoint` allows you to pause the execution at specific components, inspect the pipeline state, and resume execution from saved snapshots. This feature works for any regular component as well as an `Agent` component. You can set a `Breakpoint` on any component in a pipeline with a specific visit count. When triggered, the system stops the execution of the `Pipeline` and captures a snapshot of the current pipeline state. The state can be saved to a JSON file when snapshot file saving is enabled, see [Snapshot file saving](#snapshot-file-saving) below. You can inspect and modify the snapshot and use it to resume execution from the exact point where it stopped. @@ -115,7 +115,7 @@ print(result["llm"]["replies"]) You can also set breakpoints in an Agent component. An Agent supports two types of breakpoints: -1. **Chat Generator Breakpoint**: Pauses before LLM calls. +1. **Chat Generator Breakpoint**: Pauses before LLM calls. 2. **Tool Invoker Breakpoint**: Pauses before any tool execution. A `ChatGenerator` breakpoint is defined as shown below. You need to define a `Breakpoint` as for a pipeline breakpoint and then an `AgentBreakpoint` where you pass the breakpoint defined before and the name of Agent component. @@ -148,7 +148,7 @@ agent_breakpoint = AgentBreakpoint(break_point=tool_bp, agent_name="my_agent") ### Resuming Agent Execution -When an Agent breakpoint is triggered, you can resume execution using the saved snapshot. Similar to the regular component in a pipeline, pass the JSON file with the snapshot to the `run()` method of the pipeline. +When an Agent breakpoint is triggered, you can resume execution using the saved snapshot. Similar to the regular component in a pipeline, pass the JSON file with the snapshot to the `run()` method of the pipeline. ```python from haystack.core.pipeline.breakpoint import load_pipeline_snapshot diff --git a/docs-website/docs/development/deployment/kubernetes.mdx b/docs-website/docs/development/deployment/kubernetes.mdx index 929ca4a99e1..3d0babc5112 100644 --- a/docs-website/docs/development/deployment/kubernetes.mdx +++ b/docs-website/docs/development/deployment/kubernetes.mdx @@ -55,7 +55,7 @@ spec: ``` -After applying the above to an existing Kubernetes cluster, a `hayhooks` Pod will show up as a Service called `haystack-service`. +After applying the above to an existing Kubernetes cluster, a `hayhooks` Pod will show up as a Service called `haystack-service`. Note that the `Service` defined above is of type `ClusterIP`. That means it's exposed only _inside_ the Kubernetes cluster. To expose the Hayhooks API to the _outside_ world as well, you need a `NodePort` or `Ingress` resource. As an alternative, it's also possible to use [Port Forwarding](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) to access the `Service` locally. diff --git a/docs-website/docs/development/logging.mdx b/docs-website/docs/development/logging.mdx index e9c5cf01450..7ebe6524e9b 100644 --- a/docs-website/docs/development/logging.mdx +++ b/docs-website/docs/development/logging.mdx @@ -17,9 +17,9 @@ Haystack logs through Python’s standard library. This gives you full flexibili ### Changing the Log Level -By default, Haystack's logging level is set to `WARNING`. To display more information, you can change it to `INFO`. This way, not only warnings but also information messages are displayed in the console output. +By default, Haystack's logging level is set to `WARNING`. To display more information, you can change it to `INFO`. This way, not only warnings but also information messages are displayed in the console output. -To change the logging level to `INFO`, run: +To change the logging level to `INFO`, run: ```python import logging diff --git a/docs-website/docs/document-stores/azureaisearchdocumentstore.mdx b/docs-website/docs/document-stores/azureaisearchdocumentstore.mdx index 21532869c2a..73a1b7c43aa 100644 --- a/docs-website/docs/document-stores/azureaisearchdocumentstore.mdx +++ b/docs-website/docs/document-stores/azureaisearchdocumentstore.mdx @@ -26,7 +26,7 @@ A Document Store for storing and retrieval from Azure AI Search Index. This integration requires you to have an active Azure subscription with a deployed [Azure AI Search](https://azure.microsoft.com/en-us/products/ai-services/ai-search) service. -Once you have the subscription, install the `azure-ai-search-haystack` integration: +Once you have the subscription, install the `azure-ai-search-haystack` integration: ```python pip install azure-ai-search-haystack @@ -36,7 +36,7 @@ To use the `AzureAISearchDocumentStore`, you need to provide a search service en During initialization the Document Store will either retrieve the existing search index for the given `index_name` or create a new one if it doesn't already exist. Note that one of the limitations of `AzureAISearchDocumentStore` is that the fields of the Azure search index cannot be modified through the API after creation. Therefore, any additional fields beyond the default ones must be provided as `metadata_fields` during the Document Store's initialization. However, if needed, [Azure AI portal](https://azure.microsoft.com/) can be used to modify the fields without deleting the index. -It is recommended to pass authentication data through `AZURE_AI_SEARCH_API_KEY` and `AZURE_AI_SEARCH_ENDPOINT` before running the following example. +It is recommended to pass authentication data through `AZURE_AI_SEARCH_API_KEY` and `AZURE_AI_SEARCH_ENDPOINT` before running the following example. ```python from haystack_integrations.document_stores.azure_ai_search import ( diff --git a/docs-website/docs/document-stores/chromadocumentstore.mdx b/docs-website/docs/document-stores/chromadocumentstore.mdx index 7fe5c144b6a..bfbc1b9182e 100644 --- a/docs-website/docs/document-stores/chromadocumentstore.mdx +++ b/docs-website/docs/document-stores/chromadocumentstore.mdx @@ -15,7 +15,7 @@ slug: "/chromadocumentstore" -[Chroma](https://docs.trychroma.com/) is an open source vector database capable of storing collections of documents along with their metadata, creating embeddings for documents and queries, and searching the collections filtering by document metadata or content. Additionally, Chroma supports multi-modal embedding functions. +[Chroma](https://docs.trychroma.com/) is an open source vector database capable of storing collections of documents along with their metadata, creating embeddings for documents and queries, and searching the collections filtering by document metadata or content. Additionally, Chroma supports multi-modal embedding functions. Chroma can be used in-memory, as an embedded database, or in a client-server fashion. When running in-memory, Chroma can still keep its contents on disk across different sessions. This allows users to quickly put together prototypes using the in-memory version and later move to production, where the client-server version is deployed. @@ -27,7 +27,7 @@ First, install the Chroma integration, which will install Haystack and Chroma if pip install chroma-haystack ``` -To store data in Chroma, create a `ChromaDocumentStore` instance and write documents with: +To store data in Chroma, create a `ChromaDocumentStore` instance and write documents with: ```python from haystack_integrations.document_stores.chroma import ChromaDocumentStore @@ -47,7 +47,7 @@ In this case, since we didn’t pass any embeddings along with our documents, Ch ### Connection Options -1. **In-Memory Mode (Local)**: Chroma can be set up as a local Document Store for fast and lightweight usage. You can use this option during development or small-scale experiments. Set up a local in-memory instance of `ChromaDocumentStore` like this: +1. **In-Memory Mode (Local)**: Chroma can be set up as a local Document Store for fast and lightweight usage. You can use this option during development or small-scale experiments. Set up a local in-memory instance of `ChromaDocumentStore` like this: ```python from haystack_integrations.document_stores.chroma import ChromaDocumentStore diff --git a/docs-website/docs/document-stores/elasticsearch-document-store.mdx b/docs-website/docs/document-stores/elasticsearch-document-store.mdx index eab82b2b06c..36e7b6e22cc 100644 --- a/docs-website/docs/document-stores/elasticsearch-document-store.mdx +++ b/docs-website/docs/document-stores/elasticsearch-document-store.mdx @@ -24,7 +24,7 @@ It features the approximate nearest neighbours (ANN) search. ### Initialization -[Install](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html) Elasticsearch and then [start](https://www.elastic.co/guide/en/elasticsearch/reference/current/starting-elasticsearch.html) an instance. Haystack supports Elasticsearch 8. +[Install](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html) Elasticsearch and then [start](https://www.elastic.co/guide/en/elasticsearch/reference/current/starting-elasticsearch.html) an instance. Haystack supports Elasticsearch 8. If you have Docker set up, we recommend pulling the Docker image and running it. diff --git a/docs-website/docs/document-stores/opensearch-document-store.mdx b/docs-website/docs/document-stores/opensearch-document-store.mdx index eb952e32cf7..a522953dfeb 100644 --- a/docs-website/docs/document-stores/opensearch-document-store.mdx +++ b/docs-website/docs/document-stores/opensearch-document-store.mdx @@ -18,7 +18,7 @@ A Document Store for storing and retrieval from OpenSearch. -OpenSearch is a fully open source search and analytics engine for use cases such as log analytics, real-time application monitoring, and clickstream analysis. For more information, see the [OpenSearch documentation](https://opensearch.org/docs/). +OpenSearch is a fully open source search and analytics engine for use cases such as log analytics, real-time application monitoring, and clickstream analysis. For more information, see the [OpenSearch documentation](https://opensearch.org/docs/). This Document Store is great if you want to evaluate the performance of different retrieval options (dense vs. sparse). It’s compatible with the Amazon OpenSearch Service. diff --git a/docs-website/docs/optimization/evaluation.mdx b/docs-website/docs/optimization/evaluation.mdx index 79b7af03e3d..72e690719b6 100644 --- a/docs-website/docs/optimization/evaluation.mdx +++ b/docs-website/docs/optimization/evaluation.mdx @@ -51,7 +51,7 @@ Model-based evaluation uses LLMs with prompt instructions or smaller fine-tuned To learn more about our integration with the Ragas and DeepEval evaluation frameworks, head over to the [RagasEvaluator](../pipeline-components/evaluators/ragasevaluator.mdx) and [DeepEvalEvaluator](../pipeline-components/evaluators/deepevalevaluator.mdx) component docs. -To get started using practical examples, check out our evaluation tutorial or the respective cookbooks below. +To get started using practical examples, check out our evaluation tutorial or the respective cookbooks below. ## Additional References diff --git a/docs-website/docs/optimization/evaluation/model-based-evaluation.mdx b/docs-website/docs/optimization/evaluation/model-based-evaluation.mdx index 9e1dbb89c11..e71af8d4d34 100644 --- a/docs-website/docs/optimization/evaluation/model-based-evaluation.mdx +++ b/docs-website/docs/optimization/evaluation/model-based-evaluation.mdx @@ -96,7 +96,7 @@ Faithfulness, also called groundedness, evaluates to what extent a generated ans #### [SASEvaluator](../../pipeline-components/evaluators/sasevaluator.mdx) (Semantic Answer Similarity) -Semantic answer similarity uses a transformer-based, cross-encoder architecture to evaluate the semantic similarity of two answers rather than their lexical overlap. While F1 and EM would both score _one hundred percent_ as sharing zero similarity with _100 %_, SAS is trained to assign a high score to such cases. SAS is particularly useful for seeking out cases where F1 doesn't give a good indication of the validity of a predicted answer. You can read more about SAS in [Semantic Answer Similarity for Evaluating Question-Answering Models paper](https://arxiv.org/abs/2108.06130). +Semantic answer similarity uses a transformer-based, cross-encoder architecture to evaluate the semantic similarity of two answers rather than their lexical overlap. While F1 and EM would both score _one hundred percent_ as sharing zero similarity with _100 %_, SAS is trained to assign a high score to such cases. SAS is particularly useful for seeking out cases where F1 doesn't give a good indication of the validity of a predicted answer. You can read more about SAS in [Semantic Answer Similarity for Evaluating Question-Answering Models paper](https://arxiv.org/abs/2108.06130). ### Evaluation Framework Integrations diff --git a/docs-website/docs/optimization/evaluation/statistical-evaluation.mdx b/docs-website/docs/optimization/evaluation/statistical-evaluation.mdx index 5fc39c80ed2..012972221eb 100644 --- a/docs-website/docs/optimization/evaluation/statistical-evaluation.mdx +++ b/docs-website/docs/optimization/evaluation/statistical-evaluation.mdx @@ -30,20 +30,20 @@ There are two ways of performing model-based evaluation in Haystack, both of whi Recall measures how often the correct document was among the retrieved documents over a set of queries. For a single query, the output is binary: either the correct document is contained in the selection, or it is not. Over the entire dataset, the recall score amounts to a number between zero (no query retrieved the right document) and one (all queries retrieved the right documents). -In some scenarios, there can be multiple correct documents for one query. The metric `recall_single_hit` considers whether at least one of the correct documents is retrieved, whereas `recall_multi_hit` takes into account how many of the multiple correct documents for one query are retrieved. +In some scenarios, there can be multiple correct documents for one query. The metric `recall_single_hit` considers whether at least one of the correct documents is retrieved, whereas `recall_multi_hit` takes into account how many of the multiple correct documents for one query are retrieved. -Note that recall is affected by the number of documents that the Retriever returns. If the Retriever returns few documents, it means that it is difficult to retrieve the correct documents. Make sure to set the Retriever's `top_k` to an appropriate value in the pipeline that you're evaluating. +Note that recall is affected by the number of documents that the Retriever returns. If the Retriever returns few documents, it means that it is difficult to retrieve the correct documents. Make sure to set the Retriever's `top_k` to an appropriate value in the pipeline that you're evaluating. ### [DocumentMRREvaluator](../../pipeline-components/evaluators/documentmrrevaluator.mdx) (Mean Reciprocal Rank) -In contrast to the recall metric, mean reciprocal rank takes the position of the top correctly retrieved document (the “rank”) into account. It does this to account for the fact that a query elicits multiple responses of varying relevance. Like recall, MRR can be a value between zero (no matches) and one (the system retrieved a correct document for all queries as the top result). For more details, check out [Mean Reciprocal Rank wiki page](https://en.wikipedia.org/wiki/Mean_reciprocal_rank). +In contrast to the recall metric, mean reciprocal rank takes the position of the top correctly retrieved document (the “rank”) into account. It does this to account for the fact that a query elicits multiple responses of varying relevance. Like recall, MRR can be a value between zero (no matches) and one (the system retrieved a correct document for all queries as the top result). For more details, check out [Mean Reciprocal Rank wiki page](https://en.wikipedia.org/wiki/Mean_reciprocal_rank). ### [DocumentMAPEvaluator](../../pipeline-components/evaluators/documentmapevaluator.mdx) (Mean Average Precision) -Mean average precision is similar to mean reciprocal rank but takes into account the position of every correctly retrieved document. Like MRR, mAP can be a value between zero (no matches) and one (the system retrieved correct documents for all top results). mAP is particularly useful in cases where there is more than one correct answer to be retrieved. For more details, check out [Mean Average Precision wiki page](https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Mean_average_precision). +Mean average precision is similar to mean reciprocal rank but takes into account the position of every correctly retrieved document. Like MRR, mAP can be a value between zero (no matches) and one (the system retrieved correct documents for all top results). mAP is particularly useful in cases where there is more than one correct answer to be retrieved. For more details, check out [Mean Average Precision wiki page](https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Mean_average_precision). ## Statistical Evaluation of Extracted or Generated Answers ### [AnswerExactMatchEvaluator](../../pipeline-components/evaluators/answerexactmatchevaluator.mdx) -Exact match measures the proportion of cases where the predicted Answer is identical to the correct Answer. For example, for the annotated question-answer pair “What is Haystack?" + "A question answering library in Python”, even a predicted answer like “A Python question answering library” would yield a zero score because it does not match the expected answer 100%. \ No newline at end of file +Exact match measures the proportion of cases where the predicted Answer is identical to the correct Answer. For example, for the annotated question-answer pair “What is Haystack?" + "A question answering library in Python”, even a predicted answer like “A Python question answering library” would yield a zero score because it does not match the expected answer 100%. diff --git a/docs-website/docs/overview/migration.mdx b/docs-website/docs/overview/migration.mdx index b3984016ab1..6a66bf7c561 100644 --- a/docs-website/docs/overview/migration.mdx +++ b/docs-website/docs/overview/migration.mdx @@ -194,7 +194,7 @@ If you need help migrating a 1.x node without a 2.x counterpart, open an [issue] | Docs2Answers | Converts Documents into Answers. **Example usage:** When using REST API for document retrieval. REST API expects Answer as output, you can use `Doc2Answer` as the last node to convert the retrieved documents to answers. | [AnswerBuilder](../pipeline-components/builders/answerbuilder.mdx) | | JoinAnswers | Takes answers returned by multiple components and joins them in a single list of answers. **Example usage:** For running queries on different document types (for example, tables and text), where the documents are routed to different readers, and each reader returns a separate list of answers. | [AnswerJoiner](../pipeline-components/joiners/answerjoiner.mdx) | | JoinDocuments | Takes documents returned by different components and joins them to form one list of documents. **Example usage:** In document retrieval pipelines, where there are different types of documents, each routed to a different Retriever. Each Retriever returns a separate list of documents, and you can join them into one list using `JoinDocuments`. | [DocumentJoiner](../pipeline-components/joiners/documentjoiner.mdx) | -| Shaper | Currently functions mostly as `PromptNode` helper making sure the `PromptNode` input or output is correct. **Example usage:** In a question answering pipeline using `PromptNode`, where the `PromptTemplate` expects questions as input, while Haystack pipelines use query. You can use Shaper to rename queries to questions. | Prompt [Builders](../pipeline-components/builders.mdx) | +| Shaper | Currently functions mostly as `PromptNode` helper making sure the `PromptNode` input or output is correct. **Example usage:** In a question answering pipeline using `PromptNode`, where the `PromptTemplate` expects questions as input, while Haystack pipelines use query. You can use Shaper to rename queries to questions. | Prompt [Builders](../pipeline-components/builders.mdx) | | Summarizer | Creates an overview of a document. **Example usage:** To get a glimpse of the documents the Retriever is returning. | Prompt [Builders](../pipeline-components/builders.mdx) with dedicated prompt, [Generators](../pipeline-components/generators.mdx) | | TransformersImageToText | Generates captions for images. **Example usage:** Automatically generate captions for a list of images that you can later use in your knowledge base. | [VertexAIImageQA](../pipeline-components/generators/vertexaiimageqa.mdx) | | Translator | Translates text from one language into another. **Example usage:** Running searches on documents in other languages. | Prompt [Builders](../pipeline-components/builders.mdx) with dedicated prompt, [Generators](../pipeline-components/generators.mdx) | diff --git a/docs-website/docs/pipeline-components/builders/chatpromptbuilder.mdx b/docs-website/docs/pipeline-components/builders/chatpromptbuilder.mdx index cce7b879bd8..d4368b3fd54 100644 --- a/docs-website/docs/pipeline-components/builders/chatpromptbuilder.mdx +++ b/docs-website/docs/pipeline-components/builders/chatpromptbuilder.mdx @@ -29,7 +29,7 @@ The `ChatPromptBuilder` component creates prompts using static or dynamic templa To use it, start by providing a list of `ChatMessage` objects or a special string as the template. -[`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) is a data class that includes message content, a role (who generated the message, such as `user`, `assistant`, `system`, `tool`), and optional metadata. +[`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) is a data class that includes message content, a role (who generated the message, such as `user`, `assistant`, `system`, `tool`), and optional metadata. The builder looks for placeholders in the template and identifies the required variables. You can also list these variables manually. During runtime, the `run` method takes the template and the variables, fills in the placeholders, and returns the completed prompt. If required variables are missing. If the template is invalid, the builder raises an error. diff --git a/docs-website/docs/pipeline-components/connectors/jinareaderconnector.mdx b/docs-website/docs/pipeline-components/connectors/jinareaderconnector.mdx index 17690458438..ec22e149370 100644 --- a/docs-website/docs/pipeline-components/connectors/jinareaderconnector.mdx +++ b/docs-website/docs/pipeline-components/connectors/jinareaderconnector.mdx @@ -42,13 +42,13 @@ You can additionally control the response format from the Jina Reader API using ### Authorization -The component uses a `JINA_API_KEY` environment variable by default. Otherwise, you can pass a Jina API key at initialization with `api_key` like this: +The component uses a `JINA_API_KEY` environment variable by default. Otherwise, you can pass a Jina API key at initialization with `api_key` like this: ```python ranker = JinaRanker(api_key=Secret.from_token("")) ``` -To get your API key, head to Jina AI’s [website](https://jina.ai/reranker/). +To get your API key, head to Jina AI’s [website](https://jina.ai/reranker/). ### Installation diff --git a/docs-website/docs/pipeline-components/connectors/langfuseconnector.mdx b/docs-website/docs/pipeline-components/connectors/langfuseconnector.mdx index 2ff3eae835f..c0963f6b0c0 100644 --- a/docs-website/docs/pipeline-components/connectors/langfuseconnector.mdx +++ b/docs-website/docs/pipeline-components/connectors/langfuseconnector.mdx @@ -47,7 +47,7 @@ These are the things that you need before working with LangfuseConnector: ### Installation -First, install `langfuse-haystack` package to use the `LangfuseConnector`: +First, install `langfuse-haystack` package to use the `LangfuseConnector`: ```shell pip install langfuse-haystack diff --git a/docs-website/docs/pipeline-components/converters/azureocrdocumentconverter.mdx b/docs-website/docs/pipeline-components/converters/azureocrdocumentconverter.mdx index 593f7b15d92..e0af422e037 100644 --- a/docs-website/docs/pipeline-components/converters/azureocrdocumentconverter.mdx +++ b/docs-website/docs/pipeline-components/converters/azureocrdocumentconverter.mdx @@ -27,11 +27,11 @@ description: "`AzureOCRDocumentConverter` converts files to documents using Azur `AzureOCRDocumentConverter` takes a list of file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects as input and uses Azure services to convert the files to a list of documents. Optionally, metadata can be attached to the documents through the `meta` input parameter. You need an active Azure account and a Document Intelligence or Cognitive Services resource to use this integration. Follow the steps described in the Azure [documentation](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/quickstarts/get-started-sdks-rest-api) to set up your resource. -The component uses an `AZURE_AI_API_KEY` environment variable by default. Otherwise, you can pass an `api_key` at initialization – see code examples below. +The component uses an `AZURE_AI_API_KEY` environment variable by default. Otherwise, you can pass an `api_key` at initialization – see code examples below. When you initialize the component, you can optionally set the `model_id`, which refers to the model you want to use. Please refer to [Azure documentation](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/choose-model-feature) for a list of available models. The default model is `"prebuilt-read"`. -The `AzureOCRDocumentConverter` doesn’t extract the tables from a file as plain text but generates separate `Document` objects of type `table` that maintain the two-dimensional structure of the tables. +The `AzureOCRDocumentConverter` doesn’t extract the tables from a file as plain text but generates separate `Document` objects of type `table` that maintain the two-dimensional structure of the tables. ## Usage diff --git a/docs-website/docs/pipeline-components/converters/docxtodocument.mdx b/docs-website/docs/pipeline-components/converters/docxtodocument.mdx index f22cdb39e1a..f02ff77748e 100644 --- a/docs-website/docs/pipeline-components/converters/docxtodocument.mdx +++ b/docs-website/docs/pipeline-components/converters/docxtodocument.mdx @@ -24,7 +24,7 @@ Convert DOCX files to documents. ## Overview -The `DOCXToDocument` component converts DOCX files into documents. It takes a list of file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects as input and outputs the converted result as a list of documents. By defining the table format (CSV or Markdown), you can use this component to extract tables in your DOCX files. Optionally, you can attach metadata to the documents through the `meta` input parameter. +The `DOCXToDocument` component converts DOCX files into documents. It takes a list of file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects as input and outputs the converted result as a list of documents. By defining the table format (CSV or Markdown), you can use this component to extract tables in your DOCX files. Optionally, you can attach metadata to the documents through the `meta` input parameter. ## Usage diff --git a/docs-website/docs/pipeline-components/converters/msgtodocument.mdx b/docs-website/docs/pipeline-components/converters/msgtodocument.mdx index 7c38ca4c1d3..6ef03d07aeb 100644 --- a/docs-website/docs/pipeline-components/converters/msgtodocument.mdx +++ b/docs-website/docs/pipeline-components/converters/msgtodocument.mdx @@ -28,7 +28,7 @@ The `MSGToDocument` component converts Microsoft Outlook `.msg` files into docum ## Usage -First, install the `python-oxmsg` package to start using this converter: +First, install the `python-oxmsg` package to start using this converter: ``` pip install python-oxmsg diff --git a/docs-website/docs/pipeline-components/converters/multifileconverter.mdx b/docs-website/docs/pipeline-components/converters/multifileconverter.mdx index d4fdfc2e002..c339ff5d9dc 100644 --- a/docs-website/docs/pipeline-components/converters/multifileconverter.mdx +++ b/docs-website/docs/pipeline-components/converters/multifileconverter.mdx @@ -13,9 +13,9 @@ Converts CSV, DOCX, HTML, JSON, MD, PPTX, PDF, TXT, and XSLX files to documents. | | | | --- | --- | -| **Most common position in a pipeline** | Before PreProcessors , or right at the beginning of an indexing pipeline | -| **Mandatory run variables** | `sources`: A list of file paths or ByteStream objects | -| **Output variables** | `documents`: A list of converted documents

`unclassified`: A list of uncategorized file paths or byte streams | +| **Most common position in a pipeline** | Before PreProcessors , or right at the beginning of an indexing pipeline | +| **Mandatory run variables** | `sources`: A list of file paths or ByteStream objects | +| **Output variables** | `documents`: A list of converted documents

`unclassified`: A list of uncategorized file paths or byte streams | | **API reference** | [Converters](/reference/converters-api) | | **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/converters/multi_file_converter.py | | **Package name** | `haystack-ai` | @@ -30,7 +30,7 @@ It is a SuperComponent that combines a [`FileTypeRouter`](../routers/filetyperou ### Parameters -To initialize `MultiFileConverter`, there are no mandatory parameters. Optionally, you can provide `encoding` and `json_content_key` parameters. +To initialize `MultiFileConverter`, there are no mandatory parameters. Optionally, you can provide `encoding` and `json_content_key` parameters. The `json_content_key` parameter lets you specify for the JSON files which key in the extracted data will be the document's content. The parameter is passed on to the underlying [`JSONConverter`](jsonconverter.mdx) component. @@ -38,7 +38,7 @@ The `encoding` parameter lets you specify the default encoding of the TXT, CSV, ## Usage -Install dependencies for all supported file types to use the `MultiFileConverter`: +Install dependencies for all supported file types to use the `MultiFileConverter`: ```shell pip install pypdf markdown-it-py mdit_plain trafilatura python-pptx python-docx jq openpyxl tabulate pandas @@ -55,7 +55,7 @@ converter.run(sources=["test.txt", "test.pdf"], meta={}) ### In a pipeline -You can also use `MultiFileConverter` in your indexing pipeline. +You can also use `MultiFileConverter` in your indexing pipeline. ```python from haystack import Pipeline diff --git a/docs-website/docs/pipeline-components/converters/pdfminertodocument.mdx b/docs-website/docs/pipeline-components/converters/pdfminertodocument.mdx index 133b8563d7b..0057a9b073b 100644 --- a/docs-website/docs/pipeline-components/converters/pdfminertodocument.mdx +++ b/docs-website/docs/pipeline-components/converters/pdfminertodocument.mdx @@ -24,9 +24,9 @@ A component that converts complex PDF files to documents using pdfminer argument ## Overview -The `PDFMinerToDocument` component converts PDF files into documents using [PDFMiner](https://pdfminersix.readthedocs.io/en/latest/) extraction tool arguments. +The `PDFMinerToDocument` component converts PDF files into documents using [PDFMiner](https://pdfminersix.readthedocs.io/en/latest/) extraction tool arguments. -You can use it in an indexing pipeline to index the contents of a PDF file in a Document Store. It takes a list of file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream)objects as input and outputs the converted result as a list of documents. Optionally, you can attach metadata to the documents through the `meta` input parameter. +You can use it in an indexing pipeline to index the contents of a PDF file in a Document Store. It takes a list of file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream)objects as input and outputs the converted result as a list of documents. Optionally, you can attach metadata to the documents through the `meta` input parameter. When initializing the component, you can adjust several parameters to fit your PDF. See the full parameter list and descriptions in our [API reference](/reference/converters-api#pdfminertodocument). diff --git a/docs-website/docs/pipeline-components/converters/pptxtodocument.mdx b/docs-website/docs/pipeline-components/converters/pptxtodocument.mdx index c1d33120f5f..da24b09338c 100644 --- a/docs-website/docs/pipeline-components/converters/pptxtodocument.mdx +++ b/docs-website/docs/pipeline-components/converters/pptxtodocument.mdx @@ -24,7 +24,7 @@ Convert PPTX files to documents. ## Overview -The `PPTXToDocument` component converts PPTX files into documents. It takes a list of file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects as input and outputs the converted result as a list of documents. Optionally, you can attach metadata to the documents through the `meta` input parameter. +The `PPTXToDocument` component converts PPTX files into documents. It takes a list of file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects as input and outputs the converted result as a list of documents. Optionally, you can attach metadata to the documents through the `meta` input parameter. ## Usage diff --git a/docs-website/docs/pipeline-components/converters/unstructuredfileconverter.mdx b/docs-website/docs/pipeline-components/converters/unstructuredfileconverter.mdx index 07b00ca2aea..ac8b0033f3a 100644 --- a/docs-website/docs/pipeline-components/converters/unstructuredfileconverter.mdx +++ b/docs-website/docs/pipeline-components/converters/unstructuredfileconverter.mdx @@ -26,7 +26,7 @@ Use this component to convert text files and directories to a document. `UnstructuredFileConverter` converts files and directories into documents using the Unstructured API. -[Unstructured](https://docs.unstructured.io/) provides a series of tools to do ETL for LLMs. The `UnstructuredFileConverter` calls the Unstructured API that extracts text and other information from a vast range of file [formats](https://docs.unstructured.io/api-reference/api-services/overview#supported-file-types). +[Unstructured](https://docs.unstructured.io/) provides a series of tools to do ETL for LLMs. The `UnstructuredFileConverter` calls the Unstructured API that extracts text and other information from a vast range of file [formats](https://docs.unstructured.io/api-reference/api-services/overview#supported-file-types). This Converter supports different modes for creating documents from the elements returned by Unstructured: @@ -56,7 +56,7 @@ There are free and paid versions of Unstructured API: **Free Unstructured API** > ❗️ The API keys for the free and paid versions are different and cannot be used interchangeably. -Regardless of the chosen tier, we recommend to set the Unstructured API key as an environment variable `UNSTRUCTURED_API_KEY`: +Regardless of the chosen tier, we recommend to set the Unstructured API key as an environment variable `UNSTRUCTURED_API_KEY`: ```shell export UNSTRUCTURED_API_KEY=your_api_key diff --git a/docs-website/docs/pipeline-components/converters/xlsxtodocument.mdx b/docs-website/docs/pipeline-components/converters/xlsxtodocument.mdx index 279362196ed..4f17b203669 100644 --- a/docs-website/docs/pipeline-components/converters/xlsxtodocument.mdx +++ b/docs-website/docs/pipeline-components/converters/xlsxtodocument.mdx @@ -24,7 +24,7 @@ Converts Excel files into documents. ## Overview -The `XLSXToDocument` component converts XLSX files into Haystack Documents with a CSV (default) or Markdown format. It takes a list of file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects as input and outputs the converted result as a list of documents. Optionally, you can attach metadata to the documents through the `meta` input parameter. +The `XLSXToDocument` component converts XLSX files into Haystack Documents with a CSV (default) or Markdown format. It takes a list of file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects as input and outputs the converted result as a list of documents. Optionally, you can attach metadata to the documents through the `meta` input parameter. To see the additional parameters that you can specify with the component initialization, check out the [API Reference](/reference/converters-api#xlsxtodocument). diff --git a/docs-website/docs/pipeline-components/embedders/amazonbedrockdocumentembedder.mdx b/docs-website/docs/pipeline-components/embedders/amazonbedrockdocumentembedder.mdx index 41722672aef..b256b548f26 100644 --- a/docs-website/docs/pipeline-components/embedders/amazonbedrockdocumentembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/amazonbedrockdocumentembedder.mdx @@ -25,7 +25,7 @@ This component computes embeddings for documents using models through Amazon Bed ## Overview -[Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) is a fully managed service that makes language models from leading AI startups and Amazon available for your use through a unified API. +[Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) is a fully managed service that makes language models from leading AI startups and Amazon available for your use through a unified API. Supported models are `amazon.titan-embed-text-v1`, `cohere.embed-english-v3`, `cohere.embed-multilingual-v3`, and `amazon.titan-embed-text-v2:0`. @@ -39,7 +39,7 @@ This component should be used to embed a list of documents. To embed a string, y ### Authentication `AmazonBedrockDocumentEmbedder` uses AWS for authentication. You can either provide credentials as parameters directly to the component or use the AWS CLI and authenticate through your IAM. For more information on how to set up an IAM identity-based policy, see the [official documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html). -To initialize `AmazonBedrockDocumentEmbedder` and authenticate by providing credentials, provide the `model_name`, as well as `aws_access_key_id`, `aws_secret_access_key` and `aws_region_name`. Other parameters are optional. You can check them out in our [API reference](/reference/integrations-amazon-bedrock#amazonbedrockdocumentembedder). +To initialize `AmazonBedrockDocumentEmbedder` and authenticate by providing credentials, provide the `model_name`, as well as `aws_access_key_id`, `aws_secret_access_key` and `aws_region_name`. Other parameters are optional. You can check them out in our [API reference](/reference/integrations-amazon-bedrock#amazonbedrockdocumentembedder). ### Model-specific parameters diff --git a/docs-website/docs/pipeline-components/embedders/amazonbedrockdocumentimageembedder.mdx b/docs-website/docs/pipeline-components/embedders/amazonbedrockdocumentimageembedder.mdx index 9dd468495c2..0f6947692d3 100644 --- a/docs-website/docs/pipeline-components/embedders/amazonbedrockdocumentimageembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/amazonbedrockdocumentimageembedder.mdx @@ -13,8 +13,8 @@ description: "`AmazonBedrockDocumentImageEmbedder` computes image embeddings for | | | | --- | --- | -| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline | -| **Mandatory init variables** | `model`: The multimodal embedding model to use.

`aws_access_key_id`: AWS access key ID. Can be set with `AWS_ACCESS_KEY_ID` env var.

`aws_secret_access_key`: AWS secret access key. Can be set with `AWS_SECRET_ACCESS_KEY` env var.

`aws_region_name`: AWS region name. Can be set with `AWS_DEFAULT_REGION` env var. | +| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline | +| **Mandatory init variables** | `model`: The multimodal embedding model to use.

`aws_access_key_id`: AWS access key ID. Can be set with `AWS_ACCESS_KEY_ID` env var.

`aws_secret_access_key`: AWS secret access key. Can be set with `AWS_SECRET_ACCESS_KEY` env var.

`aws_region_name`: AWS region name. Can be set with `AWS_DEFAULT_REGION` env var. | | **Mandatory run variables** | `documents`: A list of documents, with a meta field containing an image file path | | **Output variables** | `documents`: A list of documents (enriched with embeddings) | | **API reference** | [Amazon Bedrock](/reference/integrations-amazon-bedrock) | @@ -45,9 +45,9 @@ pip install amazon-bedrock-haystack ### Authentication -`AmazonBedrockDocumentImageEmbedder` uses AWS for authentication. You can either provide credentials as parameters directly to the component or use the AWS CLI and authenticate through your IAM. For more information on how to set up an IAM identity-based policy, see the [official documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html). +`AmazonBedrockDocumentImageEmbedder` uses AWS for authentication. You can either provide credentials as parameters directly to the component or use the AWS CLI and authenticate through your IAM. For more information on how to set up an IAM identity-based policy, see the [official documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html). -To initialize `AmazonBedrockDocumentImageEmbedder` and authenticate by providing credentials, provide the `model` name, as well as `aws_access_key_id`, `aws_secret_access_key`, and `aws_region_name`. Other parameters are optional, you can check them out in our [API reference](/reference/integrations-amazon-bedrock#amazonbedrocktextembedder). +To initialize `AmazonBedrockDocumentImageEmbedder` and authenticate by providing credentials, provide the `model` name, as well as `aws_access_key_id`, `aws_secret_access_key`, and `aws_region_name`. Other parameters are optional, you can check them out in our [API reference](/reference/integrations-amazon-bedrock#amazonbedrocktextembedder). ### Model-specific parameters diff --git a/docs-website/docs/pipeline-components/embedders/amazonbedrocktextembedder.mdx b/docs-website/docs/pipeline-components/embedders/amazonbedrocktextembedder.mdx index 7a040dee523..cbdd306331a 100644 --- a/docs-website/docs/pipeline-components/embedders/amazonbedrocktextembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/amazonbedrocktextembedder.mdx @@ -25,16 +25,16 @@ This component computes embeddings for text (such as a query) using models throu ## Overview -[Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) is a fully managed service that makes language models from leading AI startups and Amazon available for your use through a unified API. +[Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) is a fully managed service that makes language models from leading AI startups and Amazon available for your use through a unified API. Supported models are `amazon.titan-embed-text-v1`, `cohere.embed-english-v3` and `cohere.embed-multilingual-v3`. -Use `AmazonBedrockTextEmbedder` to embed a simple string (such as a query) into a vector. Use the [`AmazonBedrockDocumentEmbedder`](amazonbedrockdocumentembedder.mdx) to enrich the documents with the computed embedding, also known as vector. +Use `AmazonBedrockTextEmbedder` to embed a simple string (such as a query) into a vector. Use the [`AmazonBedrockDocumentEmbedder`](amazonbedrockdocumentembedder.mdx) to enrich the documents with the computed embedding, also known as vector. ### Authentication `AmazonBedrockTextEmbedder` uses AWS for authentication. You can either provide credentials as parameters directly to the component or use the AWS CLI and authenticate through your IAM. For more information on how to set up an IAM identity-based policy, see the [official documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html). -To initialize `AmazonBedrockTextEmbedder` and authenticate by providing credentials, provide the `model` name, as well as `aws_access_key_id`, `aws_secret_access_key`, and `aws_region_name`. Other parameters are optional, you can check them out in our [API reference](/reference/integrations-amazon-bedrock#amazonbedrocktextembedder). +To initialize `AmazonBedrockTextEmbedder` and authenticate by providing credentials, provide the `model` name, as well as `aws_access_key_id`, `aws_secret_access_key`, and `aws_region_name`. Other parameters are optional, you can check them out in our [API reference](/reference/integrations-amazon-bedrock#amazonbedrocktextembedder). ### Model-specific parameters diff --git a/docs-website/docs/pipeline-components/embedders/azureopenaidocumentembedder.mdx b/docs-website/docs/pipeline-components/embedders/azureopenaidocumentembedder.mdx index 3e9f7530b94..ec7b7f1ff52 100644 --- a/docs-website/docs/pipeline-components/embedders/azureopenaidocumentembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/azureopenaidocumentembedder.mdx @@ -33,7 +33,7 @@ This component should be used to embed a list of documents. To embed a string, y To work with Azure components, you will need an Azure OpenAI API key, as well as an Azure OpenAI Endpoint. You can learn more about them in Azure [documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference). -The component uses `AZURE_OPENAI_API_KEY` or `AZURE_OPENAI_AD_TOKEN` environment variables by default. Otherwise, you can pass `api_key` or `azure_ad_token` at initialization: +The component uses `AZURE_OPENAI_API_KEY` or `AZURE_OPENAI_AD_TOKEN` environment variables by default. Otherwise, you can pass `api_key` or `azure_ad_token` at initialization: ```python client = AzureOpenAIDocumentEmbedder( diff --git a/docs-website/docs/pipeline-components/embedders/azureopenaitextembedder.mdx b/docs-website/docs/pipeline-components/embedders/azureopenaitextembedder.mdx index 9db3d57ad43..4a6f501f4ab 100644 --- a/docs-website/docs/pipeline-components/embedders/azureopenaitextembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/azureopenaitextembedder.mdx @@ -33,7 +33,7 @@ Use `AzureOpenAITextEmbedder` to embed a simple string (such as a query) into a To work with Azure components, you will need an Azure OpenAI API key, as well as an Azure OpenAI Endpoint. You can learn more about them in Azure [documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference). -The component uses `AZURE_OPENAI_API_KEY` or `AZURE_OPENAI_AD_TOKEN` environment variables by default. Otherwise, you can pass `api_key` or `azure_ad_token` at initialization: +The component uses `AZURE_OPENAI_API_KEY` or `AZURE_OPENAI_AD_TOKEN` environment variables by default. Otherwise, you can pass `api_key` or `azure_ad_token` at initialization: ```python client = AzureOpenAITextEmbedder( diff --git a/docs-website/docs/pipeline-components/embedders/coheredocumentembedder.mdx b/docs-website/docs/pipeline-components/embedders/coheredocumentembedder.mdx index e7dec100520..ded2a422068 100644 --- a/docs-website/docs/pipeline-components/embedders/coheredocumentembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/coheredocumentembedder.mdx @@ -40,7 +40,7 @@ To start using this integration with Haystack, install it with: pip install cohere-haystack ``` -The component uses a `COHERE_API_KEY` or `CO_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`: +The component uses a `COHERE_API_KEY` or `CO_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`: ```python embedder = CohereDocumentEmbedder(api_key=Secret.from_token("")) diff --git a/docs-website/docs/pipeline-components/embedders/coheredocumentimageembedder.mdx b/docs-website/docs/pipeline-components/embedders/coheredocumentimageembedder.mdx index 8220dfb5b9c..f39afd6a3cf 100644 --- a/docs-website/docs/pipeline-components/embedders/coheredocumentimageembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/coheredocumentimageembedder.mdx @@ -13,8 +13,8 @@ description: "`CohereDocumentImageEmbedder` computes the image embeddings of a l | | | | --- | --- | -| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline | -| **Mandatory init variables** | `api_key`: The Cohere API key. Can be set with `COHERE_API_KEY` or `CO_API_KEY` env var. | +| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline | +| **Mandatory init variables** | `api_key`: The Cohere API key. Can be set with `COHERE_API_KEY` or `CO_API_KEY` env var. | | **Mandatory run variables** | `documents`: A list of documents, with a meta field containing an image file path | | **Output variables** | `documents`: A list of documents (enriched with embeddings) | | **API reference** | [Cohere](/reference/integrations-cohere) | @@ -43,19 +43,19 @@ pip install cohere-haystack ### Authentication -The component uses a `COHERE_API_KEY` or `CO_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with a [Secret](../../concepts/secret-management.mdx) and `Secret.from_token`  method: +The component uses a `COHERE_API_KEY` or `CO_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with a [Secret](../../concepts/secret-management.mdx) and `Secret.from_token` method: ```python embedder = CohereTextEmbedder(api_key=Secret.from_token("")) ``` -To get a Cohere API key, head over to https://cohere.com/. +To get a Cohere API key, head over to https://cohere.com/. ## Usage ### On its own -Remember to set `COHERE_API_KEY` as an environment variable first. +Remember to set `COHERE_API_KEY` as an environment variable first. ```python from haystack import Document diff --git a/docs-website/docs/pipeline-components/embedders/coheretextembedder.mdx b/docs-website/docs/pipeline-components/embedders/coheretextembedder.mdx index 249b2cdc597..901f033d723 100644 --- a/docs-website/docs/pipeline-components/embedders/coheretextembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/coheretextembedder.mdx @@ -14,7 +14,7 @@ This component transforms a string into a vector that captures its semantics usi | | | | --- | --- | | **Most common position in a pipeline** | Before an embedding [Retriever](../retrievers.mdx) in a query/RAG pipeline | -| **Mandatory init variables** | `api_key`: The Cohere API key. Can be set with `COHERE_API_KEY` or `CO_API_KEY` env var. | +| **Mandatory init variables** | `api_key`: The Cohere API key. Can be set with `COHERE_API_KEY` or `CO_API_KEY` env var. | | **Mandatory run variables** | `text`: A string | | **Output variables** | `embedding`: A list of float numbers (vectors)

`meta`: A dictionary of metadata strings | | **API reference** | [Cohere](/reference/integrations-cohere) | @@ -38,7 +38,7 @@ To start using this integration with Haystack, install it with: pip install cohere-haystack ``` -The component uses a `COHERE_API_KEY` or `CO_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with a [Secret](../../concepts/secret-management.mdx) and `Secret.from_token` static method: +The component uses a `COHERE_API_KEY` or `CO_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with a [Secret](../../concepts/secret-management.mdx) and `Secret.from_token` static method: ```python embedder = CohereTextEmbedder(api_key=Secret.from_token("")) diff --git a/docs-website/docs/pipeline-components/embedders/huggingfaceapidocumentembedder.mdx b/docs-website/docs/pipeline-components/embedders/huggingfaceapidocumentembedder.mdx index d35f8ef3c34..e65cb1da30f 100644 --- a/docs-website/docs/pipeline-components/embedders/huggingfaceapidocumentembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/huggingfaceapidocumentembedder.mdx @@ -13,7 +13,7 @@ Use this component to compute document embeddings using various Hugging Face API | | | | --- | --- | -| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx)  in an indexing pipeline | +| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline | | **Mandatory init variables** | `api_type`: The type of Hugging Face API to use

`api_params`: A dictionary with one of the following keys:

- `model`: Hugging Face model ID. Required when `api_type` is `SERVERLESS_INFERENCE_API`.**OR** - `url`: URL of the inference endpoint. Required when `api_type` is `INFERENCE_ENDPOINTS` or `TEXT_EMBEDDINGS_INFERENCE`.

`token`: The Hugging Face API token. Can be set with `HF_API_TOKEN` or `HF_TOKEN` env var. | | **Mandatory run variables** | `documents`: A list of documents to be embedded | | **Output variables** | `documents`: A list of documents to be embedded (enriched with embeddings) | @@ -35,7 +35,7 @@ Use this component to compute document embeddings using various Hugging Face API This component should be used to embed a list of documents. To embed a string, use [`HuggingFaceAPITextEmbedder`](huggingfaceapitextembedder.mdx). ::: -The component uses a `HF_API_TOKEN` environment variable by default. Otherwise, you can pass a Hugging Face API token at initialization with `token` – see code examples below. +The component uses a `HF_API_TOKEN` environment variable by default. Otherwise, you can pass a Hugging Face API token at initialization with `token` – see code examples below. The token is needed: - If you use the Serverless Inference API, or diff --git a/docs-website/docs/pipeline-components/embedders/huggingfaceapitextembedder.mdx b/docs-website/docs/pipeline-components/embedders/huggingfaceapitextembedder.mdx index 436c92512b5..4771689db5e 100644 --- a/docs-website/docs/pipeline-components/embedders/huggingfaceapitextembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/huggingfaceapitextembedder.mdx @@ -35,7 +35,7 @@ Use this component to embed strings using various Hugging Face APIs. This component should be used to embed plain text. To embed a list of documents, use [`HuggingFaceAPIDocumentEmbedder`](huggingfaceapidocumentembedder.mdx). ::: -The component uses a `HF_API_TOKEN` environment variable by default. Otherwise, you can pass a Hugging Face API token at initialization with `token` – see code examples below. +The component uses a `HF_API_TOKEN` environment variable by default. Otherwise, you can pass a Hugging Face API token at initialization with `token` – see code examples below. The token is needed: - If you use the Serverless Inference API, or diff --git a/docs-website/docs/pipeline-components/embedders/jinadocumentimageembedder.mdx b/docs-website/docs/pipeline-components/embedders/jinadocumentimageembedder.mdx index 595f5a9dfac..64118b5e867 100644 --- a/docs-website/docs/pipeline-components/embedders/jinadocumentimageembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/jinadocumentimageembedder.mdx @@ -13,8 +13,8 @@ description: "`JinaDocumentImageEmbedder` computes the image embeddings of a lis | | | | --- | --- | -| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline | -| **Mandatory init variables** | `api_key`: The Jina API key. Can be set with `JINA_API_KEY` env var. | +| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline | +| **Mandatory init variables** | `api_key`: The Jina API key. Can be set with `JINA_API_KEY` env var. | | **Mandatory run variables** | `documents`: A list of documents, with a meta field containing an image file path | | **Output variables** | `documents`: A list of documents (enriched with embeddings) | | **API reference** | [Jina](/reference/integrations-jina) | @@ -47,19 +47,19 @@ pip install jina-haystack ### Authentication -The component uses a `JINA_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with a [Secret](../../concepts/secret-management.mdx) and `Secret.from_token`  method: +The component uses a `JINA_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with a [Secret](../../concepts/secret-management.mdx) and `Secret.from_token` method: ```python embedder = JinaDocumentImageEmbedder(api_key=Secret.from_token("")) ``` -To get a Cohere API key, head over to https://jina.ai/embeddings/. +To get a Cohere API key, head over to https://jina.ai/embeddings/. ## Usage ### On its own -Remember to set `JINA_API_KEY` as an environment variable first. +Remember to set `JINA_API_KEY` as an environment variable first. ```python from haystack import Document diff --git a/docs-website/docs/pipeline-components/embedders/mistraldocumentembedder.mdx b/docs-website/docs/pipeline-components/embedders/mistraldocumentembedder.mdx index 7de70fa54a3..b8a413a4861 100644 --- a/docs-website/docs/pipeline-components/embedders/mistraldocumentembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/mistraldocumentembedder.mdx @@ -37,7 +37,7 @@ To start using this integration with Haystack, install it with: pip install mistral-haystack ``` -`MistralDocumentEmbedder` needs a Mistral API key to work. It uses an `MISTRAL_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`: +`MistralDocumentEmbedder` needs a Mistral API key to work. It uses an `MISTRAL_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`: ```python embedder = MistralDocumentEmbedder( diff --git a/docs-website/docs/pipeline-components/embedders/mistraltextembedder.mdx b/docs-website/docs/pipeline-components/embedders/mistraltextembedder.mdx index bab018a7498..155c5990842 100644 --- a/docs-website/docs/pipeline-components/embedders/mistraltextembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/mistraltextembedder.mdx @@ -37,7 +37,7 @@ To start using this integration with Haystack, install it with: pip install mistral-haystack ``` -`MistralTextEmbedder` needs a Mistral API key to work. It uses a `MISTRAL_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`: +`MistralTextEmbedder` needs a Mistral API key to work. It uses a `MISTRAL_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`: ```python embedder = MistralTextEmbedder( diff --git a/docs-website/docs/pipeline-components/embedders/openaidocumentembedder.mdx b/docs-website/docs/pipeline-components/embedders/openaidocumentembedder.mdx index 66775d90bca..4d67118dd68 100644 --- a/docs-website/docs/pipeline-components/embedders/openaidocumentembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/openaidocumentembedder.mdx @@ -31,7 +31,7 @@ To see the list of compatible OpenAI embedding models, head over to OpenAI [docu This component should be used to embed a list of documents. To embed a string, use the [OpenAITextEmbedder](openaitextembedder.mdx). -The component uses an `OPENAI_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`: +The component uses an `OPENAI_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`: ``` embedder = OpenAIDocumentEmbedder(api_key=Secret.from_token("")) diff --git a/docs-website/docs/pipeline-components/embedders/openaitextembedder.mdx b/docs-website/docs/pipeline-components/embedders/openaitextembedder.mdx index c2ad6425d54..3ba9b928d45 100644 --- a/docs-website/docs/pipeline-components/embedders/openaitextembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/openaitextembedder.mdx @@ -31,7 +31,7 @@ To see the list of compatible OpenAI embedding models, head over to OpenAI [docu Use `OpenAITextEmbedder` to embed a simple string (such as a query) into a vector. For embedding lists of documents, use the [OpenAIDocumentEmbedder](openaidocumentembedder.mdx), which enriches the document with the computed embedding, also known as vector. -The component uses an `OPENAI_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`: +The component uses an `OPENAI_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`: ```python embedder = OpenAITextEmbedder(api_key=Secret.from_token("")) diff --git a/docs-website/docs/pipeline-components/embedders/optimumdocumentembedder.mdx b/docs-website/docs/pipeline-components/embedders/optimumdocumentembedder.mdx index 27d0b84868e..d4fa4e33ff4 100644 --- a/docs-website/docs/pipeline-components/embedders/optimumdocumentembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/optimumdocumentembedder.mdx @@ -13,7 +13,7 @@ A component to compute documents’ embeddings using models loaded with the Hugg | | | | --- | --- | -| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx)  in an indexing pipeline | +| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline | | **Mandatory run variables** | `documents`: A list of documents | | **Output variables** | `documents`: A list of documents enriched with embeddings | | **API reference** | [Optimum](/reference/integrations-optimum) | @@ -42,7 +42,7 @@ Find all the available mode details in our Optimum [API Reference](/reference/in Authentication with a Hugging Face API Token is only required to access private or gated models through Serverless Inference API or the Inference Endpoints. -The component uses an `HF_API_TOKEN` or `HF_TOKEN` environment variable, or you can pass a Hugging Face API token at initialization. See our [Secret Management](../../concepts/secret-management.mdx) page for more information. +The component uses an `HF_API_TOKEN` or `HF_TOKEN` environment variable, or you can pass a Hugging Face API token at initialization. See our [Secret Management](../../concepts/secret-management.mdx) page for more information. ## Usage diff --git a/docs-website/docs/pipeline-components/embedders/optimumtextembedder.mdx b/docs-website/docs/pipeline-components/embedders/optimumtextembedder.mdx index 89b24fad927..008b5bf1b6e 100644 --- a/docs-website/docs/pipeline-components/embedders/optimumtextembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/optimumtextembedder.mdx @@ -42,7 +42,7 @@ Find all the available mode details in our Optimum [API Reference](/reference/in Authentication with a Hugging Face API Token is only required to access private or gated models through Serverless Inference API or the Inference Endpoints. -The component uses an `HF_API_TOKEN` or `HF_TOKEN` environment variable, or you can pass a Hugging Face API token at initialization. See our [Secret Management](../../concepts/secret-management.mdx) page for more information. +The component uses an `HF_API_TOKEN` or `HF_TOKEN` environment variable, or you can pass a Hugging Face API token at initialization. See our [Secret Management](../../concepts/secret-management.mdx) page for more information. ## Usage diff --git a/docs-website/docs/pipeline-components/embedders/sentencetransformersdocumentembedder.mdx b/docs-website/docs/pipeline-components/embedders/sentencetransformersdocumentembedder.mdx index 4026598a1b1..e64fe00b80d 100644 --- a/docs-website/docs/pipeline-components/embedders/sentencetransformersdocumentembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/sentencetransformersdocumentembedder.mdx @@ -32,7 +32,7 @@ The vectors computed by this component are necessary to perform embedding retrie Authentication with a Hugging Face API Token is only required to access private or gated models through Serverless Inference API or the Inference Endpoints. -The component uses an `HF_API_TOKEN` or `HF_TOKEN` environment variable, or you can pass a Hugging Face API token at initialization. See our [Secret Management](../../concepts/secret-management.mdx) page for more information. +The component uses an `HF_API_TOKEN` or `HF_TOKEN` environment variable, or you can pass a Hugging Face API token at initialization. See our [Secret Management](../../concepts/secret-management.mdx) page for more information. ```python document_embedder = SentenceTransformersDocumentEmbedder( diff --git a/docs-website/docs/pipeline-components/embedders/sentencetransformersdocumentimageembedder.mdx b/docs-website/docs/pipeline-components/embedders/sentencetransformersdocumentimageembedder.mdx index c4d562d0ebc..6acdc58d479 100644 --- a/docs-website/docs/pipeline-components/embedders/sentencetransformersdocumentimageembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/sentencetransformersdocumentimageembedder.mdx @@ -13,7 +13,7 @@ description: "`SentenceTransformersDocumentImageEmbedder` computes the image emb | | | | --- | --- | -| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline | +| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline | | **Mandatory init variables** | `token` (only for private models): The Hugging Face API token. Can be set with `HF_API_TOKEN` or `HF_TOKEN` env var. | | **Mandatory run variables** | `documents`: A list of documents, with a meta field containing an image file path | | **Output variables** | `documents`: A list of documents (enriched with embeddings) | @@ -39,7 +39,7 @@ Additionally, you can select the backend to use for the Sentence Transformers mo Authentication with a Hugging Face API Token is only required to access private or gated models. -The component uses an `HF_API_TOKEN` or `HF_TOKEN` environment variable, or you can pass a Hugging Face API token at initialization. See our [Secret Management](../../concepts/secret-management.mdx) page for more information. +The component uses an `HF_API_TOKEN` or `HF_TOKEN` environment variable, or you can pass a Hugging Face API token at initialization. See our [Secret Management](../../concepts/secret-management.mdx) page for more information. ### Compatible Models diff --git a/docs-website/docs/pipeline-components/embedders/sentencetransformerstextembedder.mdx b/docs-website/docs/pipeline-components/embedders/sentencetransformerstextembedder.mdx index ef5eecf2b94..2ad9d61d94d 100644 --- a/docs-website/docs/pipeline-components/embedders/sentencetransformerstextembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/sentencetransformerstextembedder.mdx @@ -32,7 +32,7 @@ This component should be used to embed a simple string (such as a query) into a Authentication with a Hugging Face API Token is only required to access private or gated models through Serverless Inference API or the Inference Endpoints. -The component uses an `HF_API_TOKEN` or `HF_TOKEN` environment variable, or you can pass a Hugging Face API token at initialization. See our [Secret Management](../../concepts/secret-management.mdx) page for more information. +The component uses an `HF_API_TOKEN` or `HF_TOKEN` environment variable, or you can pass a Hugging Face API token at initialization. See our [Secret Management](../../concepts/secret-management.mdx) page for more information. ```python text_embedder = SentenceTransformersTextEmbedder( diff --git a/docs-website/docs/pipeline-components/embedders/stackitdocumentembedder.mdx b/docs-website/docs/pipeline-components/embedders/stackitdocumentembedder.mdx index 3449977c5cb..f3347507ff4 100644 --- a/docs-website/docs/pipeline-components/embedders/stackitdocumentembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/stackitdocumentembedder.mdx @@ -13,7 +13,7 @@ This component enables document embedding using the STACKIT API. | | | | --- | --- | -| **Most common position in a pipeline** | Before a [DocumentWriter](../writers/documentwriter.mdx) in an indexing pipeline | +| **Most common position in a pipeline** | Before a [DocumentWriter](../writers/documentwriter.mdx) in an indexing pipeline | | **Mandatory init variables** | `model`: The model used through the STACKIT API | | **Mandatory run variables** | `documents`: A list of documents to be embedded | | **Output variables** | `documents`: A list of documents enriched with embeddings | @@ -31,17 +31,17 @@ This component enables document embedding using the STACKIT API. To use the `STACKITDocumentEmbedder`, ensure you have set a `STACKIT_API_KEY` as an environment variable. Alternatively, provide the API key as an environment variable with a different name or a token by setting `api_key` and using Haystack’s [secret management](../../concepts/secret-management.mdx). -Set your preferred supported model with the `model` parameter when initializing the component. See the full list of all supported models on the [STACKIT website](https://docs.stackit.cloud/stackit/en/models-licenses-319914532.html). +Set your preferred supported model with the `model` parameter when initializing the component. See the full list of all supported models on the [STACKIT website](https://docs.stackit.cloud/stackit/en/models-licenses-319914532.html). Optionally, you can change the default `api_base_url`, which is `"https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1"`. You can pass any text generation parameters valid for the STACKIT Chat Completion API directly to this component with the `generation_kwargs` parameter in the init or run methods. -Then component needs a list of documents as input to operate. +Then component needs a list of documents as input to operate. ## Usage -Install the `stackit-haystack` package to use the `STACKITDocumentEmbedder` and set an environment variable called `STACKIT_API_KEY` to your API key. +Install the `stackit-haystack` package to use the `STACKITDocumentEmbedder` and set an environment variable called `STACKIT_API_KEY` to your API key. ```shell pip install stackit-haystack @@ -64,7 +64,7 @@ print(result["documents"][0].embedding) ### In a pipeline -You can also use `STACKITDocumentEmbedder` in your pipeline in a following way. +You can also use `STACKITDocumentEmbedder` in your pipeline in a following way. ```python from haystack import Document diff --git a/docs-website/docs/pipeline-components/embedders/stackittextembedder.mdx b/docs-website/docs/pipeline-components/embedders/stackittextembedder.mdx index d0211ac127f..47d4a079a91 100644 --- a/docs-website/docs/pipeline-components/embedders/stackittextembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/stackittextembedder.mdx @@ -13,7 +13,7 @@ This component enables text embedding using the STACKIT API. | | | | --- | --- | -| **Most common position in a pipeline** | Before an embedding [Retriever](../retrievers.mdx) in a query/RAG pipeline | +| **Most common position in a pipeline** | Before an embedding [Retriever](../retrievers.mdx) in a query/RAG pipeline | | **Mandatory init variables** | `model`: The model used through the STACKIT API | | **Mandatory run variables** | `text`: A string | | **Output variables** | `embedding`: A list of float numbers | @@ -31,17 +31,17 @@ This component enables text embedding using the STACKIT API. To use the `STACKITTextEmbedder`, ensure you have set a `STACKIT_API_KEY` as an environment variable. Alternatively, provide the API key as an environment variable with a different name or a token by setting `api_key` and using Haystack’s [secret management](../../concepts/secret-management.mdx). -Set your preferred supported model with the `model` parameter when initializing the component. See the full list of all supported models on the [STACKIT website](https://docs.stackit.cloud/stackit/en/models-licenses-319914532.html). +Set your preferred supported model with the `model` parameter when initializing the component. See the full list of all supported models on the [STACKIT website](https://docs.stackit.cloud/stackit/en/models-licenses-319914532.html). Optionally, you can change the default `api_base_url`, which is `"https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1"`. You can pass any text generation parameters valid for the STACKIT Chat Completion API directly to this component with the `generation_kwargs` parameter in the init or run methods. -The component needs a text input to operate. +The component needs a text input to operate. ## Usage -Install the `stackit-haystack` package to use the `STACKITTextEmbedder` and set an environment variable called `STACKIT_API_KEY` to your API key. +Install the `stackit-haystack` package to use the `STACKITTextEmbedder` and set an environment variable called `STACKIT_API_KEY` to your API key. ```shell pip install stackit-haystack @@ -61,7 +61,7 @@ print(text_embedder.run("I love pizza!")) ### In a pipeline -You can also use `STACKITTextEmbedder` in your pipeline. +You can also use `STACKITTextEmbedder` in your pipeline. ```python from haystack import Document diff --git a/docs-website/docs/pipeline-components/embedders/vertexaitextembedder.mdx b/docs-website/docs/pipeline-components/embedders/vertexaitextembedder.mdx index acce411de44..da379fccba6 100644 --- a/docs-website/docs/pipeline-components/embedders/vertexaitextembedder.mdx +++ b/docs-website/docs/pipeline-components/embedders/vertexaitextembedder.mdx @@ -20,7 +20,7 @@ We recommend switching to the new [GoogleGenAITextEmbedder](googlegenaitextembed | | | | --- | --- | -| **Most common position in a pipeline** | Before an embedding [Retriever](../retrievers.mdx) in a query/RAG pipeline | +| **Most common position in a pipeline** | Before an embedding [Retriever](../retrievers.mdx) in a query/RAG pipeline | | **Mandatory init variables** | `model`: The model used through the VertexAI Embeddings API | | **Mandatory run variables** | `text`: A string | | **Output variables** | `embedding`: A list of float numbers | diff --git a/docs-website/docs/pipeline-components/evaluators/documentndcgevaluator.mdx b/docs-website/docs/pipeline-components/evaluators/documentndcgevaluator.mdx index 0a0f5702158..da9ac35040c 100644 --- a/docs-website/docs/pipeline-components/evaluators/documentndcgevaluator.mdx +++ b/docs-website/docs/pipeline-components/evaluators/documentndcgevaluator.mdx @@ -24,17 +24,17 @@ The `DocumentNDCGEvaluator` evaluates documents retrieved by Haystack pipelines ## Overview -You can use the `DocumentNDCGEvaluator` component to evaluate documents retrieved by a Haystack pipeline, such as a RAG pipeline, against ground truth labels. A higher NDCG is better and indicates that relevant documents appear at an earlier position in the list of retrieved documents. +You can use the `DocumentNDCGEvaluator` component to evaluate documents retrieved by a Haystack pipeline, such as a RAG pipeline, against ground truth labels. A higher NDCG is better and indicates that relevant documents appear at an earlier position in the list of retrieved documents. If the ground truth documents have scores, a higher NDCG indicates that documents with a higher score appear at an earlier position in the list of retrieved documents. If the ground truth documents have no scores, binary relevance is assumed, meaning that all ground truth documents are equally relevant, and the order in which they are in the list of retrieved documents does not matter for the NDCG. -No parameters are required to initialize a `DocumentNDCGEvaluator`. +No parameters are required to initialize a `DocumentNDCGEvaluator`. ## Usage ### On its own -Below is an example where we use the `DocumentNDCGEvaluator` to evaluate documents retrieved for a query. There are two ground truth documents and three retrieved documents. All ground truth documents are retrieved, but one non-relevant document is ranked higher than one of the ground truth documents, which lowers the NDCG score. +Below is an example where we use the `DocumentNDCGEvaluator` to evaluate documents retrieved for a query. There are two ground truth documents and three retrieved documents. All ground truth documents are retrieved, but one non-relevant document is ranked higher than one of the ground truth documents, which lowers the NDCG score. ```python from haystack import Document diff --git a/docs-website/docs/pipeline-components/extractors/namedentityextractor.mdx b/docs-website/docs/pipeline-components/extractors/namedentityextractor.mdx index 6516cab7b8e..39f3203a3a1 100644 --- a/docs-website/docs/pipeline-components/extractors/namedentityextractor.mdx +++ b/docs-website/docs/pipeline-components/extractors/namedentityextractor.mdx @@ -54,7 +54,7 @@ extractor = NamedEntityExtractor(backend="hugging_face", model="dslim/bert-base- extractor = NamedEntityExtractor(backend="spacy", model="en_core_web_sm") ``` -`NamedEntityExtractor` accepts a list of `Documents` as its input. The extractor annotates the raw text in the documents and stores the annotations in the document's `meta` dictionary under the `named_entities` key. +`NamedEntityExtractor` accepts a list of `Documents` as its input. The extractor annotates the raw text in the documents and stores the annotations in the document's `meta` dictionary under the `named_entities` key. ```python from haystack.dataclasses import Document diff --git a/docs-website/docs/pipeline-components/generators/amazonbedrockchatgenerator.mdx b/docs-website/docs/pipeline-components/generators/amazonbedrockchatgenerator.mdx index 793ed014446..84fda75f77d 100644 --- a/docs-website/docs/pipeline-components/generators/amazonbedrockchatgenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/amazonbedrockchatgenerator.mdx @@ -13,7 +13,7 @@ This component enables chat completion using models through Amazon Bedrock servi | | | | --- | --- | -| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) | +| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) | | **Mandatory init variables** | `model`: The model to use

`aws_access_key_id`: AWS access key ID. Can be set with `AWS_ACCESS_KEY_ID` env var.

`aws_secret_access_key`: AWS secret access key. Can be set with `AWS_SECRET_ACCESS_KEY` env var.

`aws_region_name`: AWS region name. Can be set with `AWS_DEFAULT_REGION` env var. | | **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) instances | | **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects

`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on | @@ -23,7 +23,7 @@ This component enables chat completion using models through Amazon Bedrock servi -[Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) is a fully managed service that makes high-performing foundation models from leading AI startups and Amazon available through a unified API. You can choose from various foundation models to find the one best suited for your use case. +[Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) is a fully managed service that makes high-performing foundation models from leading AI startups and Amazon available through a unified API. You can choose from various foundation models to find the one best suited for your use case. `AmazonBedrockChatGenerator` enables chat completion using chat models from Amazon, Anthropic, Cohere, Meta, Mistral, and more with a single component. diff --git a/docs-website/docs/pipeline-components/generators/amazonbedrockgenerator.mdx b/docs-website/docs/pipeline-components/generators/amazonbedrockgenerator.mdx index 3e7f9652ef0..247873977e8 100644 --- a/docs-website/docs/pipeline-components/generators/amazonbedrockgenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/amazonbedrockgenerator.mdx @@ -23,7 +23,7 @@ This component enables text generation using models through Amazon Bedrock servi -[Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) is a fully managed service that makes high-performing foundation models from leading AI startups and Amazon available through a unified API. You can choose from various foundation models to find the one best suited for your use case. +[Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) is a fully managed service that makes high-performing foundation models from leading AI startups and Amazon available through a unified API. You can choose from various foundation models to find the one best suited for your use case. `AmazonBedrockGenerator` enables text generation using models from AI21 Labs, Anthropic, Cohere, Meta, Stability AI, and Amazon with a single component. diff --git a/docs-website/docs/pipeline-components/generators/anthropicchatgenerator.mdx b/docs-website/docs/pipeline-components/generators/anthropicchatgenerator.mdx index fc4dc37e5ec..cc825996a41 100644 --- a/docs-website/docs/pipeline-components/generators/anthropicchatgenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/anthropicchatgenerator.mdx @@ -13,9 +13,9 @@ This component enables chat completions using Anthropic large language models (L | | | | --- | --- | -| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) | +| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) | | **Mandatory init variables** | `api_key`: An Anthropic API key. Can be set with `ANTHROPIC_API_KEY` env var. | -| **Mandatory run variables** | `messages` A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx)  objects | +| **Mandatory run variables** | `messages` A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects | | **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects

`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on | | **API reference** | [Anthropic](/reference/integrations-anthropic) | | **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/anthropic | @@ -25,20 +25,20 @@ This component enables chat completions using Anthropic large language models (L ## Overview -This integration supports Anthropic `chat` models such as `claude-3-5-sonnet-20240620`,`claude-3-opus-20240229`, `claude-3-haiku-20240307`, and similar. Check out the most recent full list in [Anthropic documentation](https://docs.anthropic.com/en/docs/about-claude/models). +This integration supports Anthropic `chat` models such as `claude-3-5-sonnet-20240620`,`claude-3-opus-20240229`, `claude-3-haiku-20240307`, and similar. Check out the most recent full list in [Anthropic documentation](https://docs.anthropic.com/en/docs/about-claude/models). ### Parameters -`AnthropicChatGenerator` needs an Anthropic API key to work. You can provide this key in: +`AnthropicChatGenerator` needs an Anthropic API key to work. You can provide this key in: -- The `ANTHROPIC_API_KEY` environment variable (recommended) -- The `api_key` init parameter and Haystack [Secret](../../concepts/secret-management.mdx) API: `Secret.from_token("your-api-key-here")` +- The `ANTHROPIC_API_KEY` environment variable (recommended) +- The `api_key` init parameter and Haystack [Secret](../../concepts/secret-management.mdx) API: `Secret.from_token("your-api-key-here")` Set your preferred Anthropic model with the `model` parameter when initializing the component. -`AnthropicChatGenerator` requires a prompt to generate text, but you can pass any text generation parameters available in the Anthropic [Messaging API](https://docs.anthropic.com/en/api/messages) method directly to this component using the `generation_kwargs` parameter, both at initialization and when running the component. For more details on the parameters supported by the Anthropic API, see the [Anthropic documentation](https://docs.anthropic.com). +`AnthropicChatGenerator` requires a prompt to generate text, but you can pass any text generation parameters available in the Anthropic [Messaging API](https://docs.anthropic.com/en/api/messages) method directly to this component using the `generation_kwargs` parameter, both at initialization and when running the component. For more details on the parameters supported by the Anthropic API, see the [Anthropic documentation](https://docs.anthropic.com). -Finally, the component needs a list of `ChatMessage` objects to operate. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata. +Finally, the component needs a list of `ChatMessage` objects to operate. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata. Only text input modality is supported at this time. @@ -132,7 +132,7 @@ For more details, refer to Anthropic's [documentation](https://docs.anthropic.co ## Usage -Install the`anthropic-haystack` package to use the `AnthropicChatGenerator`: +Install the`anthropic-haystack` package to use the `AnthropicChatGenerator`: ```shell pip install anthropic-haystack @@ -170,7 +170,7 @@ print(response) ### In a pipeline -You can also use `AnthropicChatGenerator`with the Anthropic chat models in your pipeline. +You can also use `AnthropicChatGenerator`with the Anthropic chat models in your pipeline. ```python from haystack import Pipeline diff --git a/docs-website/docs/pipeline-components/generators/anthropicgenerator.mdx b/docs-website/docs/pipeline-components/generators/anthropicgenerator.mdx index f21aefe82c6..a6c48b5a25d 100644 --- a/docs-website/docs/pipeline-components/generators/anthropicgenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/anthropicgenerator.mdx @@ -13,7 +13,7 @@ This component enables text completions using Anthropic large language models (L | | | | --- | --- | -| **Most common position in a pipeline** | After a [PromptBuilder](../builders/promptbuilder.mdx) | +| **Most common position in a pipeline** | After a [PromptBuilder](../builders/promptbuilder.mdx) | | **Mandatory init variables** | `api_key`: An Anthropic API key. Can be set with `ANTHROPIC_API_KEY` env var. | | **Mandatory run variables** | `prompt`: A string containing the prompt for the LLM | | **Output variables** | `replies`: A list of strings with all the replies generated by the LLM

`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on | @@ -25,20 +25,20 @@ This component enables text completions using Anthropic large language models (L ## Overview -This integration supports Anthropic models such as `claude-3-5-sonnet-20240620`,`claude-3-opus-20240229`, `claude-3-haiku-20240307`, and similar. Although these LLMs are called chat models, the main prompt interface works with the string prompts. Check out the most recent full list in the [Anthropic documentation](https://docs.anthropic.com/en/docs/about-claude/models). +This integration supports Anthropic models such as `claude-3-5-sonnet-20240620`,`claude-3-opus-20240229`, `claude-3-haiku-20240307`, and similar. Although these LLMs are called chat models, the main prompt interface works with the string prompts. Check out the most recent full list in the [Anthropic documentation](https://docs.anthropic.com/en/docs/about-claude/models). ### Parameters -`AnthropicGenerator` needs an Anthropic API key to work. You can provide this key in: +`AnthropicGenerator` needs an Anthropic API key to work. You can provide this key in: -- The `ANTHROPIC_API_KEY` environment variable (recommended) -- The `api_key` init parameter and Haystack [Secret](../../concepts/secret-management.mdx) API: `Secret.from_token("your-api-key-here")` +- The `ANTHROPIC_API_KEY` environment variable (recommended) +- The `api_key` init parameter and Haystack [Secret](../../concepts/secret-management.mdx) API: `Secret.from_token("your-api-key-here")` Set your preferred Anthropic model in the `model` parameter when initializing the component. -`AnthropicGenerator` requires a prompt to generate text, but you can pass any text generation parameters available in the Anthropic [Messaging API](https://docs.anthropic.com/en/api/messages) method directly to this component using the `generation_kwargs` parameter, both at initialization and to `run()` method. For more details on the parameters supported by the Anthropic API, see [Anthropic documentation](https://docs.anthropic.com). +`AnthropicGenerator` requires a prompt to generate text, but you can pass any text generation parameters available in the Anthropic [Messaging API](https://docs.anthropic.com/en/api/messages) method directly to this component using the `generation_kwargs` parameter, both at initialization and to `run()` method. For more details on the parameters supported by the Anthropic API, see [Anthropic documentation](https://docs.anthropic.com). -Finally, the component run method requires a single string prompt to generate text. +Finally, the component run method requires a single string prompt to generate text. ### Streaming @@ -46,7 +46,7 @@ This Generator supports [streaming](guides-to-generators/choosing-the-right-gene ## Usage -Install the `anthropic-haystack` package to use the `AnthropicGenerator`: +Install the `anthropic-haystack` package to use the `AnthropicGenerator`: ```shell pip install anthropic-haystack @@ -63,7 +63,7 @@ print(generator.run("What's Natural Language Processing? Be brief.")) ### In a pipeline -You can also use `AnthropicGenerator` with the Anthropic models in your pipeline. +You can also use `AnthropicGenerator` with the Anthropic models in your pipeline. ```python from haystack import Pipeline diff --git a/docs-website/docs/pipeline-components/generators/anthropicvertexchatgenerator.mdx b/docs-website/docs/pipeline-components/generators/anthropicvertexchatgenerator.mdx index a40b038c11b..e769e64ddb5 100644 --- a/docs-website/docs/pipeline-components/generators/anthropicvertexchatgenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/anthropicvertexchatgenerator.mdx @@ -26,7 +26,7 @@ This component enables chat completions using AnthropicVertex API. ## Overview `AnthropicVertexChatGenerator` enables text generation using state-of-the-art Claude 3 LLMs using the Anthropic Vertex AI API. -It supports `Claude 3.5 Sonnet`, `Claude 3 Opus`, `Claude 3 Sonnet`, and `Claude 3 Haiku` models, that are accessible through the Vertex AI API endpoint. For more details about the models, refer to [Anthropic Vertex AI documentation](https://docs.anthropic.com/en/api/claude-on-vertex-ai). +It supports `Claude 3.5 Sonnet`, `Claude 3 Opus`, `Claude 3 Sonnet`, and `Claude 3 Haiku` models, that are accessible through the Vertex AI API endpoint. For more details about the models, refer to [Anthropic Vertex AI documentation](https://docs.anthropic.com/en/api/claude-on-vertex-ai). ### Parameters @@ -34,16 +34,16 @@ To use the `AnthropicVertexChatGenerator`, ensure you have a GCP project with Ve You can provide these keys in the following ways: -- The `REGION` and `PROJECT_ID` environment variables (recommended) -- The `region` and `project_id` init parameters +- The `REGION` and `PROJECT_ID` environment variables (recommended) +- The `region` and `project_id` init parameters Before making requests, you may need to authenticate with GCP using `gcloud auth login`. -Set your preferred supported Anthropic model with the `model` parameter when initializing the component. Additionally, ensure that the desired Anthropic model is activated in the Vertex AI Model Garden. +Set your preferred supported Anthropic model with the `model` parameter when initializing the component. Additionally, ensure that the desired Anthropic model is activated in the Vertex AI Model Garden. -`AnthropicVertexChatGenerator` requires a prompt to generate text, but you can pass any text generation parameters available in the Anthropic [Messaging API](https://docs.anthropic.com/en/api/messages) method directly to this component using the `generation_kwargs` parameter, both at initialization and when running the component. For more details on the parameters supported by the Anthropic API, see the [Anthropic documentation](https://docs.anthropic.com/). +`AnthropicVertexChatGenerator` requires a prompt to generate text, but you can pass any text generation parameters available in the Anthropic [Messaging API](https://docs.anthropic.com/en/api/messages) method directly to this component using the `generation_kwargs` parameter, both at initialization and when running the component. For more details on the parameters supported by the Anthropic API, see the [Anthropic documentation](https://docs.anthropic.com/). -Finally, the component needs a list of `ChatMessage` objects to operate. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata. +Finally, the component needs a list of `ChatMessage` objects to operate. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata. Only text input modality is supported at this time. @@ -79,7 +79,7 @@ Prompt caching is a feature for Anthropic LLMs that stores large text inputs for This feature is particularly useful for coding assistants that need full codebase context and for processing large documents. It can help reduce costs and improve response times. -Here's an example of an instance of `AnthropicVertexChatGenerator` being initialized with prompt caching and tagging a message to be cached: +Here's an example of an instance of `AnthropicVertexChatGenerator` being initialized with prompt caching and tagging a message to be cached: ```python from haystack_integrations.components.generators.anthropic import ( @@ -117,11 +117,11 @@ result = claude_llm.run(messages) # and so on, either invoking component directly or in the pipeline ``` -For more details, refer to Anthropic's [documentation](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) and integration [examples](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/anthropic/example). +For more details, refer to Anthropic's [documentation](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) and integration [examples](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/anthropic/example). ## Usage -Install the`anthropic-haystack` package to use the `AnthropicVertexChatGenerator`: +Install the`anthropic-haystack` package to use the `AnthropicVertexChatGenerator`: ```shell pip install anthropic-haystack @@ -148,7 +148,7 @@ print(response) ### In a pipeline -You can also use `AnthropicVertexChatGenerator`with the Anthropic chat models in your pipeline. +You can also use `AnthropicVertexChatGenerator`with the Anthropic chat models in your pipeline. ```python from haystack import Pipeline diff --git a/docs-website/docs/pipeline-components/generators/azureopenaichatgenerator.mdx b/docs-website/docs/pipeline-components/generators/azureopenaichatgenerator.mdx index d886817beb6..39625b9e748 100644 --- a/docs-website/docs/pipeline-components/generators/azureopenaichatgenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/azureopenaichatgenerator.mdx @@ -13,7 +13,7 @@ This component enables chat completion using OpenAI’s large language models (L | | | | --- | --- | -| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) | +| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) | | **Mandatory init variables** | `api_key`: The Azure OpenAI API key. Can be set with `AZURE_OPENAI_API_KEY` env var.

`azure_ad_token`: Microsoft Entra ID token. Can be set with `AZURE_OPENAI_AD_TOKEN` env var. | | **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat or a plain string | | **Output variables** | `replies`: A list of alternative replies of the LLM to the input chat | @@ -29,7 +29,7 @@ This component enables chat completion using OpenAI’s large language models (L To work with Azure components, you will need an Azure OpenAI API key, as well as an Azure OpenAI Endpoint. You can learn more about them in Azure [documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference). -The component uses `AZURE_OPENAI_API_KEY` and `AZURE_OPENAI_AD_TOKEN` environment variables by default. Otherwise, you can pass `api_key` and `azure_ad_token` at initialization: +The component uses `AZURE_OPENAI_API_KEY` and `AZURE_OPENAI_AD_TOKEN` environment variables by default. Otherwise, you can pass `api_key` and `azure_ad_token` at initialization: ```python client = AzureOpenAIChatGenerator( diff --git a/docs-website/docs/pipeline-components/generators/azureopenaigenerator.mdx b/docs-website/docs/pipeline-components/generators/azureopenaigenerator.mdx index 70609181419..19362895501 100644 --- a/docs-website/docs/pipeline-components/generators/azureopenaigenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/azureopenaigenerator.mdx @@ -29,7 +29,7 @@ This component enables text generation using OpenAI's large language models (LLM To work with Azure components, you will need an Azure OpenAI API key, as well as an Azure OpenAI Endpoint. You can learn more about them in Azure [documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference). -The component uses `AZURE_OPENAI_API_KEY` and `AZURE_OPENAI_AD_TOKEN` environment variables by default. Otherwise, you can pass `api_key` and `azure_ad_token` at initialization: +The component uses `AZURE_OPENAI_API_KEY` and `AZURE_OPENAI_AD_TOKEN` environment variables by default. Otherwise, you can pass `api_key` and `azure_ad_token` at initialization: ```python client = AzureOpenAIGenerator( diff --git a/docs-website/docs/pipeline-components/generators/dalleimagegenerator.mdx b/docs-website/docs/pipeline-components/generators/dalleimagegenerator.mdx index 862c814a7c5..25c49e4eb10 100644 --- a/docs-website/docs/pipeline-components/generators/dalleimagegenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/dalleimagegenerator.mdx @@ -29,7 +29,7 @@ The `DALLEImageGenerator` component generates images using OpenAI's image genera By default, the component uses the `gpt-image-2` model, `"auto"` quality, and 1024x1024 resolution. You can change these parameters using `model` (during component initialization), `quality`, and `size` (during component initialization or run) parameters. -`DALLEImageGenerator` needs an OpenAI key to work. It uses an `OPENAI_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`: +`DALLEImageGenerator` needs an OpenAI key to work. It uses an `OPENAI_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`: ``` image_generator = DALLEImageGenerator(api_key=Secret.from_token("")) diff --git a/docs-website/docs/pipeline-components/generators/googleaigeminichatgenerator.mdx b/docs-website/docs/pipeline-components/generators/googleaigeminichatgenerator.mdx index 40e266934f7..b8fb7cdfb65 100644 --- a/docs-website/docs/pipeline-components/generators/googleaigeminichatgenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/googleaigeminichatgenerator.mdx @@ -20,7 +20,7 @@ We recommend switching to the new [GoogleGenAIChatGenerator](googlegenaichatgene | | | | --- | --- | -| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) | +| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) | | **Mandatory init variables** | `api_key`: A Google AI Studio API key. Can be set with `GOOGLE_API_KEY` env var. | | **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat | | **Output variables** | `replies`: A list of alternative replies of the model to the input chat | diff --git a/docs-website/docs/pipeline-components/generators/googlegenaichatgenerator.mdx b/docs-website/docs/pipeline-components/generators/googlegenaichatgenerator.mdx index a55cd0a6bf7..fd451956f15 100644 --- a/docs-website/docs/pipeline-components/generators/googlegenaichatgenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/googlegenaichatgenerator.mdx @@ -13,7 +13,7 @@ This component enables chat completion using Google Gemini models through Google | | | | --- | --- | -| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) | +| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) | | **Mandatory init variables** | `api_key`: A Google API key. Can be set with `GOOGLE_API_KEY` env var. | | **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat | | **Output variables** | `replies`: A list of alternative replies of the model to the input chat | diff --git a/docs-website/docs/pipeline-components/generators/huggingfacelocalchatgenerator.mdx b/docs-website/docs/pipeline-components/generators/huggingfacelocalchatgenerator.mdx index 9301ad65b1c..39636c9b5c9 100644 --- a/docs-website/docs/pipeline-components/generators/huggingfacelocalchatgenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/huggingfacelocalchatgenerator.mdx @@ -39,7 +39,7 @@ This component is designed for chat completion, not for text generation. If you If a string is passed to `messages`, it is converted into a list containing a single `ChatMessage` with the `user` role. -For remote file authorization, this component uses a `HF_API_TOKEN` environment variable by default. Otherwise, you can pass a Hugging Face API token at initialization with `token`: +For remote file authorization, this component uses a `HF_API_TOKEN` environment variable by default. Otherwise, you can pass a Hugging Face API token at initialization with `token`: ```python local_generator = HuggingFaceLocalChatGenerator( diff --git a/docs-website/docs/pipeline-components/generators/huggingfacelocalgenerator.mdx b/docs-website/docs/pipeline-components/generators/huggingfacelocalgenerator.mdx index 264dd06ec92..de8025bc7ad 100644 --- a/docs-website/docs/pipeline-components/generators/huggingfacelocalgenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/huggingfacelocalgenerator.mdx @@ -32,7 +32,7 @@ Keep in mind that if LLMs run locally, you may need a powerful machine to run th This component is designed for text generation, not for chat. If you want to use Hugging Face LLMs for chat, consider using [`HuggingFaceLocalChatGenerator`](huggingfacelocalchatgenerator.mdx) instead. ::: -For remote files authorization, this component uses a `HF_API_TOKEN` environment variable by default. Otherwise, you can pass a Hugging Face API token at initialization with `token`: +For remote files authorization, this component uses a `HF_API_TOKEN` environment variable by default. Otherwise, you can pass a Hugging Face API token at initialization with `token`: ```python local_generator = HuggingFaceLocalGenerator(token=Secret.from_token("")) diff --git a/docs-website/docs/pipeline-components/generators/llamastackchatgenerator.mdx b/docs-website/docs/pipeline-components/generators/llamastackchatgenerator.mdx index fb9fea78979..a38ee675464 100644 --- a/docs-website/docs/pipeline-components/generators/llamastackchatgenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/llamastackchatgenerator.mdx @@ -13,7 +13,7 @@ This component enables chat completions using any model made available by infere | | | | --- | --- | -| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) | +| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) | | **Mandatory init variables** | `model`: The name of the model to use for chat completion.
This depends on the inference provider used for the Llama Stack Server. | | **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat | | **Output variables** | `replies`: A list of alternative replies of the model to the input chat | diff --git a/docs-website/docs/pipeline-components/generators/metallamachatgenerator.mdx b/docs-website/docs/pipeline-components/generators/metallamachatgenerator.mdx index 704eb50e510..bfe800fb0d5 100644 --- a/docs-website/docs/pipeline-components/generators/metallamachatgenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/metallamachatgenerator.mdx @@ -75,7 +75,7 @@ For more details on working with tools, see the [Tool](../../tools/tool.mdx) and To use this integration, you must have a Meta Llama API key. You can provide it with the `LLAMA_API_KEY` environment variable or by using a [Secret](../../concepts/secret-management.mdx). -Then, install the `meta-llama-haystack` integration: +Then, install the `meta-llama-haystack` integration: ```shell pip install meta-llama-haystack diff --git a/docs-website/docs/pipeline-components/generators/openaichatgenerator.mdx b/docs-website/docs/pipeline-components/generators/openaichatgenerator.mdx index e4b7b47bd94..85ab1a0d3b2 100644 --- a/docs-website/docs/pipeline-components/generators/openaichatgenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/openaichatgenerator.mdx @@ -27,7 +27,7 @@ description: "`OpenAIChatGenerator` enables chat completion using OpenAI’s lar `OpenAIChatGenerator` supports OpenAI models starting from gpt-3.5-turbo and later (gpt-4, gpt-4-turbo, and so on). -`OpenAIChatGenerator` needs an OpenAI key to work. It uses an ` OPENAI_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`: +`OpenAIChatGenerator` needs an OpenAI key to work. It uses an ` OPENAI_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`: ```python generator = OpenAIChatGenerator(model="gpt-4o-mini") diff --git a/docs-website/docs/pipeline-components/generators/openaigenerator.mdx b/docs-website/docs/pipeline-components/generators/openaigenerator.mdx index 0c6cf0e7d0d..2b3df2313d4 100644 --- a/docs-website/docs/pipeline-components/generators/openaigenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/openaigenerator.mdx @@ -27,7 +27,7 @@ description: "`OpenAIGenerator` enables text generation using OpenAI's large lan `OpenAIGenerator` supports OpenAI models starting from gpt-3.5-turbo and later (gpt-4, gpt-4-turbo, and so on). -`OpenAIGenerator` needs an OpenAI key to work. It uses an `OPENAI_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`: +`OpenAIGenerator` needs an OpenAI key to work. It uses an `OPENAI_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`: ``` generator = OpenAIGenerator(api_key=Secret.from_token(""), model="gpt-4o-mini") diff --git a/docs-website/docs/pipeline-components/generators/openrouterchatgenerator.mdx b/docs-website/docs/pipeline-components/generators/openrouterchatgenerator.mdx index ebb384a5591..3b5f9e59559 100644 --- a/docs-website/docs/pipeline-components/generators/openrouterchatgenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/openrouterchatgenerator.mdx @@ -67,7 +67,7 @@ For more details on working with tools, see the [Tool](../../tools/tool.mdx) and To use this integration, you must have an active OpenRouter subscription with sufficient credits and an API key. You can provide it with the `OPENROUTER_API_KEY` environment variable or by using a [Secret](../../concepts/secret-management.mdx). -Then, install the `openrouter-haystack` integration: +Then, install the `openrouter-haystack` integration: ```shell pip install openrouter-haystack diff --git a/docs-website/docs/pipeline-components/generators/stackitchatgenerator.mdx b/docs-website/docs/pipeline-components/generators/stackitchatgenerator.mdx index 744dbd58f34..9cc73547f02 100644 --- a/docs-website/docs/pipeline-components/generators/stackitchatgenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/stackitchatgenerator.mdx @@ -15,8 +15,8 @@ This component enables chat completions using the STACKIT API. | --- | --- | | **Most common position in a pipeline** | After a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) | | **Mandatory init variables** | `model`: The model used through the STACKIT API | -| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx)  objects | -| **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects

`meta`: A list of dictionaries with the metadata associated with each reply (such as token count, finish reason, and so on) | +| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects | +| **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects

`meta`: A list of dictionaries with the metadata associated with each reply (such as token count, finish reason, and so on) | | **API reference** | [STACKIT](/reference/integrations-stackit) | | **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/stackit | | **Package name** | `stackit-haystack` | @@ -32,21 +32,21 @@ This component enables chat completions using the STACKIT API. To use the `STACKITChatGenerator`, ensure you have set a `STACKIT_API_KEY` as an environment variable. Alternatively, provide the API key as another environment variable or a token by setting `api_key` and using Haystack’s [secret management](../../concepts/secret-management.mdx). -Set your preferred supported model with the `model` parameter when initializing the component. See the full list of all supported models on the [STACKIT website](https://docs.stackit.cloud/stackit/en/models-licenses-319914532.html). +Set your preferred supported model with the `model` parameter when initializing the component. See the full list of all supported models on the [STACKIT website](https://docs.stackit.cloud/stackit/en/models-licenses-319914532.html). Optionally, you can change the default `api_base_url`, which is `"https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1"`. You can pass any text generation parameters valid for the STACKIT Chat Completion API directly to this component with the `generation_kwargs` parameter in the init or run methods. -The component needs a list of `ChatMessage` objects to run. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata. Find out more about it [ChatMessage documentation](../../concepts/data-classes/chatmessage.mdx). +The component needs a list of `ChatMessage` objects to run. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata. Find out more about it [ChatMessage documentation](../../concepts/data-classes/chatmessage.mdx). ### Streaming -This ChatGenerator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly into the output. To do so, pass a function to the `streaming_callback` init parameter. +This ChatGenerator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly into the output. To do so, pass a function to the `streaming_callback` init parameter. ## Usage -Install the `stackit-haystack` package to use the `STACKITChatGenerator`: +Install the `stackit-haystack` package to use the `STACKITChatGenerator`: ```shell pip install stackit-haystack @@ -85,7 +85,7 @@ print(response) ### In a pipeline -You can also use `STACKITChatGenerator` in your pipeline. +You can also use `STACKITChatGenerator` in your pipeline. ```python from haystack import Pipeline diff --git a/docs-website/docs/pipeline-components/generators/togetheraichatgenerator.mdx b/docs-website/docs/pipeline-components/generators/togetheraichatgenerator.mdx index b5ecfa92a41..a80f8841288 100644 --- a/docs-website/docs/pipeline-components/generators/togetheraichatgenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/togetheraichatgenerator.mdx @@ -69,7 +69,7 @@ For more details on working with tools, see the [Tool](../../tools/tool.mdx) and ### Streaming -`TogetherAIChatGenerator` supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) responses from the LLM, allowing tokens to be emitted as they are generated. To enable streaming, pass a callable to the `streaming_callback` parameter during initialization. +`TogetherAIChatGenerator` supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) responses from the LLM, allowing tokens to be emitted as they are generated. To enable streaming, pass a callable to the `streaming_callback` parameter during initialization. ## Usage diff --git a/docs-website/docs/pipeline-components/generators/vertexaigeminichatgenerator.mdx b/docs-website/docs/pipeline-components/generators/vertexaigeminichatgenerator.mdx index 2d601cef711..8cccd8aad9a 100644 --- a/docs-website/docs/pipeline-components/generators/vertexaigeminichatgenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/vertexaigeminichatgenerator.mdx @@ -20,7 +20,7 @@ We recommend switching to the new [GoogleGenAIChatGenerator](googlegenaichatgene | | | | --- | --- | -| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) | +| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) | | **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat | | **Output variables** | `replies`: A list of alternative replies of the model to the input chat | | **API reference** | [Google Vertex](/reference/integrations-google-vertex) | diff --git a/docs-website/docs/pipeline-components/joiners/branchjoiner.mdx b/docs-website/docs/pipeline-components/joiners/branchjoiner.mdx index 0c392752296..21b7159450e 100644 --- a/docs-website/docs/pipeline-components/joiners/branchjoiner.mdx +++ b/docs-website/docs/pipeline-components/joiners/branchjoiner.mdx @@ -31,7 +31,7 @@ Use this component to join different branches of a pipeline into a single output `BranchJoiner` receives multiple data connections of the same type from other components and passes the first value it receives to its single output. This makes it essential for closing loops in pipelines or reconciling multiple branches from a decision component. -`BranchJoiner` can handle only one input of one data type, declared in the `__init__` function. It ensures that the data type remains consistent across the pipeline branches. If more than one value is received for the input when `run` is invoked, the component will raise an error: +`BranchJoiner` can handle only one input of one data type, declared in the `__init__` function. It ensures that the data type remains consistent across the pipeline branches. If more than one value is received for the input when `run` is invoked, the component will raise an error: ```python from haystack.components.joiners import BranchJoiner diff --git a/docs-website/docs/pipeline-components/joiners/listjoiner.mdx b/docs-website/docs/pipeline-components/joiners/listjoiner.mdx index c556c22bb87..d7ec2f4c199 100644 --- a/docs-website/docs/pipeline-components/joiners/listjoiner.mdx +++ b/docs-website/docs/pipeline-components/joiners/listjoiner.mdx @@ -13,7 +13,7 @@ A component that joins multiple lists into a single flat list. | | | | --- | --- | -| **Most common position in a pipeline** | In indexing and query pipelines, after components that return lists of documents such as multiple [Retrievers](../retrievers.mdx) or multiple [Converters](../converters.mdx) | +| **Most common position in a pipeline** | In indexing and query pipelines, after components that return lists of documents such as multiple [Retrievers](../retrievers.mdx) or multiple [Converters](../converters.mdx) | | **Mandatory run variables** | `values`: The dictionary of lists to be joined | | **Output variables** | `values`: A dictionary with a `values` key containing the joined list | | **API reference** | [Joiners](/reference/joiners-api) | diff --git a/docs-website/docs/pipeline-components/preprocessors/documentpreprocessor.mdx b/docs-website/docs/pipeline-components/preprocessors/documentpreprocessor.mdx index cd274c26a3e..d29b138cdb7 100644 --- a/docs-website/docs/pipeline-components/preprocessors/documentpreprocessor.mdx +++ b/docs-website/docs/pipeline-components/preprocessors/documentpreprocessor.mdx @@ -13,7 +13,7 @@ Divides a list of text documents into a list of shorter text documents and then | | | | --- | --- | -| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx)  | +| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) | | **Mandatory run variables** | `documents`: A list of documents | | **Output variables** | `documents`: A list of split and cleaned documents | | **API reference** | [PreProcessors](/reference/preprocessors-api) | @@ -52,7 +52,7 @@ print(result["documents"]) ### In a pipeline -You can use the `DocumentPreprocessor` in your indexing pipeline. The example below requires installing additional dependencies for the `MultiFileConverter`: +You can use the `DocumentPreprocessor` in your indexing pipeline. The example below requires installing additional dependencies for the `MultiFileConverter`: ```shell pip install pypdf markdown-it-py mdit_plain trafilatura python-pptx python-docx jq openpyxl tabulate pandas diff --git a/docs-website/docs/pipeline-components/preprocessors/hierarchicaldocumentsplitter.mdx b/docs-website/docs/pipeline-components/preprocessors/hierarchicaldocumentsplitter.mdx index f47722064b8..c64024e6a23 100644 --- a/docs-website/docs/pipeline-components/preprocessors/hierarchicaldocumentsplitter.mdx +++ b/docs-website/docs/pipeline-components/preprocessors/hierarchicaldocumentsplitter.mdx @@ -40,8 +40,8 @@ This descending order of sizes builds the hierarchy. These additional parameters can be set when the component is initialized: -- `split_by` can be `"word"` (default), `"sentence"`, `"passage"`, `"page"`. -- `split_overlap` is an integer indicating the number of overlapping words, sentences, or passages between chunks, 0 being the default. +- `split_by` can be `"word"` (default), `"sentence"`, `"passage"`, `"page"`. +- `split_overlap` is an integer indicating the number of overlapping words, sentences, or passages between chunks, 0 being the default. ## Usage diff --git a/docs-website/docs/pipeline-components/preprocessors/recursivesplitter.mdx b/docs-website/docs/pipeline-components/preprocessors/recursivesplitter.mdx index 51f12d638d7..8b76c633525 100644 --- a/docs-website/docs/pipeline-components/preprocessors/recursivesplitter.mdx +++ b/docs-website/docs/pipeline-components/preprocessors/recursivesplitter.mdx @@ -57,7 +57,7 @@ print(doc_chunks["documents"]) ### In a pipeline -Here's how you can use `RecursiveSplitter` in an indexing pipeline: +Here's how you can use `RecursiveSplitter` in an indexing pipeline: ```python from pathlib import Path diff --git a/docs-website/docs/pipeline-components/rankers/amazonbedrockranker.mdx b/docs-website/docs/pipeline-components/rankers/amazonbedrockranker.mdx index e65a346ef33..2eb6416ae90 100644 --- a/docs-website/docs/pipeline-components/rankers/amazonbedrockranker.mdx +++ b/docs-website/docs/pipeline-components/rankers/amazonbedrockranker.mdx @@ -25,9 +25,9 @@ Use this component to rank documents based on their similarity to the query usin ## Overview -`AmazonBedrockRanker` ranks documents based on semantic relevance to a specified query. It uses Amazon Bedrock Rerank API. This list of all supported models can be found in Amazon’s [documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/rerank-supported.html). The default model for this Ranker is `cohere.rerank-v3-5:0`. +`AmazonBedrockRanker` ranks documents based on semantic relevance to a specified query. It uses Amazon Bedrock Rerank API. This list of all supported models can be found in Amazon’s [documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/rerank-supported.html). The default model for this Ranker is `cohere.rerank-v3-5:0`. -You can also specify the `top_k` parameter to set the maximum number of documents to return. +You can also specify the `top_k` parameter to set the maximum number of documents to return. ### Installation diff --git a/docs-website/docs/pipeline-components/rankers/cohereranker.mdx b/docs-website/docs/pipeline-components/rankers/cohereranker.mdx index fb5a3b035ad..ed103554974 100644 --- a/docs-website/docs/pipeline-components/rankers/cohereranker.mdx +++ b/docs-website/docs/pipeline-components/rankers/cohereranker.mdx @@ -14,7 +14,7 @@ Use this component to rank documents based on their similarity to the query usin | | | | --- | --- | | **Most common position in a pipeline** | In a query pipeline, after a component that returns a list of documents such as a [Retriever](../retrievers.mdx) | -| **Mandatory init variables** | `api_key`: The Cohere API key. Can be set with `COHERE_API_KEY` or `CO_API_KEY` env var. | +| **Mandatory init variables** | `api_key`: The Cohere API key. Can be set with `COHERE_API_KEY` or `CO_API_KEY` env var. | | **Mandatory run variables** | `documents`: A list of document objects

`query`: A query string

`top_k`: The maximum number of documents to return | | **Output variables** | `documents`: A list of document objects | | **API reference** | [Cohere](/reference/integrations-cohere) | @@ -35,7 +35,7 @@ To start using this integration with Haystack, install it with: pip install cohere-haystack ``` -The component uses a `COHERE_API_KEY` or `CO_API_KEY` environment variable by default. Otherwise, you can pass a Cohere API key at initialization with `api_key` like this: +The component uses a `COHERE_API_KEY` or `CO_API_KEY` environment variable by default. Otherwise, you can pass a Cohere API key at initialization with `api_key` like this: ```python ranker = CohereRanker(api_key=Secret.from_token("")) diff --git a/docs-website/docs/pipeline-components/rankers/jinaranker.mdx b/docs-website/docs/pipeline-components/rankers/jinaranker.mdx index 6583471da40..9dd98f55fc6 100644 --- a/docs-website/docs/pipeline-components/rankers/jinaranker.mdx +++ b/docs-website/docs/pipeline-components/rankers/jinaranker.mdx @@ -29,7 +29,7 @@ Use this component to rank documents based on their similarity to the query usin Additionally, you can use the optional `top_k` and `score_threshold` parameters with `JinaRanker` : -- The Ranker's `top_k` is the number of documents it returns (if it's the last component in the pipeline) or forwards to the next component. +- The Ranker's `top_k` is the number of documents it returns (if it's the last component in the pipeline) or forwards to the next component. - If you set the `score_threshold` for the Ranker, it will only return documents with a similarity score (computed by the Jina AI model) above this threshold. ### Installation @@ -42,7 +42,7 @@ pip install jina-haystack ### Authorization -The component uses a `JINA_API_KEY` environment variable by default. Otherwise, you can pass a Jina API key at initialization with `api_key` like this: +The component uses a `JINA_API_KEY` environment variable by default. Otherwise, you can pass a Jina API key at initialization with `api_key` like this: ```python ranker = JinaRanker(api_key=Secret.from_token("")) @@ -54,9 +54,9 @@ To get your API key, head to Jina AI’s [website](https://jina.ai/reranker/). ### On its own -You can use `JinaRanker` outside of a pipeline to order documents based on your query. +You can use `JinaRanker` outside of a pipeline to order documents based on your query. -To run the Ranker, pass a query, provide the documents, and set the number of documents to return in the `top_k` parameter. +To run the Ranker, pass a query, provide the documents, and set the number of documents to return in the `top_k` parameter. ```python from haystack import Document diff --git a/docs-website/docs/pipeline-components/rankers/nvidiaranker.mdx b/docs-website/docs/pipeline-components/rankers/nvidiaranker.mdx index dec34af6b08..9a7ea6b0d08 100644 --- a/docs-website/docs/pipeline-components/rankers/nvidiaranker.mdx +++ b/docs-website/docs/pipeline-components/rankers/nvidiaranker.mdx @@ -37,7 +37,7 @@ To start using this integration with Haystack, install it with: pip install nvidia-haystack ``` -The component uses an `NVIDIA_API_KEY` environment variable by default. Otherwise, you can pass an Nvidia API key at initialization with `api_key` like this: +The component uses an `NVIDIA_API_KEY` environment variable by default. Otherwise, you can pass an Nvidia API key at initialization with `api_key` like this: ```python ranker = NvidiaRanker(api_key=Secret.from_token("")) diff --git a/docs-website/docs/pipeline-components/rankers/sentencetransformerssimilarityranker.mdx b/docs-website/docs/pipeline-components/rankers/sentencetransformerssimilarityranker.mdx index 8e5f7cb26e1..7ef043908cc 100644 --- a/docs-website/docs/pipeline-components/rankers/sentencetransformerssimilarityranker.mdx +++ b/docs-website/docs/pipeline-components/rankers/sentencetransformerssimilarityranker.mdx @@ -37,7 +37,7 @@ Additionally, you can select the backend to use for the Sentence Transformers mo ### Authorization -The component uses a `HF_API_TOKEN` environment variable by default. Otherwise, you can pass a Hugging Face API token at initialization with [Secret](../../concepts/secret-management.mdx) `token`: +The component uses a `HF_API_TOKEN` environment variable by default. Otherwise, you can pass a Hugging Face API token at initialization with [Secret](../../concepts/secret-management.mdx) `token`: ```python ranker = SentenceTransformersSimilarityRanker(token=Secret.from_token("")) diff --git a/docs-website/docs/pipeline-components/rankers/transformerssimilarityranker.mdx b/docs-website/docs/pipeline-components/rankers/transformerssimilarityranker.mdx index 3eb3a95ad10..508ee021260 100644 --- a/docs-website/docs/pipeline-components/rankers/transformerssimilarityranker.mdx +++ b/docs-website/docs/pipeline-components/rankers/transformerssimilarityranker.mdx @@ -41,7 +41,7 @@ You can also set the `device` parameter to use HF models on your CPU or GPU. ### Authorization -The component uses a `HF_API_TOKEN` environment variable by default. Otherwise, you can pass a Hugging Face API token at initialization with `token` – see code examples below. +The component uses a `HF_API_TOKEN` environment variable by default. Otherwise, you can pass a Hugging Face API token at initialization with `token` – see code examples below. ```python ranker = TransformersSimilarityRanker(token=Secret.from_token("")) diff --git a/docs-website/docs/pipeline-components/retrievers.mdx b/docs-website/docs/pipeline-components/retrievers.mdx index f6e2f19e100..8932ab0561d 100644 --- a/docs-website/docs/pipeline-components/retrievers.mdx +++ b/docs-website/docs/pipeline-components/retrievers.mdx @@ -36,7 +36,7 @@ Main features: ### Dense Embedding-Based Retrievers -Dense embedding-based Retrievers work with embeddings, which are vector representations of words that capture their semantics. Dense Retrievers need an [Embedder](embedders.mdx) first to turn the documents and the query into vectors. Then, they calculate the vector similarity of the query and each document in the Document Store to fetch the most relevant documents. +Dense embedding-based Retrievers work with embeddings, which are vector representations of words that capture their semantics. Dense Retrievers need an [Embedder](embedders.mdx) first to turn the documents and the query into vectors. Then, they calculate the vector similarity of the query and each document in the Document Store to fetch the most relevant documents. Main features: @@ -61,7 +61,7 @@ Main features: ### Filter Retriever -`FilterRetriever` is a special kind of Retriever that can work with all Document Stores and retrieves all documents that match the provided filters. +`FilterRetriever` is a special kind of Retriever that can work with all Document Stores and retrieves all documents that match the provided filters. For more information, read this Retriever's [documentation page](retrievers/filterretriever.mdx). diff --git a/docs-website/docs/pipeline-components/retrievers/azureaisearchbm25retriever.mdx b/docs-website/docs/pipeline-components/retrievers/azureaisearchbm25retriever.mdx index 9cdf5103663..793ead852b3 100644 --- a/docs-website/docs/pipeline-components/retrievers/azureaisearchbm25retriever.mdx +++ b/docs-website/docs/pipeline-components/retrievers/azureaisearchbm25retriever.mdx @@ -15,7 +15,7 @@ A keyword-based Retriever that fetches documents matching a query from the Azure | | | | --- | --- | -| **Most common position in a pipeline** | 1. Before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. Before an [`ExtractiveReader`](../readers/extractivereader.mdx) in an extractive QA pipeline | +| **Most common position in a pipeline** | 1. Before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. Before an [`ExtractiveReader`](../readers/extractivereader.mdx) in an extractive QA pipeline | | **Mandatory init variables** | `document_store`: An instance of [`AzureAISearchDocumentStore`](../../document-stores/azureaisearchdocumentstore.mdx) | | **Mandatory run variables** | `query`: A string | | **Output variables** | `documents`: A list of documents (matching the query) | @@ -29,11 +29,11 @@ A keyword-based Retriever that fetches documents matching a query from the Azure The `AzureAISearchBM25Retriever` is a keyword-based Retriever designed to fetch documents that match a query from an `AzureAISearchDocumentStore`. It uses the BM25 algorithm which calculates a weighted word overlap between the query and the documents to determine their similarity. The Retriever accepts textual query but you can also provide a combination of terms with boolean operators. Some examples of valid queries could be `"pool"`, `"pool spa"`, and `"pool spa +airport"`. -In addition to the `query`, the `AzureAISearchBM25Retriever` accepts other optional parameters, including `top_k` (the maximum number of documents to retrieve) and `filters` to narrow down the search space. +In addition to the `query`, the `AzureAISearchBM25Retriever` accepts other optional parameters, including `top_k` (the maximum number of documents to retrieve) and `filters` to narrow down the search space. If your search index includes a [semantic configuration](https://learn.microsoft.com/en-us/azure/search/semantic-how-to-query-request), you can enable semantic ranking to apply it to the Retriever's results. For more details, refer to the [Azure AI documentation](https://learn.microsoft.com/en-us/azure/search/hybrid-search-how-to-query#semantic-hybrid-search). -If you want a combination of BM25 and vector retrieval, use the `AzureAISearchHybridRetriever`, which uses both vector search and BM25 search to match documents and query. +If you want a combination of BM25 and vector retrieval, use the `AzureAISearchHybridRetriever`, which uses both vector search and BM25 search to match documents and query. ## Usage @@ -78,7 +78,7 @@ retriever.run(query="How many languages are spoken around the world today?") ### In a RAG pipeline -The below example shows how to use the `AzureAISearchBM25Retriever` in a RAG pipeline. Set your `OPENAI_API_KEY` as an environment variable and then run the following code: +The below example shows how to use the `AzureAISearchBM25Retriever` in a RAG pipeline. Set your `OPENAI_API_KEY` as an environment variable and then run the following code: ```python diff --git a/docs-website/docs/pipeline-components/retrievers/azureaisearchembeddingretriever.mdx b/docs-website/docs/pipeline-components/retrievers/azureaisearchembeddingretriever.mdx index 7fb26bedbe8..4396d04a05f 100644 --- a/docs-website/docs/pipeline-components/retrievers/azureaisearchembeddingretriever.mdx +++ b/docs-website/docs/pipeline-components/retrievers/azureaisearchembeddingretriever.mdx @@ -15,7 +15,7 @@ This Retriever accepts the embeddings of a single query as input and returns a l | | | | --- | --- | -| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the embedding retrieval pipeline 3. After a Text Embedder and before an [`ExtractiveReader`](../readers/extractivereader.mdx) in an extractive QA pipeline | +| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the embedding retrieval pipeline 3. After a Text Embedder and before an [`ExtractiveReader`](../readers/extractivereader.mdx) in an extractive QA pipeline | | **Mandatory init variables** | `document_store`: An instance of [`AzureAISearchDocumentStore`](../../document-stores/azureaisearchdocumentstore.mdx) | | **Mandatory run variables** | `query_embedding`: A list of floats | | **Output variables** | `documents`: A list of documents | @@ -27,13 +27,13 @@ This Retriever accepts the embeddings of a single query as input and returns a l ## Overview -The `AzureAISearchEmbeddingRetriever` is an embedding-based Retriever compatible with the `AzureAISearchDocumentStore`. It compares the query and document embeddings and fetches the most relevant documents from the `AzureAISearchDocumentStore` based on the outcome. +The `AzureAISearchEmbeddingRetriever` is an embedding-based Retriever compatible with the `AzureAISearchDocumentStore`. It compares the query and document embeddings and fetches the most relevant documents from the `AzureAISearchDocumentStore` based on the outcome. -The query needs to be embedded before being passed to this component. For example, you could use a Text [Embedder](../embedders.mdx) component. +The query needs to be embedded before being passed to this component. For example, you could use a Text [Embedder](../embedders.mdx) component. By default, the `AzureAISearchDocumentStore` uses the [HNSW algorithm](https://learn.microsoft.com/en-us/azure/search/vector-search-overview#nearest-neighbors-search) with cosine similarity to handle vector searches. The vector configuration is set during the initialization of the document store and can be customized by providing the `vector_search_configuration` parameter. -In addition to the `query_embedding`, the `AzureAISearchEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of documents to retrieve) and `filters` to narrow down the search space. +In addition to the `query_embedding`, the `AzureAISearchEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of documents to retrieve) and `filters` to narrow down the search space. :::info[Semantic Ranking] @@ -74,11 +74,11 @@ retriever.run(query_embedding=[0.1] * 384) ### In a pipeline -Here is how you could use the `AzureAISearchEmbeddingRetriever` in a pipeline. In this example, you would create two pipelines: an indexing one and a querying one. +Here is how you could use the `AzureAISearchEmbeddingRetriever` in a pipeline. In this example, you would create two pipelines: an indexing one and a querying one. In the indexing pipeline, the documents are passed to the Document Embedder and then written into the Document Store. -Then, in the querying pipeline, we use a Text Embedder to get the vector representation of the input query that will be then passed to the `AzureAISearchEmbeddingRetriever` to get the results. +Then, in the querying pipeline, we use a Text Embedder to get the vector representation of the input query that will be then passed to the `AzureAISearchEmbeddingRetriever` to get the results. ```python from haystack import Document, Pipeline diff --git a/docs-website/docs/pipeline-components/retrievers/azureaisearchhybridretriever.mdx b/docs-website/docs/pipeline-components/retrievers/azureaisearchhybridretriever.mdx index fe532a10cc3..368b72d1141 100644 --- a/docs-website/docs/pipeline-components/retrievers/azureaisearchhybridretriever.mdx +++ b/docs-website/docs/pipeline-components/retrievers/azureaisearchhybridretriever.mdx @@ -15,7 +15,7 @@ This Retriever combines embedding-based retrieval and BM25 text search search to | | | | --- | --- | -| **Most common position in a pipeline** | 1. After a TextEmbedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in a hybrid search pipeline 3. After a TextEmbedder and before an [`ExtractiveReader`](../readers/extractivereader.mdx) in an extractive QA pipeline | +| **Most common position in a pipeline** | 1. After a TextEmbedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in a hybrid search pipeline 3. After a TextEmbedder and before an [`ExtractiveReader`](../readers/extractivereader.mdx) in an extractive QA pipeline | | **Mandatory init variables** | `document_store`: An instance of [`AzureAISearchDocumentStore`](../../document-stores/azureaisearchdocumentstore.mdx) | | **Mandatory run variables** | `query`: A string

`query_embedding`: A list of floats | | **Output variables** | `documents`: A list of documents (matching the query) | diff --git a/docs-website/docs/pipeline-components/retrievers/chromaembeddingretriever.mdx b/docs-website/docs/pipeline-components/retrievers/chromaembeddingretriever.mdx index d108a3985bd..dd8765e2992 100644 --- a/docs-website/docs/pipeline-components/retrievers/chromaembeddingretriever.mdx +++ b/docs-website/docs/pipeline-components/retrievers/chromaembeddingretriever.mdx @@ -35,7 +35,7 @@ In addition to the `query_embedding`, the `ChromaEmbeddingRetriever` accepts oth #### On its own -This Retriever needs the `ChromaDocumentStore` and indexed documents to run. +This Retriever needs the `ChromaDocumentStore` and indexed documents to run. ```python from haystack_integrations.document_stores.chroma import ChromaDocumentStore diff --git a/docs-website/docs/pipeline-components/retrievers/chromaqueryretriever.mdx b/docs-website/docs/pipeline-components/retrievers/chromaqueryretriever.mdx index bfd11fa8057..98665328a93 100644 --- a/docs-website/docs/pipeline-components/retrievers/chromaqueryretriever.mdx +++ b/docs-website/docs/pipeline-components/retrievers/chromaqueryretriever.mdx @@ -33,7 +33,7 @@ Chroma will create the embedding for the query using its [embedding function](ht #### On its own -This Retriever needs the `ChromaDocumentStore` and indexed documents to run. +This Retriever needs the `ChromaDocumentStore` and indexed documents to run. ```python from haystack_integrations.document_stores.chroma import ChromaDocumentStore diff --git a/docs-website/docs/pipeline-components/retrievers/elasticsearchbm25retriever.mdx b/docs-website/docs/pipeline-components/retrievers/elasticsearchbm25retriever.mdx index cfbd1d357b5..5ddf0fb253b 100644 --- a/docs-website/docs/pipeline-components/retrievers/elasticsearchbm25retriever.mdx +++ b/docs-website/docs/pipeline-components/retrievers/elasticsearchbm25retriever.mdx @@ -36,7 +36,7 @@ If you want a semantic match between a query and documents, you can use `Elastic ## Installation -[Install](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html) Elasticsearch and then [start](https://www.elastic.co/guide/en/elasticsearch/reference/current/starting-elasticsearch.html) an instance. Haystack supports Elasticsearch 8. +[Install](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html) Elasticsearch and then [start](https://www.elastic.co/guide/en/elasticsearch/reference/current/starting-elasticsearch.html) an instance. Haystack supports Elasticsearch 8. If you have Docker set up, we recommend pulling the Docker image and running it. diff --git a/docs-website/docs/pipeline-components/retrievers/elasticsearchembeddingretriever.mdx b/docs-website/docs/pipeline-components/retrievers/elasticsearchembeddingretriever.mdx index b059e951455..f08452d5e44 100644 --- a/docs-website/docs/pipeline-components/retrievers/elasticsearchembeddingretriever.mdx +++ b/docs-website/docs/pipeline-components/retrievers/elasticsearchembeddingretriever.mdx @@ -37,7 +37,7 @@ The `embedding_similarity_function` to use for embedding retrieval must be defin ## Installation -[Install](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html) Elasticsearch and then [start](https://www.elastic.co/guide/en/elasticsearch/reference/current/starting-elasticsearch.html) an instance. Haystack supports Elasticsearch 8. +[Install](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html) Elasticsearch and then [start](https://www.elastic.co/guide/en/elasticsearch/reference/current/starting-elasticsearch.html) an instance. Haystack supports Elasticsearch 8. If you have Docker set up, we recommend pulling the Docker image and running it. diff --git a/docs-website/docs/pipeline-components/retrievers/pgvectorkeywordretriever.mdx b/docs-website/docs/pipeline-components/retrievers/pgvectorkeywordretriever.mdx index e0a4ffc709b..b87f7b50ca0 100644 --- a/docs-website/docs/pipeline-components/retrievers/pgvectorkeywordretriever.mdx +++ b/docs-website/docs/pipeline-components/retrievers/pgvectorkeywordretriever.mdx @@ -33,7 +33,7 @@ For more details, see [Postgres documentation](https://www.postgresql.org/docs/c Keep in mind that, unlike similar components such as `ElasticsearchBM25Retriever`, this Retriever does not apply fuzzy search out of the box, so it’s necessary to carefully formulate the query in order to avoid getting zero results. -In addition to the `query`, the `PgvectorKeywordRetriever` accepts other optional parameters, including `top_k` (the maximum number of documents to retrieve) and `filters` to narrow the search space. +In addition to the `query`, the `PgvectorKeywordRetriever` accepts other optional parameters, including `top_k` (the maximum number of documents to retrieve) and `filters` to narrow the search space. ### Installation @@ -75,7 +75,7 @@ retriever.run(query="my nice query") The prerequisites necessary for running this code are: -- Set an environment variable `OPENAI_API_KEY` with your OpenAI API key. +- Set an environment variable `OPENAI_API_KEY` with your OpenAI API key. - Set an environment variable `PG_CONN_STR` with the connection string to your PostgreSQL database. ```python diff --git a/docs-website/docs/pipeline-components/retrievers/qdrantsparseembeddingretriever.mdx b/docs-website/docs/pipeline-components/retrievers/qdrantsparseembeddingretriever.mdx index 3f0b0588556..116262decdc 100644 --- a/docs-website/docs/pipeline-components/retrievers/qdrantsparseembeddingretriever.mdx +++ b/docs-website/docs/pipeline-components/retrievers/qdrantsparseembeddingretriever.mdx @@ -31,7 +31,7 @@ It compares the query and document sparse embeddings and, based on the outcome, When using the `QdrantSparseEmbeddingRetriever`, make sure it has the query and document sparse embeddings available. You can do so by adding a sparse document Embedder to your indexing pipeline and a sparse text Embedder to your query pipeline. -In addition to the `query_sparse_embedding`, the `QdrantSparseEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of documents to retrieve) and `filters` to narrow down the search space. +In addition to the `query_sparse_embedding`, the `QdrantSparseEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of documents to retrieve) and `filters` to narrow down the search space. :::note[Sparse Embedding Support] diff --git a/docs-website/docs/pipeline-components/routers/llmmessagesrouter.mdx b/docs-website/docs/pipeline-components/routers/llmmessagesrouter.mdx index 370f2f2cb0c..943506f33a3 100644 --- a/docs-website/docs/pipeline-components/routers/llmmessagesrouter.mdx +++ b/docs-website/docs/pipeline-components/routers/llmmessagesrouter.mdx @@ -39,13 +39,13 @@ Each pattern is evaluated in order, and the first match determines the output. T Optionally, you can provide a `system_prompt` to guide the classification behavior of the LLM. In this case as well, we recommend checking the model card to discover customization options. -To see the full list of parameters, check out our [API reference](/reference/routers-api#llmmessagesrouter). +To see the full list of parameters, check out our [API reference](/reference/routers-api#llmmessagesrouter). ## Usage ### On its own -Below is an example of using `LLMMessagesRouter` to route Chat Messages to two output connections based on safety classification. Messages that don’t match any pattern are routed to `unmatched`. +Below is an example of using `LLMMessagesRouter` to route Chat Messages to two output connections based on safety classification. Messages that don’t match any pattern are routed to `unmatched`. We use Llama Guard 4 for content moderation. To use this model with the Hugging Face API, you need to [request access](https://huggingface.co/meta-llama/Llama-Guard-4-12B) and set the `HF_TOKEN` environment variable. diff --git a/docs-website/docs/pipeline-components/samplers/toppsampler.mdx b/docs-website/docs/pipeline-components/samplers/toppsampler.mdx index 3e1ab2e3b23..e0cca93bbca 100644 --- a/docs-website/docs/pipeline-components/samplers/toppsampler.mdx +++ b/docs-website/docs/pipeline-components/samplers/toppsampler.mdx @@ -27,9 +27,9 @@ Uses nucleus sampling to filter documents. Top-P (nucleus) sampling is a method that helps identify and select a subset of documents based on their cumulative probabilities. Instead of choosing a fixed number of documents, this method focuses on a specified percentage of the highest cumulative probabilities within a list of documents. To put it simply, `TopPSampler` provides a way to efficiently select the most relevant documents based on their similarity to a given query. -The practical goal of the `TopPSampler` is to return a list of documents that, in sum, have a score larger than the `top_p` value. So, for example, when `top_p` is set to a high value, more documents will be returned, which can result in more varied outputs. The value is typically set between 0 and 1. By default, the component uses documents' `score` fields to look at the similarity scores. +The practical goal of the `TopPSampler` is to return a list of documents that, in sum, have a score larger than the `top_p` value. So, for example, when `top_p` is set to a high value, more documents will be returned, which can result in more varied outputs. The value is typically set between 0 and 1. By default, the component uses documents' `score` fields to look at the similarity scores. -The component’s `run()` method takes in a set of documents, calculates the similarity scores between the query and the documents, and then filters the documents based on the cumulative probability of these scores. +The component’s `run()` method takes in a set of documents, calculates the similarity scores between the query and the documents, and then filters the documents based on the cumulative probability of these scores. ## Usage diff --git a/docs-website/docs/pipeline-components/tools/toolinvoker.mdx b/docs-website/docs/pipeline-components/tools/toolinvoker.mdx index 634bfd72179..a1014c28cb1 100644 --- a/docs-website/docs/pipeline-components/tools/toolinvoker.mdx +++ b/docs-website/docs/pipeline-components/tools/toolinvoker.mdx @@ -2,12 +2,12 @@ title: "ToolInvoker" id: toolinvoker slug: "/toolinvoker" -description: "This component is designed to execute tool calls prepared by language models. It acts as a bridge between the language model's output and the actual execution of functions or tools that perform specific tasks." +description: "This component is designed to execute tool calls prepared by language models. It acts as a bridge between the language model's output and the actual execution of functions or tools that perform specific tasks." --- # ToolInvoker -This component is designed to execute tool calls prepared by language models. It acts as a bridge between the language model's output and the actual execution of functions or tools that perform specific tasks. +This component is designed to execute tool calls prepared by language models. It acts as a bridge between the language model's output and the actual execution of functions or tools that perform specific tasks.
@@ -25,7 +25,7 @@ This component is designed to execute tool calls prepared by language m ## Overview -A `ToolInvoker` is a component that processes `ChatMessage` objects containing tool calls. It invokes the corresponding tools and returns the results as a list of `ChatMessage` objects. Each tool is defined with a name, description, parameters, and a function that performs the task. The `ToolInvoker` manages these tools and handles the invocation process. +A `ToolInvoker` is a component that processes `ChatMessage` objects containing tool calls. It invokes the corresponding tools and returns the results as a list of `ChatMessage` objects. Each tool is defined with a name, description, parameters, and a function that performs the task. The `ToolInvoker` manages these tools and handles the invocation process. You can pass multiple tools to the `ToolInvoker` component, and it will automatically choose the right tool to call based on tool calls produced by a Language Model. diff --git a/docs-website/docs/pipeline-components/validators/jsonschemavalidator.mdx b/docs-website/docs/pipeline-components/validators/jsonschemavalidator.mdx index aa6ae6496fd..12c6eca7dbc 100644 --- a/docs-website/docs/pipeline-components/validators/jsonschemavalidator.mdx +++ b/docs-website/docs/pipeline-components/validators/jsonschemavalidator.mdx @@ -14,7 +14,7 @@ Use this component to ensure that an LLM-generated chat message JSON adheres to | | | | --- | --- | | **Most common position in a pipeline** | After a [Generator](../generators.mdx) | -| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) instances to be validated – the last message in this list is the one that is validated | +| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) instances to be validated – the last message in this list is the one that is validated | | **Output variables** | `validated`: A list of messages if the last message is valid

`validation_error`: A list of messages if the last message is invalid | | **API reference** | [Validators](/reference/validators-api) | | **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/validators/json_schema.py | @@ -24,7 +24,7 @@ Use this component to ensure that an LLM-generated chat message JSON adheres to ## Overview -`JsonSchemaValidator` checks the JSON content of a `ChatMessage` against a given [JSON Schema](https://json-schema.org/). If a message's JSON content follows the provided schema, it's moved to the `validated` output. If not, it's moved to the `validation_error`output. When there's an error, the component uses either the provided custom `error_template` or a default template to create the error message. These error `ChatMessages` can be used in Haystack recovery loops. +`JsonSchemaValidator` checks the JSON content of a `ChatMessage` against a given [JSON Schema](https://json-schema.org/). If a message's JSON content follows the provided schema, it's moved to the `validated` output. If not, it's moved to the `validation_error`output. When there's an error, the component uses either the provided custom `error_template` or a default template to create the error message. These error `ChatMessages` can be used in Haystack recovery loops. ## Usage diff --git a/docs-website/docs/pipeline-components/websearch/searchapiwebsearch.mdx b/docs-website/docs/pipeline-components/websearch/searchapiwebsearch.mdx index 75f91256075..1ba64a1f50f 100644 --- a/docs-website/docs/pipeline-components/websearch/searchapiwebsearch.mdx +++ b/docs-website/docs/pipeline-components/websearch/searchapiwebsearch.mdx @@ -29,7 +29,7 @@ When you give `SearchApiWebSearch` a query, it returns a list of the URLs most r To search the content of the web pages, use the [`LinkContentFetcher`](../fetchers/linkcontentfetcher.mdx) component. -`SearchApiWebSearch` requires a [SearchApi](https://www.searchapi.io) key to work. It uses a `SEARCHAPI_API_KEY` environment variable by default. Otherwise, you can pass an `api_key` at initialization – see code examples below. +`SearchApiWebSearch` requires a [SearchApi](https://www.searchapi.io) key to work. It uses a `SEARCHAPI_API_KEY` environment variable by default. Otherwise, you can pass an `api_key` at initialization – see code examples below. :::info[Alternative search] @@ -59,7 +59,7 @@ response = web_search.run(query) ### In a pipeline -Here’s an example of a RAG pipeline where we use a `SearchApiWebSearch` to look up the answer to the query. The resulting documents are then passed to `LinkContentFetcher` to get the full text from the URLs. Finally, `ChatPromptBuilder` and `OpenAIChatGenerator` work together to form the final answer. +Here’s an example of a RAG pipeline where we use a `SearchApiWebSearch` to look up the answer to the query. The resulting documents are then passed to `LinkContentFetcher` to get the full text from the URLs. Finally, `ChatPromptBuilder` and `OpenAIChatGenerator` work together to form the final answer. ```python from haystack import Pipeline diff --git a/docs-website/docs/pipeline-components/websearch/serperdevwebsearch.mdx b/docs-website/docs/pipeline-components/websearch/serperdevwebsearch.mdx index 3964e01794e..c544478b3bc 100644 --- a/docs-website/docs/pipeline-components/websearch/serperdevwebsearch.mdx +++ b/docs-website/docs/pipeline-components/websearch/serperdevwebsearch.mdx @@ -29,7 +29,7 @@ When you give `SerperDevWebSearch` a query, it returns a list of the URLs most r To search the content of the web pages, use the [`LinkContentFetcher`](../fetchers/linkcontentfetcher.mdx) component. -`SerperDevWebSearch` requires a [SerperDev](https://serper.dev/) key to work. It uses a `SERPERDEV_API_KEY` environment variable by default. Otherwise, you can pass an `api_key` at initialization – see code examples below. +`SerperDevWebSearch` requires a [SerperDev](https://serper.dev/) key to work. It uses a `SERPERDEV_API_KEY` environment variable by default. Otherwise, you can pass an `api_key` at initialization – see code examples below. :::info[Alternative search] @@ -60,7 +60,7 @@ response = web_search.run(query) ### In a pipeline -Here’s an example of a RAG pipeline where we use a `SerperDevWebSearch` to look up the answer to the query. The resulting documents are then passed to `LinkContentFetcher` to get the full text from the URLs. Finally, `ChatPromptBuilder` and `OpenAIChatGenerator` work together to form the final answer. +Here’s an example of a RAG pipeline where we use a `SerperDevWebSearch` to look up the answer to the query. The resulting documents are then passed to `LinkContentFetcher` to get the full text from the URLs. Finally, `ChatPromptBuilder` and `OpenAIChatGenerator` work together to form the final answer. ```python from haystack import Pipeline diff --git a/docs-website/docs/tools/toolset.mdx b/docs-website/docs/tools/toolset.mdx index 48200d803a9..6c590e07332 100644 --- a/docs-website/docs/tools/toolset.mdx +++ b/docs-website/docs/tools/toolset.mdx @@ -22,7 +22,7 @@ Group multiple Tools into a single unit. ## Overview -A `Toolset` groups multiple Tool instances into a single manageable unit. It simplifies passing tools to components like Chat Generators, [`ToolInvoker`](../pipeline-components/tools/toolinvoker.mdx), or [`Agent`](../pipeline-components/agents-1/agent.mdx), and supports filtering, serialization, and reuse. +A `Toolset` groups multiple Tool instances into a single manageable unit. It simplifies passing tools to components like Chat Generators, [`ToolInvoker`](../pipeline-components/tools/toolinvoker.mdx), or [`Agent`](../pipeline-components/agents-1/agent.mdx), and supports filtering, serialization, and reuse. Additionally, by subclassing `Toolset`, you can create implementations that dynamically load tools from external sources like OpenAPI URLs, MCP servers, or other resources.