Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 25 additions & 7 deletions docs-website/docs/concepts/device-management.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ description: "This page discusses the concept of device management in the contex

This page discusses the concept of device management in the context of Haystack.

Many Haystack components, such as `HuggingFaceLocalGenerator` , `AzureOpenAIGenerator`, and others, allow users the ability to pick and choose which language model is to be queried and executed. For components that interface with cloud-based services, the service provider automatically takes care of the details of provisioning the requisite hardware (like GPUs). However, if you wish to use models on your local machine, you’ll need to figure out how to deploy them on your hardware. Further complicating things, different ML libraries have different APIs to launch models on specific devices.
Many Haystack components, such as `TransformersChatGenerator`, `AzureOpenAIChatGenerator`, and others, allow users the ability to pick and choose which language model is to be queried and executed. For components that interface with cloud-based services, the service provider automatically takes care of the details of provisioning the requisite hardware (like GPUs). However, if you wish to use models on your local machine, you’ll need to figure out how to deploy them on your hardware. Further complicating things, different ML libraries have different APIs to launch models on specific devices.

To make the process of running inference on local models as straightforward as possible, Haystack uses a framework-agnostic device management implementation. Exposing devices through this interface means you no longer need to worry about library-specific invocations and device representations.

Expand All @@ -31,21 +31,35 @@ Find the full code for the abstractions above in the Haystack GitHub [repo](http

## Usage

:::info
The examples below use the [`TransformersChatGenerator`](../pipeline-components/generators/transformerschatgenerator.mdx), which is part of the `transformers-haystack` integration. Install it with:

```bash
pip install transformers-haystack
```
:::

To use a single device for inference, use either the `ComponentDevice.from_single` or `ComponentDevice.from_str` class method:

```python
from haystack.utils import ComponentDevice, Device
from haystack_integrations.components.generators.transformers import (
TransformersChatGenerator,
)

device = ComponentDevice.from_single(Device.gpu(id=1))
# Alternatively, use a PyTorch device string
device = ComponentDevice.from_str("cuda:1")
generator = HuggingFaceLocalGenerator(model="llama2", device=device)
generator = TransformersChatGenerator(model="Qwen/Qwen3-0.6B", device=device)
```

To use multiple devices, use the `ComponentDevice.from_multiple` class method:

```python
from haystack.utils import ComponentDevice, Device, DeviceMap
from haystack_integrations.components.generators.transformers import (
TransformersChatGenerator,
)

device_map = DeviceMap(
{
Expand All @@ -56,7 +70,7 @@ device_map = DeviceMap(
},
)
device = ComponentDevice.from_multiple(device_map)
generator = HuggingFaceLocalGenerator(model="llama2", device=device)
generator = TransformersChatGenerator(model="Qwen/Qwen3-0.6B", device=device)
```

### Integrating Devices in Custom Components
Expand Down Expand Up @@ -115,13 +129,17 @@ c = MyComponent(device=ComponentDevice.from_multiple(DeviceMap({
})))
```

If the component’s backend provides a more specialized API to manage devices, it could add an additional init parameter that acts as a conduit. For instance, `HuggingFaceLocalGenerator` exposes a `huggingface_pipeline_kwargs` parameter through which Hugging Face-specific `device_map` arguments can be passed:
If the component’s backend provides a more specialized API to manage devices, it could add an additional init parameter that acts as a conduit. For instance, `TransformersChatGenerator` exposes a `huggingface_pipeline_kwargs` parameter through which Hugging Face-specific `device_map` arguments can be passed:

```python
generator = HuggingFaceLocalGenerator(
model="llama2",
from haystack_integrations.components.generators.transformers import (
TransformersChatGenerator,
)

generator = TransformersChatGenerator(
model="Qwen/Qwen3-0.6B",
huggingface_pipeline_kwargs={"device_map": "balanced"},
)
```

In such cases, ensure that the parameter precedence and selection behavior is clearly documented. In the case of `HuggingFaceLocalGenerator`, the device map passed through the `huggingface_pipeline_kwargs` parameter overrides the explicit `device` parameter and is documented as such.
In such cases, ensure that the parameter precedence and selection behavior is clearly documented. In the case of `TransformersChatGenerator`, the device map passed through the `huggingface_pipeline_kwargs` parameter overrides the explicit `device` parameter and is documented as such.
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ The Transformer models used in Haystack are designed to be run on GPU-accelerate

Once you have GPU enabled on your machine, you can set the `device` on which a given model for a component is loaded.

For example, to load a model for the `HuggingFaceLocalGenerator`, set `device="ComponentDevice.from_single(Device.gpu(id=0))` or `device = ComponentDevice.from_str("cuda:0")` when initializing.
For example, to load a model for the `TransformersChatGenerator`, set `device=ComponentDevice.from_single(Device.gpu(id=0))` or `device = ComponentDevice.from_str("cuda:0")` when initializing.

You can find more information on the [Device management](../concepts/device-management.mdx) page.

Expand Down
2 changes: 1 addition & 1 deletion docs-website/docs/overview/migration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ If you need help migrating a 1.x node without a 2.x counterpart, open an [issue]
| Crawler | Scrapes text from websites. **Example usage:** To run searches on your website content. | Not Available |
| DocumentClassifier | Classifies documents by attaching metadata to them. **Example usage:** Labeling documents by their characteristic (for example, sentiment). | [TransformersZeroShotDocumentClassifier](../pipeline-components/classifiers/transformerszeroshotdocumentclassifier.mdx) |
| DocumentLanguageClassifier | Detects the language of the documents you pass to it and adds it to the document metadata. | [DocumentLanguageClassifier](../pipeline-components/classifiers/documentlanguageclassifier.mdx) |
| EntityExtractor | Extracts predefined entities out of a piece of text. **Example usage:** Named entity extraction (NER). | [NamedEntityExtractor](../pipeline-components/extractors/namedentityextractor.mdx) |
| EntityExtractor | Extracts predefined entities out of a piece of text. **Example usage:** Named entity extraction (NER). | [NamedEntityExtractor](../pipeline-components/extractors/transformersnamedentityextractor.mdx) |
| FileClassifier | Distinguishes between text, PDF, Markdown, Docx, and HTML files. **Example usage:** Routing files to appropriate converters (for example, it routes PDF files to `PDFToTextConverter`). | [FileTypeRouter](../pipeline-components/routers/filetyperouter.mdx) |
| FileConverter | Cleans and splits documents in different formats. **Example usage:** In indexing pipelines, extracting text from a file and casting it into the Document class format. | [Converters](../pipeline-components/converters.mdx) |
| PreProcessor | Cleans and splits documents. **Example usage:** Normalizing white spaces, getting rid of headers and footers, splitting documents into smaller ones. | [PreProcessors](../pipeline-components/preprocessors.mdx) |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ Below is an example of a RAG pipeline where we use a `PromptBuilder` to render a
```python
from haystack import Pipeline, Document
from haystack.utils import Secret
from haystack.components.generators import OpenAIGenerator
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.components.builders.prompt_builder import PromptBuilder

# in a real world use case documents could come from a retriever, web, or any other source
Expand All @@ -179,7 +179,7 @@ prompt_template = """
p = Pipeline()
p.add_component(instance=PromptBuilder(template=prompt_template), name="prompt_builder")
p.add_component(
instance=OpenAIGenerator(api_key=Secret.from_env_var("OPENAI_API_KEY")),
instance=OpenAIChatGenerator(api_key=Secret.from_env_var("OPENAI_API_KEY")),
name="llm",
)
p.connect("prompt_builder", "llm")
Expand Down Expand Up @@ -278,20 +278,21 @@ components:
model: gpt-5-mini
organization: null
streaming_callback: null
system_prompt: null
timeout: null
type: haystack.components.generators.openai.OpenAIGenerator
tools: null
tools_strict: false
type: haystack.components.generators.chat.openai.OpenAIChatGenerator
prompt_builder:
init_parameters:
required_variables: null
template: "\n Given these documents, answer the question.\\nDocuments:\n\
\ {% for doc in documents %}\n {{ doc.content }}\n {% endfor\
\ %}\n\n \\nQuestion: {{query}}\n \\nAnswer:\n "
required_variables: '*'
template: "\n Given these documents, answer the question.\nDocuments:\n \
\ {% for doc in documents %}\n {{ doc.content }}\n {% endfor %}\n\
\n \nQuestion: {{query}}\n \nAnswer:\n "
variables: null
type: haystack.components.builders.prompt_builder.PromptBuilder
connection_type_validation: true
connections:
- receiver: llm.prompt
- receiver: llm.messages
sender: prompt_builder.prompt
max_runs_per_component: 100
metadata: {}
Expand Down
1 change: 0 additions & 1 deletion docs-website/docs/pipeline-components/extractors.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ slug: "/extractors"
| --- | --- |
| [LLMDocumentContentExtractor](extractors/llmdocumentcontentextractor.mdx) | Extracts textual content from image-based documents using a vision-enabled Large Language Model (LLM). |
| [LLMMetadataExtractor](extractors/llmmetadataextractor.mdx) | Extracts metadata from documents using a Large Language Model. The metadata is extracted by providing a prompt to a LLM that generates it. |
| [NamedEntityExtractor](extractors/namedentityextractor.mdx) | Extracts predefined entities out of a piece of text and writes them into documents' meta field. |
| [PresidioEntityExtractor](extractors/presidioentityextractor.mdx) | Detects PII in Documents and stores entities as structured metadata, without modifying the text. Powered by Microsoft Presidio. |
| [RegexTextExtractor](extractors/regextextextractor.mdx) | Extracts text from chat messages or strings using a regular expression pattern. |
| [SpacyNamedEntityExtractor](extractors/spacynamedentityextractor.mdx) | Extracts predefined entities out of a piece of text and writes them into documents' meta field. Uses a spaCy model. |
Expand Down

This file was deleted.

Loading