diff --git a/docs-website/reference/integrations-api/optimum.md b/docs-website/reference/integrations-api/optimum.md index da0d85c035..e823475a86 100644 --- a/docs-website/reference/integrations-api/optimum.md +++ b/docs-website/reference/integrations-api/optimum.md @@ -5,134 +5,96 @@ description: "Optimum integration for Haystack" slug: "/integrations-optimum" --- - -## Module haystack\_integrations.components.embedders.optimum.optimization - - +## haystack_integrations.components.embedders.optimum.optimization ### OptimumEmbedderOptimizationMode -[ONXX Optimization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) -support by the Optimum Embedders. - - - -#### O1 - -Basic general optimizations. - - - -#### O2 - -Basic and extended general optimizations, transformers-specific fusions. +Bases: Enum - +ONNX Optimization modes supported by the Optimum Embedders. -#### O3 +See [Optimum ONNX optimization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) +for more details. -Same as O2 with Gelu approximation. - - - -#### O4 - -Same as O3 with mixed precision. - - - -#### OptimumEmbedderOptimizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderOptimizationMode" +from_str(string: str) -> OptimumEmbedderOptimizationMode ``` Create an optimization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Optimization mode. +**Returns:** - +- OptimumEmbedderOptimizationMode – Optimization mode. ### OptimumEmbedderOptimizationConfig Configuration for Optimum Embedder Optimization. -**Arguments**: +**Parameters:** -- `mode`: Optimization mode. -- `for_gpu`: Whether to optimize for GPUs. +- **mode** (OptimumEmbedderOptimizationMode) – Optimization mode. +- **for_gpu** (bool) – Whether to optimize for GPUs. - - -#### OptimumEmbedderOptimizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> OptimizationConfig +to_optimum_config() -> OptimizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: - -Optimum configuration. +**Returns:** - +- OptimizationConfig – Optimum configuration. -#### OptimumEmbedderOptimizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumEmbedderOptimizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderOptimizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderOptimizationConfig ``` Create an optimization configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Optimization configuration. +**Returns:** - +- OptimumEmbedderOptimizationConfig – Optimization configuration. -## Module haystack\_integrations.components.embedders.optimum.optimum\_document\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_document_embedder ### OptimumDocumentEmbedder -A component for computing `Document` embeddings using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component for computing `Document` embeddings using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. The embedding of each Document is stored in the `embedding` field of the Document. Usage example: + ```python from haystack.dataclasses import Document from haystack_integrations.components.embedders.optimum import OptimumDocumentEmbedder @@ -140,7 +102,7 @@ from haystack_integrations.components.embedders.optimum import OptimumDocumentEm doc = Document(content="I love pizza!") document_embedder = OptimumDocumentEmbedder(model="sentence-transformers/all-mpnet-base-v2") -document_embedder.warm_up() +# Components warm up automatically on first run. result = document_embedder.run([doc]) print(result["documents"][0].embedding) @@ -148,482 +110,398 @@ print(result["documents"][0].embedding) # [0.017020374536514282, -0.023255806416273117, ...] ``` - - -#### OptimumDocumentEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__(model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig - | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig - | None = None, - batch_size: int = 32, - progress_bar: bool = True, - meta_fields_to_embed: list[str] | None = None, - embedding_separator: str = "\n") -> None +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, + batch_size: int = 32, + progress_bar: bool = True, + meta_fields_to_embed: list[str] | None = None, + embedding_separator: str = "\n", +) -> None ``` Create a OptimumDocumentEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. -- `batch_size`: Number of Documents to encode at once. -- `progress_bar`: Whether to show a progress bar or not. -- `meta_fields_to_embed`: List of meta fields that should be embedded along with the Document text. -- `embedding_separator`: Separator used to concatenate the meta fields to the Document text. - - - -#### OptimumDocumentEmbedder.warm\_up +**Parameters:** + +- **model** (str) – A string representing the model id on HF Hub. + +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. + +- **prefix** (str) – A string to add to the beginning of each text. + +- **suffix** (str) – A string to add to the end of each text. + +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. + +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. + + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: + + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` + +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. + +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. + +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. + +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. + +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. + +- **batch_size** (int) – Number of Documents to encode at once. + +- **progress_bar** (bool) – Whether to show a progress bar or not. + +- **meta_fields_to_embed** (list\[str\] | None) – List of meta fields that should be embedded along with the Document text. + +- **embedding_separator** (str) – Separator used to concatenate the meta fields to the Document text. + +#### warm_up ```python -def warm_up() -> None +warm_up() -> None ``` Initializes the component. - - -#### OptimumDocumentEmbedder.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Serializes the component to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumDocumentEmbedder.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumDocumentEmbedder" +from_dict(data: dict[str, Any]) -> OptimumDocumentEmbedder ``` Deserializes the component from a dictionary. -**Arguments**: - -- `data`: The dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -The deserialized component. +**Returns:** - +- OptimumDocumentEmbedder – The deserialized component. -#### OptimumDocumentEmbedder.run +#### run ```python -@component.output_types(documents=list[Document]) -def run(documents: list[Document]) -> dict[str, list[Document]] +run(documents: list[Document]) -> dict[str, list[Document]] ``` Embed a list of Documents. The embedding of each Document is stored in the `embedding` field of the Document. -**Arguments**: - -- `documents`: A list of Documents to embed. +**Parameters:** -**Raises**: +- **documents** (list\[Document\]) – A list of Documents to embed. -- `TypeError`: If the input is not a list of Documents. +**Returns:** -**Returns**: +- dict\[str, list\[Document\]\] – The updated Documents with their embeddings. -The updated Documents with their embeddings. +**Raises:** - +- TypeError – If the input is not a list of Documents. -## Module haystack\_integrations.components.embedders.optimum.optimum\_text\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_text_embedder ### OptimumTextEmbedder -A component to embed text using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component to embed text using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. Usage example: + ```python from haystack_integrations.components.embedders.optimum import OptimumTextEmbedder text_to_embed = "I love pizza!" text_embedder = OptimumTextEmbedder(model="sentence-transformers/all-mpnet-base-v2") -text_embedder.warm_up() +# Components warm up automatically on first run. print(text_embedder.run(text_to_embed)) # {'embedding': [-0.07804739475250244, 0.1498992145061493,, ...]} ``` - - -#### OptimumTextEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__( - model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig | None = None) +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, +) -> None ``` Create a OptimumTextEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. - - - -#### OptimumTextEmbedder.warm\_up - -```python -def warm_up() -``` +**Parameters:** -Initializes the component. +- **model** (str) – A string representing the model id on HF Hub. - +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. -#### OptimumTextEmbedder.to\_dict +- **prefix** (str) – A string to add to the beginning of each text. -```python -def to_dict() -> dict[str, Any] -``` +- **suffix** (str) – A string to add to the end of each text. -Serializes the component to a dictionary. +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. -**Returns**: +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. -Dictionary with serialized data. + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: - - -#### OptimumTextEmbedder.from\_dict - -```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumTextEmbedder" -``` - -Deserializes the component from a dictionary. + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` -**Arguments**: +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `data`: The dictionary to deserialize from. +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. -**Returns**: +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. -The deserialized component. +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. - +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. -#### OptimumTextEmbedder.run +#### warm_up ```python -@component.output_types(embedding=list[float]) -def run(text: str) -> dict[str, list[float]] +warm_up() -> None ``` -Embed a string. - -**Arguments**: - -- `text`: The text to embed. - -**Raises**: - -- `TypeError`: If the input is not a string. - -**Returns**: +Initializes the component. -The embeddings of the text. +#### to_dict - +```python +to_dict() -> dict[str, Any] +``` -## Module haystack\_integrations.components.embedders.optimum.pooling +Serializes the component to a dictionary. - +**Returns:** -### OptimumEmbedderPooling +- dict\[str, Any\] – Dictionary with serialized data. -Pooling modes support by the Optimum Embedders. +#### from_dict - +```python +from_dict(data: dict[str, Any]) -> OptimumTextEmbedder +``` -#### CLS +Deserializes the component from a dictionary. -Perform CLS Pooling on the output of the embedding model -using the first token (CLS token). +**Parameters:** - +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -#### MEAN +**Returns:** -Perform Mean Pooling on the output of the embedding model. +- OptimumTextEmbedder – The deserialized component. - +#### run -#### MAX +```python +run(text: str) -> dict[str, list[float]] +``` -Perform Max Pooling on the output of the embedding model -using the maximum value in each dimension over all the tokens. +Embed a string. - +**Parameters:** -#### MEAN\_SQRT\_LEN +- **text** (str) – The text to embed. -Perform mean-pooling on the output of the embedding model but -divide by the square root of the sequence length. +**Returns:** - +- dict\[str, list\[float\]\] – The embeddings of the text. -#### WEIGHTED\_MEAN +**Raises:** -Perform weighted (position) mean pooling on the output of the -embedding model. +- TypeError – If the input is not a string. - +## haystack_integrations.components.embedders.optimum.pooling -#### LAST\_TOKEN +### OptimumEmbedderPooling -Perform Last Token Pooling on the output of the embedding model. +Bases: Enum - +Pooling modes support by the Optimum Embedders. -#### OptimumEmbedderPooling.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderPooling" +from_str(string: str) -> OptimumEmbedderPooling ``` Create a pooling mode from a string. -**Arguments**: +**Parameters:** -- `string`: String to convert. +- **string** (str) – String to convert. -**Returns**: +**Returns:** -Pooling mode. +- OptimumEmbedderPooling – Pooling mode. - - -## Module haystack\_integrations.components.embedders.optimum.quantization - - +## haystack_integrations.components.embedders.optimum.quantization ### OptimumEmbedderQuantizationMode -[Dynamic Quantization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) -support by the Optimum Embedders. - - - -#### ARM64 - -Quantization for the ARM64 architecture. - - +Bases: Enum -#### AVX2 +Dynamic Quantization modes supported by the Optimum Embedders. -Quantization with AVX-2 instructions. +See [Optimum ONNX quantization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) +for more details. - - -#### AVX512 - -Quantization with AVX-512 instructions. - - - -#### AVX512\_VNNI - -Quantization with AVX-512 and VNNI instructions. - - - -#### OptimumEmbedderQuantizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderQuantizationMode" +from_str(string: str) -> OptimumEmbedderQuantizationMode ``` Create an quantization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Quantization mode. +**Returns:** - +- OptimumEmbedderQuantizationMode – Quantization mode. ### OptimumEmbedderQuantizationConfig Configuration for Optimum Embedder Quantization. -**Arguments**: - -- `mode`: Quantization mode. -- `per_channel`: Whether to apply per-channel quantization. +**Parameters:** - +- **mode** (OptimumEmbedderQuantizationMode) – Quantization mode. +- **per_channel** (bool) – Whether to apply per-channel quantization. -#### OptimumEmbedderQuantizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> QuantizationConfig +to_optimum_config() -> QuantizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: +**Returns:** -Optimum configuration. +- QuantizationConfig – Optimum configuration. - - -#### OptimumEmbedderQuantizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: - -Dictionary with serialized data. +**Returns:** - +- dict\[str, Any\] – Dictionary with serialized data. -#### OptimumEmbedderQuantizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderQuantizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderQuantizationConfig ``` Create a configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Quantization configuration. +**Returns:** +- OptimumEmbedderQuantizationConfig – Quantization configuration. diff --git a/docs-website/reference_versioned_docs/version-2.18/integrations-api/optimum.md b/docs-website/reference_versioned_docs/version-2.18/integrations-api/optimum.md index 21dd9995df..e823475a86 100644 --- a/docs-website/reference_versioned_docs/version-2.18/integrations-api/optimum.md +++ b/docs-website/reference_versioned_docs/version-2.18/integrations-api/optimum.md @@ -5,134 +5,96 @@ description: "Optimum integration for Haystack" slug: "/integrations-optimum" --- - -## Module haystack\_integrations.components.embedders.optimum.optimization - - +## haystack_integrations.components.embedders.optimum.optimization ### OptimumEmbedderOptimizationMode -[ONXX Optimization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) -support by the Optimum Embedders. - - - -#### O1 - -Basic general optimizations. - - - -#### O2 - -Basic and extended general optimizations, transformers-specific fusions. +Bases: Enum - +ONNX Optimization modes supported by the Optimum Embedders. -#### O3 +See [Optimum ONNX optimization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) +for more details. -Same as O2 with Gelu approximation. - - - -#### O4 - -Same as O3 with mixed precision. - - - -#### OptimumEmbedderOptimizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderOptimizationMode" +from_str(string: str) -> OptimumEmbedderOptimizationMode ``` Create an optimization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Optimization mode. +**Returns:** - +- OptimumEmbedderOptimizationMode – Optimization mode. ### OptimumEmbedderOptimizationConfig Configuration for Optimum Embedder Optimization. -**Arguments**: +**Parameters:** -- `mode`: Optimization mode. -- `for_gpu`: Whether to optimize for GPUs. +- **mode** (OptimumEmbedderOptimizationMode) – Optimization mode. +- **for_gpu** (bool) – Whether to optimize for GPUs. - - -#### OptimumEmbedderOptimizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> OptimizationConfig +to_optimum_config() -> OptimizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: - -Optimum configuration. +**Returns:** - +- OptimizationConfig – Optimum configuration. -#### OptimumEmbedderOptimizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumEmbedderOptimizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderOptimizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderOptimizationConfig ``` Create an optimization configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Optimization configuration. +**Returns:** - +- OptimumEmbedderOptimizationConfig – Optimization configuration. -## Module haystack\_integrations.components.embedders.optimum.optimum\_document\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_document_embedder ### OptimumDocumentEmbedder -A component for computing `Document` embeddings using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component for computing `Document` embeddings using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. The embedding of each Document is stored in the `embedding` field of the Document. Usage example: + ```python from haystack.dataclasses import Document from haystack_integrations.components.embedders.optimum import OptimumDocumentEmbedder @@ -140,7 +102,7 @@ from haystack_integrations.components.embedders.optimum import OptimumDocumentEm doc = Document(content="I love pizza!") document_embedder = OptimumDocumentEmbedder(model="sentence-transformers/all-mpnet-base-v2") -document_embedder.warm_up() +# Components warm up automatically on first run. result = document_embedder.run([doc]) print(result["documents"][0].embedding) @@ -148,481 +110,398 @@ print(result["documents"][0].embedding) # [0.017020374536514282, -0.023255806416273117, ...] ``` - - -#### OptimumDocumentEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__(model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig - | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig - | None = None, - batch_size: int = 32, - progress_bar: bool = True, - meta_fields_to_embed: list[str] | None = None, - embedding_separator: str = "\n") -> None +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, + batch_size: int = 32, + progress_bar: bool = True, + meta_fields_to_embed: list[str] | None = None, + embedding_separator: str = "\n", +) -> None ``` Create a OptimumDocumentEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. -- `batch_size`: Number of Documents to encode at once. -- `progress_bar`: Whether to show a progress bar or not. -- `meta_fields_to_embed`: List of meta fields that should be embedded along with the Document text. -- `embedding_separator`: Separator used to concatenate the meta fields to the Document text. - - - -#### OptimumDocumentEmbedder.warm\_up +**Parameters:** + +- **model** (str) – A string representing the model id on HF Hub. + +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. + +- **prefix** (str) – A string to add to the beginning of each text. + +- **suffix** (str) – A string to add to the end of each text. + +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. + +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. + + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: + + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` + +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. + +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. + +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. + +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. + +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. + +- **batch_size** (int) – Number of Documents to encode at once. + +- **progress_bar** (bool) – Whether to show a progress bar or not. + +- **meta_fields_to_embed** (list\[str\] | None) – List of meta fields that should be embedded along with the Document text. + +- **embedding_separator** (str) – Separator used to concatenate the meta fields to the Document text. + +#### warm_up ```python -def warm_up() -> None +warm_up() -> None ``` Initializes the component. - - -#### OptimumDocumentEmbedder.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Serializes the component to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumDocumentEmbedder.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumDocumentEmbedder" +from_dict(data: dict[str, Any]) -> OptimumDocumentEmbedder ``` Deserializes the component from a dictionary. -**Arguments**: - -- `data`: The dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -The deserialized component. +**Returns:** - +- OptimumDocumentEmbedder – The deserialized component. -#### OptimumDocumentEmbedder.run +#### run ```python -@component.output_types(documents=list[Document]) -def run(documents: list[Document]) -> dict[str, list[Document]] +run(documents: list[Document]) -> dict[str, list[Document]] ``` Embed a list of Documents. The embedding of each Document is stored in the `embedding` field of the Document. -**Arguments**: - -- `documents`: A list of Documents to embed. +**Parameters:** -**Raises**: +- **documents** (list\[Document\]) – A list of Documents to embed. -- `TypeError`: If the input is not a list of Documents. +**Returns:** -**Returns**: +- dict\[str, list\[Document\]\] – The updated Documents with their embeddings. -The updated Documents with their embeddings. +**Raises:** - +- TypeError – If the input is not a list of Documents. -## Module haystack\_integrations.components.embedders.optimum.optimum\_text\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_text_embedder ### OptimumTextEmbedder -A component to embed text using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component to embed text using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. Usage example: + ```python from haystack_integrations.components.embedders.optimum import OptimumTextEmbedder text_to_embed = "I love pizza!" text_embedder = OptimumTextEmbedder(model="sentence-transformers/all-mpnet-base-v2") -text_embedder.warm_up() +# Components warm up automatically on first run. print(text_embedder.run(text_to_embed)) # {'embedding': [-0.07804739475250244, 0.1498992145061493,, ...]} ``` - - -#### OptimumTextEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__( - model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig | None = None) +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, +) -> None ``` Create a OptimumTextEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. - - - -#### OptimumTextEmbedder.warm\_up +**Parameters:** -```python -def warm_up() -``` +- **model** (str) – A string representing the model id on HF Hub. -Initializes the component. +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. - +- **prefix** (str) – A string to add to the beginning of each text. -#### OptimumTextEmbedder.to\_dict +- **suffix** (str) – A string to add to the end of each text. -```python -def to_dict() -> dict[str, Any] -``` - -Serializes the component to a dictionary. +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. -**Returns**: +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. -Dictionary with serialized data. + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: - - -#### OptimumTextEmbedder.from\_dict - -```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumTextEmbedder" -``` - -Deserializes the component from a dictionary. + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` -**Arguments**: +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `data`: The dictionary to deserialize from. +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. -**Returns**: +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. -The deserialized component. +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. - +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. -#### OptimumTextEmbedder.run +#### warm_up ```python -@component.output_types(embedding=list[float]) -def run(text: str) -> dict[str, list[float]] +warm_up() -> None ``` -Embed a string. - -**Arguments**: - -- `text`: The text to embed. - -**Raises**: - -- `TypeError`: If the input is not a string. - -**Returns**: +Initializes the component. -The embeddings of the text. +#### to_dict - +```python +to_dict() -> dict[str, Any] +``` -## Module haystack\_integrations.components.embedders.optimum.pooling +Serializes the component to a dictionary. - +**Returns:** -### OptimumEmbedderPooling +- dict\[str, Any\] – Dictionary with serialized data. -Pooling modes support by the Optimum Embedders. +#### from_dict - +```python +from_dict(data: dict[str, Any]) -> OptimumTextEmbedder +``` -#### CLS +Deserializes the component from a dictionary. -Perform CLS Pooling on the output of the embedding model -using the first token (CLS token). +**Parameters:** - +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -#### MEAN +**Returns:** -Perform Mean Pooling on the output of the embedding model. +- OptimumTextEmbedder – The deserialized component. - +#### run -#### MAX +```python +run(text: str) -> dict[str, list[float]] +``` -Perform Max Pooling on the output of the embedding model -using the maximum value in each dimension over all the tokens. +Embed a string. - +**Parameters:** -#### MEAN\_SQRT\_LEN +- **text** (str) – The text to embed. -Perform mean-pooling on the output of the embedding model but -divide by the square root of the sequence length. +**Returns:** - +- dict\[str, list\[float\]\] – The embeddings of the text. -#### WEIGHTED\_MEAN +**Raises:** -Perform weighted (position) mean pooling on the output of the -embedding model. +- TypeError – If the input is not a string. - +## haystack_integrations.components.embedders.optimum.pooling -#### LAST\_TOKEN +### OptimumEmbedderPooling -Perform Last Token Pooling on the output of the embedding model. +Bases: Enum - +Pooling modes support by the Optimum Embedders. -#### OptimumEmbedderPooling.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderPooling" +from_str(string: str) -> OptimumEmbedderPooling ``` Create a pooling mode from a string. -**Arguments**: - -- `string`: String to convert. - -**Returns**: +**Parameters:** -Pooling mode. +- **string** (str) – String to convert. - +**Returns:** -## Module haystack\_integrations.components.embedders.optimum.quantization +- OptimumEmbedderPooling – Pooling mode. - +## haystack_integrations.components.embedders.optimum.quantization ### OptimumEmbedderQuantizationMode -[Dynamic Quantization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) -support by the Optimum Embedders. - - - -#### ARM64 - -Quantization for the ARM64 architecture. +Bases: Enum - +Dynamic Quantization modes supported by the Optimum Embedders. -#### AVX2 +See [Optimum ONNX quantization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) +for more details. -Quantization with AVX-2 instructions. - - - -#### AVX512 - -Quantization with AVX-512 instructions. - - - -#### AVX512\_VNNI - -Quantization with AVX-512 and VNNI instructions. - - - -#### OptimumEmbedderQuantizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderQuantizationMode" +from_str(string: str) -> OptimumEmbedderQuantizationMode ``` Create an quantization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Quantization mode. +**Returns:** - +- OptimumEmbedderQuantizationMode – Quantization mode. ### OptimumEmbedderQuantizationConfig Configuration for Optimum Embedder Quantization. -**Arguments**: +**Parameters:** -- `mode`: Quantization mode. -- `per_channel`: Whether to apply per-channel quantization. +- **mode** (OptimumEmbedderQuantizationMode) – Quantization mode. +- **per_channel** (bool) – Whether to apply per-channel quantization. - - -#### OptimumEmbedderQuantizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> QuantizationConfig +to_optimum_config() -> QuantizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: - -Optimum configuration. +**Returns:** - +- QuantizationConfig – Optimum configuration. -#### OptimumEmbedderQuantizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: - -Dictionary with serialized data. +**Returns:** - +- dict\[str, Any\] – Dictionary with serialized data. -#### OptimumEmbedderQuantizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderQuantizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderQuantizationConfig ``` Create a configuration from a dictionary. -**Arguments**: +**Parameters:** -- `data`: Dictionary to deserialize from. +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -**Returns**: +**Returns:** -Quantization configuration. +- OptimumEmbedderQuantizationConfig – Quantization configuration. diff --git a/docs-website/reference_versioned_docs/version-2.19/integrations-api/optimum.md b/docs-website/reference_versioned_docs/version-2.19/integrations-api/optimum.md index da0d85c035..e823475a86 100644 --- a/docs-website/reference_versioned_docs/version-2.19/integrations-api/optimum.md +++ b/docs-website/reference_versioned_docs/version-2.19/integrations-api/optimum.md @@ -5,134 +5,96 @@ description: "Optimum integration for Haystack" slug: "/integrations-optimum" --- - -## Module haystack\_integrations.components.embedders.optimum.optimization - - +## haystack_integrations.components.embedders.optimum.optimization ### OptimumEmbedderOptimizationMode -[ONXX Optimization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) -support by the Optimum Embedders. - - - -#### O1 - -Basic general optimizations. - - - -#### O2 - -Basic and extended general optimizations, transformers-specific fusions. +Bases: Enum - +ONNX Optimization modes supported by the Optimum Embedders. -#### O3 +See [Optimum ONNX optimization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) +for more details. -Same as O2 with Gelu approximation. - - - -#### O4 - -Same as O3 with mixed precision. - - - -#### OptimumEmbedderOptimizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderOptimizationMode" +from_str(string: str) -> OptimumEmbedderOptimizationMode ``` Create an optimization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Optimization mode. +**Returns:** - +- OptimumEmbedderOptimizationMode – Optimization mode. ### OptimumEmbedderOptimizationConfig Configuration for Optimum Embedder Optimization. -**Arguments**: +**Parameters:** -- `mode`: Optimization mode. -- `for_gpu`: Whether to optimize for GPUs. +- **mode** (OptimumEmbedderOptimizationMode) – Optimization mode. +- **for_gpu** (bool) – Whether to optimize for GPUs. - - -#### OptimumEmbedderOptimizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> OptimizationConfig +to_optimum_config() -> OptimizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: - -Optimum configuration. +**Returns:** - +- OptimizationConfig – Optimum configuration. -#### OptimumEmbedderOptimizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumEmbedderOptimizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderOptimizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderOptimizationConfig ``` Create an optimization configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Optimization configuration. +**Returns:** - +- OptimumEmbedderOptimizationConfig – Optimization configuration. -## Module haystack\_integrations.components.embedders.optimum.optimum\_document\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_document_embedder ### OptimumDocumentEmbedder -A component for computing `Document` embeddings using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component for computing `Document` embeddings using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. The embedding of each Document is stored in the `embedding` field of the Document. Usage example: + ```python from haystack.dataclasses import Document from haystack_integrations.components.embedders.optimum import OptimumDocumentEmbedder @@ -140,7 +102,7 @@ from haystack_integrations.components.embedders.optimum import OptimumDocumentEm doc = Document(content="I love pizza!") document_embedder = OptimumDocumentEmbedder(model="sentence-transformers/all-mpnet-base-v2") -document_embedder.warm_up() +# Components warm up automatically on first run. result = document_embedder.run([doc]) print(result["documents"][0].embedding) @@ -148,482 +110,398 @@ print(result["documents"][0].embedding) # [0.017020374536514282, -0.023255806416273117, ...] ``` - - -#### OptimumDocumentEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__(model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig - | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig - | None = None, - batch_size: int = 32, - progress_bar: bool = True, - meta_fields_to_embed: list[str] | None = None, - embedding_separator: str = "\n") -> None +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, + batch_size: int = 32, + progress_bar: bool = True, + meta_fields_to_embed: list[str] | None = None, + embedding_separator: str = "\n", +) -> None ``` Create a OptimumDocumentEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. -- `batch_size`: Number of Documents to encode at once. -- `progress_bar`: Whether to show a progress bar or not. -- `meta_fields_to_embed`: List of meta fields that should be embedded along with the Document text. -- `embedding_separator`: Separator used to concatenate the meta fields to the Document text. - - - -#### OptimumDocumentEmbedder.warm\_up +**Parameters:** + +- **model** (str) – A string representing the model id on HF Hub. + +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. + +- **prefix** (str) – A string to add to the beginning of each text. + +- **suffix** (str) – A string to add to the end of each text. + +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. + +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. + + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: + + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` + +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. + +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. + +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. + +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. + +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. + +- **batch_size** (int) – Number of Documents to encode at once. + +- **progress_bar** (bool) – Whether to show a progress bar or not. + +- **meta_fields_to_embed** (list\[str\] | None) – List of meta fields that should be embedded along with the Document text. + +- **embedding_separator** (str) – Separator used to concatenate the meta fields to the Document text. + +#### warm_up ```python -def warm_up() -> None +warm_up() -> None ``` Initializes the component. - - -#### OptimumDocumentEmbedder.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Serializes the component to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumDocumentEmbedder.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumDocumentEmbedder" +from_dict(data: dict[str, Any]) -> OptimumDocumentEmbedder ``` Deserializes the component from a dictionary. -**Arguments**: - -- `data`: The dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -The deserialized component. +**Returns:** - +- OptimumDocumentEmbedder – The deserialized component. -#### OptimumDocumentEmbedder.run +#### run ```python -@component.output_types(documents=list[Document]) -def run(documents: list[Document]) -> dict[str, list[Document]] +run(documents: list[Document]) -> dict[str, list[Document]] ``` Embed a list of Documents. The embedding of each Document is stored in the `embedding` field of the Document. -**Arguments**: - -- `documents`: A list of Documents to embed. +**Parameters:** -**Raises**: +- **documents** (list\[Document\]) – A list of Documents to embed. -- `TypeError`: If the input is not a list of Documents. +**Returns:** -**Returns**: +- dict\[str, list\[Document\]\] – The updated Documents with their embeddings. -The updated Documents with their embeddings. +**Raises:** - +- TypeError – If the input is not a list of Documents. -## Module haystack\_integrations.components.embedders.optimum.optimum\_text\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_text_embedder ### OptimumTextEmbedder -A component to embed text using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component to embed text using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. Usage example: + ```python from haystack_integrations.components.embedders.optimum import OptimumTextEmbedder text_to_embed = "I love pizza!" text_embedder = OptimumTextEmbedder(model="sentence-transformers/all-mpnet-base-v2") -text_embedder.warm_up() +# Components warm up automatically on first run. print(text_embedder.run(text_to_embed)) # {'embedding': [-0.07804739475250244, 0.1498992145061493,, ...]} ``` - - -#### OptimumTextEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__( - model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig | None = None) +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, +) -> None ``` Create a OptimumTextEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. - - - -#### OptimumTextEmbedder.warm\_up - -```python -def warm_up() -``` +**Parameters:** -Initializes the component. +- **model** (str) – A string representing the model id on HF Hub. - +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. -#### OptimumTextEmbedder.to\_dict +- **prefix** (str) – A string to add to the beginning of each text. -```python -def to_dict() -> dict[str, Any] -``` +- **suffix** (str) – A string to add to the end of each text. -Serializes the component to a dictionary. +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. -**Returns**: +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. -Dictionary with serialized data. + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: - - -#### OptimumTextEmbedder.from\_dict - -```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumTextEmbedder" -``` - -Deserializes the component from a dictionary. + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` -**Arguments**: +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `data`: The dictionary to deserialize from. +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. -**Returns**: +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. -The deserialized component. +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. - +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. -#### OptimumTextEmbedder.run +#### warm_up ```python -@component.output_types(embedding=list[float]) -def run(text: str) -> dict[str, list[float]] +warm_up() -> None ``` -Embed a string. - -**Arguments**: - -- `text`: The text to embed. - -**Raises**: - -- `TypeError`: If the input is not a string. - -**Returns**: +Initializes the component. -The embeddings of the text. +#### to_dict - +```python +to_dict() -> dict[str, Any] +``` -## Module haystack\_integrations.components.embedders.optimum.pooling +Serializes the component to a dictionary. - +**Returns:** -### OptimumEmbedderPooling +- dict\[str, Any\] – Dictionary with serialized data. -Pooling modes support by the Optimum Embedders. +#### from_dict - +```python +from_dict(data: dict[str, Any]) -> OptimumTextEmbedder +``` -#### CLS +Deserializes the component from a dictionary. -Perform CLS Pooling on the output of the embedding model -using the first token (CLS token). +**Parameters:** - +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -#### MEAN +**Returns:** -Perform Mean Pooling on the output of the embedding model. +- OptimumTextEmbedder – The deserialized component. - +#### run -#### MAX +```python +run(text: str) -> dict[str, list[float]] +``` -Perform Max Pooling on the output of the embedding model -using the maximum value in each dimension over all the tokens. +Embed a string. - +**Parameters:** -#### MEAN\_SQRT\_LEN +- **text** (str) – The text to embed. -Perform mean-pooling on the output of the embedding model but -divide by the square root of the sequence length. +**Returns:** - +- dict\[str, list\[float\]\] – The embeddings of the text. -#### WEIGHTED\_MEAN +**Raises:** -Perform weighted (position) mean pooling on the output of the -embedding model. +- TypeError – If the input is not a string. - +## haystack_integrations.components.embedders.optimum.pooling -#### LAST\_TOKEN +### OptimumEmbedderPooling -Perform Last Token Pooling on the output of the embedding model. +Bases: Enum - +Pooling modes support by the Optimum Embedders. -#### OptimumEmbedderPooling.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderPooling" +from_str(string: str) -> OptimumEmbedderPooling ``` Create a pooling mode from a string. -**Arguments**: +**Parameters:** -- `string`: String to convert. +- **string** (str) – String to convert. -**Returns**: +**Returns:** -Pooling mode. +- OptimumEmbedderPooling – Pooling mode. - - -## Module haystack\_integrations.components.embedders.optimum.quantization - - +## haystack_integrations.components.embedders.optimum.quantization ### OptimumEmbedderQuantizationMode -[Dynamic Quantization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) -support by the Optimum Embedders. - - - -#### ARM64 - -Quantization for the ARM64 architecture. - - +Bases: Enum -#### AVX2 +Dynamic Quantization modes supported by the Optimum Embedders. -Quantization with AVX-2 instructions. +See [Optimum ONNX quantization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) +for more details. - - -#### AVX512 - -Quantization with AVX-512 instructions. - - - -#### AVX512\_VNNI - -Quantization with AVX-512 and VNNI instructions. - - - -#### OptimumEmbedderQuantizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderQuantizationMode" +from_str(string: str) -> OptimumEmbedderQuantizationMode ``` Create an quantization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Quantization mode. +**Returns:** - +- OptimumEmbedderQuantizationMode – Quantization mode. ### OptimumEmbedderQuantizationConfig Configuration for Optimum Embedder Quantization. -**Arguments**: - -- `mode`: Quantization mode. -- `per_channel`: Whether to apply per-channel quantization. +**Parameters:** - +- **mode** (OptimumEmbedderQuantizationMode) – Quantization mode. +- **per_channel** (bool) – Whether to apply per-channel quantization. -#### OptimumEmbedderQuantizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> QuantizationConfig +to_optimum_config() -> QuantizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: +**Returns:** -Optimum configuration. +- QuantizationConfig – Optimum configuration. - - -#### OptimumEmbedderQuantizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: - -Dictionary with serialized data. +**Returns:** - +- dict\[str, Any\] – Dictionary with serialized data. -#### OptimumEmbedderQuantizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderQuantizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderQuantizationConfig ``` Create a configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Quantization configuration. +**Returns:** +- OptimumEmbedderQuantizationConfig – Quantization configuration. diff --git a/docs-website/reference_versioned_docs/version-2.20/integrations-api/optimum.md b/docs-website/reference_versioned_docs/version-2.20/integrations-api/optimum.md index da0d85c035..e823475a86 100644 --- a/docs-website/reference_versioned_docs/version-2.20/integrations-api/optimum.md +++ b/docs-website/reference_versioned_docs/version-2.20/integrations-api/optimum.md @@ -5,134 +5,96 @@ description: "Optimum integration for Haystack" slug: "/integrations-optimum" --- - -## Module haystack\_integrations.components.embedders.optimum.optimization - - +## haystack_integrations.components.embedders.optimum.optimization ### OptimumEmbedderOptimizationMode -[ONXX Optimization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) -support by the Optimum Embedders. - - - -#### O1 - -Basic general optimizations. - - - -#### O2 - -Basic and extended general optimizations, transformers-specific fusions. +Bases: Enum - +ONNX Optimization modes supported by the Optimum Embedders. -#### O3 +See [Optimum ONNX optimization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) +for more details. -Same as O2 with Gelu approximation. - - - -#### O4 - -Same as O3 with mixed precision. - - - -#### OptimumEmbedderOptimizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderOptimizationMode" +from_str(string: str) -> OptimumEmbedderOptimizationMode ``` Create an optimization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Optimization mode. +**Returns:** - +- OptimumEmbedderOptimizationMode – Optimization mode. ### OptimumEmbedderOptimizationConfig Configuration for Optimum Embedder Optimization. -**Arguments**: +**Parameters:** -- `mode`: Optimization mode. -- `for_gpu`: Whether to optimize for GPUs. +- **mode** (OptimumEmbedderOptimizationMode) – Optimization mode. +- **for_gpu** (bool) – Whether to optimize for GPUs. - - -#### OptimumEmbedderOptimizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> OptimizationConfig +to_optimum_config() -> OptimizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: - -Optimum configuration. +**Returns:** - +- OptimizationConfig – Optimum configuration. -#### OptimumEmbedderOptimizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumEmbedderOptimizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderOptimizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderOptimizationConfig ``` Create an optimization configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Optimization configuration. +**Returns:** - +- OptimumEmbedderOptimizationConfig – Optimization configuration. -## Module haystack\_integrations.components.embedders.optimum.optimum\_document\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_document_embedder ### OptimumDocumentEmbedder -A component for computing `Document` embeddings using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component for computing `Document` embeddings using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. The embedding of each Document is stored in the `embedding` field of the Document. Usage example: + ```python from haystack.dataclasses import Document from haystack_integrations.components.embedders.optimum import OptimumDocumentEmbedder @@ -140,7 +102,7 @@ from haystack_integrations.components.embedders.optimum import OptimumDocumentEm doc = Document(content="I love pizza!") document_embedder = OptimumDocumentEmbedder(model="sentence-transformers/all-mpnet-base-v2") -document_embedder.warm_up() +# Components warm up automatically on first run. result = document_embedder.run([doc]) print(result["documents"][0].embedding) @@ -148,482 +110,398 @@ print(result["documents"][0].embedding) # [0.017020374536514282, -0.023255806416273117, ...] ``` - - -#### OptimumDocumentEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__(model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig - | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig - | None = None, - batch_size: int = 32, - progress_bar: bool = True, - meta_fields_to_embed: list[str] | None = None, - embedding_separator: str = "\n") -> None +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, + batch_size: int = 32, + progress_bar: bool = True, + meta_fields_to_embed: list[str] | None = None, + embedding_separator: str = "\n", +) -> None ``` Create a OptimumDocumentEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. -- `batch_size`: Number of Documents to encode at once. -- `progress_bar`: Whether to show a progress bar or not. -- `meta_fields_to_embed`: List of meta fields that should be embedded along with the Document text. -- `embedding_separator`: Separator used to concatenate the meta fields to the Document text. - - - -#### OptimumDocumentEmbedder.warm\_up +**Parameters:** + +- **model** (str) – A string representing the model id on HF Hub. + +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. + +- **prefix** (str) – A string to add to the beginning of each text. + +- **suffix** (str) – A string to add to the end of each text. + +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. + +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. + + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: + + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` + +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. + +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. + +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. + +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. + +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. + +- **batch_size** (int) – Number of Documents to encode at once. + +- **progress_bar** (bool) – Whether to show a progress bar or not. + +- **meta_fields_to_embed** (list\[str\] | None) – List of meta fields that should be embedded along with the Document text. + +- **embedding_separator** (str) – Separator used to concatenate the meta fields to the Document text. + +#### warm_up ```python -def warm_up() -> None +warm_up() -> None ``` Initializes the component. - - -#### OptimumDocumentEmbedder.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Serializes the component to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumDocumentEmbedder.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumDocumentEmbedder" +from_dict(data: dict[str, Any]) -> OptimumDocumentEmbedder ``` Deserializes the component from a dictionary. -**Arguments**: - -- `data`: The dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -The deserialized component. +**Returns:** - +- OptimumDocumentEmbedder – The deserialized component. -#### OptimumDocumentEmbedder.run +#### run ```python -@component.output_types(documents=list[Document]) -def run(documents: list[Document]) -> dict[str, list[Document]] +run(documents: list[Document]) -> dict[str, list[Document]] ``` Embed a list of Documents. The embedding of each Document is stored in the `embedding` field of the Document. -**Arguments**: - -- `documents`: A list of Documents to embed. +**Parameters:** -**Raises**: +- **documents** (list\[Document\]) – A list of Documents to embed. -- `TypeError`: If the input is not a list of Documents. +**Returns:** -**Returns**: +- dict\[str, list\[Document\]\] – The updated Documents with their embeddings. -The updated Documents with their embeddings. +**Raises:** - +- TypeError – If the input is not a list of Documents. -## Module haystack\_integrations.components.embedders.optimum.optimum\_text\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_text_embedder ### OptimumTextEmbedder -A component to embed text using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component to embed text using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. Usage example: + ```python from haystack_integrations.components.embedders.optimum import OptimumTextEmbedder text_to_embed = "I love pizza!" text_embedder = OptimumTextEmbedder(model="sentence-transformers/all-mpnet-base-v2") -text_embedder.warm_up() +# Components warm up automatically on first run. print(text_embedder.run(text_to_embed)) # {'embedding': [-0.07804739475250244, 0.1498992145061493,, ...]} ``` - - -#### OptimumTextEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__( - model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig | None = None) +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, +) -> None ``` Create a OptimumTextEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. - - - -#### OptimumTextEmbedder.warm\_up - -```python -def warm_up() -``` +**Parameters:** -Initializes the component. +- **model** (str) – A string representing the model id on HF Hub. - +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. -#### OptimumTextEmbedder.to\_dict +- **prefix** (str) – A string to add to the beginning of each text. -```python -def to_dict() -> dict[str, Any] -``` +- **suffix** (str) – A string to add to the end of each text. -Serializes the component to a dictionary. +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. -**Returns**: +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. -Dictionary with serialized data. + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: - - -#### OptimumTextEmbedder.from\_dict - -```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumTextEmbedder" -``` - -Deserializes the component from a dictionary. + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` -**Arguments**: +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `data`: The dictionary to deserialize from. +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. -**Returns**: +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. -The deserialized component. +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. - +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. -#### OptimumTextEmbedder.run +#### warm_up ```python -@component.output_types(embedding=list[float]) -def run(text: str) -> dict[str, list[float]] +warm_up() -> None ``` -Embed a string. - -**Arguments**: - -- `text`: The text to embed. - -**Raises**: - -- `TypeError`: If the input is not a string. - -**Returns**: +Initializes the component. -The embeddings of the text. +#### to_dict - +```python +to_dict() -> dict[str, Any] +``` -## Module haystack\_integrations.components.embedders.optimum.pooling +Serializes the component to a dictionary. - +**Returns:** -### OptimumEmbedderPooling +- dict\[str, Any\] – Dictionary with serialized data. -Pooling modes support by the Optimum Embedders. +#### from_dict - +```python +from_dict(data: dict[str, Any]) -> OptimumTextEmbedder +``` -#### CLS +Deserializes the component from a dictionary. -Perform CLS Pooling on the output of the embedding model -using the first token (CLS token). +**Parameters:** - +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -#### MEAN +**Returns:** -Perform Mean Pooling on the output of the embedding model. +- OptimumTextEmbedder – The deserialized component. - +#### run -#### MAX +```python +run(text: str) -> dict[str, list[float]] +``` -Perform Max Pooling on the output of the embedding model -using the maximum value in each dimension over all the tokens. +Embed a string. - +**Parameters:** -#### MEAN\_SQRT\_LEN +- **text** (str) – The text to embed. -Perform mean-pooling on the output of the embedding model but -divide by the square root of the sequence length. +**Returns:** - +- dict\[str, list\[float\]\] – The embeddings of the text. -#### WEIGHTED\_MEAN +**Raises:** -Perform weighted (position) mean pooling on the output of the -embedding model. +- TypeError – If the input is not a string. - +## haystack_integrations.components.embedders.optimum.pooling -#### LAST\_TOKEN +### OptimumEmbedderPooling -Perform Last Token Pooling on the output of the embedding model. +Bases: Enum - +Pooling modes support by the Optimum Embedders. -#### OptimumEmbedderPooling.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderPooling" +from_str(string: str) -> OptimumEmbedderPooling ``` Create a pooling mode from a string. -**Arguments**: +**Parameters:** -- `string`: String to convert. +- **string** (str) – String to convert. -**Returns**: +**Returns:** -Pooling mode. +- OptimumEmbedderPooling – Pooling mode. - - -## Module haystack\_integrations.components.embedders.optimum.quantization - - +## haystack_integrations.components.embedders.optimum.quantization ### OptimumEmbedderQuantizationMode -[Dynamic Quantization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) -support by the Optimum Embedders. - - - -#### ARM64 - -Quantization for the ARM64 architecture. - - +Bases: Enum -#### AVX2 +Dynamic Quantization modes supported by the Optimum Embedders. -Quantization with AVX-2 instructions. +See [Optimum ONNX quantization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) +for more details. - - -#### AVX512 - -Quantization with AVX-512 instructions. - - - -#### AVX512\_VNNI - -Quantization with AVX-512 and VNNI instructions. - - - -#### OptimumEmbedderQuantizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderQuantizationMode" +from_str(string: str) -> OptimumEmbedderQuantizationMode ``` Create an quantization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Quantization mode. +**Returns:** - +- OptimumEmbedderQuantizationMode – Quantization mode. ### OptimumEmbedderQuantizationConfig Configuration for Optimum Embedder Quantization. -**Arguments**: - -- `mode`: Quantization mode. -- `per_channel`: Whether to apply per-channel quantization. +**Parameters:** - +- **mode** (OptimumEmbedderQuantizationMode) – Quantization mode. +- **per_channel** (bool) – Whether to apply per-channel quantization. -#### OptimumEmbedderQuantizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> QuantizationConfig +to_optimum_config() -> QuantizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: +**Returns:** -Optimum configuration. +- QuantizationConfig – Optimum configuration. - - -#### OptimumEmbedderQuantizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: - -Dictionary with serialized data. +**Returns:** - +- dict\[str, Any\] – Dictionary with serialized data. -#### OptimumEmbedderQuantizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderQuantizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderQuantizationConfig ``` Create a configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Quantization configuration. +**Returns:** +- OptimumEmbedderQuantizationConfig – Quantization configuration. diff --git a/docs-website/reference_versioned_docs/version-2.21/integrations-api/optimum.md b/docs-website/reference_versioned_docs/version-2.21/integrations-api/optimum.md index da0d85c035..e823475a86 100644 --- a/docs-website/reference_versioned_docs/version-2.21/integrations-api/optimum.md +++ b/docs-website/reference_versioned_docs/version-2.21/integrations-api/optimum.md @@ -5,134 +5,96 @@ description: "Optimum integration for Haystack" slug: "/integrations-optimum" --- - -## Module haystack\_integrations.components.embedders.optimum.optimization - - +## haystack_integrations.components.embedders.optimum.optimization ### OptimumEmbedderOptimizationMode -[ONXX Optimization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) -support by the Optimum Embedders. - - - -#### O1 - -Basic general optimizations. - - - -#### O2 - -Basic and extended general optimizations, transformers-specific fusions. +Bases: Enum - +ONNX Optimization modes supported by the Optimum Embedders. -#### O3 +See [Optimum ONNX optimization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) +for more details. -Same as O2 with Gelu approximation. - - - -#### O4 - -Same as O3 with mixed precision. - - - -#### OptimumEmbedderOptimizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderOptimizationMode" +from_str(string: str) -> OptimumEmbedderOptimizationMode ``` Create an optimization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Optimization mode. +**Returns:** - +- OptimumEmbedderOptimizationMode – Optimization mode. ### OptimumEmbedderOptimizationConfig Configuration for Optimum Embedder Optimization. -**Arguments**: +**Parameters:** -- `mode`: Optimization mode. -- `for_gpu`: Whether to optimize for GPUs. +- **mode** (OptimumEmbedderOptimizationMode) – Optimization mode. +- **for_gpu** (bool) – Whether to optimize for GPUs. - - -#### OptimumEmbedderOptimizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> OptimizationConfig +to_optimum_config() -> OptimizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: - -Optimum configuration. +**Returns:** - +- OptimizationConfig – Optimum configuration. -#### OptimumEmbedderOptimizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumEmbedderOptimizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderOptimizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderOptimizationConfig ``` Create an optimization configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Optimization configuration. +**Returns:** - +- OptimumEmbedderOptimizationConfig – Optimization configuration. -## Module haystack\_integrations.components.embedders.optimum.optimum\_document\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_document_embedder ### OptimumDocumentEmbedder -A component for computing `Document` embeddings using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component for computing `Document` embeddings using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. The embedding of each Document is stored in the `embedding` field of the Document. Usage example: + ```python from haystack.dataclasses import Document from haystack_integrations.components.embedders.optimum import OptimumDocumentEmbedder @@ -140,7 +102,7 @@ from haystack_integrations.components.embedders.optimum import OptimumDocumentEm doc = Document(content="I love pizza!") document_embedder = OptimumDocumentEmbedder(model="sentence-transformers/all-mpnet-base-v2") -document_embedder.warm_up() +# Components warm up automatically on first run. result = document_embedder.run([doc]) print(result["documents"][0].embedding) @@ -148,482 +110,398 @@ print(result["documents"][0].embedding) # [0.017020374536514282, -0.023255806416273117, ...] ``` - - -#### OptimumDocumentEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__(model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig - | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig - | None = None, - batch_size: int = 32, - progress_bar: bool = True, - meta_fields_to_embed: list[str] | None = None, - embedding_separator: str = "\n") -> None +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, + batch_size: int = 32, + progress_bar: bool = True, + meta_fields_to_embed: list[str] | None = None, + embedding_separator: str = "\n", +) -> None ``` Create a OptimumDocumentEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. -- `batch_size`: Number of Documents to encode at once. -- `progress_bar`: Whether to show a progress bar or not. -- `meta_fields_to_embed`: List of meta fields that should be embedded along with the Document text. -- `embedding_separator`: Separator used to concatenate the meta fields to the Document text. - - - -#### OptimumDocumentEmbedder.warm\_up +**Parameters:** + +- **model** (str) – A string representing the model id on HF Hub. + +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. + +- **prefix** (str) – A string to add to the beginning of each text. + +- **suffix** (str) – A string to add to the end of each text. + +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. + +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. + + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: + + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` + +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. + +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. + +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. + +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. + +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. + +- **batch_size** (int) – Number of Documents to encode at once. + +- **progress_bar** (bool) – Whether to show a progress bar or not. + +- **meta_fields_to_embed** (list\[str\] | None) – List of meta fields that should be embedded along with the Document text. + +- **embedding_separator** (str) – Separator used to concatenate the meta fields to the Document text. + +#### warm_up ```python -def warm_up() -> None +warm_up() -> None ``` Initializes the component. - - -#### OptimumDocumentEmbedder.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Serializes the component to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumDocumentEmbedder.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumDocumentEmbedder" +from_dict(data: dict[str, Any]) -> OptimumDocumentEmbedder ``` Deserializes the component from a dictionary. -**Arguments**: - -- `data`: The dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -The deserialized component. +**Returns:** - +- OptimumDocumentEmbedder – The deserialized component. -#### OptimumDocumentEmbedder.run +#### run ```python -@component.output_types(documents=list[Document]) -def run(documents: list[Document]) -> dict[str, list[Document]] +run(documents: list[Document]) -> dict[str, list[Document]] ``` Embed a list of Documents. The embedding of each Document is stored in the `embedding` field of the Document. -**Arguments**: - -- `documents`: A list of Documents to embed. +**Parameters:** -**Raises**: +- **documents** (list\[Document\]) – A list of Documents to embed. -- `TypeError`: If the input is not a list of Documents. +**Returns:** -**Returns**: +- dict\[str, list\[Document\]\] – The updated Documents with their embeddings. -The updated Documents with their embeddings. +**Raises:** - +- TypeError – If the input is not a list of Documents. -## Module haystack\_integrations.components.embedders.optimum.optimum\_text\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_text_embedder ### OptimumTextEmbedder -A component to embed text using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component to embed text using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. Usage example: + ```python from haystack_integrations.components.embedders.optimum import OptimumTextEmbedder text_to_embed = "I love pizza!" text_embedder = OptimumTextEmbedder(model="sentence-transformers/all-mpnet-base-v2") -text_embedder.warm_up() +# Components warm up automatically on first run. print(text_embedder.run(text_to_embed)) # {'embedding': [-0.07804739475250244, 0.1498992145061493,, ...]} ``` - - -#### OptimumTextEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__( - model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig | None = None) +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, +) -> None ``` Create a OptimumTextEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. - - - -#### OptimumTextEmbedder.warm\_up - -```python -def warm_up() -``` +**Parameters:** -Initializes the component. +- **model** (str) – A string representing the model id on HF Hub. - +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. -#### OptimumTextEmbedder.to\_dict +- **prefix** (str) – A string to add to the beginning of each text. -```python -def to_dict() -> dict[str, Any] -``` +- **suffix** (str) – A string to add to the end of each text. -Serializes the component to a dictionary. +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. -**Returns**: +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. -Dictionary with serialized data. + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: - - -#### OptimumTextEmbedder.from\_dict - -```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumTextEmbedder" -``` - -Deserializes the component from a dictionary. + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` -**Arguments**: +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `data`: The dictionary to deserialize from. +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. -**Returns**: +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. -The deserialized component. +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. - +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. -#### OptimumTextEmbedder.run +#### warm_up ```python -@component.output_types(embedding=list[float]) -def run(text: str) -> dict[str, list[float]] +warm_up() -> None ``` -Embed a string. - -**Arguments**: - -- `text`: The text to embed. - -**Raises**: - -- `TypeError`: If the input is not a string. - -**Returns**: +Initializes the component. -The embeddings of the text. +#### to_dict - +```python +to_dict() -> dict[str, Any] +``` -## Module haystack\_integrations.components.embedders.optimum.pooling +Serializes the component to a dictionary. - +**Returns:** -### OptimumEmbedderPooling +- dict\[str, Any\] – Dictionary with serialized data. -Pooling modes support by the Optimum Embedders. +#### from_dict - +```python +from_dict(data: dict[str, Any]) -> OptimumTextEmbedder +``` -#### CLS +Deserializes the component from a dictionary. -Perform CLS Pooling on the output of the embedding model -using the first token (CLS token). +**Parameters:** - +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -#### MEAN +**Returns:** -Perform Mean Pooling on the output of the embedding model. +- OptimumTextEmbedder – The deserialized component. - +#### run -#### MAX +```python +run(text: str) -> dict[str, list[float]] +``` -Perform Max Pooling on the output of the embedding model -using the maximum value in each dimension over all the tokens. +Embed a string. - +**Parameters:** -#### MEAN\_SQRT\_LEN +- **text** (str) – The text to embed. -Perform mean-pooling on the output of the embedding model but -divide by the square root of the sequence length. +**Returns:** - +- dict\[str, list\[float\]\] – The embeddings of the text. -#### WEIGHTED\_MEAN +**Raises:** -Perform weighted (position) mean pooling on the output of the -embedding model. +- TypeError – If the input is not a string. - +## haystack_integrations.components.embedders.optimum.pooling -#### LAST\_TOKEN +### OptimumEmbedderPooling -Perform Last Token Pooling on the output of the embedding model. +Bases: Enum - +Pooling modes support by the Optimum Embedders. -#### OptimumEmbedderPooling.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderPooling" +from_str(string: str) -> OptimumEmbedderPooling ``` Create a pooling mode from a string. -**Arguments**: +**Parameters:** -- `string`: String to convert. +- **string** (str) – String to convert. -**Returns**: +**Returns:** -Pooling mode. +- OptimumEmbedderPooling – Pooling mode. - - -## Module haystack\_integrations.components.embedders.optimum.quantization - - +## haystack_integrations.components.embedders.optimum.quantization ### OptimumEmbedderQuantizationMode -[Dynamic Quantization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) -support by the Optimum Embedders. - - - -#### ARM64 - -Quantization for the ARM64 architecture. - - +Bases: Enum -#### AVX2 +Dynamic Quantization modes supported by the Optimum Embedders. -Quantization with AVX-2 instructions. +See [Optimum ONNX quantization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) +for more details. - - -#### AVX512 - -Quantization with AVX-512 instructions. - - - -#### AVX512\_VNNI - -Quantization with AVX-512 and VNNI instructions. - - - -#### OptimumEmbedderQuantizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderQuantizationMode" +from_str(string: str) -> OptimumEmbedderQuantizationMode ``` Create an quantization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Quantization mode. +**Returns:** - +- OptimumEmbedderQuantizationMode – Quantization mode. ### OptimumEmbedderQuantizationConfig Configuration for Optimum Embedder Quantization. -**Arguments**: - -- `mode`: Quantization mode. -- `per_channel`: Whether to apply per-channel quantization. +**Parameters:** - +- **mode** (OptimumEmbedderQuantizationMode) – Quantization mode. +- **per_channel** (bool) – Whether to apply per-channel quantization. -#### OptimumEmbedderQuantizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> QuantizationConfig +to_optimum_config() -> QuantizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: +**Returns:** -Optimum configuration. +- QuantizationConfig – Optimum configuration. - - -#### OptimumEmbedderQuantizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: - -Dictionary with serialized data. +**Returns:** - +- dict\[str, Any\] – Dictionary with serialized data. -#### OptimumEmbedderQuantizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderQuantizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderQuantizationConfig ``` Create a configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Quantization configuration. +**Returns:** +- OptimumEmbedderQuantizationConfig – Quantization configuration. diff --git a/docs-website/reference_versioned_docs/version-2.22/integrations-api/optimum.md b/docs-website/reference_versioned_docs/version-2.22/integrations-api/optimum.md index da0d85c035..e823475a86 100644 --- a/docs-website/reference_versioned_docs/version-2.22/integrations-api/optimum.md +++ b/docs-website/reference_versioned_docs/version-2.22/integrations-api/optimum.md @@ -5,134 +5,96 @@ description: "Optimum integration for Haystack" slug: "/integrations-optimum" --- - -## Module haystack\_integrations.components.embedders.optimum.optimization - - +## haystack_integrations.components.embedders.optimum.optimization ### OptimumEmbedderOptimizationMode -[ONXX Optimization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) -support by the Optimum Embedders. - - - -#### O1 - -Basic general optimizations. - - - -#### O2 - -Basic and extended general optimizations, transformers-specific fusions. +Bases: Enum - +ONNX Optimization modes supported by the Optimum Embedders. -#### O3 +See [Optimum ONNX optimization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) +for more details. -Same as O2 with Gelu approximation. - - - -#### O4 - -Same as O3 with mixed precision. - - - -#### OptimumEmbedderOptimizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderOptimizationMode" +from_str(string: str) -> OptimumEmbedderOptimizationMode ``` Create an optimization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Optimization mode. +**Returns:** - +- OptimumEmbedderOptimizationMode – Optimization mode. ### OptimumEmbedderOptimizationConfig Configuration for Optimum Embedder Optimization. -**Arguments**: +**Parameters:** -- `mode`: Optimization mode. -- `for_gpu`: Whether to optimize for GPUs. +- **mode** (OptimumEmbedderOptimizationMode) – Optimization mode. +- **for_gpu** (bool) – Whether to optimize for GPUs. - - -#### OptimumEmbedderOptimizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> OptimizationConfig +to_optimum_config() -> OptimizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: - -Optimum configuration. +**Returns:** - +- OptimizationConfig – Optimum configuration. -#### OptimumEmbedderOptimizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumEmbedderOptimizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderOptimizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderOptimizationConfig ``` Create an optimization configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Optimization configuration. +**Returns:** - +- OptimumEmbedderOptimizationConfig – Optimization configuration. -## Module haystack\_integrations.components.embedders.optimum.optimum\_document\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_document_embedder ### OptimumDocumentEmbedder -A component for computing `Document` embeddings using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component for computing `Document` embeddings using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. The embedding of each Document is stored in the `embedding` field of the Document. Usage example: + ```python from haystack.dataclasses import Document from haystack_integrations.components.embedders.optimum import OptimumDocumentEmbedder @@ -140,7 +102,7 @@ from haystack_integrations.components.embedders.optimum import OptimumDocumentEm doc = Document(content="I love pizza!") document_embedder = OptimumDocumentEmbedder(model="sentence-transformers/all-mpnet-base-v2") -document_embedder.warm_up() +# Components warm up automatically on first run. result = document_embedder.run([doc]) print(result["documents"][0].embedding) @@ -148,482 +110,398 @@ print(result["documents"][0].embedding) # [0.017020374536514282, -0.023255806416273117, ...] ``` - - -#### OptimumDocumentEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__(model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig - | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig - | None = None, - batch_size: int = 32, - progress_bar: bool = True, - meta_fields_to_embed: list[str] | None = None, - embedding_separator: str = "\n") -> None +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, + batch_size: int = 32, + progress_bar: bool = True, + meta_fields_to_embed: list[str] | None = None, + embedding_separator: str = "\n", +) -> None ``` Create a OptimumDocumentEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. -- `batch_size`: Number of Documents to encode at once. -- `progress_bar`: Whether to show a progress bar or not. -- `meta_fields_to_embed`: List of meta fields that should be embedded along with the Document text. -- `embedding_separator`: Separator used to concatenate the meta fields to the Document text. - - - -#### OptimumDocumentEmbedder.warm\_up +**Parameters:** + +- **model** (str) – A string representing the model id on HF Hub. + +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. + +- **prefix** (str) – A string to add to the beginning of each text. + +- **suffix** (str) – A string to add to the end of each text. + +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. + +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. + + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: + + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` + +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. + +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. + +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. + +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. + +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. + +- **batch_size** (int) – Number of Documents to encode at once. + +- **progress_bar** (bool) – Whether to show a progress bar or not. + +- **meta_fields_to_embed** (list\[str\] | None) – List of meta fields that should be embedded along with the Document text. + +- **embedding_separator** (str) – Separator used to concatenate the meta fields to the Document text. + +#### warm_up ```python -def warm_up() -> None +warm_up() -> None ``` Initializes the component. - - -#### OptimumDocumentEmbedder.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Serializes the component to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumDocumentEmbedder.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumDocumentEmbedder" +from_dict(data: dict[str, Any]) -> OptimumDocumentEmbedder ``` Deserializes the component from a dictionary. -**Arguments**: - -- `data`: The dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -The deserialized component. +**Returns:** - +- OptimumDocumentEmbedder – The deserialized component. -#### OptimumDocumentEmbedder.run +#### run ```python -@component.output_types(documents=list[Document]) -def run(documents: list[Document]) -> dict[str, list[Document]] +run(documents: list[Document]) -> dict[str, list[Document]] ``` Embed a list of Documents. The embedding of each Document is stored in the `embedding` field of the Document. -**Arguments**: - -- `documents`: A list of Documents to embed. +**Parameters:** -**Raises**: +- **documents** (list\[Document\]) – A list of Documents to embed. -- `TypeError`: If the input is not a list of Documents. +**Returns:** -**Returns**: +- dict\[str, list\[Document\]\] – The updated Documents with their embeddings. -The updated Documents with their embeddings. +**Raises:** - +- TypeError – If the input is not a list of Documents. -## Module haystack\_integrations.components.embedders.optimum.optimum\_text\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_text_embedder ### OptimumTextEmbedder -A component to embed text using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component to embed text using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. Usage example: + ```python from haystack_integrations.components.embedders.optimum import OptimumTextEmbedder text_to_embed = "I love pizza!" text_embedder = OptimumTextEmbedder(model="sentence-transformers/all-mpnet-base-v2") -text_embedder.warm_up() +# Components warm up automatically on first run. print(text_embedder.run(text_to_embed)) # {'embedding': [-0.07804739475250244, 0.1498992145061493,, ...]} ``` - - -#### OptimumTextEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__( - model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig | None = None) +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, +) -> None ``` Create a OptimumTextEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. - - - -#### OptimumTextEmbedder.warm\_up - -```python -def warm_up() -``` +**Parameters:** -Initializes the component. +- **model** (str) – A string representing the model id on HF Hub. - +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. -#### OptimumTextEmbedder.to\_dict +- **prefix** (str) – A string to add to the beginning of each text. -```python -def to_dict() -> dict[str, Any] -``` +- **suffix** (str) – A string to add to the end of each text. -Serializes the component to a dictionary. +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. -**Returns**: +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. -Dictionary with serialized data. + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: - - -#### OptimumTextEmbedder.from\_dict - -```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumTextEmbedder" -``` - -Deserializes the component from a dictionary. + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` -**Arguments**: +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `data`: The dictionary to deserialize from. +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. -**Returns**: +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. -The deserialized component. +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. - +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. -#### OptimumTextEmbedder.run +#### warm_up ```python -@component.output_types(embedding=list[float]) -def run(text: str) -> dict[str, list[float]] +warm_up() -> None ``` -Embed a string. - -**Arguments**: - -- `text`: The text to embed. - -**Raises**: - -- `TypeError`: If the input is not a string. - -**Returns**: +Initializes the component. -The embeddings of the text. +#### to_dict - +```python +to_dict() -> dict[str, Any] +``` -## Module haystack\_integrations.components.embedders.optimum.pooling +Serializes the component to a dictionary. - +**Returns:** -### OptimumEmbedderPooling +- dict\[str, Any\] – Dictionary with serialized data. -Pooling modes support by the Optimum Embedders. +#### from_dict - +```python +from_dict(data: dict[str, Any]) -> OptimumTextEmbedder +``` -#### CLS +Deserializes the component from a dictionary. -Perform CLS Pooling on the output of the embedding model -using the first token (CLS token). +**Parameters:** - +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -#### MEAN +**Returns:** -Perform Mean Pooling on the output of the embedding model. +- OptimumTextEmbedder – The deserialized component. - +#### run -#### MAX +```python +run(text: str) -> dict[str, list[float]] +``` -Perform Max Pooling on the output of the embedding model -using the maximum value in each dimension over all the tokens. +Embed a string. - +**Parameters:** -#### MEAN\_SQRT\_LEN +- **text** (str) – The text to embed. -Perform mean-pooling on the output of the embedding model but -divide by the square root of the sequence length. +**Returns:** - +- dict\[str, list\[float\]\] – The embeddings of the text. -#### WEIGHTED\_MEAN +**Raises:** -Perform weighted (position) mean pooling on the output of the -embedding model. +- TypeError – If the input is not a string. - +## haystack_integrations.components.embedders.optimum.pooling -#### LAST\_TOKEN +### OptimumEmbedderPooling -Perform Last Token Pooling on the output of the embedding model. +Bases: Enum - +Pooling modes support by the Optimum Embedders. -#### OptimumEmbedderPooling.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderPooling" +from_str(string: str) -> OptimumEmbedderPooling ``` Create a pooling mode from a string. -**Arguments**: +**Parameters:** -- `string`: String to convert. +- **string** (str) – String to convert. -**Returns**: +**Returns:** -Pooling mode. +- OptimumEmbedderPooling – Pooling mode. - - -## Module haystack\_integrations.components.embedders.optimum.quantization - - +## haystack_integrations.components.embedders.optimum.quantization ### OptimumEmbedderQuantizationMode -[Dynamic Quantization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) -support by the Optimum Embedders. - - - -#### ARM64 - -Quantization for the ARM64 architecture. - - +Bases: Enum -#### AVX2 +Dynamic Quantization modes supported by the Optimum Embedders. -Quantization with AVX-2 instructions. +See [Optimum ONNX quantization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) +for more details. - - -#### AVX512 - -Quantization with AVX-512 instructions. - - - -#### AVX512\_VNNI - -Quantization with AVX-512 and VNNI instructions. - - - -#### OptimumEmbedderQuantizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderQuantizationMode" +from_str(string: str) -> OptimumEmbedderQuantizationMode ``` Create an quantization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Quantization mode. +**Returns:** - +- OptimumEmbedderQuantizationMode – Quantization mode. ### OptimumEmbedderQuantizationConfig Configuration for Optimum Embedder Quantization. -**Arguments**: - -- `mode`: Quantization mode. -- `per_channel`: Whether to apply per-channel quantization. +**Parameters:** - +- **mode** (OptimumEmbedderQuantizationMode) – Quantization mode. +- **per_channel** (bool) – Whether to apply per-channel quantization. -#### OptimumEmbedderQuantizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> QuantizationConfig +to_optimum_config() -> QuantizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: +**Returns:** -Optimum configuration. +- QuantizationConfig – Optimum configuration. - - -#### OptimumEmbedderQuantizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: - -Dictionary with serialized data. +**Returns:** - +- dict\[str, Any\] – Dictionary with serialized data. -#### OptimumEmbedderQuantizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderQuantizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderQuantizationConfig ``` Create a configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Quantization configuration. +**Returns:** +- OptimumEmbedderQuantizationConfig – Quantization configuration. diff --git a/docs-website/reference_versioned_docs/version-2.23/integrations-api/optimum.md b/docs-website/reference_versioned_docs/version-2.23/integrations-api/optimum.md index da0d85c035..e823475a86 100644 --- a/docs-website/reference_versioned_docs/version-2.23/integrations-api/optimum.md +++ b/docs-website/reference_versioned_docs/version-2.23/integrations-api/optimum.md @@ -5,134 +5,96 @@ description: "Optimum integration for Haystack" slug: "/integrations-optimum" --- - -## Module haystack\_integrations.components.embedders.optimum.optimization - - +## haystack_integrations.components.embedders.optimum.optimization ### OptimumEmbedderOptimizationMode -[ONXX Optimization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) -support by the Optimum Embedders. - - - -#### O1 - -Basic general optimizations. - - - -#### O2 - -Basic and extended general optimizations, transformers-specific fusions. +Bases: Enum - +ONNX Optimization modes supported by the Optimum Embedders. -#### O3 +See [Optimum ONNX optimization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) +for more details. -Same as O2 with Gelu approximation. - - - -#### O4 - -Same as O3 with mixed precision. - - - -#### OptimumEmbedderOptimizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderOptimizationMode" +from_str(string: str) -> OptimumEmbedderOptimizationMode ``` Create an optimization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Optimization mode. +**Returns:** - +- OptimumEmbedderOptimizationMode – Optimization mode. ### OptimumEmbedderOptimizationConfig Configuration for Optimum Embedder Optimization. -**Arguments**: +**Parameters:** -- `mode`: Optimization mode. -- `for_gpu`: Whether to optimize for GPUs. +- **mode** (OptimumEmbedderOptimizationMode) – Optimization mode. +- **for_gpu** (bool) – Whether to optimize for GPUs. - - -#### OptimumEmbedderOptimizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> OptimizationConfig +to_optimum_config() -> OptimizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: - -Optimum configuration. +**Returns:** - +- OptimizationConfig – Optimum configuration. -#### OptimumEmbedderOptimizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumEmbedderOptimizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderOptimizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderOptimizationConfig ``` Create an optimization configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Optimization configuration. +**Returns:** - +- OptimumEmbedderOptimizationConfig – Optimization configuration. -## Module haystack\_integrations.components.embedders.optimum.optimum\_document\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_document_embedder ### OptimumDocumentEmbedder -A component for computing `Document` embeddings using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component for computing `Document` embeddings using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. The embedding of each Document is stored in the `embedding` field of the Document. Usage example: + ```python from haystack.dataclasses import Document from haystack_integrations.components.embedders.optimum import OptimumDocumentEmbedder @@ -140,7 +102,7 @@ from haystack_integrations.components.embedders.optimum import OptimumDocumentEm doc = Document(content="I love pizza!") document_embedder = OptimumDocumentEmbedder(model="sentence-transformers/all-mpnet-base-v2") -document_embedder.warm_up() +# Components warm up automatically on first run. result = document_embedder.run([doc]) print(result["documents"][0].embedding) @@ -148,482 +110,398 @@ print(result["documents"][0].embedding) # [0.017020374536514282, -0.023255806416273117, ...] ``` - - -#### OptimumDocumentEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__(model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig - | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig - | None = None, - batch_size: int = 32, - progress_bar: bool = True, - meta_fields_to_embed: list[str] | None = None, - embedding_separator: str = "\n") -> None +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, + batch_size: int = 32, + progress_bar: bool = True, + meta_fields_to_embed: list[str] | None = None, + embedding_separator: str = "\n", +) -> None ``` Create a OptimumDocumentEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. -- `batch_size`: Number of Documents to encode at once. -- `progress_bar`: Whether to show a progress bar or not. -- `meta_fields_to_embed`: List of meta fields that should be embedded along with the Document text. -- `embedding_separator`: Separator used to concatenate the meta fields to the Document text. - - - -#### OptimumDocumentEmbedder.warm\_up +**Parameters:** + +- **model** (str) – A string representing the model id on HF Hub. + +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. + +- **prefix** (str) – A string to add to the beginning of each text. + +- **suffix** (str) – A string to add to the end of each text. + +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. + +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. + + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: + + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` + +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. + +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. + +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. + +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. + +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. + +- **batch_size** (int) – Number of Documents to encode at once. + +- **progress_bar** (bool) – Whether to show a progress bar or not. + +- **meta_fields_to_embed** (list\[str\] | None) – List of meta fields that should be embedded along with the Document text. + +- **embedding_separator** (str) – Separator used to concatenate the meta fields to the Document text. + +#### warm_up ```python -def warm_up() -> None +warm_up() -> None ``` Initializes the component. - - -#### OptimumDocumentEmbedder.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Serializes the component to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumDocumentEmbedder.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumDocumentEmbedder" +from_dict(data: dict[str, Any]) -> OptimumDocumentEmbedder ``` Deserializes the component from a dictionary. -**Arguments**: - -- `data`: The dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -The deserialized component. +**Returns:** - +- OptimumDocumentEmbedder – The deserialized component. -#### OptimumDocumentEmbedder.run +#### run ```python -@component.output_types(documents=list[Document]) -def run(documents: list[Document]) -> dict[str, list[Document]] +run(documents: list[Document]) -> dict[str, list[Document]] ``` Embed a list of Documents. The embedding of each Document is stored in the `embedding` field of the Document. -**Arguments**: - -- `documents`: A list of Documents to embed. +**Parameters:** -**Raises**: +- **documents** (list\[Document\]) – A list of Documents to embed. -- `TypeError`: If the input is not a list of Documents. +**Returns:** -**Returns**: +- dict\[str, list\[Document\]\] – The updated Documents with their embeddings. -The updated Documents with their embeddings. +**Raises:** - +- TypeError – If the input is not a list of Documents. -## Module haystack\_integrations.components.embedders.optimum.optimum\_text\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_text_embedder ### OptimumTextEmbedder -A component to embed text using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component to embed text using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. Usage example: + ```python from haystack_integrations.components.embedders.optimum import OptimumTextEmbedder text_to_embed = "I love pizza!" text_embedder = OptimumTextEmbedder(model="sentence-transformers/all-mpnet-base-v2") -text_embedder.warm_up() +# Components warm up automatically on first run. print(text_embedder.run(text_to_embed)) # {'embedding': [-0.07804739475250244, 0.1498992145061493,, ...]} ``` - - -#### OptimumTextEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__( - model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig | None = None) +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, +) -> None ``` Create a OptimumTextEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. - - - -#### OptimumTextEmbedder.warm\_up - -```python -def warm_up() -``` +**Parameters:** -Initializes the component. +- **model** (str) – A string representing the model id on HF Hub. - +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. -#### OptimumTextEmbedder.to\_dict +- **prefix** (str) – A string to add to the beginning of each text. -```python -def to_dict() -> dict[str, Any] -``` +- **suffix** (str) – A string to add to the end of each text. -Serializes the component to a dictionary. +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. -**Returns**: +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. -Dictionary with serialized data. + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: - - -#### OptimumTextEmbedder.from\_dict - -```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumTextEmbedder" -``` - -Deserializes the component from a dictionary. + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` -**Arguments**: +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `data`: The dictionary to deserialize from. +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. -**Returns**: +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. -The deserialized component. +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. - +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. -#### OptimumTextEmbedder.run +#### warm_up ```python -@component.output_types(embedding=list[float]) -def run(text: str) -> dict[str, list[float]] +warm_up() -> None ``` -Embed a string. - -**Arguments**: - -- `text`: The text to embed. - -**Raises**: - -- `TypeError`: If the input is not a string. - -**Returns**: +Initializes the component. -The embeddings of the text. +#### to_dict - +```python +to_dict() -> dict[str, Any] +``` -## Module haystack\_integrations.components.embedders.optimum.pooling +Serializes the component to a dictionary. - +**Returns:** -### OptimumEmbedderPooling +- dict\[str, Any\] – Dictionary with serialized data. -Pooling modes support by the Optimum Embedders. +#### from_dict - +```python +from_dict(data: dict[str, Any]) -> OptimumTextEmbedder +``` -#### CLS +Deserializes the component from a dictionary. -Perform CLS Pooling on the output of the embedding model -using the first token (CLS token). +**Parameters:** - +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -#### MEAN +**Returns:** -Perform Mean Pooling on the output of the embedding model. +- OptimumTextEmbedder – The deserialized component. - +#### run -#### MAX +```python +run(text: str) -> dict[str, list[float]] +``` -Perform Max Pooling on the output of the embedding model -using the maximum value in each dimension over all the tokens. +Embed a string. - +**Parameters:** -#### MEAN\_SQRT\_LEN +- **text** (str) – The text to embed. -Perform mean-pooling on the output of the embedding model but -divide by the square root of the sequence length. +**Returns:** - +- dict\[str, list\[float\]\] – The embeddings of the text. -#### WEIGHTED\_MEAN +**Raises:** -Perform weighted (position) mean pooling on the output of the -embedding model. +- TypeError – If the input is not a string. - +## haystack_integrations.components.embedders.optimum.pooling -#### LAST\_TOKEN +### OptimumEmbedderPooling -Perform Last Token Pooling on the output of the embedding model. +Bases: Enum - +Pooling modes support by the Optimum Embedders. -#### OptimumEmbedderPooling.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderPooling" +from_str(string: str) -> OptimumEmbedderPooling ``` Create a pooling mode from a string. -**Arguments**: +**Parameters:** -- `string`: String to convert. +- **string** (str) – String to convert. -**Returns**: +**Returns:** -Pooling mode. +- OptimumEmbedderPooling – Pooling mode. - - -## Module haystack\_integrations.components.embedders.optimum.quantization - - +## haystack_integrations.components.embedders.optimum.quantization ### OptimumEmbedderQuantizationMode -[Dynamic Quantization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) -support by the Optimum Embedders. - - - -#### ARM64 - -Quantization for the ARM64 architecture. - - +Bases: Enum -#### AVX2 +Dynamic Quantization modes supported by the Optimum Embedders. -Quantization with AVX-2 instructions. +See [Optimum ONNX quantization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) +for more details. - - -#### AVX512 - -Quantization with AVX-512 instructions. - - - -#### AVX512\_VNNI - -Quantization with AVX-512 and VNNI instructions. - - - -#### OptimumEmbedderQuantizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderQuantizationMode" +from_str(string: str) -> OptimumEmbedderQuantizationMode ``` Create an quantization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Quantization mode. +**Returns:** - +- OptimumEmbedderQuantizationMode – Quantization mode. ### OptimumEmbedderQuantizationConfig Configuration for Optimum Embedder Quantization. -**Arguments**: - -- `mode`: Quantization mode. -- `per_channel`: Whether to apply per-channel quantization. +**Parameters:** - +- **mode** (OptimumEmbedderQuantizationMode) – Quantization mode. +- **per_channel** (bool) – Whether to apply per-channel quantization. -#### OptimumEmbedderQuantizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> QuantizationConfig +to_optimum_config() -> QuantizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: +**Returns:** -Optimum configuration. +- QuantizationConfig – Optimum configuration. - - -#### OptimumEmbedderQuantizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: - -Dictionary with serialized data. +**Returns:** - +- dict\[str, Any\] – Dictionary with serialized data. -#### OptimumEmbedderQuantizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderQuantizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderQuantizationConfig ``` Create a configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Quantization configuration. +**Returns:** +- OptimumEmbedderQuantizationConfig – Quantization configuration. diff --git a/docs-website/reference_versioned_docs/version-2.24/integrations-api/optimum.md b/docs-website/reference_versioned_docs/version-2.24/integrations-api/optimum.md index da0d85c035..e823475a86 100644 --- a/docs-website/reference_versioned_docs/version-2.24/integrations-api/optimum.md +++ b/docs-website/reference_versioned_docs/version-2.24/integrations-api/optimum.md @@ -5,134 +5,96 @@ description: "Optimum integration for Haystack" slug: "/integrations-optimum" --- - -## Module haystack\_integrations.components.embedders.optimum.optimization - - +## haystack_integrations.components.embedders.optimum.optimization ### OptimumEmbedderOptimizationMode -[ONXX Optimization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) -support by the Optimum Embedders. - - - -#### O1 - -Basic general optimizations. - - - -#### O2 - -Basic and extended general optimizations, transformers-specific fusions. +Bases: Enum - +ONNX Optimization modes supported by the Optimum Embedders. -#### O3 +See [Optimum ONNX optimization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) +for more details. -Same as O2 with Gelu approximation. - - - -#### O4 - -Same as O3 with mixed precision. - - - -#### OptimumEmbedderOptimizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderOptimizationMode" +from_str(string: str) -> OptimumEmbedderOptimizationMode ``` Create an optimization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Optimization mode. +**Returns:** - +- OptimumEmbedderOptimizationMode – Optimization mode. ### OptimumEmbedderOptimizationConfig Configuration for Optimum Embedder Optimization. -**Arguments**: +**Parameters:** -- `mode`: Optimization mode. -- `for_gpu`: Whether to optimize for GPUs. +- **mode** (OptimumEmbedderOptimizationMode) – Optimization mode. +- **for_gpu** (bool) – Whether to optimize for GPUs. - - -#### OptimumEmbedderOptimizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> OptimizationConfig +to_optimum_config() -> OptimizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: - -Optimum configuration. +**Returns:** - +- OptimizationConfig – Optimum configuration. -#### OptimumEmbedderOptimizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumEmbedderOptimizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderOptimizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderOptimizationConfig ``` Create an optimization configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Optimization configuration. +**Returns:** - +- OptimumEmbedderOptimizationConfig – Optimization configuration. -## Module haystack\_integrations.components.embedders.optimum.optimum\_document\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_document_embedder ### OptimumDocumentEmbedder -A component for computing `Document` embeddings using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component for computing `Document` embeddings using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. The embedding of each Document is stored in the `embedding` field of the Document. Usage example: + ```python from haystack.dataclasses import Document from haystack_integrations.components.embedders.optimum import OptimumDocumentEmbedder @@ -140,7 +102,7 @@ from haystack_integrations.components.embedders.optimum import OptimumDocumentEm doc = Document(content="I love pizza!") document_embedder = OptimumDocumentEmbedder(model="sentence-transformers/all-mpnet-base-v2") -document_embedder.warm_up() +# Components warm up automatically on first run. result = document_embedder.run([doc]) print(result["documents"][0].embedding) @@ -148,482 +110,398 @@ print(result["documents"][0].embedding) # [0.017020374536514282, -0.023255806416273117, ...] ``` - - -#### OptimumDocumentEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__(model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig - | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig - | None = None, - batch_size: int = 32, - progress_bar: bool = True, - meta_fields_to_embed: list[str] | None = None, - embedding_separator: str = "\n") -> None +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, + batch_size: int = 32, + progress_bar: bool = True, + meta_fields_to_embed: list[str] | None = None, + embedding_separator: str = "\n", +) -> None ``` Create a OptimumDocumentEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. -- `batch_size`: Number of Documents to encode at once. -- `progress_bar`: Whether to show a progress bar or not. -- `meta_fields_to_embed`: List of meta fields that should be embedded along with the Document text. -- `embedding_separator`: Separator used to concatenate the meta fields to the Document text. - - - -#### OptimumDocumentEmbedder.warm\_up +**Parameters:** + +- **model** (str) – A string representing the model id on HF Hub. + +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. + +- **prefix** (str) – A string to add to the beginning of each text. + +- **suffix** (str) – A string to add to the end of each text. + +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. + +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. + + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: + + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` + +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. + +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. + +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. + +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. + +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. + +- **batch_size** (int) – Number of Documents to encode at once. + +- **progress_bar** (bool) – Whether to show a progress bar or not. + +- **meta_fields_to_embed** (list\[str\] | None) – List of meta fields that should be embedded along with the Document text. + +- **embedding_separator** (str) – Separator used to concatenate the meta fields to the Document text. + +#### warm_up ```python -def warm_up() -> None +warm_up() -> None ``` Initializes the component. - - -#### OptimumDocumentEmbedder.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Serializes the component to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumDocumentEmbedder.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumDocumentEmbedder" +from_dict(data: dict[str, Any]) -> OptimumDocumentEmbedder ``` Deserializes the component from a dictionary. -**Arguments**: - -- `data`: The dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -The deserialized component. +**Returns:** - +- OptimumDocumentEmbedder – The deserialized component. -#### OptimumDocumentEmbedder.run +#### run ```python -@component.output_types(documents=list[Document]) -def run(documents: list[Document]) -> dict[str, list[Document]] +run(documents: list[Document]) -> dict[str, list[Document]] ``` Embed a list of Documents. The embedding of each Document is stored in the `embedding` field of the Document. -**Arguments**: - -- `documents`: A list of Documents to embed. +**Parameters:** -**Raises**: +- **documents** (list\[Document\]) – A list of Documents to embed. -- `TypeError`: If the input is not a list of Documents. +**Returns:** -**Returns**: +- dict\[str, list\[Document\]\] – The updated Documents with their embeddings. -The updated Documents with their embeddings. +**Raises:** - +- TypeError – If the input is not a list of Documents. -## Module haystack\_integrations.components.embedders.optimum.optimum\_text\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_text_embedder ### OptimumTextEmbedder -A component to embed text using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component to embed text using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. Usage example: + ```python from haystack_integrations.components.embedders.optimum import OptimumTextEmbedder text_to_embed = "I love pizza!" text_embedder = OptimumTextEmbedder(model="sentence-transformers/all-mpnet-base-v2") -text_embedder.warm_up() +# Components warm up automatically on first run. print(text_embedder.run(text_to_embed)) # {'embedding': [-0.07804739475250244, 0.1498992145061493,, ...]} ``` - - -#### OptimumTextEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__( - model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig | None = None) +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, +) -> None ``` Create a OptimumTextEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. - - - -#### OptimumTextEmbedder.warm\_up - -```python -def warm_up() -``` +**Parameters:** -Initializes the component. +- **model** (str) – A string representing the model id on HF Hub. - +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. -#### OptimumTextEmbedder.to\_dict +- **prefix** (str) – A string to add to the beginning of each text. -```python -def to_dict() -> dict[str, Any] -``` +- **suffix** (str) – A string to add to the end of each text. -Serializes the component to a dictionary. +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. -**Returns**: +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. -Dictionary with serialized data. + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: - - -#### OptimumTextEmbedder.from\_dict - -```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumTextEmbedder" -``` - -Deserializes the component from a dictionary. + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` -**Arguments**: +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `data`: The dictionary to deserialize from. +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. -**Returns**: +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. -The deserialized component. +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. - +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. -#### OptimumTextEmbedder.run +#### warm_up ```python -@component.output_types(embedding=list[float]) -def run(text: str) -> dict[str, list[float]] +warm_up() -> None ``` -Embed a string. - -**Arguments**: - -- `text`: The text to embed. - -**Raises**: - -- `TypeError`: If the input is not a string. - -**Returns**: +Initializes the component. -The embeddings of the text. +#### to_dict - +```python +to_dict() -> dict[str, Any] +``` -## Module haystack\_integrations.components.embedders.optimum.pooling +Serializes the component to a dictionary. - +**Returns:** -### OptimumEmbedderPooling +- dict\[str, Any\] – Dictionary with serialized data. -Pooling modes support by the Optimum Embedders. +#### from_dict - +```python +from_dict(data: dict[str, Any]) -> OptimumTextEmbedder +``` -#### CLS +Deserializes the component from a dictionary. -Perform CLS Pooling on the output of the embedding model -using the first token (CLS token). +**Parameters:** - +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -#### MEAN +**Returns:** -Perform Mean Pooling on the output of the embedding model. +- OptimumTextEmbedder – The deserialized component. - +#### run -#### MAX +```python +run(text: str) -> dict[str, list[float]] +``` -Perform Max Pooling on the output of the embedding model -using the maximum value in each dimension over all the tokens. +Embed a string. - +**Parameters:** -#### MEAN\_SQRT\_LEN +- **text** (str) – The text to embed. -Perform mean-pooling on the output of the embedding model but -divide by the square root of the sequence length. +**Returns:** - +- dict\[str, list\[float\]\] – The embeddings of the text. -#### WEIGHTED\_MEAN +**Raises:** -Perform weighted (position) mean pooling on the output of the -embedding model. +- TypeError – If the input is not a string. - +## haystack_integrations.components.embedders.optimum.pooling -#### LAST\_TOKEN +### OptimumEmbedderPooling -Perform Last Token Pooling on the output of the embedding model. +Bases: Enum - +Pooling modes support by the Optimum Embedders. -#### OptimumEmbedderPooling.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderPooling" +from_str(string: str) -> OptimumEmbedderPooling ``` Create a pooling mode from a string. -**Arguments**: +**Parameters:** -- `string`: String to convert. +- **string** (str) – String to convert. -**Returns**: +**Returns:** -Pooling mode. +- OptimumEmbedderPooling – Pooling mode. - - -## Module haystack\_integrations.components.embedders.optimum.quantization - - +## haystack_integrations.components.embedders.optimum.quantization ### OptimumEmbedderQuantizationMode -[Dynamic Quantization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) -support by the Optimum Embedders. - - - -#### ARM64 - -Quantization for the ARM64 architecture. - - +Bases: Enum -#### AVX2 +Dynamic Quantization modes supported by the Optimum Embedders. -Quantization with AVX-2 instructions. +See [Optimum ONNX quantization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) +for more details. - - -#### AVX512 - -Quantization with AVX-512 instructions. - - - -#### AVX512\_VNNI - -Quantization with AVX-512 and VNNI instructions. - - - -#### OptimumEmbedderQuantizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderQuantizationMode" +from_str(string: str) -> OptimumEmbedderQuantizationMode ``` Create an quantization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Quantization mode. +**Returns:** - +- OptimumEmbedderQuantizationMode – Quantization mode. ### OptimumEmbedderQuantizationConfig Configuration for Optimum Embedder Quantization. -**Arguments**: - -- `mode`: Quantization mode. -- `per_channel`: Whether to apply per-channel quantization. +**Parameters:** - +- **mode** (OptimumEmbedderQuantizationMode) – Quantization mode. +- **per_channel** (bool) – Whether to apply per-channel quantization. -#### OptimumEmbedderQuantizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> QuantizationConfig +to_optimum_config() -> QuantizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: +**Returns:** -Optimum configuration. +- QuantizationConfig – Optimum configuration. - - -#### OptimumEmbedderQuantizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: - -Dictionary with serialized data. +**Returns:** - +- dict\[str, Any\] – Dictionary with serialized data. -#### OptimumEmbedderQuantizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderQuantizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderQuantizationConfig ``` Create a configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Quantization configuration. +**Returns:** +- OptimumEmbedderQuantizationConfig – Quantization configuration. diff --git a/docs-website/reference_versioned_docs/version-2.25/integrations-api/optimum.md b/docs-website/reference_versioned_docs/version-2.25/integrations-api/optimum.md index da0d85c035..e823475a86 100644 --- a/docs-website/reference_versioned_docs/version-2.25/integrations-api/optimum.md +++ b/docs-website/reference_versioned_docs/version-2.25/integrations-api/optimum.md @@ -5,134 +5,96 @@ description: "Optimum integration for Haystack" slug: "/integrations-optimum" --- - -## Module haystack\_integrations.components.embedders.optimum.optimization - - +## haystack_integrations.components.embedders.optimum.optimization ### OptimumEmbedderOptimizationMode -[ONXX Optimization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) -support by the Optimum Embedders. - - - -#### O1 - -Basic general optimizations. - - - -#### O2 - -Basic and extended general optimizations, transformers-specific fusions. +Bases: Enum - +ONNX Optimization modes supported by the Optimum Embedders. -#### O3 +See [Optimum ONNX optimization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) +for more details. -Same as O2 with Gelu approximation. - - - -#### O4 - -Same as O3 with mixed precision. - - - -#### OptimumEmbedderOptimizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderOptimizationMode" +from_str(string: str) -> OptimumEmbedderOptimizationMode ``` Create an optimization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Optimization mode. +**Returns:** - +- OptimumEmbedderOptimizationMode – Optimization mode. ### OptimumEmbedderOptimizationConfig Configuration for Optimum Embedder Optimization. -**Arguments**: +**Parameters:** -- `mode`: Optimization mode. -- `for_gpu`: Whether to optimize for GPUs. +- **mode** (OptimumEmbedderOptimizationMode) – Optimization mode. +- **for_gpu** (bool) – Whether to optimize for GPUs. - - -#### OptimumEmbedderOptimizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> OptimizationConfig +to_optimum_config() -> OptimizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: - -Optimum configuration. +**Returns:** - +- OptimizationConfig – Optimum configuration. -#### OptimumEmbedderOptimizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumEmbedderOptimizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderOptimizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderOptimizationConfig ``` Create an optimization configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Optimization configuration. +**Returns:** - +- OptimumEmbedderOptimizationConfig – Optimization configuration. -## Module haystack\_integrations.components.embedders.optimum.optimum\_document\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_document_embedder ### OptimumDocumentEmbedder -A component for computing `Document` embeddings using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component for computing `Document` embeddings using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. The embedding of each Document is stored in the `embedding` field of the Document. Usage example: + ```python from haystack.dataclasses import Document from haystack_integrations.components.embedders.optimum import OptimumDocumentEmbedder @@ -140,7 +102,7 @@ from haystack_integrations.components.embedders.optimum import OptimumDocumentEm doc = Document(content="I love pizza!") document_embedder = OptimumDocumentEmbedder(model="sentence-transformers/all-mpnet-base-v2") -document_embedder.warm_up() +# Components warm up automatically on first run. result = document_embedder.run([doc]) print(result["documents"][0].embedding) @@ -148,482 +110,398 @@ print(result["documents"][0].embedding) # [0.017020374536514282, -0.023255806416273117, ...] ``` - - -#### OptimumDocumentEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__(model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig - | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig - | None = None, - batch_size: int = 32, - progress_bar: bool = True, - meta_fields_to_embed: list[str] | None = None, - embedding_separator: str = "\n") -> None +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, + batch_size: int = 32, + progress_bar: bool = True, + meta_fields_to_embed: list[str] | None = None, + embedding_separator: str = "\n", +) -> None ``` Create a OptimumDocumentEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. -- `batch_size`: Number of Documents to encode at once. -- `progress_bar`: Whether to show a progress bar or not. -- `meta_fields_to_embed`: List of meta fields that should be embedded along with the Document text. -- `embedding_separator`: Separator used to concatenate the meta fields to the Document text. - - - -#### OptimumDocumentEmbedder.warm\_up +**Parameters:** + +- **model** (str) – A string representing the model id on HF Hub. + +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. + +- **prefix** (str) – A string to add to the beginning of each text. + +- **suffix** (str) – A string to add to the end of each text. + +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. + +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. + + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: + + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` + +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. + +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. + +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. + +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. + +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. + +- **batch_size** (int) – Number of Documents to encode at once. + +- **progress_bar** (bool) – Whether to show a progress bar or not. + +- **meta_fields_to_embed** (list\[str\] | None) – List of meta fields that should be embedded along with the Document text. + +- **embedding_separator** (str) – Separator used to concatenate the meta fields to the Document text. + +#### warm_up ```python -def warm_up() -> None +warm_up() -> None ``` Initializes the component. - - -#### OptimumDocumentEmbedder.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Serializes the component to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumDocumentEmbedder.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumDocumentEmbedder" +from_dict(data: dict[str, Any]) -> OptimumDocumentEmbedder ``` Deserializes the component from a dictionary. -**Arguments**: - -- `data`: The dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -The deserialized component. +**Returns:** - +- OptimumDocumentEmbedder – The deserialized component. -#### OptimumDocumentEmbedder.run +#### run ```python -@component.output_types(documents=list[Document]) -def run(documents: list[Document]) -> dict[str, list[Document]] +run(documents: list[Document]) -> dict[str, list[Document]] ``` Embed a list of Documents. The embedding of each Document is stored in the `embedding` field of the Document. -**Arguments**: - -- `documents`: A list of Documents to embed. +**Parameters:** -**Raises**: +- **documents** (list\[Document\]) – A list of Documents to embed. -- `TypeError`: If the input is not a list of Documents. +**Returns:** -**Returns**: +- dict\[str, list\[Document\]\] – The updated Documents with their embeddings. -The updated Documents with their embeddings. +**Raises:** - +- TypeError – If the input is not a list of Documents. -## Module haystack\_integrations.components.embedders.optimum.optimum\_text\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_text_embedder ### OptimumTextEmbedder -A component to embed text using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component to embed text using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. Usage example: + ```python from haystack_integrations.components.embedders.optimum import OptimumTextEmbedder text_to_embed = "I love pizza!" text_embedder = OptimumTextEmbedder(model="sentence-transformers/all-mpnet-base-v2") -text_embedder.warm_up() +# Components warm up automatically on first run. print(text_embedder.run(text_to_embed)) # {'embedding': [-0.07804739475250244, 0.1498992145061493,, ...]} ``` - - -#### OptimumTextEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__( - model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig | None = None) +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, +) -> None ``` Create a OptimumTextEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. - - - -#### OptimumTextEmbedder.warm\_up - -```python -def warm_up() -``` +**Parameters:** -Initializes the component. +- **model** (str) – A string representing the model id on HF Hub. - +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. -#### OptimumTextEmbedder.to\_dict +- **prefix** (str) – A string to add to the beginning of each text. -```python -def to_dict() -> dict[str, Any] -``` +- **suffix** (str) – A string to add to the end of each text. -Serializes the component to a dictionary. +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. -**Returns**: +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. -Dictionary with serialized data. + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: - - -#### OptimumTextEmbedder.from\_dict - -```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumTextEmbedder" -``` - -Deserializes the component from a dictionary. + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` -**Arguments**: +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `data`: The dictionary to deserialize from. +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. -**Returns**: +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. -The deserialized component. +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. - +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. -#### OptimumTextEmbedder.run +#### warm_up ```python -@component.output_types(embedding=list[float]) -def run(text: str) -> dict[str, list[float]] +warm_up() -> None ``` -Embed a string. - -**Arguments**: - -- `text`: The text to embed. - -**Raises**: - -- `TypeError`: If the input is not a string. - -**Returns**: +Initializes the component. -The embeddings of the text. +#### to_dict - +```python +to_dict() -> dict[str, Any] +``` -## Module haystack\_integrations.components.embedders.optimum.pooling +Serializes the component to a dictionary. - +**Returns:** -### OptimumEmbedderPooling +- dict\[str, Any\] – Dictionary with serialized data. -Pooling modes support by the Optimum Embedders. +#### from_dict - +```python +from_dict(data: dict[str, Any]) -> OptimumTextEmbedder +``` -#### CLS +Deserializes the component from a dictionary. -Perform CLS Pooling on the output of the embedding model -using the first token (CLS token). +**Parameters:** - +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -#### MEAN +**Returns:** -Perform Mean Pooling on the output of the embedding model. +- OptimumTextEmbedder – The deserialized component. - +#### run -#### MAX +```python +run(text: str) -> dict[str, list[float]] +``` -Perform Max Pooling on the output of the embedding model -using the maximum value in each dimension over all the tokens. +Embed a string. - +**Parameters:** -#### MEAN\_SQRT\_LEN +- **text** (str) – The text to embed. -Perform mean-pooling on the output of the embedding model but -divide by the square root of the sequence length. +**Returns:** - +- dict\[str, list\[float\]\] – The embeddings of the text. -#### WEIGHTED\_MEAN +**Raises:** -Perform weighted (position) mean pooling on the output of the -embedding model. +- TypeError – If the input is not a string. - +## haystack_integrations.components.embedders.optimum.pooling -#### LAST\_TOKEN +### OptimumEmbedderPooling -Perform Last Token Pooling on the output of the embedding model. +Bases: Enum - +Pooling modes support by the Optimum Embedders. -#### OptimumEmbedderPooling.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderPooling" +from_str(string: str) -> OptimumEmbedderPooling ``` Create a pooling mode from a string. -**Arguments**: +**Parameters:** -- `string`: String to convert. +- **string** (str) – String to convert. -**Returns**: +**Returns:** -Pooling mode. +- OptimumEmbedderPooling – Pooling mode. - - -## Module haystack\_integrations.components.embedders.optimum.quantization - - +## haystack_integrations.components.embedders.optimum.quantization ### OptimumEmbedderQuantizationMode -[Dynamic Quantization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) -support by the Optimum Embedders. - - - -#### ARM64 - -Quantization for the ARM64 architecture. - - +Bases: Enum -#### AVX2 +Dynamic Quantization modes supported by the Optimum Embedders. -Quantization with AVX-2 instructions. +See [Optimum ONNX quantization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) +for more details. - - -#### AVX512 - -Quantization with AVX-512 instructions. - - - -#### AVX512\_VNNI - -Quantization with AVX-512 and VNNI instructions. - - - -#### OptimumEmbedderQuantizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderQuantizationMode" +from_str(string: str) -> OptimumEmbedderQuantizationMode ``` Create an quantization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Quantization mode. +**Returns:** - +- OptimumEmbedderQuantizationMode – Quantization mode. ### OptimumEmbedderQuantizationConfig Configuration for Optimum Embedder Quantization. -**Arguments**: - -- `mode`: Quantization mode. -- `per_channel`: Whether to apply per-channel quantization. +**Parameters:** - +- **mode** (OptimumEmbedderQuantizationMode) – Quantization mode. +- **per_channel** (bool) – Whether to apply per-channel quantization. -#### OptimumEmbedderQuantizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> QuantizationConfig +to_optimum_config() -> QuantizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: +**Returns:** -Optimum configuration. +- QuantizationConfig – Optimum configuration. - - -#### OptimumEmbedderQuantizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: - -Dictionary with serialized data. +**Returns:** - +- dict\[str, Any\] – Dictionary with serialized data. -#### OptimumEmbedderQuantizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderQuantizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderQuantizationConfig ``` Create a configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Quantization configuration. +**Returns:** +- OptimumEmbedderQuantizationConfig – Quantization configuration. diff --git a/docs-website/reference_versioned_docs/version-2.26/integrations-api/optimum.md b/docs-website/reference_versioned_docs/version-2.26/integrations-api/optimum.md index da0d85c035..e823475a86 100644 --- a/docs-website/reference_versioned_docs/version-2.26/integrations-api/optimum.md +++ b/docs-website/reference_versioned_docs/version-2.26/integrations-api/optimum.md @@ -5,134 +5,96 @@ description: "Optimum integration for Haystack" slug: "/integrations-optimum" --- - -## Module haystack\_integrations.components.embedders.optimum.optimization - - +## haystack_integrations.components.embedders.optimum.optimization ### OptimumEmbedderOptimizationMode -[ONXX Optimization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) -support by the Optimum Embedders. - - - -#### O1 - -Basic general optimizations. - - - -#### O2 - -Basic and extended general optimizations, transformers-specific fusions. +Bases: Enum - +ONNX Optimization modes supported by the Optimum Embedders. -#### O3 +See [Optimum ONNX optimization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) +for more details. -Same as O2 with Gelu approximation. - - - -#### O4 - -Same as O3 with mixed precision. - - - -#### OptimumEmbedderOptimizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderOptimizationMode" +from_str(string: str) -> OptimumEmbedderOptimizationMode ``` Create an optimization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Optimization mode. +**Returns:** - +- OptimumEmbedderOptimizationMode – Optimization mode. ### OptimumEmbedderOptimizationConfig Configuration for Optimum Embedder Optimization. -**Arguments**: +**Parameters:** -- `mode`: Optimization mode. -- `for_gpu`: Whether to optimize for GPUs. +- **mode** (OptimumEmbedderOptimizationMode) – Optimization mode. +- **for_gpu** (bool) – Whether to optimize for GPUs. - - -#### OptimumEmbedderOptimizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> OptimizationConfig +to_optimum_config() -> OptimizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: - -Optimum configuration. +**Returns:** - +- OptimizationConfig – Optimum configuration. -#### OptimumEmbedderOptimizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumEmbedderOptimizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderOptimizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderOptimizationConfig ``` Create an optimization configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Optimization configuration. +**Returns:** - +- OptimumEmbedderOptimizationConfig – Optimization configuration. -## Module haystack\_integrations.components.embedders.optimum.optimum\_document\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_document_embedder ### OptimumDocumentEmbedder -A component for computing `Document` embeddings using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component for computing `Document` embeddings using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. The embedding of each Document is stored in the `embedding` field of the Document. Usage example: + ```python from haystack.dataclasses import Document from haystack_integrations.components.embedders.optimum import OptimumDocumentEmbedder @@ -140,7 +102,7 @@ from haystack_integrations.components.embedders.optimum import OptimumDocumentEm doc = Document(content="I love pizza!") document_embedder = OptimumDocumentEmbedder(model="sentence-transformers/all-mpnet-base-v2") -document_embedder.warm_up() +# Components warm up automatically on first run. result = document_embedder.run([doc]) print(result["documents"][0].embedding) @@ -148,482 +110,398 @@ print(result["documents"][0].embedding) # [0.017020374536514282, -0.023255806416273117, ...] ``` - - -#### OptimumDocumentEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__(model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig - | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig - | None = None, - batch_size: int = 32, - progress_bar: bool = True, - meta_fields_to_embed: list[str] | None = None, - embedding_separator: str = "\n") -> None +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, + batch_size: int = 32, + progress_bar: bool = True, + meta_fields_to_embed: list[str] | None = None, + embedding_separator: str = "\n", +) -> None ``` Create a OptimumDocumentEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. -- `batch_size`: Number of Documents to encode at once. -- `progress_bar`: Whether to show a progress bar or not. -- `meta_fields_to_embed`: List of meta fields that should be embedded along with the Document text. -- `embedding_separator`: Separator used to concatenate the meta fields to the Document text. - - - -#### OptimumDocumentEmbedder.warm\_up +**Parameters:** + +- **model** (str) – A string representing the model id on HF Hub. + +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. + +- **prefix** (str) – A string to add to the beginning of each text. + +- **suffix** (str) – A string to add to the end of each text. + +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. + +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. + + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: + + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` + +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. + +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. + +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. + +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. + +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. + +- **batch_size** (int) – Number of Documents to encode at once. + +- **progress_bar** (bool) – Whether to show a progress bar or not. + +- **meta_fields_to_embed** (list\[str\] | None) – List of meta fields that should be embedded along with the Document text. + +- **embedding_separator** (str) – Separator used to concatenate the meta fields to the Document text. + +#### warm_up ```python -def warm_up() -> None +warm_up() -> None ``` Initializes the component. - - -#### OptimumDocumentEmbedder.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Serializes the component to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumDocumentEmbedder.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumDocumentEmbedder" +from_dict(data: dict[str, Any]) -> OptimumDocumentEmbedder ``` Deserializes the component from a dictionary. -**Arguments**: - -- `data`: The dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -The deserialized component. +**Returns:** - +- OptimumDocumentEmbedder – The deserialized component. -#### OptimumDocumentEmbedder.run +#### run ```python -@component.output_types(documents=list[Document]) -def run(documents: list[Document]) -> dict[str, list[Document]] +run(documents: list[Document]) -> dict[str, list[Document]] ``` Embed a list of Documents. The embedding of each Document is stored in the `embedding` field of the Document. -**Arguments**: - -- `documents`: A list of Documents to embed. +**Parameters:** -**Raises**: +- **documents** (list\[Document\]) – A list of Documents to embed. -- `TypeError`: If the input is not a list of Documents. +**Returns:** -**Returns**: +- dict\[str, list\[Document\]\] – The updated Documents with their embeddings. -The updated Documents with their embeddings. +**Raises:** - +- TypeError – If the input is not a list of Documents. -## Module haystack\_integrations.components.embedders.optimum.optimum\_text\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_text_embedder ### OptimumTextEmbedder -A component to embed text using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component to embed text using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. Usage example: + ```python from haystack_integrations.components.embedders.optimum import OptimumTextEmbedder text_to_embed = "I love pizza!" text_embedder = OptimumTextEmbedder(model="sentence-transformers/all-mpnet-base-v2") -text_embedder.warm_up() +# Components warm up automatically on first run. print(text_embedder.run(text_to_embed)) # {'embedding': [-0.07804739475250244, 0.1498992145061493,, ...]} ``` - - -#### OptimumTextEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__( - model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig | None = None) +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, +) -> None ``` Create a OptimumTextEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. - - - -#### OptimumTextEmbedder.warm\_up - -```python -def warm_up() -``` +**Parameters:** -Initializes the component. +- **model** (str) – A string representing the model id on HF Hub. - +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. -#### OptimumTextEmbedder.to\_dict +- **prefix** (str) – A string to add to the beginning of each text. -```python -def to_dict() -> dict[str, Any] -``` +- **suffix** (str) – A string to add to the end of each text. -Serializes the component to a dictionary. +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. -**Returns**: +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. -Dictionary with serialized data. + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: - - -#### OptimumTextEmbedder.from\_dict - -```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumTextEmbedder" -``` - -Deserializes the component from a dictionary. + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` -**Arguments**: +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `data`: The dictionary to deserialize from. +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. -**Returns**: +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. -The deserialized component. +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. - +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. -#### OptimumTextEmbedder.run +#### warm_up ```python -@component.output_types(embedding=list[float]) -def run(text: str) -> dict[str, list[float]] +warm_up() -> None ``` -Embed a string. - -**Arguments**: - -- `text`: The text to embed. - -**Raises**: - -- `TypeError`: If the input is not a string. - -**Returns**: +Initializes the component. -The embeddings of the text. +#### to_dict - +```python +to_dict() -> dict[str, Any] +``` -## Module haystack\_integrations.components.embedders.optimum.pooling +Serializes the component to a dictionary. - +**Returns:** -### OptimumEmbedderPooling +- dict\[str, Any\] – Dictionary with serialized data. -Pooling modes support by the Optimum Embedders. +#### from_dict - +```python +from_dict(data: dict[str, Any]) -> OptimumTextEmbedder +``` -#### CLS +Deserializes the component from a dictionary. -Perform CLS Pooling on the output of the embedding model -using the first token (CLS token). +**Parameters:** - +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -#### MEAN +**Returns:** -Perform Mean Pooling on the output of the embedding model. +- OptimumTextEmbedder – The deserialized component. - +#### run -#### MAX +```python +run(text: str) -> dict[str, list[float]] +``` -Perform Max Pooling on the output of the embedding model -using the maximum value in each dimension over all the tokens. +Embed a string. - +**Parameters:** -#### MEAN\_SQRT\_LEN +- **text** (str) – The text to embed. -Perform mean-pooling on the output of the embedding model but -divide by the square root of the sequence length. +**Returns:** - +- dict\[str, list\[float\]\] – The embeddings of the text. -#### WEIGHTED\_MEAN +**Raises:** -Perform weighted (position) mean pooling on the output of the -embedding model. +- TypeError – If the input is not a string. - +## haystack_integrations.components.embedders.optimum.pooling -#### LAST\_TOKEN +### OptimumEmbedderPooling -Perform Last Token Pooling on the output of the embedding model. +Bases: Enum - +Pooling modes support by the Optimum Embedders. -#### OptimumEmbedderPooling.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderPooling" +from_str(string: str) -> OptimumEmbedderPooling ``` Create a pooling mode from a string. -**Arguments**: +**Parameters:** -- `string`: String to convert. +- **string** (str) – String to convert. -**Returns**: +**Returns:** -Pooling mode. +- OptimumEmbedderPooling – Pooling mode. - - -## Module haystack\_integrations.components.embedders.optimum.quantization - - +## haystack_integrations.components.embedders.optimum.quantization ### OptimumEmbedderQuantizationMode -[Dynamic Quantization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) -support by the Optimum Embedders. - - - -#### ARM64 - -Quantization for the ARM64 architecture. - - +Bases: Enum -#### AVX2 +Dynamic Quantization modes supported by the Optimum Embedders. -Quantization with AVX-2 instructions. +See [Optimum ONNX quantization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) +for more details. - - -#### AVX512 - -Quantization with AVX-512 instructions. - - - -#### AVX512\_VNNI - -Quantization with AVX-512 and VNNI instructions. - - - -#### OptimumEmbedderQuantizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderQuantizationMode" +from_str(string: str) -> OptimumEmbedderQuantizationMode ``` Create an quantization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Quantization mode. +**Returns:** - +- OptimumEmbedderQuantizationMode – Quantization mode. ### OptimumEmbedderQuantizationConfig Configuration for Optimum Embedder Quantization. -**Arguments**: - -- `mode`: Quantization mode. -- `per_channel`: Whether to apply per-channel quantization. +**Parameters:** - +- **mode** (OptimumEmbedderQuantizationMode) – Quantization mode. +- **per_channel** (bool) – Whether to apply per-channel quantization. -#### OptimumEmbedderQuantizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> QuantizationConfig +to_optimum_config() -> QuantizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: +**Returns:** -Optimum configuration. +- QuantizationConfig – Optimum configuration. - - -#### OptimumEmbedderQuantizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: - -Dictionary with serialized data. +**Returns:** - +- dict\[str, Any\] – Dictionary with serialized data. -#### OptimumEmbedderQuantizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderQuantizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderQuantizationConfig ``` Create a configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Quantization configuration. +**Returns:** +- OptimumEmbedderQuantizationConfig – Quantization configuration. diff --git a/docs-website/reference_versioned_docs/version-2.27/integrations-api/optimum.md b/docs-website/reference_versioned_docs/version-2.27/integrations-api/optimum.md index da0d85c035..e823475a86 100644 --- a/docs-website/reference_versioned_docs/version-2.27/integrations-api/optimum.md +++ b/docs-website/reference_versioned_docs/version-2.27/integrations-api/optimum.md @@ -5,134 +5,96 @@ description: "Optimum integration for Haystack" slug: "/integrations-optimum" --- - -## Module haystack\_integrations.components.embedders.optimum.optimization - - +## haystack_integrations.components.embedders.optimum.optimization ### OptimumEmbedderOptimizationMode -[ONXX Optimization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) -support by the Optimum Embedders. - - - -#### O1 - -Basic general optimizations. - - - -#### O2 - -Basic and extended general optimizations, transformers-specific fusions. +Bases: Enum - +ONNX Optimization modes supported by the Optimum Embedders. -#### O3 +See [Optimum ONNX optimization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) +for more details. -Same as O2 with Gelu approximation. - - - -#### O4 - -Same as O3 with mixed precision. - - - -#### OptimumEmbedderOptimizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderOptimizationMode" +from_str(string: str) -> OptimumEmbedderOptimizationMode ``` Create an optimization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Optimization mode. +**Returns:** - +- OptimumEmbedderOptimizationMode – Optimization mode. ### OptimumEmbedderOptimizationConfig Configuration for Optimum Embedder Optimization. -**Arguments**: +**Parameters:** -- `mode`: Optimization mode. -- `for_gpu`: Whether to optimize for GPUs. +- **mode** (OptimumEmbedderOptimizationMode) – Optimization mode. +- **for_gpu** (bool) – Whether to optimize for GPUs. - - -#### OptimumEmbedderOptimizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> OptimizationConfig +to_optimum_config() -> OptimizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: - -Optimum configuration. +**Returns:** - +- OptimizationConfig – Optimum configuration. -#### OptimumEmbedderOptimizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumEmbedderOptimizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderOptimizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderOptimizationConfig ``` Create an optimization configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Optimization configuration. +**Returns:** - +- OptimumEmbedderOptimizationConfig – Optimization configuration. -## Module haystack\_integrations.components.embedders.optimum.optimum\_document\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_document_embedder ### OptimumDocumentEmbedder -A component for computing `Document` embeddings using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component for computing `Document` embeddings using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. The embedding of each Document is stored in the `embedding` field of the Document. Usage example: + ```python from haystack.dataclasses import Document from haystack_integrations.components.embedders.optimum import OptimumDocumentEmbedder @@ -140,7 +102,7 @@ from haystack_integrations.components.embedders.optimum import OptimumDocumentEm doc = Document(content="I love pizza!") document_embedder = OptimumDocumentEmbedder(model="sentence-transformers/all-mpnet-base-v2") -document_embedder.warm_up() +# Components warm up automatically on first run. result = document_embedder.run([doc]) print(result["documents"][0].embedding) @@ -148,482 +110,398 @@ print(result["documents"][0].embedding) # [0.017020374536514282, -0.023255806416273117, ...] ``` - - -#### OptimumDocumentEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__(model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig - | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig - | None = None, - batch_size: int = 32, - progress_bar: bool = True, - meta_fields_to_embed: list[str] | None = None, - embedding_separator: str = "\n") -> None +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, + batch_size: int = 32, + progress_bar: bool = True, + meta_fields_to_embed: list[str] | None = None, + embedding_separator: str = "\n", +) -> None ``` Create a OptimumDocumentEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. -- `batch_size`: Number of Documents to encode at once. -- `progress_bar`: Whether to show a progress bar or not. -- `meta_fields_to_embed`: List of meta fields that should be embedded along with the Document text. -- `embedding_separator`: Separator used to concatenate the meta fields to the Document text. - - - -#### OptimumDocumentEmbedder.warm\_up +**Parameters:** + +- **model** (str) – A string representing the model id on HF Hub. + +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. + +- **prefix** (str) – A string to add to the beginning of each text. + +- **suffix** (str) – A string to add to the end of each text. + +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. + +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. + + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: + + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` + +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. + +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. + +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. + +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. + +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. + +- **batch_size** (int) – Number of Documents to encode at once. + +- **progress_bar** (bool) – Whether to show a progress bar or not. + +- **meta_fields_to_embed** (list\[str\] | None) – List of meta fields that should be embedded along with the Document text. + +- **embedding_separator** (str) – Separator used to concatenate the meta fields to the Document text. + +#### warm_up ```python -def warm_up() -> None +warm_up() -> None ``` Initializes the component. - - -#### OptimumDocumentEmbedder.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Serializes the component to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumDocumentEmbedder.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumDocumentEmbedder" +from_dict(data: dict[str, Any]) -> OptimumDocumentEmbedder ``` Deserializes the component from a dictionary. -**Arguments**: - -- `data`: The dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -The deserialized component. +**Returns:** - +- OptimumDocumentEmbedder – The deserialized component. -#### OptimumDocumentEmbedder.run +#### run ```python -@component.output_types(documents=list[Document]) -def run(documents: list[Document]) -> dict[str, list[Document]] +run(documents: list[Document]) -> dict[str, list[Document]] ``` Embed a list of Documents. The embedding of each Document is stored in the `embedding` field of the Document. -**Arguments**: - -- `documents`: A list of Documents to embed. +**Parameters:** -**Raises**: +- **documents** (list\[Document\]) – A list of Documents to embed. -- `TypeError`: If the input is not a list of Documents. +**Returns:** -**Returns**: +- dict\[str, list\[Document\]\] – The updated Documents with their embeddings. -The updated Documents with their embeddings. +**Raises:** - +- TypeError – If the input is not a list of Documents. -## Module haystack\_integrations.components.embedders.optimum.optimum\_text\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_text_embedder ### OptimumTextEmbedder -A component to embed text using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component to embed text using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. Usage example: + ```python from haystack_integrations.components.embedders.optimum import OptimumTextEmbedder text_to_embed = "I love pizza!" text_embedder = OptimumTextEmbedder(model="sentence-transformers/all-mpnet-base-v2") -text_embedder.warm_up() +# Components warm up automatically on first run. print(text_embedder.run(text_to_embed)) # {'embedding': [-0.07804739475250244, 0.1498992145061493,, ...]} ``` - - -#### OptimumTextEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__( - model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig | None = None) +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, +) -> None ``` Create a OptimumTextEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. - - - -#### OptimumTextEmbedder.warm\_up - -```python -def warm_up() -``` +**Parameters:** -Initializes the component. +- **model** (str) – A string representing the model id on HF Hub. - +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. -#### OptimumTextEmbedder.to\_dict +- **prefix** (str) – A string to add to the beginning of each text. -```python -def to_dict() -> dict[str, Any] -``` +- **suffix** (str) – A string to add to the end of each text. -Serializes the component to a dictionary. +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. -**Returns**: +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. -Dictionary with serialized data. + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: - - -#### OptimumTextEmbedder.from\_dict - -```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumTextEmbedder" -``` - -Deserializes the component from a dictionary. + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` -**Arguments**: +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `data`: The dictionary to deserialize from. +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. -**Returns**: +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. -The deserialized component. +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. - +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. -#### OptimumTextEmbedder.run +#### warm_up ```python -@component.output_types(embedding=list[float]) -def run(text: str) -> dict[str, list[float]] +warm_up() -> None ``` -Embed a string. - -**Arguments**: - -- `text`: The text to embed. - -**Raises**: - -- `TypeError`: If the input is not a string. - -**Returns**: +Initializes the component. -The embeddings of the text. +#### to_dict - +```python +to_dict() -> dict[str, Any] +``` -## Module haystack\_integrations.components.embedders.optimum.pooling +Serializes the component to a dictionary. - +**Returns:** -### OptimumEmbedderPooling +- dict\[str, Any\] – Dictionary with serialized data. -Pooling modes support by the Optimum Embedders. +#### from_dict - +```python +from_dict(data: dict[str, Any]) -> OptimumTextEmbedder +``` -#### CLS +Deserializes the component from a dictionary. -Perform CLS Pooling on the output of the embedding model -using the first token (CLS token). +**Parameters:** - +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -#### MEAN +**Returns:** -Perform Mean Pooling on the output of the embedding model. +- OptimumTextEmbedder – The deserialized component. - +#### run -#### MAX +```python +run(text: str) -> dict[str, list[float]] +``` -Perform Max Pooling on the output of the embedding model -using the maximum value in each dimension over all the tokens. +Embed a string. - +**Parameters:** -#### MEAN\_SQRT\_LEN +- **text** (str) – The text to embed. -Perform mean-pooling on the output of the embedding model but -divide by the square root of the sequence length. +**Returns:** - +- dict\[str, list\[float\]\] – The embeddings of the text. -#### WEIGHTED\_MEAN +**Raises:** -Perform weighted (position) mean pooling on the output of the -embedding model. +- TypeError – If the input is not a string. - +## haystack_integrations.components.embedders.optimum.pooling -#### LAST\_TOKEN +### OptimumEmbedderPooling -Perform Last Token Pooling on the output of the embedding model. +Bases: Enum - +Pooling modes support by the Optimum Embedders. -#### OptimumEmbedderPooling.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderPooling" +from_str(string: str) -> OptimumEmbedderPooling ``` Create a pooling mode from a string. -**Arguments**: +**Parameters:** -- `string`: String to convert. +- **string** (str) – String to convert. -**Returns**: +**Returns:** -Pooling mode. +- OptimumEmbedderPooling – Pooling mode. - - -## Module haystack\_integrations.components.embedders.optimum.quantization - - +## haystack_integrations.components.embedders.optimum.quantization ### OptimumEmbedderQuantizationMode -[Dynamic Quantization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) -support by the Optimum Embedders. - - - -#### ARM64 - -Quantization for the ARM64 architecture. - - +Bases: Enum -#### AVX2 +Dynamic Quantization modes supported by the Optimum Embedders. -Quantization with AVX-2 instructions. +See [Optimum ONNX quantization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) +for more details. - - -#### AVX512 - -Quantization with AVX-512 instructions. - - - -#### AVX512\_VNNI - -Quantization with AVX-512 and VNNI instructions. - - - -#### OptimumEmbedderQuantizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderQuantizationMode" +from_str(string: str) -> OptimumEmbedderQuantizationMode ``` Create an quantization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Quantization mode. +**Returns:** - +- OptimumEmbedderQuantizationMode – Quantization mode. ### OptimumEmbedderQuantizationConfig Configuration for Optimum Embedder Quantization. -**Arguments**: - -- `mode`: Quantization mode. -- `per_channel`: Whether to apply per-channel quantization. +**Parameters:** - +- **mode** (OptimumEmbedderQuantizationMode) – Quantization mode. +- **per_channel** (bool) – Whether to apply per-channel quantization. -#### OptimumEmbedderQuantizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> QuantizationConfig +to_optimum_config() -> QuantizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: +**Returns:** -Optimum configuration. +- QuantizationConfig – Optimum configuration. - - -#### OptimumEmbedderQuantizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: - -Dictionary with serialized data. +**Returns:** - +- dict\[str, Any\] – Dictionary with serialized data. -#### OptimumEmbedderQuantizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderQuantizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderQuantizationConfig ``` Create a configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Quantization configuration. +**Returns:** +- OptimumEmbedderQuantizationConfig – Quantization configuration. diff --git a/docs-website/reference_versioned_docs/version-2.28/integrations-api/optimum.md b/docs-website/reference_versioned_docs/version-2.28/integrations-api/optimum.md index da0d85c035..e823475a86 100644 --- a/docs-website/reference_versioned_docs/version-2.28/integrations-api/optimum.md +++ b/docs-website/reference_versioned_docs/version-2.28/integrations-api/optimum.md @@ -5,134 +5,96 @@ description: "Optimum integration for Haystack" slug: "/integrations-optimum" --- - -## Module haystack\_integrations.components.embedders.optimum.optimization - - +## haystack_integrations.components.embedders.optimum.optimization ### OptimumEmbedderOptimizationMode -[ONXX Optimization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) -support by the Optimum Embedders. - - - -#### O1 - -Basic general optimizations. - - - -#### O2 - -Basic and extended general optimizations, transformers-specific fusions. +Bases: Enum - +ONNX Optimization modes supported by the Optimum Embedders. -#### O3 +See [Optimum ONNX optimization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) +for more details. -Same as O2 with Gelu approximation. - - - -#### O4 - -Same as O3 with mixed precision. - - - -#### OptimumEmbedderOptimizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderOptimizationMode" +from_str(string: str) -> OptimumEmbedderOptimizationMode ``` Create an optimization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Optimization mode. +**Returns:** - +- OptimumEmbedderOptimizationMode – Optimization mode. ### OptimumEmbedderOptimizationConfig Configuration for Optimum Embedder Optimization. -**Arguments**: +**Parameters:** -- `mode`: Optimization mode. -- `for_gpu`: Whether to optimize for GPUs. +- **mode** (OptimumEmbedderOptimizationMode) – Optimization mode. +- **for_gpu** (bool) – Whether to optimize for GPUs. - - -#### OptimumEmbedderOptimizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> OptimizationConfig +to_optimum_config() -> OptimizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: - -Optimum configuration. +**Returns:** - +- OptimizationConfig – Optimum configuration. -#### OptimumEmbedderOptimizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumEmbedderOptimizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderOptimizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderOptimizationConfig ``` Create an optimization configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Optimization configuration. +**Returns:** - +- OptimumEmbedderOptimizationConfig – Optimization configuration. -## Module haystack\_integrations.components.embedders.optimum.optimum\_document\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_document_embedder ### OptimumDocumentEmbedder -A component for computing `Document` embeddings using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component for computing `Document` embeddings using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. The embedding of each Document is stored in the `embedding` field of the Document. Usage example: + ```python from haystack.dataclasses import Document from haystack_integrations.components.embedders.optimum import OptimumDocumentEmbedder @@ -140,7 +102,7 @@ from haystack_integrations.components.embedders.optimum import OptimumDocumentEm doc = Document(content="I love pizza!") document_embedder = OptimumDocumentEmbedder(model="sentence-transformers/all-mpnet-base-v2") -document_embedder.warm_up() +# Components warm up automatically on first run. result = document_embedder.run([doc]) print(result["documents"][0].embedding) @@ -148,482 +110,398 @@ print(result["documents"][0].embedding) # [0.017020374536514282, -0.023255806416273117, ...] ``` - - -#### OptimumDocumentEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__(model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig - | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig - | None = None, - batch_size: int = 32, - progress_bar: bool = True, - meta_fields_to_embed: list[str] | None = None, - embedding_separator: str = "\n") -> None +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, + batch_size: int = 32, + progress_bar: bool = True, + meta_fields_to_embed: list[str] | None = None, + embedding_separator: str = "\n", +) -> None ``` Create a OptimumDocumentEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. -- `batch_size`: Number of Documents to encode at once. -- `progress_bar`: Whether to show a progress bar or not. -- `meta_fields_to_embed`: List of meta fields that should be embedded along with the Document text. -- `embedding_separator`: Separator used to concatenate the meta fields to the Document text. - - - -#### OptimumDocumentEmbedder.warm\_up +**Parameters:** + +- **model** (str) – A string representing the model id on HF Hub. + +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. + +- **prefix** (str) – A string to add to the beginning of each text. + +- **suffix** (str) – A string to add to the end of each text. + +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. + +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. + + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: + + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` + +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. + +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. + +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. + +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. + +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. + +- **batch_size** (int) – Number of Documents to encode at once. + +- **progress_bar** (bool) – Whether to show a progress bar or not. + +- **meta_fields_to_embed** (list\[str\] | None) – List of meta fields that should be embedded along with the Document text. + +- **embedding_separator** (str) – Separator used to concatenate the meta fields to the Document text. + +#### warm_up ```python -def warm_up() -> None +warm_up() -> None ``` Initializes the component. - - -#### OptimumDocumentEmbedder.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Serializes the component to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumDocumentEmbedder.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumDocumentEmbedder" +from_dict(data: dict[str, Any]) -> OptimumDocumentEmbedder ``` Deserializes the component from a dictionary. -**Arguments**: - -- `data`: The dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -The deserialized component. +**Returns:** - +- OptimumDocumentEmbedder – The deserialized component. -#### OptimumDocumentEmbedder.run +#### run ```python -@component.output_types(documents=list[Document]) -def run(documents: list[Document]) -> dict[str, list[Document]] +run(documents: list[Document]) -> dict[str, list[Document]] ``` Embed a list of Documents. The embedding of each Document is stored in the `embedding` field of the Document. -**Arguments**: - -- `documents`: A list of Documents to embed. +**Parameters:** -**Raises**: +- **documents** (list\[Document\]) – A list of Documents to embed. -- `TypeError`: If the input is not a list of Documents. +**Returns:** -**Returns**: +- dict\[str, list\[Document\]\] – The updated Documents with their embeddings. -The updated Documents with their embeddings. +**Raises:** - +- TypeError – If the input is not a list of Documents. -## Module haystack\_integrations.components.embedders.optimum.optimum\_text\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_text_embedder ### OptimumTextEmbedder -A component to embed text using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component to embed text using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. Usage example: + ```python from haystack_integrations.components.embedders.optimum import OptimumTextEmbedder text_to_embed = "I love pizza!" text_embedder = OptimumTextEmbedder(model="sentence-transformers/all-mpnet-base-v2") -text_embedder.warm_up() +# Components warm up automatically on first run. print(text_embedder.run(text_to_embed)) # {'embedding': [-0.07804739475250244, 0.1498992145061493,, ...]} ``` - - -#### OptimumTextEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__( - model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig | None = None) +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, +) -> None ``` Create a OptimumTextEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. - - - -#### OptimumTextEmbedder.warm\_up - -```python -def warm_up() -``` +**Parameters:** -Initializes the component. +- **model** (str) – A string representing the model id on HF Hub. - +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. -#### OptimumTextEmbedder.to\_dict +- **prefix** (str) – A string to add to the beginning of each text. -```python -def to_dict() -> dict[str, Any] -``` +- **suffix** (str) – A string to add to the end of each text. -Serializes the component to a dictionary. +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. -**Returns**: +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. -Dictionary with serialized data. + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: - - -#### OptimumTextEmbedder.from\_dict - -```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumTextEmbedder" -``` - -Deserializes the component from a dictionary. + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` -**Arguments**: +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `data`: The dictionary to deserialize from. +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. -**Returns**: +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. -The deserialized component. +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. - +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. -#### OptimumTextEmbedder.run +#### warm_up ```python -@component.output_types(embedding=list[float]) -def run(text: str) -> dict[str, list[float]] +warm_up() -> None ``` -Embed a string. - -**Arguments**: - -- `text`: The text to embed. - -**Raises**: - -- `TypeError`: If the input is not a string. - -**Returns**: +Initializes the component. -The embeddings of the text. +#### to_dict - +```python +to_dict() -> dict[str, Any] +``` -## Module haystack\_integrations.components.embedders.optimum.pooling +Serializes the component to a dictionary. - +**Returns:** -### OptimumEmbedderPooling +- dict\[str, Any\] – Dictionary with serialized data. -Pooling modes support by the Optimum Embedders. +#### from_dict - +```python +from_dict(data: dict[str, Any]) -> OptimumTextEmbedder +``` -#### CLS +Deserializes the component from a dictionary. -Perform CLS Pooling on the output of the embedding model -using the first token (CLS token). +**Parameters:** - +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -#### MEAN +**Returns:** -Perform Mean Pooling on the output of the embedding model. +- OptimumTextEmbedder – The deserialized component. - +#### run -#### MAX +```python +run(text: str) -> dict[str, list[float]] +``` -Perform Max Pooling on the output of the embedding model -using the maximum value in each dimension over all the tokens. +Embed a string. - +**Parameters:** -#### MEAN\_SQRT\_LEN +- **text** (str) – The text to embed. -Perform mean-pooling on the output of the embedding model but -divide by the square root of the sequence length. +**Returns:** - +- dict\[str, list\[float\]\] – The embeddings of the text. -#### WEIGHTED\_MEAN +**Raises:** -Perform weighted (position) mean pooling on the output of the -embedding model. +- TypeError – If the input is not a string. - +## haystack_integrations.components.embedders.optimum.pooling -#### LAST\_TOKEN +### OptimumEmbedderPooling -Perform Last Token Pooling on the output of the embedding model. +Bases: Enum - +Pooling modes support by the Optimum Embedders. -#### OptimumEmbedderPooling.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderPooling" +from_str(string: str) -> OptimumEmbedderPooling ``` Create a pooling mode from a string. -**Arguments**: +**Parameters:** -- `string`: String to convert. +- **string** (str) – String to convert. -**Returns**: +**Returns:** -Pooling mode. +- OptimumEmbedderPooling – Pooling mode. - - -## Module haystack\_integrations.components.embedders.optimum.quantization - - +## haystack_integrations.components.embedders.optimum.quantization ### OptimumEmbedderQuantizationMode -[Dynamic Quantization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) -support by the Optimum Embedders. - - - -#### ARM64 - -Quantization for the ARM64 architecture. - - +Bases: Enum -#### AVX2 +Dynamic Quantization modes supported by the Optimum Embedders. -Quantization with AVX-2 instructions. +See [Optimum ONNX quantization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) +for more details. - - -#### AVX512 - -Quantization with AVX-512 instructions. - - - -#### AVX512\_VNNI - -Quantization with AVX-512 and VNNI instructions. - - - -#### OptimumEmbedderQuantizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderQuantizationMode" +from_str(string: str) -> OptimumEmbedderQuantizationMode ``` Create an quantization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Quantization mode. +**Returns:** - +- OptimumEmbedderQuantizationMode – Quantization mode. ### OptimumEmbedderQuantizationConfig Configuration for Optimum Embedder Quantization. -**Arguments**: - -- `mode`: Quantization mode. -- `per_channel`: Whether to apply per-channel quantization. +**Parameters:** - +- **mode** (OptimumEmbedderQuantizationMode) – Quantization mode. +- **per_channel** (bool) – Whether to apply per-channel quantization. -#### OptimumEmbedderQuantizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> QuantizationConfig +to_optimum_config() -> QuantizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: +**Returns:** -Optimum configuration. +- QuantizationConfig – Optimum configuration. - - -#### OptimumEmbedderQuantizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: - -Dictionary with serialized data. +**Returns:** - +- dict\[str, Any\] – Dictionary with serialized data. -#### OptimumEmbedderQuantizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderQuantizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderQuantizationConfig ``` Create a configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Quantization configuration. +**Returns:** +- OptimumEmbedderQuantizationConfig – Quantization configuration. diff --git a/docs-website/reference_versioned_docs/version-2.29/integrations-api/optimum.md b/docs-website/reference_versioned_docs/version-2.29/integrations-api/optimum.md index da0d85c035..e823475a86 100644 --- a/docs-website/reference_versioned_docs/version-2.29/integrations-api/optimum.md +++ b/docs-website/reference_versioned_docs/version-2.29/integrations-api/optimum.md @@ -5,134 +5,96 @@ description: "Optimum integration for Haystack" slug: "/integrations-optimum" --- - -## Module haystack\_integrations.components.embedders.optimum.optimization - - +## haystack_integrations.components.embedders.optimum.optimization ### OptimumEmbedderOptimizationMode -[ONXX Optimization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) -support by the Optimum Embedders. - - - -#### O1 - -Basic general optimizations. - - - -#### O2 - -Basic and extended general optimizations, transformers-specific fusions. +Bases: Enum - +ONNX Optimization modes supported by the Optimum Embedders. -#### O3 +See [Optimum ONNX optimization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) +for more details. -Same as O2 with Gelu approximation. - - - -#### O4 - -Same as O3 with mixed precision. - - - -#### OptimumEmbedderOptimizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderOptimizationMode" +from_str(string: str) -> OptimumEmbedderOptimizationMode ``` Create an optimization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Optimization mode. +**Returns:** - +- OptimumEmbedderOptimizationMode – Optimization mode. ### OptimumEmbedderOptimizationConfig Configuration for Optimum Embedder Optimization. -**Arguments**: +**Parameters:** -- `mode`: Optimization mode. -- `for_gpu`: Whether to optimize for GPUs. +- **mode** (OptimumEmbedderOptimizationMode) – Optimization mode. +- **for_gpu** (bool) – Whether to optimize for GPUs. - - -#### OptimumEmbedderOptimizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> OptimizationConfig +to_optimum_config() -> OptimizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: - -Optimum configuration. +**Returns:** - +- OptimizationConfig – Optimum configuration. -#### OptimumEmbedderOptimizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumEmbedderOptimizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderOptimizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderOptimizationConfig ``` Create an optimization configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Optimization configuration. +**Returns:** - +- OptimumEmbedderOptimizationConfig – Optimization configuration. -## Module haystack\_integrations.components.embedders.optimum.optimum\_document\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_document_embedder ### OptimumDocumentEmbedder -A component for computing `Document` embeddings using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component for computing `Document` embeddings using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. The embedding of each Document is stored in the `embedding` field of the Document. Usage example: + ```python from haystack.dataclasses import Document from haystack_integrations.components.embedders.optimum import OptimumDocumentEmbedder @@ -140,7 +102,7 @@ from haystack_integrations.components.embedders.optimum import OptimumDocumentEm doc = Document(content="I love pizza!") document_embedder = OptimumDocumentEmbedder(model="sentence-transformers/all-mpnet-base-v2") -document_embedder.warm_up() +# Components warm up automatically on first run. result = document_embedder.run([doc]) print(result["documents"][0].embedding) @@ -148,482 +110,398 @@ print(result["documents"][0].embedding) # [0.017020374536514282, -0.023255806416273117, ...] ``` - - -#### OptimumDocumentEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__(model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig - | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig - | None = None, - batch_size: int = 32, - progress_bar: bool = True, - meta_fields_to_embed: list[str] | None = None, - embedding_separator: str = "\n") -> None +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, + batch_size: int = 32, + progress_bar: bool = True, + meta_fields_to_embed: list[str] | None = None, + embedding_separator: str = "\n", +) -> None ``` Create a OptimumDocumentEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. -- `batch_size`: Number of Documents to encode at once. -- `progress_bar`: Whether to show a progress bar or not. -- `meta_fields_to_embed`: List of meta fields that should be embedded along with the Document text. -- `embedding_separator`: Separator used to concatenate the meta fields to the Document text. - - - -#### OptimumDocumentEmbedder.warm\_up +**Parameters:** + +- **model** (str) – A string representing the model id on HF Hub. + +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. + +- **prefix** (str) – A string to add to the beginning of each text. + +- **suffix** (str) – A string to add to the end of each text. + +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. + +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. + + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: + + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` + +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. + +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. + +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. + +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. + +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. + +- **batch_size** (int) – Number of Documents to encode at once. + +- **progress_bar** (bool) – Whether to show a progress bar or not. + +- **meta_fields_to_embed** (list\[str\] | None) – List of meta fields that should be embedded along with the Document text. + +- **embedding_separator** (str) – Separator used to concatenate the meta fields to the Document text. + +#### warm_up ```python -def warm_up() -> None +warm_up() -> None ``` Initializes the component. - - -#### OptimumDocumentEmbedder.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Serializes the component to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumDocumentEmbedder.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumDocumentEmbedder" +from_dict(data: dict[str, Any]) -> OptimumDocumentEmbedder ``` Deserializes the component from a dictionary. -**Arguments**: - -- `data`: The dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -The deserialized component. +**Returns:** - +- OptimumDocumentEmbedder – The deserialized component. -#### OptimumDocumentEmbedder.run +#### run ```python -@component.output_types(documents=list[Document]) -def run(documents: list[Document]) -> dict[str, list[Document]] +run(documents: list[Document]) -> dict[str, list[Document]] ``` Embed a list of Documents. The embedding of each Document is stored in the `embedding` field of the Document. -**Arguments**: - -- `documents`: A list of Documents to embed. +**Parameters:** -**Raises**: +- **documents** (list\[Document\]) – A list of Documents to embed. -- `TypeError`: If the input is not a list of Documents. +**Returns:** -**Returns**: +- dict\[str, list\[Document\]\] – The updated Documents with their embeddings. -The updated Documents with their embeddings. +**Raises:** - +- TypeError – If the input is not a list of Documents. -## Module haystack\_integrations.components.embedders.optimum.optimum\_text\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_text_embedder ### OptimumTextEmbedder -A component to embed text using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component to embed text using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. Usage example: + ```python from haystack_integrations.components.embedders.optimum import OptimumTextEmbedder text_to_embed = "I love pizza!" text_embedder = OptimumTextEmbedder(model="sentence-transformers/all-mpnet-base-v2") -text_embedder.warm_up() +# Components warm up automatically on first run. print(text_embedder.run(text_to_embed)) # {'embedding': [-0.07804739475250244, 0.1498992145061493,, ...]} ``` - - -#### OptimumTextEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__( - model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig | None = None) +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, +) -> None ``` Create a OptimumTextEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. - - - -#### OptimumTextEmbedder.warm\_up - -```python -def warm_up() -``` +**Parameters:** -Initializes the component. +- **model** (str) – A string representing the model id on HF Hub. - +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. -#### OptimumTextEmbedder.to\_dict +- **prefix** (str) – A string to add to the beginning of each text. -```python -def to_dict() -> dict[str, Any] -``` +- **suffix** (str) – A string to add to the end of each text. -Serializes the component to a dictionary. +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. -**Returns**: +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. -Dictionary with serialized data. + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: - - -#### OptimumTextEmbedder.from\_dict - -```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumTextEmbedder" -``` - -Deserializes the component from a dictionary. + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` -**Arguments**: +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `data`: The dictionary to deserialize from. +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. -**Returns**: +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. -The deserialized component. +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. - +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. -#### OptimumTextEmbedder.run +#### warm_up ```python -@component.output_types(embedding=list[float]) -def run(text: str) -> dict[str, list[float]] +warm_up() -> None ``` -Embed a string. - -**Arguments**: - -- `text`: The text to embed. - -**Raises**: - -- `TypeError`: If the input is not a string. - -**Returns**: +Initializes the component. -The embeddings of the text. +#### to_dict - +```python +to_dict() -> dict[str, Any] +``` -## Module haystack\_integrations.components.embedders.optimum.pooling +Serializes the component to a dictionary. - +**Returns:** -### OptimumEmbedderPooling +- dict\[str, Any\] – Dictionary with serialized data. -Pooling modes support by the Optimum Embedders. +#### from_dict - +```python +from_dict(data: dict[str, Any]) -> OptimumTextEmbedder +``` -#### CLS +Deserializes the component from a dictionary. -Perform CLS Pooling on the output of the embedding model -using the first token (CLS token). +**Parameters:** - +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -#### MEAN +**Returns:** -Perform Mean Pooling on the output of the embedding model. +- OptimumTextEmbedder – The deserialized component. - +#### run -#### MAX +```python +run(text: str) -> dict[str, list[float]] +``` -Perform Max Pooling on the output of the embedding model -using the maximum value in each dimension over all the tokens. +Embed a string. - +**Parameters:** -#### MEAN\_SQRT\_LEN +- **text** (str) – The text to embed. -Perform mean-pooling on the output of the embedding model but -divide by the square root of the sequence length. +**Returns:** - +- dict\[str, list\[float\]\] – The embeddings of the text. -#### WEIGHTED\_MEAN +**Raises:** -Perform weighted (position) mean pooling on the output of the -embedding model. +- TypeError – If the input is not a string. - +## haystack_integrations.components.embedders.optimum.pooling -#### LAST\_TOKEN +### OptimumEmbedderPooling -Perform Last Token Pooling on the output of the embedding model. +Bases: Enum - +Pooling modes support by the Optimum Embedders. -#### OptimumEmbedderPooling.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderPooling" +from_str(string: str) -> OptimumEmbedderPooling ``` Create a pooling mode from a string. -**Arguments**: +**Parameters:** -- `string`: String to convert. +- **string** (str) – String to convert. -**Returns**: +**Returns:** -Pooling mode. +- OptimumEmbedderPooling – Pooling mode. - - -## Module haystack\_integrations.components.embedders.optimum.quantization - - +## haystack_integrations.components.embedders.optimum.quantization ### OptimumEmbedderQuantizationMode -[Dynamic Quantization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) -support by the Optimum Embedders. - - - -#### ARM64 - -Quantization for the ARM64 architecture. - - +Bases: Enum -#### AVX2 +Dynamic Quantization modes supported by the Optimum Embedders. -Quantization with AVX-2 instructions. +See [Optimum ONNX quantization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) +for more details. - - -#### AVX512 - -Quantization with AVX-512 instructions. - - - -#### AVX512\_VNNI - -Quantization with AVX-512 and VNNI instructions. - - - -#### OptimumEmbedderQuantizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderQuantizationMode" +from_str(string: str) -> OptimumEmbedderQuantizationMode ``` Create an quantization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Quantization mode. +**Returns:** - +- OptimumEmbedderQuantizationMode – Quantization mode. ### OptimumEmbedderQuantizationConfig Configuration for Optimum Embedder Quantization. -**Arguments**: - -- `mode`: Quantization mode. -- `per_channel`: Whether to apply per-channel quantization. +**Parameters:** - +- **mode** (OptimumEmbedderQuantizationMode) – Quantization mode. +- **per_channel** (bool) – Whether to apply per-channel quantization. -#### OptimumEmbedderQuantizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> QuantizationConfig +to_optimum_config() -> QuantizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: +**Returns:** -Optimum configuration. +- QuantizationConfig – Optimum configuration. - - -#### OptimumEmbedderQuantizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: - -Dictionary with serialized data. +**Returns:** - +- dict\[str, Any\] – Dictionary with serialized data. -#### OptimumEmbedderQuantizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderQuantizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderQuantizationConfig ``` Create a configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Quantization configuration. +**Returns:** +- OptimumEmbedderQuantizationConfig – Quantization configuration. diff --git a/docs-website/reference_versioned_docs/version-2.30/integrations-api/optimum.md b/docs-website/reference_versioned_docs/version-2.30/integrations-api/optimum.md index da0d85c035..e823475a86 100644 --- a/docs-website/reference_versioned_docs/version-2.30/integrations-api/optimum.md +++ b/docs-website/reference_versioned_docs/version-2.30/integrations-api/optimum.md @@ -5,134 +5,96 @@ description: "Optimum integration for Haystack" slug: "/integrations-optimum" --- - -## Module haystack\_integrations.components.embedders.optimum.optimization - - +## haystack_integrations.components.embedders.optimum.optimization ### OptimumEmbedderOptimizationMode -[ONXX Optimization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) -support by the Optimum Embedders. - - - -#### O1 - -Basic general optimizations. - - - -#### O2 - -Basic and extended general optimizations, transformers-specific fusions. +Bases: Enum - +ONNX Optimization modes supported by the Optimum Embedders. -#### O3 +See [Optimum ONNX optimization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) +for more details. -Same as O2 with Gelu approximation. - - - -#### O4 - -Same as O3 with mixed precision. - - - -#### OptimumEmbedderOptimizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderOptimizationMode" +from_str(string: str) -> OptimumEmbedderOptimizationMode ``` Create an optimization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Optimization mode. +**Returns:** - +- OptimumEmbedderOptimizationMode – Optimization mode. ### OptimumEmbedderOptimizationConfig Configuration for Optimum Embedder Optimization. -**Arguments**: +**Parameters:** -- `mode`: Optimization mode. -- `for_gpu`: Whether to optimize for GPUs. +- **mode** (OptimumEmbedderOptimizationMode) – Optimization mode. +- **for_gpu** (bool) – Whether to optimize for GPUs. - - -#### OptimumEmbedderOptimizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> OptimizationConfig +to_optimum_config() -> OptimizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: - -Optimum configuration. +**Returns:** - +- OptimizationConfig – Optimum configuration. -#### OptimumEmbedderOptimizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumEmbedderOptimizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderOptimizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderOptimizationConfig ``` Create an optimization configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Optimization configuration. +**Returns:** - +- OptimumEmbedderOptimizationConfig – Optimization configuration. -## Module haystack\_integrations.components.embedders.optimum.optimum\_document\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_document_embedder ### OptimumDocumentEmbedder -A component for computing `Document` embeddings using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component for computing `Document` embeddings using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. The embedding of each Document is stored in the `embedding` field of the Document. Usage example: + ```python from haystack.dataclasses import Document from haystack_integrations.components.embedders.optimum import OptimumDocumentEmbedder @@ -140,7 +102,7 @@ from haystack_integrations.components.embedders.optimum import OptimumDocumentEm doc = Document(content="I love pizza!") document_embedder = OptimumDocumentEmbedder(model="sentence-transformers/all-mpnet-base-v2") -document_embedder.warm_up() +# Components warm up automatically on first run. result = document_embedder.run([doc]) print(result["documents"][0].embedding) @@ -148,482 +110,398 @@ print(result["documents"][0].embedding) # [0.017020374536514282, -0.023255806416273117, ...] ``` - - -#### OptimumDocumentEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__(model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig - | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig - | None = None, - batch_size: int = 32, - progress_bar: bool = True, - meta_fields_to_embed: list[str] | None = None, - embedding_separator: str = "\n") -> None +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, + batch_size: int = 32, + progress_bar: bool = True, + meta_fields_to_embed: list[str] | None = None, + embedding_separator: str = "\n", +) -> None ``` Create a OptimumDocumentEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. -- `batch_size`: Number of Documents to encode at once. -- `progress_bar`: Whether to show a progress bar or not. -- `meta_fields_to_embed`: List of meta fields that should be embedded along with the Document text. -- `embedding_separator`: Separator used to concatenate the meta fields to the Document text. - - - -#### OptimumDocumentEmbedder.warm\_up +**Parameters:** + +- **model** (str) – A string representing the model id on HF Hub. + +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. + +- **prefix** (str) – A string to add to the beginning of each text. + +- **suffix** (str) – A string to add to the end of each text. + +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. + +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. + + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: + + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` + +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. + +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. + +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. + +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. + +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. + +- **batch_size** (int) – Number of Documents to encode at once. + +- **progress_bar** (bool) – Whether to show a progress bar or not. + +- **meta_fields_to_embed** (list\[str\] | None) – List of meta fields that should be embedded along with the Document text. + +- **embedding_separator** (str) – Separator used to concatenate the meta fields to the Document text. + +#### warm_up ```python -def warm_up() -> None +warm_up() -> None ``` Initializes the component. - - -#### OptimumDocumentEmbedder.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Serializes the component to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumDocumentEmbedder.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumDocumentEmbedder" +from_dict(data: dict[str, Any]) -> OptimumDocumentEmbedder ``` Deserializes the component from a dictionary. -**Arguments**: - -- `data`: The dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -The deserialized component. +**Returns:** - +- OptimumDocumentEmbedder – The deserialized component. -#### OptimumDocumentEmbedder.run +#### run ```python -@component.output_types(documents=list[Document]) -def run(documents: list[Document]) -> dict[str, list[Document]] +run(documents: list[Document]) -> dict[str, list[Document]] ``` Embed a list of Documents. The embedding of each Document is stored in the `embedding` field of the Document. -**Arguments**: - -- `documents`: A list of Documents to embed. +**Parameters:** -**Raises**: +- **documents** (list\[Document\]) – A list of Documents to embed. -- `TypeError`: If the input is not a list of Documents. +**Returns:** -**Returns**: +- dict\[str, list\[Document\]\] – The updated Documents with their embeddings. -The updated Documents with their embeddings. +**Raises:** - +- TypeError – If the input is not a list of Documents. -## Module haystack\_integrations.components.embedders.optimum.optimum\_text\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_text_embedder ### OptimumTextEmbedder -A component to embed text using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component to embed text using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. Usage example: + ```python from haystack_integrations.components.embedders.optimum import OptimumTextEmbedder text_to_embed = "I love pizza!" text_embedder = OptimumTextEmbedder(model="sentence-transformers/all-mpnet-base-v2") -text_embedder.warm_up() +# Components warm up automatically on first run. print(text_embedder.run(text_to_embed)) # {'embedding': [-0.07804739475250244, 0.1498992145061493,, ...]} ``` - - -#### OptimumTextEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__( - model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig | None = None) +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, +) -> None ``` Create a OptimumTextEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. - - - -#### OptimumTextEmbedder.warm\_up - -```python -def warm_up() -``` +**Parameters:** -Initializes the component. +- **model** (str) – A string representing the model id on HF Hub. - +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. -#### OptimumTextEmbedder.to\_dict +- **prefix** (str) – A string to add to the beginning of each text. -```python -def to_dict() -> dict[str, Any] -``` +- **suffix** (str) – A string to add to the end of each text. -Serializes the component to a dictionary. +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. -**Returns**: +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. -Dictionary with serialized data. + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: - - -#### OptimumTextEmbedder.from\_dict - -```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumTextEmbedder" -``` - -Deserializes the component from a dictionary. + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` -**Arguments**: +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `data`: The dictionary to deserialize from. +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. -**Returns**: +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. -The deserialized component. +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. - +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. -#### OptimumTextEmbedder.run +#### warm_up ```python -@component.output_types(embedding=list[float]) -def run(text: str) -> dict[str, list[float]] +warm_up() -> None ``` -Embed a string. - -**Arguments**: - -- `text`: The text to embed. - -**Raises**: - -- `TypeError`: If the input is not a string. - -**Returns**: +Initializes the component. -The embeddings of the text. +#### to_dict - +```python +to_dict() -> dict[str, Any] +``` -## Module haystack\_integrations.components.embedders.optimum.pooling +Serializes the component to a dictionary. - +**Returns:** -### OptimumEmbedderPooling +- dict\[str, Any\] – Dictionary with serialized data. -Pooling modes support by the Optimum Embedders. +#### from_dict - +```python +from_dict(data: dict[str, Any]) -> OptimumTextEmbedder +``` -#### CLS +Deserializes the component from a dictionary. -Perform CLS Pooling on the output of the embedding model -using the first token (CLS token). +**Parameters:** - +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -#### MEAN +**Returns:** -Perform Mean Pooling on the output of the embedding model. +- OptimumTextEmbedder – The deserialized component. - +#### run -#### MAX +```python +run(text: str) -> dict[str, list[float]] +``` -Perform Max Pooling on the output of the embedding model -using the maximum value in each dimension over all the tokens. +Embed a string. - +**Parameters:** -#### MEAN\_SQRT\_LEN +- **text** (str) – The text to embed. -Perform mean-pooling on the output of the embedding model but -divide by the square root of the sequence length. +**Returns:** - +- dict\[str, list\[float\]\] – The embeddings of the text. -#### WEIGHTED\_MEAN +**Raises:** -Perform weighted (position) mean pooling on the output of the -embedding model. +- TypeError – If the input is not a string. - +## haystack_integrations.components.embedders.optimum.pooling -#### LAST\_TOKEN +### OptimumEmbedderPooling -Perform Last Token Pooling on the output of the embedding model. +Bases: Enum - +Pooling modes support by the Optimum Embedders. -#### OptimumEmbedderPooling.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderPooling" +from_str(string: str) -> OptimumEmbedderPooling ``` Create a pooling mode from a string. -**Arguments**: +**Parameters:** -- `string`: String to convert. +- **string** (str) – String to convert. -**Returns**: +**Returns:** -Pooling mode. +- OptimumEmbedderPooling – Pooling mode. - - -## Module haystack\_integrations.components.embedders.optimum.quantization - - +## haystack_integrations.components.embedders.optimum.quantization ### OptimumEmbedderQuantizationMode -[Dynamic Quantization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) -support by the Optimum Embedders. - - - -#### ARM64 - -Quantization for the ARM64 architecture. - - +Bases: Enum -#### AVX2 +Dynamic Quantization modes supported by the Optimum Embedders. -Quantization with AVX-2 instructions. +See [Optimum ONNX quantization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) +for more details. - - -#### AVX512 - -Quantization with AVX-512 instructions. - - - -#### AVX512\_VNNI - -Quantization with AVX-512 and VNNI instructions. - - - -#### OptimumEmbedderQuantizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderQuantizationMode" +from_str(string: str) -> OptimumEmbedderQuantizationMode ``` Create an quantization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Quantization mode. +**Returns:** - +- OptimumEmbedderQuantizationMode – Quantization mode. ### OptimumEmbedderQuantizationConfig Configuration for Optimum Embedder Quantization. -**Arguments**: - -- `mode`: Quantization mode. -- `per_channel`: Whether to apply per-channel quantization. +**Parameters:** - +- **mode** (OptimumEmbedderQuantizationMode) – Quantization mode. +- **per_channel** (bool) – Whether to apply per-channel quantization. -#### OptimumEmbedderQuantizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> QuantizationConfig +to_optimum_config() -> QuantizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: +**Returns:** -Optimum configuration. +- QuantizationConfig – Optimum configuration. - - -#### OptimumEmbedderQuantizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: - -Dictionary with serialized data. +**Returns:** - +- dict\[str, Any\] – Dictionary with serialized data. -#### OptimumEmbedderQuantizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderQuantizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderQuantizationConfig ``` Create a configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Quantization configuration. +**Returns:** +- OptimumEmbedderQuantizationConfig – Quantization configuration. diff --git a/docs-website/reference_versioned_docs/version-2.31/integrations-api/optimum.md b/docs-website/reference_versioned_docs/version-2.31/integrations-api/optimum.md index da0d85c035..e823475a86 100644 --- a/docs-website/reference_versioned_docs/version-2.31/integrations-api/optimum.md +++ b/docs-website/reference_versioned_docs/version-2.31/integrations-api/optimum.md @@ -5,134 +5,96 @@ description: "Optimum integration for Haystack" slug: "/integrations-optimum" --- - -## Module haystack\_integrations.components.embedders.optimum.optimization - - +## haystack_integrations.components.embedders.optimum.optimization ### OptimumEmbedderOptimizationMode -[ONXX Optimization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) -support by the Optimum Embedders. - - - -#### O1 - -Basic general optimizations. - - - -#### O2 - -Basic and extended general optimizations, transformers-specific fusions. +Bases: Enum - +ONNX Optimization modes supported by the Optimum Embedders. -#### O3 +See [Optimum ONNX optimization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) +for more details. -Same as O2 with Gelu approximation. - - - -#### O4 - -Same as O3 with mixed precision. - - - -#### OptimumEmbedderOptimizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderOptimizationMode" +from_str(string: str) -> OptimumEmbedderOptimizationMode ``` Create an optimization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Optimization mode. +**Returns:** - +- OptimumEmbedderOptimizationMode – Optimization mode. ### OptimumEmbedderOptimizationConfig Configuration for Optimum Embedder Optimization. -**Arguments**: +**Parameters:** -- `mode`: Optimization mode. -- `for_gpu`: Whether to optimize for GPUs. +- **mode** (OptimumEmbedderOptimizationMode) – Optimization mode. +- **for_gpu** (bool) – Whether to optimize for GPUs. - - -#### OptimumEmbedderOptimizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> OptimizationConfig +to_optimum_config() -> OptimizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: - -Optimum configuration. +**Returns:** - +- OptimizationConfig – Optimum configuration. -#### OptimumEmbedderOptimizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumEmbedderOptimizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderOptimizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderOptimizationConfig ``` Create an optimization configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Optimization configuration. +**Returns:** - +- OptimumEmbedderOptimizationConfig – Optimization configuration. -## Module haystack\_integrations.components.embedders.optimum.optimum\_document\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_document_embedder ### OptimumDocumentEmbedder -A component for computing `Document` embeddings using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component for computing `Document` embeddings using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. The embedding of each Document is stored in the `embedding` field of the Document. Usage example: + ```python from haystack.dataclasses import Document from haystack_integrations.components.embedders.optimum import OptimumDocumentEmbedder @@ -140,7 +102,7 @@ from haystack_integrations.components.embedders.optimum import OptimumDocumentEm doc = Document(content="I love pizza!") document_embedder = OptimumDocumentEmbedder(model="sentence-transformers/all-mpnet-base-v2") -document_embedder.warm_up() +# Components warm up automatically on first run. result = document_embedder.run([doc]) print(result["documents"][0].embedding) @@ -148,482 +110,398 @@ print(result["documents"][0].embedding) # [0.017020374536514282, -0.023255806416273117, ...] ``` - - -#### OptimumDocumentEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__(model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig - | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig - | None = None, - batch_size: int = 32, - progress_bar: bool = True, - meta_fields_to_embed: list[str] | None = None, - embedding_separator: str = "\n") -> None +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, + batch_size: int = 32, + progress_bar: bool = True, + meta_fields_to_embed: list[str] | None = None, + embedding_separator: str = "\n", +) -> None ``` Create a OptimumDocumentEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. -- `batch_size`: Number of Documents to encode at once. -- `progress_bar`: Whether to show a progress bar or not. -- `meta_fields_to_embed`: List of meta fields that should be embedded along with the Document text. -- `embedding_separator`: Separator used to concatenate the meta fields to the Document text. - - - -#### OptimumDocumentEmbedder.warm\_up +**Parameters:** + +- **model** (str) – A string representing the model id on HF Hub. + +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. + +- **prefix** (str) – A string to add to the beginning of each text. + +- **suffix** (str) – A string to add to the end of each text. + +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. + +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. + + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: + + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` + +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. + +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. + +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. + +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. + +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. + +- **batch_size** (int) – Number of Documents to encode at once. + +- **progress_bar** (bool) – Whether to show a progress bar or not. + +- **meta_fields_to_embed** (list\[str\] | None) – List of meta fields that should be embedded along with the Document text. + +- **embedding_separator** (str) – Separator used to concatenate the meta fields to the Document text. + +#### warm_up ```python -def warm_up() -> None +warm_up() -> None ``` Initializes the component. - - -#### OptimumDocumentEmbedder.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Serializes the component to a dictionary. -**Returns**: +**Returns:** -Dictionary with serialized data. +- dict\[str, Any\] – Dictionary with serialized data. - - -#### OptimumDocumentEmbedder.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumDocumentEmbedder" +from_dict(data: dict[str, Any]) -> OptimumDocumentEmbedder ``` Deserializes the component from a dictionary. -**Arguments**: - -- `data`: The dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -The deserialized component. +**Returns:** - +- OptimumDocumentEmbedder – The deserialized component. -#### OptimumDocumentEmbedder.run +#### run ```python -@component.output_types(documents=list[Document]) -def run(documents: list[Document]) -> dict[str, list[Document]] +run(documents: list[Document]) -> dict[str, list[Document]] ``` Embed a list of Documents. The embedding of each Document is stored in the `embedding` field of the Document. -**Arguments**: - -- `documents`: A list of Documents to embed. +**Parameters:** -**Raises**: +- **documents** (list\[Document\]) – A list of Documents to embed. -- `TypeError`: If the input is not a list of Documents. +**Returns:** -**Returns**: +- dict\[str, list\[Document\]\] – The updated Documents with their embeddings. -The updated Documents with their embeddings. +**Raises:** - +- TypeError – If the input is not a list of Documents. -## Module haystack\_integrations.components.embedders.optimum.optimum\_text\_embedder - - +## haystack_integrations.components.embedders.optimum.optimum_text_embedder ### OptimumTextEmbedder -A component to embed text using models loaded with the -[HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library, -leveraging the ONNX runtime for high-speed inference. +A component to embed text using models loaded with the HuggingFace Optimum library. + +Uses the [HuggingFace Optimum](https://huggingface.co/docs/optimum/index) library and leverages the ONNX +runtime for high-speed inference. Usage example: + ```python from haystack_integrations.components.embedders.optimum import OptimumTextEmbedder text_to_embed = "I love pizza!" text_embedder = OptimumTextEmbedder(model="sentence-transformers/all-mpnet-base-v2") -text_embedder.warm_up() +# Components warm up automatically on first run. print(text_embedder.run(text_to_embed)) # {'embedding': [-0.07804739475250244, 0.1498992145061493,, ...]} ``` - - -#### OptimumTextEmbedder.\_\_init\_\_ +#### __init__ ```python -def __init__( - model: str = "sentence-transformers/all-mpnet-base-v2", - token: Secret | None = Secret.from_env_var("HF_API_TOKEN", - strict=False), - prefix: str = "", - suffix: str = "", - normalize_embeddings: bool = True, - onnx_execution_provider: str = "CPUExecutionProvider", - pooling_mode: str | OptimumEmbedderPooling | None = None, - model_kwargs: dict[str, Any] | None = None, - working_dir: str | None = None, - optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, - quantizer_settings: OptimumEmbedderQuantizationConfig | None = None) +__init__( + model: str = "sentence-transformers/all-mpnet-base-v2", + token: Secret | None = Secret.from_env_var("HF_API_TOKEN", strict=False), + prefix: str = "", + suffix: str = "", + normalize_embeddings: bool = True, + onnx_execution_provider: str = "CPUExecutionProvider", + pooling_mode: str | OptimumEmbedderPooling | None = None, + model_kwargs: dict[str, Any] | None = None, + working_dir: str | None = None, + optimizer_settings: OptimumEmbedderOptimizationConfig | None = None, + quantizer_settings: OptimumEmbedderQuantizationConfig | None = None, +) -> None ``` Create a OptimumTextEmbedder component. -**Arguments**: - -- `model`: A string representing the model id on HF Hub. -- `token`: The HuggingFace token to use as HTTP bearer authorization. -- `prefix`: A string to add to the beginning of each text. -- `suffix`: A string to add to the end of each text. -- `normalize_embeddings`: Whether to normalize the embeddings to unit length. -- `onnx_execution_provider`: The [execution provider](https://onnxruntime.ai/docs/execution-providers/) -to use for ONNX models. - -Note: Using the TensorRT execution provider -TensorRT requires to build its inference engine ahead of inference, -which takes some time due to the model optimization and nodes fusion. -To avoid rebuilding the engine every time the model is loaded, ONNX -Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` -and `trt_engine_cache_path`. We recommend setting these two provider -options using the `model_kwargs` parameter, when using the TensorRT execution provider. -The usage is as follows: -```python -embedder = OptimumDocumentEmbedder( - model="sentence-transformers/all-mpnet-base-v2", - onnx_execution_provider="TensorrtExecutionProvider", - model_kwargs={ - "provider_options": { - "trt_engine_cache_enable": True, - "trt_engine_cache_path": "tmp/trt_cache", - } - }, -) -``` -- `pooling_mode`: The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `model_kwargs`: Dictionary containing additional keyword arguments to pass to the model. -In case of duplication, these kwargs override `model`, `onnx_execution_provider` -and `token` initialization parameters. -- `working_dir`: The directory to use for storing intermediate files -generated during model optimization/quantization. Required -for optimization and quantization. -- `optimizer_settings`: Configuration for Optimum Embedder Optimization. -If `None`, no additional optimization is be applied. -- `quantizer_settings`: Configuration for Optimum Embedder Quantization. -If `None`, no quantization is be applied. - - - -#### OptimumTextEmbedder.warm\_up - -```python -def warm_up() -``` +**Parameters:** -Initializes the component. +- **model** (str) – A string representing the model id on HF Hub. - +- **token** (Secret | None) – The HuggingFace token to use as HTTP bearer authorization. -#### OptimumTextEmbedder.to\_dict +- **prefix** (str) – A string to add to the beginning of each text. -```python -def to_dict() -> dict[str, Any] -``` +- **suffix** (str) – A string to add to the end of each text. -Serializes the component to a dictionary. +- **normalize_embeddings** (bool) – Whether to normalize the embeddings to unit length. -**Returns**: +- **onnx_execution_provider** (str) – The [execution provider](https://onnxruntime.ai/docs/execution-providers/) + to use for ONNX models. -Dictionary with serialized data. + Note: Using the TensorRT execution provider + TensorRT requires to build its inference engine ahead of inference, + which takes some time due to the model optimization and nodes fusion. + To avoid rebuilding the engine every time the model is loaded, ONNX + Runtime provides a pair of options to save the engine: `trt_engine_cache_enable` + and `trt_engine_cache_path`. We recommend setting these two provider + options using the `model_kwargs` parameter, when using the TensorRT execution provider. + The usage is as follows: - - -#### OptimumTextEmbedder.from\_dict - -```python -@classmethod -def from_dict(cls, data: dict[str, Any]) -> "OptimumTextEmbedder" -``` - -Deserializes the component from a dictionary. + ```python + embedder = OptimumDocumentEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + onnx_execution_provider="TensorrtExecutionProvider", + model_kwargs={ + "provider_options": { + "trt_engine_cache_enable": True, + "trt_engine_cache_path": "tmp/trt_cache", + } + }, + ) + ``` -**Arguments**: +- **pooling_mode** (str | OptimumEmbedderPooling | None) – The pooling mode to use. When `None`, pooling mode will be inferred from the model config. -- `data`: The dictionary to deserialize from. +- **model_kwargs** (dict\[str, Any\] | None) – Dictionary containing additional keyword arguments to pass to the model. + In case of duplication, these kwargs override `model`, `onnx_execution_provider` + and `token` initialization parameters. -**Returns**: +- **working_dir** (str | None) – The directory to use for storing intermediate files + generated during model optimization/quantization. Required + for optimization and quantization. -The deserialized component. +- **optimizer_settings** (OptimumEmbedderOptimizationConfig | None) – Configuration for Optimum Embedder Optimization. + If `None`, no additional optimization is be applied. - +- **quantizer_settings** (OptimumEmbedderQuantizationConfig | None) – Configuration for Optimum Embedder Quantization. + If `None`, no quantization is be applied. -#### OptimumTextEmbedder.run +#### warm_up ```python -@component.output_types(embedding=list[float]) -def run(text: str) -> dict[str, list[float]] +warm_up() -> None ``` -Embed a string. - -**Arguments**: - -- `text`: The text to embed. - -**Raises**: - -- `TypeError`: If the input is not a string. - -**Returns**: +Initializes the component. -The embeddings of the text. +#### to_dict - +```python +to_dict() -> dict[str, Any] +``` -## Module haystack\_integrations.components.embedders.optimum.pooling +Serializes the component to a dictionary. - +**Returns:** -### OptimumEmbedderPooling +- dict\[str, Any\] – Dictionary with serialized data. -Pooling modes support by the Optimum Embedders. +#### from_dict - +```python +from_dict(data: dict[str, Any]) -> OptimumTextEmbedder +``` -#### CLS +Deserializes the component from a dictionary. -Perform CLS Pooling on the output of the embedding model -using the first token (CLS token). +**Parameters:** - +- **data** (dict\[str, Any\]) – The dictionary to deserialize from. -#### MEAN +**Returns:** -Perform Mean Pooling on the output of the embedding model. +- OptimumTextEmbedder – The deserialized component. - +#### run -#### MAX +```python +run(text: str) -> dict[str, list[float]] +``` -Perform Max Pooling on the output of the embedding model -using the maximum value in each dimension over all the tokens. +Embed a string. - +**Parameters:** -#### MEAN\_SQRT\_LEN +- **text** (str) – The text to embed. -Perform mean-pooling on the output of the embedding model but -divide by the square root of the sequence length. +**Returns:** - +- dict\[str, list\[float\]\] – The embeddings of the text. -#### WEIGHTED\_MEAN +**Raises:** -Perform weighted (position) mean pooling on the output of the -embedding model. +- TypeError – If the input is not a string. - +## haystack_integrations.components.embedders.optimum.pooling -#### LAST\_TOKEN +### OptimumEmbedderPooling -Perform Last Token Pooling on the output of the embedding model. +Bases: Enum - +Pooling modes support by the Optimum Embedders. -#### OptimumEmbedderPooling.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderPooling" +from_str(string: str) -> OptimumEmbedderPooling ``` Create a pooling mode from a string. -**Arguments**: +**Parameters:** -- `string`: String to convert. +- **string** (str) – String to convert. -**Returns**: +**Returns:** -Pooling mode. +- OptimumEmbedderPooling – Pooling mode. - - -## Module haystack\_integrations.components.embedders.optimum.quantization - - +## haystack_integrations.components.embedders.optimum.quantization ### OptimumEmbedderQuantizationMode -[Dynamic Quantization modes](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) -support by the Optimum Embedders. - - - -#### ARM64 - -Quantization for the ARM64 architecture. - - +Bases: Enum -#### AVX2 +Dynamic Quantization modes supported by the Optimum Embedders. -Quantization with AVX-2 instructions. +See [Optimum ONNX quantization docs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization) +for more details. - - -#### AVX512 - -Quantization with AVX-512 instructions. - - - -#### AVX512\_VNNI - -Quantization with AVX-512 and VNNI instructions. - - - -#### OptimumEmbedderQuantizationMode.from\_str +#### from_str ```python -@classmethod -def from_str(cls, string: str) -> "OptimumEmbedderQuantizationMode" +from_str(string: str) -> OptimumEmbedderQuantizationMode ``` Create an quantization mode from a string. -**Arguments**: - -- `string`: String to convert. +**Parameters:** -**Returns**: +- **string** (str) – String to convert. -Quantization mode. +**Returns:** - +- OptimumEmbedderQuantizationMode – Quantization mode. ### OptimumEmbedderQuantizationConfig Configuration for Optimum Embedder Quantization. -**Arguments**: - -- `mode`: Quantization mode. -- `per_channel`: Whether to apply per-channel quantization. +**Parameters:** - +- **mode** (OptimumEmbedderQuantizationMode) – Quantization mode. +- **per_channel** (bool) – Whether to apply per-channel quantization. -#### OptimumEmbedderQuantizationConfig.to\_optimum\_config +#### to_optimum_config ```python -def to_optimum_config() -> QuantizationConfig +to_optimum_config() -> QuantizationConfig ``` Convert the configuration to a Optimum configuration. -**Returns**: +**Returns:** -Optimum configuration. +- QuantizationConfig – Optimum configuration. - - -#### OptimumEmbedderQuantizationConfig.to\_dict +#### to_dict ```python -def to_dict() -> dict[str, Any] +to_dict() -> dict[str, Any] ``` Convert the configuration to a dictionary. -**Returns**: - -Dictionary with serialized data. +**Returns:** - +- dict\[str, Any\] – Dictionary with serialized data. -#### OptimumEmbedderQuantizationConfig.from\_dict +#### from_dict ```python -@classmethod -def from_dict(cls, data: dict[str, - Any]) -> "OptimumEmbedderQuantizationConfig" +from_dict(data: dict[str, Any]) -> OptimumEmbedderQuantizationConfig ``` Create a configuration from a dictionary. -**Arguments**: - -- `data`: Dictionary to deserialize from. +**Parameters:** -**Returns**: +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. -Quantization configuration. +**Returns:** +- OptimumEmbedderQuantizationConfig – Quantization configuration.