Skip to content

Commit c96ce98

Browse files
divyaruhilsjrl
divyaruhil
and
sjrl
authored
feat: Adding watsonx support in Haystack (#1949)
* Add WatsonX integration * pre-commit fix * Remove file from commit * adding py.typed files * adding CHANGELOG.md * adding workflow for watsonx * update pyproject.toml * addressing review comments * Update integrations/watsonx/pyproject.toml Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com> * Update integrations/watsonx/pyproject.toml Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com> * Fix pyproject.toml and get hatch run fmt-check to pass * Fix hatch run test:types * Fix workflow * More fixes to types, ruff linting, workflow and pyproject.toml * PR comments and typing fixes * Also pin pandas to get it to run on python 3.13 * Addressing review comments * fixing formatting errors * fixing error * fixing build error * fixing build error * Removing Space id not required * addressing review comments * Updates * More updates * Rename to make component names consistent * Slight changes --------- Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com> Co-authored-by: Sebastian Husch Lee <sjrl423@gmail.com>
1 parent 97b5bce commit c96ce98

25 files changed

Lines changed: 3653 additions & 3 deletions

‎.github/labeler.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -164,6 +164,11 @@ integration:unstructured-fileconverter:
164164
- any-glob-to-any-file: "integrations/unstructured/**/*"
165165
- any-glob-to-any-file: ".github/workflows/unstructured.yml"
166166

167+
integration:watsonx:
168+
- changed-files:
169+
- any-glob-to-any-file: "integrations/watsonx/**/*"
170+
- any-glob-to-any-file: ".github/workflows/watsonx.yml"
171+
167172
integration:weaviate:
168173
- changed-files:
169174
- any-glob-to-any-file: "integrations/weaviate/**/*"

‎.github/workflows/watsonx.yml‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
# This workflow comes from https://github.com/ofek/hatch-mypyc
2+
# https://github.com/ofek/hatch-mypyc/blob/5a198c0ba8660494d02716cfc9d79ce4adfb1442/.github/workflows/test.yml
3+
name: Test / watsonx
4+
5+
on:
6+
schedule:
7+
- cron: "0 0 * * *"
8+
pull_request:
9+
paths:
10+
- "integrations/watsonx/**"
11+
- "!integrations/watsonx/*.md"
12+
- ".github/workflows/watsonx.yml"
13+
14+
defaults:
15+
run:
16+
working-directory: integrations/watsonx
17+
18+
concurrency:
19+
group: watsonx-${{ github.head_ref }}
20+
cancel-in-progress: true
21+
22+
env:
23+
PYTHONUNBUFFERED: "1"
24+
FORCE_COLOR: "1"
25+
WATSONX_API_KEY: ${{ secrets.WATSONX_API_KEY }}
26+
WATSONX_PROJECT_ID: ${{ secrets.WATSONX_PROJECT_ID }}
27+
28+
jobs:
29+
run:
30+
name: Python ${{ matrix.python-version }} on ${{ startsWith(matrix.os, 'macos-') && 'macOS' || startsWith(matrix.os, 'windows-') && 'Windows' || 'Linux' }}
31+
runs-on: ${{ matrix.os }}
32+
strategy:
33+
fail-fast: false
34+
matrix:
35+
os: [ubuntu-latest, windows-latest, macos-latest]
36+
python-version: ["3.10", "3.13"]
37+
38+
steps:
39+
- name: Support longpaths
40+
if: matrix.os == 'windows-latest'
41+
working-directory: .
42+
run: git config --system core.longpaths true
43+
44+
- uses: actions/checkout@v4
45+
46+
- name: Set up Python ${{ matrix.python-version }}
47+
uses: actions/setup-python@v5
48+
with:
49+
python-version: ${{ matrix.python-version }}
50+
51+
- name: Install Hatch
52+
run: pip install --upgrade hatch
53+
54+
- name: Lint
55+
if: matrix.python-version == '3.10' && runner.os == 'Linux'
56+
run: hatch run fmt-check && hatch run test:types
57+
58+
- name: Generate docs
59+
if: matrix.python-version == '3.10' && runner.os == 'Linux'
60+
run: hatch run docs
61+
62+
- name: Run tests
63+
run: hatch run test:cov-retry
64+
65+
- name: Run unit tests with lowest direct dependencies
66+
run: |
67+
hatch run uv pip compile pyproject.toml --resolution lowest-direct --output-file requirements_lowest_direct.txt
68+
hatch run uv pip install -r requirements_lowest_direct.txt
69+
hatch run test:unit
70+
71+
- name: Nightly - run unit tests with Haystack main branch
72+
if: github.event_name == 'schedule'
73+
run: |
74+
hatch env prune
75+
hatch run uv pip install git+https://github.com/deepset-ai/haystack.git@main
76+
hatch run test:unit
77+
78+
- name: Send event to Datadog for nightly failures
79+
if: failure() && github.event_name == 'schedule'
80+
uses: ./.github/actions/send_failure
81+
with:
82+
title: |
83+
Core integrations nightly tests failure: ${{ github.workflow }}
84+
api-key: ${{ secrets.CORE_DATADOG_API_KEY }}

‎README.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -42,13 +42,13 @@ Please check out our [Contribution Guidelines](CONTRIBUTING.md) for all the deta
4242
| [jina-haystack](integrations/jina/) | Connector, Embedder, Ranker | [![PyPI - Version](https://img.shields.io/pypi/v/jina-haystack.svg)](https://pypi.org/project/jina-haystack) | [![Test / jina](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/jina.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/jina.yml) |
4343
| [langfuse-haystack](integrations/langfuse/) | Tracer | [![PyPI - Version](https://img.shields.io/pypi/v/langfuse-haystack.svg?color=orange)](https://pypi.org/project/langfuse-haystack) | [![Test / langfuse](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/langfuse.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/langfuse.yml) |
4444
| [llama-cpp-haystack](integrations/llama_cpp/) | Generator | [![PyPI - Version](https://img.shields.io/pypi/v/llama-cpp-haystack.svg?color=orange)](https://pypi.org/project/llama-cpp-haystack) | [![Test / llama-cpp](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/llama_cpp.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/llama_cpp.yml) |
45-
| [mcp-haystack](integrations/mcp/) | Tool | [![PyPI - Version](https://img.shields.io/pypi/v/mcp-haystack.svg?color=orange)](https://pypi.org/project/mcp-haystack) | [![Test / mcp](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/mcp.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/mcp.yml) |
46-
| [meta-llama-haystack](integrations/meta_llama/) | Generator | [![PyPI - Version](https://img.shields.io/pypi/v/meta-llama-haystack.svg?color=orange)](https://pypi.org/project/meta-llama-haystack) | [![Test / meta_llama](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/meta_llama.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/meta_llama.yml) |
45+
| [mcp-haystack](integrations/mcp/) | Tool | [![PyPI - Version](https://img.shields.io/pypi/v/mcp-haystack.svg?color=orange)](https://pypi.org/project/mcp-haystack) | [![Test / mcp](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/mcp.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/mcp.yml) |
46+
| [meta-llama-haystack](integrations/meta_llama/) | Generator | [![PyPI - Version](https://img.shields.io/pypi/v/meta-llama-haystack.svg?color=orange)](https://pypi.org/project/meta-llama-haystack) | [![Test / meta_llama](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/meta_llama.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/meta_llama.yml) |
4747
| [mistral-haystack](integrations/mistral/) | Embedder, Generator | [![PyPI - Version](https://img.shields.io/pypi/v/mistral-haystack.svg)](https://pypi.org/project/mistral-haystack) | [![Test / mistral](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/mistral.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/mistral.yml) |
4848
| [mongodb-atlas-haystack](integrations/mongodb_atlas/) | Document Store | [![PyPI - Version](https://img.shields.io/pypi/v/mongodb-atlas-haystack.svg?color=orange)](https://pypi.org/project/mongodb-atlas-haystack) | [![Test / mongodb-atlas](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/mongodb_atlas.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/mongodb_atlas.yml) |
4949
| [nvidia-haystack](integrations/nvidia/) | Embedder, Generator, Ranker | [![PyPI - Version](https://img.shields.io/pypi/v/nvidia-haystack.svg?color=orange)](https://pypi.org/project/nvidia-haystack) | [![Test / nvidia](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/nvidia.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/nvidia.yml) |
5050
| [ollama-haystack](integrations/ollama/) | Embedder, Generator | [![PyPI - Version](https://img.shields.io/pypi/v/ollama-haystack.svg?color=orange)](https://pypi.org/project/ollama-haystack) | [![Test / ollama](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/ollama.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/ollama.yml) |
51-
| [openrouter-haystack](integrations/openrouter/) | Generator | [![PyPI - Version](https://img.shields.io/pypi/v/openrouter-haystack.svg)](https://pypi.org/project/openrouter-haystack) | [![Test / openrouter](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/openrouter.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/openrouter.yml) |
51+
| [openrouter-haystack](integrations/openrouter/) | Generator | [![PyPI - Version](https://img.shields.io/pypi/v/openrouter-haystack.svg)](https://pypi.org/project/openrouter-haystack) | [![Test / openrouter](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/openrouter.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/openrouter.yml) |
5252
| [opensearch-haystack](integrations/opensearch/) | Document Store | [![PyPI - Version](https://img.shields.io/pypi/v/opensearch-haystack.svg)](https://pypi.org/project/opensearch-haystack) | [![Test / opensearch](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/opensearch.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/opensearch.yml) |
5353
| [optimum-haystack](integrations/optimum/) | Embedder | [![PyPI - Version](https://img.shields.io/pypi/v/optimum-haystack.svg)](https://pypi.org/project/optimum-haystack) | [![Test / optimum](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/optimum.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/optimum.yml) |
5454
| [pinecone-haystack](integrations/pinecone/) | Document Store | [![PyPI - Version](https://img.shields.io/pypi/v/pinecone-haystack.svg?color=orange)](https://pypi.org/project/pinecone-haystack) | [![Test / pinecone](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/pinecone.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/pinecone.yml) |
@@ -58,6 +58,7 @@ Please check out our [Contribution Guidelines](CONTRIBUTING.md) for all the deta
5858
| [snowflake-haystack](integrations/snowflake/) | Retriever | [![PyPI - Version](https://img.shields.io/pypi/v/snowflake-haystack.svg)](https://pypi.org/project/snowflake-haystack) | [![Test / snowflake](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/snowflake.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/snowflake.yml) |
5959
| [stackit-haystack](integrations/stackit/) | Embedder, Generator | [![PyPI - Version](https://img.shields.io/pypi/v/stackit-haystack.svg)](https://pypi.org/project/stackit-haystack) | [![Test / stackit](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/stackit.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/stackit.yml) |
6060
| [unstructured-fileconverter-haystack](integrations/unstructured/) | File converter | [![PyPI - Version](https://img.shields.io/pypi/v/unstructured-fileconverter-haystack.svg)](https://pypi.org/project/unstructured-fileconverter-haystack) | [![Test / unstructured](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/unstructured.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/unstructured.yml) |
61+
| [watsonx-haystack](integrations/watsonx/) | Embedder, Generator | [![PyPI - Version](https://img.shields.io/pypi/v/watsonx-haystack.svg?color=orange)](https://pypi.org/project/watsonx-haystack) | [![Test / watsonx](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/watsonx.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/watsonx.yml) | |
6162
| [weaviate-haystack](integrations/weaviate/) | Document Store | [![PyPI - Version](https://img.shields.io/pypi/v/weaviate-haystack.svg)](https://pypi.org/project/weaviate-haystack) | [![Test / weaviate](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/weaviate.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/weaviate.yml) |
6263
| [weave-haystack](integrations/weights_and_biases_weave/) | Tracer | [![PyPI - Version](https://img.shields.io/pypi/v/weave-haystack.svg)](https://pypi.org/project/weave-haystack) | [![Test / weights_and_biases_weave](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/weights_and_biases_weave.yml/badge.svg)](https://github.com/deepset-ai/haystack-core-integrations/actions/workflows/weights_and_biases_weave.yml) |
6364
| [instructor-embedders-haystack](integrations/instructor_embedders/) | Embedder | [![PyPI - Version](https://img.shields.io/pypi/v/instructor-embedders-haystack.svg)](https://pypi.org/project/instructor-embedders-haystack) | [Staged](https://docs.haystack.deepset.ai/docs/breaking-change-policy#discontinuing-an-integration) |

‎integrations/watsonx/CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
# Changelog

0 commit comments

Comments
 (0)