Updated examples to support haystack breaking changes

Author: shyam-cb

README.md

@@ -19,11 +19,9 @@
 
 ----
 
-**Table of Contents**
+# Table of Contents
 
-- [Breaking Changes in Version 2.0.0](#breaking-changes-in-version-200)
 - [Overview](#overview)
-- [Choosing the Right Document Store](#choosing-the-right-document-store)
 - [Installation](#installation)
 - [Usage](#usage)
   - [Running Couchbase](#running-couchbase)
@@ -32,7 +30,7 @@
   - [More Examples](#more-examples)
 - [License](#license)
 
-## Breaking Changes in Version 2.0.0
+### Breaking Changes in Version 2.0.0
 
 > **Important Note:**  
 > In version 2.0.0, the following component names have been changed:
@@ -44,7 +42,7 @@
 
 ## Overview
 
-An integration of [Couchbase](https://www.couchbase.com) NoSQL database with [Haystack v2.0](https://docs.haystack.deepset.ai/v2.0/docs/intro)
+An integration of [Couchbase](https://www.couchbase.com) NoSQL database with [Haystack](https://docs.haystack.deepset.ai/docs/intro)
 by [deepset](https://www.deepset.ai). Couchbase supports three types of [vector indexes](https://docs.couchbase.com/server/current/vector-search/vector-search.html) for AI applications, and this library provides document stores for two of them:
 
 ### Document Stores
@@ -62,7 +60,7 @@ from couchbase_haystack import CouchbaseSearchDocumentStore, CouchbaseQueryDocum
 
 ### Retrievers
 
-In addition to the document stores, the library includes the following [retriever components](https://docs.haystack.deepset.ai/v2.0/docs/retrievers):
+In addition to the document stores, the library includes the following [retriever components](https://docs.haystack.deepset.ai/docs/retrievers):
 
 - **`CouchbaseSearchEmbeddingRetriever`** - Works with `CouchbaseSearchDocumentStore` to perform hybrid searches combining vector similarity with full-text and geospatial queries.
 
@@ -72,7 +70,7 @@ The `couchbase-haystack` library uses the [Couchbase Python SDK](https://docs.co
 
 Both document stores store Documents as JSON documents in Couchbase. Embeddings are stored as part of the document, with indexing and querying managed by different Couchbase services depending on the document store type.
 
-## Choosing the Right Document Store
+### Choosing the Right Document Store
 
 Couchbase supports three types of vector indexes. This library currently supports two of them:
 
@@ -274,7 +272,7 @@ The full list of parameters accepted by `CouchbaseSearchDocumentStore` can be fo
 
 #### Indexing Documents with CouchbaseSearchDocumentStore
 
-With Haystack you can use [DocumentWriter](https://docs.haystack.deepset.ai/v2.0/docs/documentwriter) component to write Documents into a Document Store. In the example below we construct pipeline to write documents to Couchbase using `CouchbaseSearchDocumentStore`:
+With Haystack you can use [DocumentWriter](https://docs.haystack.deepset.ai/docs/documentwriter) component to write Documents into a Document Store. In the example below we construct pipeline to write documents to Couchbase using `CouchbaseSearchDocumentStore`:
 
 ```python
 from haystack import Document
@@ -609,7 +607,7 @@ You can find more examples in the [examples](examples) directory:
 #### Search-based (FTS) Examples
 
 - [examples/search/indexing_pipeline.py](examples/search/indexing_pipeline.py) - Indexing documents using `CouchbaseSearchDocumentStore`
-- [examples/search/rag_pipeline.py](examples/search/rag_pipeline.py) - RAG pipeline using `CouchbaseSearchEmbeddingRetriever` with [HuggingFaceAPIGenerator](https://docs.haystack.deepset.ai/v2.0/docs/huggingfacetgigenerator)
+- [examples/search/rag_pipeline.py](examples/search/rag_pipeline.py) - RAG pipeline using `CouchbaseSearchEmbeddingRetriever` with [HuggingFaceAPIGenerator](https://docs.haystack.deepset.ai/v2.20/docs/huggingfacetgigenerator)
 
 #### GSI-based Examples
 

examples/gsi/indexing_pipeline.py

@@ -20,14 +20,16 @@
     CouchbaseQueryDocumentStore,
     CouchbaseQueryOptions,
     QueryVectorSearchType,
+    CouchbaseClusterOptions,
 )
+from couchbase.options import KnownConfigProfiles, QueryOptions
 
 logger = logging.getLogger(__name__)
 
 
 def fetch_archive_from_http(url: str, output_dir: str):
     if Path(output_dir).is_dir():
-        logger.warn(f"'{output_dir}' directory already exists. Skipping data download")
+        logger.warning(f"'{output_dir}' directory already exists. Skipping data download")
         return
 
     with requests.get(url, timeout=10, stream=True) as response:
@@ -50,14 +52,15 @@ def fetch_archive_from_http(url: str, output_dir: str):
 #     --env COUCHBASE_ADMINISTRATOR_PASSWORD=passw0rd \
 #     couchbase:enterprise-8.0.0
 
-bucket_name = "haystack_bucket_name"
-scope_name = "haystack_scope_name"
-collection_name = "haystack_collection_name"
-index_name = "haystack_index_name"
+bucket_name = os.getenv("BUCKET_NAME")
+scope_name = os.getenv("SCOPE_NAME")
+collection_name = os.getenv("COLLECTION_NAME")
+index_name = os.getenv("INDEX_NAME")
 
 document_store = CouchbaseQueryDocumentStore(
     cluster_connection_string=Secret.from_env_var("CONNECTION_STRING"),
-    authenticator=CouchbasePasswordAuthenticator(username=Secret.from_token("username"), password=Secret.from_token("password")),
+    authenticator=CouchbasePasswordAuthenticator(username=Secret.from_env_var("USER_NAME"), password=Secret.from_env_var("PASSWORD")),
+    cluster_options=CouchbaseClusterOptions(profile=KnownConfigProfiles.WanDevelopment),
     bucket=bucket_name,
     scope=scope_name,
     collection=collection_name,
@@ -73,7 +76,7 @@ def fetch_archive_from_http(url: str, output_dir: str):
 p = Pipeline()
 p.add_component("text_file_converter", TextFileToDocument())
 p.add_component("cleaner", DocumentCleaner())
-p.add_component("splitter", DocumentSplitter(split_by="sentence", split_length=250, split_overlap=30))
+p.add_component("splitter", DocumentSplitter(split_by="word", split_length=250, split_overlap=30))
 p.add_component("embedder", SentenceTransformersDocumentEmbedder(model="sentence-transformers/all-MiniLM-L6-v2"))
 p.add_component("writer", DocumentWriter(document_store=document_store))
 
@@ -86,6 +89,8 @@ def fetch_archive_from_http(url: str, output_dir: str):
 file_paths = [docs_dir / Path(name) for name in os.listdir(docs_dir)]
 result = p.run({"text_file_converter": {"sources": file_paths}})
 
+logger.info(f"Data written to Couchbase: {result}")
+
 # Assuming you have a Docker container running, navigate to <http://localhost:8091>
 # to open the Couchbase Web Console and explore your data.
 
@@ -94,8 +99,9 @@ def fetch_archive_from_http(url: str, output_dir: str):
 # this is a current limitation of the couchbase gsi vector index
 cfg = {
     "dimension": 384,
-    "train_list": 500,
     "description": "IVF,PQ32x8",
     "similarity": "L2",
 }
-document_store.scope.query(f"Create Index {index_name} ON {collection_name} (embedding vector) USING GSI WITH {json.dumps(cfg)}")
+document_store.scope.query(f"Create Index {index_name} ON {collection_name} (embedding vector) USING GSI WITH {json.dumps(cfg)}", QueryOptions(timeout=timedelta(seconds=300))).execute()
+
+logger.info(f"Index created: {index_name}")

examples/gsi/rag_pipeline.py

@@ -1,18 +1,20 @@
-from couchbase.options import KnownConfigProfiles
+import os
 from haystack import GeneratedAnswer, Pipeline
 from haystack.components.builders.answer_builder import AnswerBuilder
-from haystack.components.builders.prompt_builder import PromptBuilder
+from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
+from haystack.dataclasses import ChatMessage
 from haystack.components.embedders import SentenceTransformersTextEmbedder
-from haystack.components.generators import HuggingFaceAPIGenerator
+from haystack.components.generators.chat import HuggingFaceAPIChatGenerator
+from haystack.utils.hf import HFGenerationAPIType
 from haystack.utils import Secret
-
 from couchbase_haystack import (
     CouchbaseClusterOptions,
     CouchbasePasswordAuthenticator,
     CouchbaseQueryDocumentStore,
     CouchbaseQueryEmbeddingRetriever,
     QueryVectorSearchType,
 )
+from couchbase.options import KnownConfigProfiles
 
 # Load HF Token from environment variables.
 HF_TOKEN = Secret.from_env_var("HF_API_TOKEN")
@@ -26,51 +28,49 @@
 #     couchbase:enterprise-8.0.0
 
 document_store = CouchbaseQueryDocumentStore(
-    cluster_connection_string=Secret.from_token("localhost"),
-    authenticator=CouchbasePasswordAuthenticator(username=Secret.from_token("username"), password=Secret.from_token("password")),
+    cluster_connection_string=Secret.from_env_var("CONNECTION_STRING"),
+    authenticator=CouchbasePasswordAuthenticator(username=Secret.from_env_var("USER_NAME"), password=Secret.from_env_var("PASSWORD")),
     cluster_options=CouchbaseClusterOptions(
         profile=KnownConfigProfiles.WanDevelopment,
     ),
-    bucket="haystack_bucket_name",
-    scope="haystack_scope_name",
-    collection="haystack_collection_name",
+    bucket=os.getenv("BUCKET_NAME"),
+    scope=os.getenv("SCOPE_NAME"),
+    collection=os.getenv("COLLECTION_NAME"),
     search_type=QueryVectorSearchType.ANN,
     similarity="L2",
     nprobes=10,
 )
 
-# Build a RAG pipeline with a Retriever to get relevant documents to the query and a HuggingFaceTGIGenerator
+# Build a RAG pipeline with a Retriever to get relevant documents to the query and a HuggingFaceAPIChatGenerator
 # interacting with LLMs using a custom prompt.
-prompt_template = """
-Given these documents, answer the question.\nDocuments:
+prompt_messages = [
+    ChatMessage.from_system("You are a helpful assistant that answers questions based on the provided documents."),
+    ChatMessage.from_user("""Given these documents, answer the question.
+Documents:
 {% for doc in documents %}
     {{ doc.content }}
 {% endfor %}
 
-\nQuestion: {{question}}
-\nAnswer:
-"""
+Question: {{question}}
+Answer:""")
+]
 rag_pipeline = Pipeline()
 rag_pipeline.add_component(
     "query_embedder",
     SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2", progress_bar=False),
 )
 rag_pipeline.add_component("retriever", CouchbaseQueryEmbeddingRetriever(document_store=document_store))
-rag_pipeline.add_component("prompt_builder", PromptBuilder(template=prompt_template))
-rag_pipeline.add_component(
-    "llm",
-    HuggingFaceAPIGenerator(
-        api_type="serverless_inference_api",
-        api_params={"model": "mistralai/Mistral-7B-v0.1"},
-    ),
-)
+rag_pipeline.add_component("prompt_builder", ChatPromptBuilder(template=prompt_messages, required_variables=["question"]))
+rag_pipeline.add_component("llm", HuggingFaceAPIChatGenerator(
+    api_type=HFGenerationAPIType.SERVERLESS_INFERENCE_API, 
+    api_params={"model": "mistralai/Mistral-7B-Instruct-v0.2"},
+))
 rag_pipeline.add_component("answer_builder", AnswerBuilder())
 
 rag_pipeline.connect("query_embedder", "retriever.query_embedding")
-rag_pipeline.connect("retriever.documents", "prompt_builder.documents")
-rag_pipeline.connect("prompt_builder.prompt", "llm.prompt")
+rag_pipeline.connect("retriever", "prompt_builder.documents")
+rag_pipeline.connect("prompt_builder", "llm")
 rag_pipeline.connect("llm.replies", "answer_builder.replies")
-rag_pipeline.connect("llm.meta", "answer_builder.meta")
 rag_pipeline.connect("retriever", "answer_builder.documents")
 
 # Ask a question on the data you just added.

examples/search/rag_pipeline.py

@@ -1,8 +1,11 @@
+import os
 from haystack import GeneratedAnswer, Pipeline
 from haystack.components.builders.answer_builder import AnswerBuilder
-from haystack.components.builders.prompt_builder import PromptBuilder
+from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
+from haystack.dataclasses import ChatMessage
 from haystack.components.embedders import SentenceTransformersTextEmbedder
-from haystack.components.generators import HuggingFaceAPIGenerator
+from haystack.components.generators.chat import HuggingFaceAPIChatGenerator
+from haystack.utils.hf import HFGenerationAPIType
 from haystack.utils import Secret
 
 from couchbase_haystack import CouchbasePasswordAuthenticator, CouchbaseSearchDocumentStore, CouchbaseSearchEmbeddingRetriever
@@ -19,46 +22,44 @@
 #     couchbase:enterprise-7.6.2
 
 document_store = CouchbaseSearchDocumentStore(
-    cluster_connection_string=Secret.from_token("localhost"),
-    authenticator=CouchbasePasswordAuthenticator(username=Secret.from_token("username"), password=Secret.from_token("password")),
-    bucket="haystack_bucket_name",
-    scope="haystack_scope_name",
-    collection="haystack_collection_name",
-    vector_search_index="vector_search_index",
+    cluster_connection_string=Secret.from_env_var("CONNECTION_STRING"),
+    authenticator=CouchbasePasswordAuthenticator(username=Secret.from_env_var("USER_NAME"), password=Secret.from_env_var("PASSWORD")),
+    bucket=os.getenv("BUCKET_NAME"),
+    scope=os.getenv("SCOPE_NAME"),
+    collection=os.getenv("COLLECTION_NAME"),
+    vector_search_index="vector_search",
 )
 
 # Build a RAG pipeline with a Retriever to get relevant documents to the query and a HuggingFaceTGIGenerator
 # interacting with LLMs using a custom prompt.
-prompt_template = """
-Given these documents, answer the question.\nDocuments:
+prompt_messages = [
+    ChatMessage.from_system("You are a helpful assistant that answers questions based on the provided documents."),
+    ChatMessage.from_user("""Given these documents, answer the question.
+Documents:
 {% for doc in documents %}
     {{ doc.content }}
 {% endfor %}
 
-\nQuestion: {{question}}
-\nAnswer:
-"""
+Question: {{question}}
+Answer:""")
+]
 rag_pipeline = Pipeline()
 rag_pipeline.add_component(
     "query_embedder",
     SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2", progress_bar=False),
 )
 rag_pipeline.add_component("retriever", CouchbaseSearchEmbeddingRetriever(document_store=document_store))
-rag_pipeline.add_component("prompt_builder", PromptBuilder(template=prompt_template))
-rag_pipeline.add_component(
-    "llm",
-    HuggingFaceAPIGenerator(
-        api_type="serverless_inference_api",
-        api_params={"model": "mistralai/Mistral-7B-v0.1"},
-    ),
-)
+rag_pipeline.add_component("prompt_builder", ChatPromptBuilder(template=prompt_messages, required_variables=["question"]))
+rag_pipeline.add_component("llm", HuggingFaceAPIChatGenerator(
+    api_type=HFGenerationAPIType.SERVERLESS_INFERENCE_API, 
+    api_params={"model": "mistralai/Mistral-7B-Instruct-v0.2"},
+))
 rag_pipeline.add_component("answer_builder", AnswerBuilder())
 
 rag_pipeline.connect("query_embedder", "retriever.query_embedding")
-rag_pipeline.connect("retriever.documents", "prompt_builder.documents")
-rag_pipeline.connect("prompt_builder.prompt", "llm.prompt")
+rag_pipeline.connect("retriever", "prompt_builder.documents")
+rag_pipeline.connect("prompt_builder", "llm")
 rag_pipeline.connect("llm.replies", "answer_builder.replies")
-rag_pipeline.connect("llm.meta", "answer_builder.meta")
 rag_pipeline.connect("retriever", "answer_builder.documents")
 
 # Ask a question on the data you just added.