diff --git a/README.md b/README.md index f152a8de..bdd58325 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,14 @@ -# Semantic Search with Couchbase Vector Store and LLM Integration +## Semantic Search with Couchbase Vector Store and LLM Integration This repository demonstrates how to build a powerful semantic search engine using Couchbase as the backend database, combined with various AI-powered embedding and language model providers such as OpenAI, Azure OpenAI, Anthropic (Claude), Cohere, Hugging Face, Jina AI, Mistral AI, and Voyage AI. Each example provides two distinct approaches: -- **FTS (Full Text Search)**: Uses Couchbase's vector search capabilities with pre-created search indices -- **GSI (Global Secondary Index)**: Leverages Couchbase's native SQL++ queries with vector similarity functions +- **Search Vector Index**: Uses Couchbase's vector search capabilities with pre-created search indices +- **Hyperscale Vector Index, Composite Vector Index**: Leverages Couchbase's native SQL++ queries with vector similarity functions Semantic search goes beyond simple keyword matching by understanding the context and meaning behind the words in a query, making it essential for applications that require intelligent information retrieval. -## Features +### Features - **Multiple Embedding Models**: Support for embeddings from OpenAI, Azure OpenAI, Anthropic (Claude), Cohere, Hugging Face, Jina AI, Mistral AI, and Voyage AI. - **Couchbase Vector Store**: Utilizes Couchbase's vector storage capabilities for efficient similarity search. @@ -16,37 +16,37 @@ Semantic search goes beyond simple keyword matching by understanding the context - **Scalable and Flexible**: Easy to switch between different embedding models and adjust the index structure accordingly. - **Caching Mechanism**: Implements `CouchbaseCache` for improved performance on repeated queries. -## Prerequisites +### Prerequisites - Python 3.8+ - Couchbase Cluster (Self Managed or Capella) version 7.6+ with [Search Service](https://docs.couchbase.com/server/current/search/search.html) - API keys for the respective AI providers (e.g., OpenAI, Azure OpenAI, Anthropic, Cohere, etc.) -## Setup +### Setup -### 1. Clone the repository: +#### 1. Clone the repository: ```bash git clone https://github.com/your-username/vector-search-cookbook.git cd vector-search-cookbook ``` -### 2. Choose Your Approach: +#### 2. Choose Your Approach: -#### For FTS (Full Text Search) Examples: -Use the provided `{model}_index.json` index definition file in each model's `fts/` directory to create a new vector search index in your Couchbase cluster. +##### For Search Vector Index Examples: +Use the provided `{model}_index.json` index definition file in each model's `search_based/` directory to create a new vector search index in your Couchbase cluster. -#### For GSI (Global Secondary Index) Examples: -No additional setup required. GSI index will be created in each model's example. +##### For Hyperscale Vector Index, Composite Vector Index Examples: +No additional setup required. Hyperscale and Composite Vector Indexes will be created in each model's example. -### 3. Run the notebook file +#### 3. Run the notebook file You can either run the notebook file on [Google Colab](https://colab.research.google.com/) or run it on your system by setting up the Python environment. -## Components +### Components -### 1. Multiple Embedding Models +#### 1. Multiple Embedding Models The system supports embeddings from various AI providers: @@ -59,25 +59,25 @@ The system supports embeddings from various AI providers: * Mistral AI * Voyage AI -### 2. Couchbase Vector Store +#### 2. Couchbase Vector Store Couchbase is used to store document embeddings and metadata. The index structure allows for efficient retrieval across different embedding types. -### 3. Retrieval-Augmented Generation (RAG) +#### 3. Retrieval-Augmented Generation (RAG) The RAG pipeline integrates with language models like GPT-4 to generate contextually relevant answers based on retrieved documents. -### 4. Semantic Search +#### 4. Semantic Search Each notebook implements a semantic search function that performs similarity searche using the appropriate embedding type and retrieves the top-k most similar documents. -### 5. Caching +#### 5. Caching The system implements caching functionality using `CouchbaseCache` to improve performance for repeated queries. -## Couchbase Vector Search Index (FTS Approach Only) +### Couchbase Search Vector Index -For FTS examples, you'll need to create a vector search index using the provided JSON configuration files. For more information on creating a vector search index, please follow the [instructions](https://docs.couchbase.com/cloud/vector-search/create-vector-search-index-ui.html). The following is an example for Azure OpenAI Model. +For Search Vector Index examples, you'll need to create a vector search index using the provided JSON configuration files. For more information on creating a vector search index, please follow the [instructions](https://docs.couchbase.com/cloud/vector-search/create-vector-search-index-ui.html). The following is an example for Azure OpenAI Model. ```json { @@ -153,4 +153,4 @@ For FTS examples, you'll need to create a vector search index using the provided }, "sourceParams": {} } -``` \ No newline at end of file +```