Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 22 additions & 22 deletions README.md
Comment thread
SanjanaBoggaramJ marked this conversation as resolved.
Comment thread
SanjanaBoggaramJ marked this conversation as resolved.
Original file line number Diff line number Diff line change
@@ -1,52 +1,52 @@
# Semantic Search with Couchbase Vector Store and LLM Integration
## Semantic Search with Couchbase Vector Store and LLM Integration

This repository demonstrates how to build a powerful semantic search engine using Couchbase as the backend database, combined with various AI-powered embedding and language model providers such as OpenAI, Azure OpenAI, Anthropic (Claude), Cohere, Hugging Face, Jina AI, Mistral AI, and Voyage AI.

Each example provides two distinct approaches:
- **FTS (Full Text Search)**: Uses Couchbase's vector search capabilities with pre-created search indices
- **GSI (Global Secondary Index)**: Leverages Couchbase's native SQL++ queries with vector similarity functions
- **Search Vector Index**: Uses Couchbase's vector search capabilities with pre-created search indices
- **Hyperscale Vector Index, Composite Vector Index**: Leverages Couchbase's native SQL++ queries with vector similarity functions

Semantic search goes beyond simple keyword matching by understanding the context and meaning behind the words in a query, making it essential for applications that require intelligent information retrieval.

## Features
### Features

- **Multiple Embedding Models**: Support for embeddings from OpenAI, Azure OpenAI, Anthropic (Claude), Cohere, Hugging Face, Jina AI, Mistral AI, and Voyage AI.
- **Couchbase Vector Store**: Utilizes Couchbase's vector storage capabilities for efficient similarity search.
- **Retrieval-Augmented Generation (RAG)**: Integrates with advanced language models like GPT-4 for generating contextually relevant responses.
- **Scalable and Flexible**: Easy to switch between different embedding models and adjust the index structure accordingly.
- **Caching Mechanism**: Implements `CouchbaseCache` for improved performance on repeated queries.

## Prerequisites
### Prerequisites

- Python 3.8+
- Couchbase Cluster (Self Managed or Capella) version 7.6+ with [Search Service](https://docs.couchbase.com/server/current/search/search.html)

- API keys for the respective AI providers (e.g., OpenAI, Azure OpenAI, Anthropic, Cohere, etc.)

## Setup
### Setup

### 1. Clone the repository:
#### 1. Clone the repository:
```bash
git clone https://github.com/your-username/vector-search-cookbook.git
cd vector-search-cookbook
```

### 2. Choose Your Approach:
#### 2. Choose Your Approach:

#### For FTS (Full Text Search) Examples:
Use the provided `{model}_index.json` index definition file in each model's `fts/` directory to create a new vector search index in your Couchbase cluster.
##### For Search Vector Index Examples:
Use the provided `{model}_index.json` index definition file in each model's `search_based/` directory to create a new vector search index in your Couchbase cluster.
Comment thread
SanjanaBoggaramJ marked this conversation as resolved.

#### For GSI (Global Secondary Index) Examples:
No additional setup required. GSI index will be created in each model's example.
##### For Hyperscale Vector Index, Composite Vector Index Examples:
No additional setup required. Hyperscale and Composite Vector Indexes will be created in each model's example.

### 3. Run the notebook file
#### 3. Run the notebook file

You can either run the notebook file on [Google Colab](https://colab.research.google.com/) or run it on your system by setting up the Python environment.


## Components
### Components

### 1. Multiple Embedding Models
#### 1. Multiple Embedding Models

The system supports embeddings from various AI providers:

Expand All @@ -59,25 +59,25 @@ The system supports embeddings from various AI providers:
* Mistral AI
* Voyage AI

### 2. Couchbase Vector Store
#### 2. Couchbase Vector Store

Couchbase is used to store document embeddings and metadata. The index structure allows for efficient retrieval across different embedding types.

### 3. Retrieval-Augmented Generation (RAG)
#### 3. Retrieval-Augmented Generation (RAG)

The RAG pipeline integrates with language models like GPT-4 to generate contextually relevant answers based on retrieved documents.

### 4. Semantic Search
#### 4. Semantic Search

Each notebook implements a semantic search function that performs similarity searche using the appropriate embedding type and retrieves the top-k most similar documents.

### 5. Caching
#### 5. Caching

The system implements caching functionality using `CouchbaseCache` to improve performance for repeated queries.

## Couchbase Vector Search Index (FTS Approach Only)
### Couchbase Search Vector Index

For FTS examples, you'll need to create a vector search index using the provided JSON configuration files. For more information on creating a vector search index, please follow the [instructions](https://docs.couchbase.com/cloud/vector-search/create-vector-search-index-ui.html). The following is an example for Azure OpenAI Model.
For Search Vector Index examples, you'll need to create a vector search index using the provided JSON configuration files. For more information on creating a vector search index, please follow the [instructions](https://docs.couchbase.com/cloud/vector-search/create-vector-search-index-ui.html). The following is an example for Azure OpenAI Model.

```json
{
Expand Down Expand Up @@ -153,4 +153,4 @@ For FTS examples, you'll need to create a vector search index using the provided
},
"sourceParams": {}
}
```
```