fix: update databricks-vector-search skill with search modes and operations docs (#318)

Add two new reference files and update existing docs for the databricks-vector-search skill:
- search-modes.md: ANN vs HYBRID vs FULL_TEXT decision guide, filter combinations, self-managed embedding patterns
- troubleshooting-and-operations.md: endpoint/index monitoring, cost optimization, capacity planning, migration workflows
- SKILL.md: expand hybrid search section, add reference file links, update latency numbers and filter syntax
- end-to-end-rag.md: align filter parameter names with databricks-vectorsearch package

Co-authored-by: Isaac

Co-authored-by: praneeth_paikray-data <praneeth.paikray@databricks.com>

Author: Praneeth16

databricks-skills/databricks-vector-search/SKILL.md

@@ -31,8 +31,8 @@ Databricks Vector Search provides managed vector similarity search with automati
 
 | Type | Latency | Capacity | Cost | Best For |
 |------|---------|----------|------|----------|
-| **Standard** | ~50-100ms | 320M vectors (768 dim) | Higher | Real-time, low-latency |
-| **Storage-Optimized** | ~250ms | 1B+ vectors (768 dim) | 7x lower | Large-scale, cost-sensitive |
+| **Standard** | 20-50ms | 320M vectors (768 dim) | Higher | Real-time, low-latency |
+| **Storage-Optimized** | 300-500ms | 1B+ vectors (768 dim) | 7x lower | Large-scale, cost-sensitive |
 
 ## Index Types
 
@@ -184,13 +184,15 @@ results = w.vector_search_indexes.query_index(
 
 ### Hybrid Search (Semantic + Keyword)
 
+Hybrid search combines vector similarity (ANN) with BM25 keyword scoring. Use it when queries contain exact terms that must match — SKUs, error codes, proper nouns, or technical terminology — where pure semantic search might miss keyword-specific results. See [search-modes.md](search-modes.md) for detailed guidance on choosing between ANN and hybrid search.
+
 ```python
 # Combines vector similarity with keyword matching
 results = w.vector_search_indexes.query_index(
     index_name="catalog.schema.my_index",
     columns=["id", "content"],
-    query_text="machine learning algorithms",
-    query_type="hybrid",  # Enable hybrid search
+    query_text="SPARK-12345 executor memory error",
+    query_type="HYBRID",
     num_results=10
 )
 ```
@@ -212,20 +214,26 @@ results = w.vector_search_indexes.query_index(
 
 ### Storage-Optimized Filters (SQL-like)
 
+Storage-Optimized endpoints use SQL-like filter syntax via the `databricks-vectorsearch` package's `filters` parameter (accepts a string):
+
 ```python
-# filter_string uses SQL-like syntax
-results = w.vector_search_indexes.query_index(
-    index_name="catalog.schema.my_index",
-    columns=["id", "content"],
+from databricks.vector_search.client import VectorSearchClient
+
+vsc = VectorSearchClient()
+index = vsc.get_index(endpoint_name="my-storage-endpoint", index_name="catalog.schema.my_index")
+
+# SQL-like filter syntax for storage-optimized endpoints
+results = index.similarity_search(
     query_text="machine learning",
+    columns=["id", "content"],
     num_results=10,
-    filter_string="category = 'ai' AND status IN ('active', 'pending')"
+    filters="category = 'ai' AND status IN ('active', 'pending')"
 )
 
 # More filter examples
-filter_string="price > 100 AND price < 500"
-filter_string="department LIKE 'eng%'"
-filter_string="created_at >= '2024-01-01'"
+# filters="price > 100 AND price < 500"
+# filters="department LIKE 'eng%'"
+# filters="created_at >= '2024-01-01'"
 ```
 
 ### Trigger Index Sync
@@ -253,6 +261,8 @@ scan_result = w.vector_search_indexes.scan_index(
 |-------|------|-------------|
 | Index Types | [index-types.md](index-types.md) | Detailed comparison of Delta Sync (managed/self-managed) vs Direct Access |
 | End-to-End RAG | [end-to-end-rag.md](end-to-end-rag.md) | Complete walkthrough: source table → endpoint → index → query → agent integration |
+| Search Modes | [search-modes.md](search-modes.md) | When to use semantic (ANN) vs hybrid search, decision guide |
+| Operations | [troubleshooting-and-operations.md](troubleshooting-and-operations.md) | Monitoring, cost optimization, capacity planning, migration |
 
 ## CLI Quick Reference
 
@@ -288,7 +298,7 @@ databricks vector-search indexes delete-index \
 |-------|----------|
 | **Index sync slow** | Use Storage-Optimized endpoints (20x faster indexing) |
 | **Query latency high** | Use Standard endpoint for <100ms latency |
-| **filters_json not working** | Storage-Optimized uses `filter_string` (SQL syntax) |
+| **filters_json not working** | Storage-Optimized uses SQL-like string filters via `databricks-vectorsearch` package's `filters` parameter |
 | **Embedding dimension mismatch** | Ensure query and index dimensions match |
 | **Index not updating** | Check pipeline_type; use sync_index() for TRIGGERED |
 | **Out of capacity** | Upgrade to Storage-Optimized (1B+ vectors) |
@@ -298,10 +308,10 @@ databricks vector-search indexes delete-index \
 
 Databricks provides built-in embedding models:
 
-| Model | Dimensions | Use Case |
-|-------|------------|----------|
-| `databricks-gte-large-en` | 1024 | English text, high quality |
-| `databricks-bge-large-en` | 1024 | English text, general |
+| Model | Dimensions | Context Window | Use Case |
+|-------|------------|----------------|----------|
+| `databricks-gte-large-en` | 1024 | 8192 tokens | English text, high quality |
+| `databricks-bge-large-en` | 1024 | 512 tokens | English text, general purpose |
 
 ```python
 # Use with managed embeddings
@@ -396,7 +406,7 @@ manage_vs_data(index_name="catalog.schema.my_index", operation="sync")
 - **Delta Sync recommended** — easier than Direct Access for most scenarios
 - **Hybrid search** — available for both Delta Sync and Direct Access indexes
 - **`columns_to_sync` matters** — only synced columns are available in query results; include all columns you need
-- **Filter syntax differs by endpoint** — Standard uses `filters_json` (dict), Storage-Optimized uses `filter_string` (SQL)
+- **Filter syntax differs by endpoint** — Standard uses dict-format filters, Storage-Optimized uses SQL-like string filters. Use the `databricks-vectorsearch` package's `filters` parameter which accepts both formats
 - **Management vs runtime** — MCP tools above handle lifecycle management; for agent tool-calling at runtime, use `VectorSearchRetrieverTool` or the Databricks managed Vector Search MCP server
 
 ## Related Skills

databricks-skills/databricks-vector-search/end-to-end-rag.md

@@ -119,13 +119,13 @@ query_vs_index(
 The filter syntax depends on the endpoint type used when creating the index.
 
 ```python
-# Storage-Optimized endpoint (used in this walkthrough): SQL-like filter_string
+# Storage-Optimized endpoint (used in this walkthrough): SQL-like filter syntax
 query_vs_index(
     index_name="catalog.schema.knowledge_base_index",
     columns=["doc_id", "title", "content"],
     query_text="How do I govern my data?",
     num_results=3,
-    filter_string="category = 'governance'"
+    filters="category = 'governance'"
 )
 
 # Standard endpoint (if you created a Standard endpoint instead): JSON filters_json
@@ -236,4 +236,4 @@ Then sync — the index automatically handles deletions via Delta change data fe
 | **"Column not found in index"** | Column must be in `columns_to_sync`. Recreate index with the column included |
 | **Embeddings not computed** | Ensure `embedding_model_endpoint_name` is a valid serving endpoint |
 | **Stale results after table update** | For TRIGGERED pipelines, you must call `sync_vs_index` manually |
-| **Filter not working** | Standard endpoints use `filters_json` (dict), Storage-Optimized use `filter_string` (SQL) |
+| **Filter not working** | Standard endpoints use dict-format filters (`filters_json`), Storage-Optimized use SQL-like string filters (`filters`) |

databricks-skills/databricks-vector-search/search-modes.md

@@ -0,0 +1,142 @@
+# Vector Search Modes
+
+Databricks Vector Search supports three search modes: **ANN** (semantic, default), **HYBRID** (semantic + keyword), and **FULL_TEXT** (keyword only, beta). ANN and HYBRID work with Delta Sync and Direct Access indexes.
+
+## Semantic Search (ANN)
+
+ANN (Approximate Nearest Neighbor) is the default search mode. It finds documents by vector similarity — matching the *meaning* of your query against stored embeddings.
+
+### When to use
+
+- Conceptual or meaning-based queries ("How do I handle errors in my pipeline?")
+- Paraphrased input where exact terms may not appear in the documents
+- Multilingual scenarios where query and document languages may differ
+- General-purpose RAG retrieval
+
+### Example
+
+```python
+# ANN is the default — no query_type parameter needed
+results = w.vector_search_indexes.query_index(
+    index_name="catalog.schema.my_index",
+    columns=["id", "content"],
+    query_text="How do I handle errors in my pipeline?",
+    num_results=5
+)
+```
+
+## Hybrid Search
+
+Hybrid search combines vector similarity (ANN) with BM25 keyword scoring. It retrieves documents that are both semantically similar *and* contain matching keywords, then merges the results.
+
+### When to use
+
+- Queries containing exact terms that must appear: SKUs, product codes, error codes, acronyms
+- Proper nouns — company names, people, specific technologies
+- Technical documentation where terminology precision matters
+- Mixed-intent queries combining concepts with specific terms
+
+### Example
+
+```python
+results = w.vector_search_indexes.query_index(
+    index_name="catalog.schema.my_index",
+    columns=["id", "content"],
+    query_text="SPARK-12345 executor memory error",
+    query_type="HYBRID",
+    num_results=10
+)
+```
+
+## Decision Guide
+
+| Mode | Best for | Trade-off | Choose when |
+|------|----------|-----------|-------------|
+| **ANN** (default) | Conceptual queries, paraphrases, meaning-based search | Fastest; may miss exact keyword matches | You want documents *about* a topic regardless of exact wording |
+| **HYBRID** | Exact terms, codes, proper nouns, mixed-intent queries | ~2x resource usage vs ANN; max 200 results | Your queries contain specific identifiers or technical terms that must appear in results |
+| **FULL_TEXT** (beta) | Pure keyword search without vector embeddings | No semantic understanding; max 200 results | You need keyword matching only, without vector similarity |
+
+**Start with ANN.** Switch to HYBRID if you notice relevant documents being missed because they don't share vocabulary with the query.
+
+## Combining Search Modes with Filters
+
+Both search modes support filters. The filter syntax depends on your endpoint type:
+
+- **Standard endpoints** → `filters` as dict (or `filters_json` as JSON string via `databricks-sdk`)
+- **Storage-Optimized endpoints** → `filters` as SQL-like string (via `databricks-vectorsearch` package)
+
+### Standard endpoint with hybrid search
+
+```python
+results = w.vector_search_indexes.query_index(
+    index_name="catalog.schema.my_index",
+    columns=["id", "content", "category"],
+    query_text="SPARK-12345 executor memory error",
+    query_type="HYBRID",
+    num_results=10,
+    filters_json='{"category": "troubleshooting", "status": ["open", "in_progress"]}'
+)
+```
+
+### Storage-Optimized endpoint with hybrid search
+
+```python
+from databricks.vector_search.client import VectorSearchClient
+
+vsc = VectorSearchClient()
+index = vsc.get_index(endpoint_name="my-storage-endpoint", index_name="catalog.schema.my_index")
+
+results = index.similarity_search(
+    query_text="SPARK-12345 executor memory error",
+    columns=["id", "content", "category"],
+    query_type="hybrid",
+    num_results=10,
+    filters="category = 'troubleshooting' AND status IN ('open', 'in_progress')"
+)
+```
+
+## Using with Pre-Computed Embeddings
+
+If you compute embeddings yourself, use `query_vector` instead of `query_text` for ANN search:
+
+```python
+# ANN with pre-computed embedding (default)
+results = w.vector_search_indexes.query_index(
+    index_name="catalog.schema.my_index",
+    columns=["id", "content"],
+    query_vector=[0.1, 0.2, 0.3, ...],  # Your embedding vector
+    num_results=10
+)
+```
+
+For **hybrid search with self-managed embeddings** (indexes without an associated model endpoint), you must provide **both** `query_vector` and `query_text`. The vector is used for the ANN component and the text for the BM25 keyword component:
+
+```python
+# HYBRID with self-managed embeddings — requires both vector AND text
+results = w.vector_search_indexes.query_index(
+    index_name="catalog.schema.my_index",
+    columns=["id", "content"],
+    query_vector=[0.1, 0.2, 0.3, ...],  # For ANN similarity
+    query_text="executor memory error",   # For BM25 keyword matching
+    query_type="HYBRID",
+    num_results=10
+)
+```
+
+**Notes:**
+- For **ANN** queries: provide either `query_text` or `query_vector`, not both.
+- For **HYBRID** queries on **managed embedding indexes**: provide only `query_text` (the system handles both components).
+- For **HYBRID** queries on **self-managed indexes without a model endpoint**: provide both `query_vector` and `query_text`.
+- When using `query_text` alone, the index must have an associated embedding model (managed embeddings or `embedding_model_endpoint_name` on a Direct Access index).
+
+## Parameter Reference
+
+| Parameter | Type | Package | Description |
+|-----------|------|---------|-------------|
+| `query_text` | `str` | Both | Text query — requires embedding model on the index |
+| `query_vector` | `list[float]` | Both | Pre-computed embedding vector |
+| `query_type` | `str` | Both | `"ANN"` (default) or `"HYBRID"` or `"FULL_TEXT"` (beta) |
+| `columns` | `list[str]` | Both | Column names to return in results |
+| `num_results` | `int` | Both | Number of results (default: 10 in `databricks-sdk`, 5 in `databricks-vectorsearch`) |
+| `filters_json` | `str` | `databricks-sdk` | JSON dict filter string (Standard endpoints) |
+| `filters` | `str` or `dict` | `databricks-vectorsearch` | Dict for Standard, SQL-like string for Storage-Optimized |