11# Vector Search: Semantic Product Discovery
22
3- A Declarative Automation Bundle demonstrating ** semantic product search** using
3+ A Declarative Automation Bundle demonstrating semantic product search using
44[ Databricks Vector Search] ( https://docs.databricks.com/en/generative-ai/vector-search.html ) .
5+ It automates the full setup — the Unity Catalog schema, the Vector Search endpoint and
6+ index, and the jobs that load and query the catalog — so a single ` databricks bundle deploy `
7+ gives you a working semantic-search example to explore and adapt.
58
6- ## The problem
9+ ## How it works
710
811Keyword search fails when shoppers use different words than what appears in product
9- descriptions. A customer searching for * "something to keep my coffee hot all day"* won't
10- match a product described as an * "insulated stainless water bottle with double-wall vacuum
11- insulation"* — even though it's the right answer.
12-
13- Semantic search using vector embeddings matches on ** meaning** , not words.
14-
15- ## How it works
12+ descriptions. A customer searching for "something to keep my coffee hot all day" won't
13+ match a product described as an "insulated stainless water bottle with double-wall vacuum
14+ insulation" even though it's the right answer. Semantic search using vector embeddings
15+ matches on meaning, not words.
1616
1717Product descriptions are embedded at upsert time by the setup job using
1818[ ` databricks-gte-large-en ` ] ( https://docs.databricks.com/en/machine-learning/foundation-models/supported-models.html ) .
@@ -27,48 +27,56 @@ product_index (Direct Access Vector Search index)
2727ranked results
2828```
2929
30- ## Bundle resources
30+ ## Project structure
3131
32- | Resource | Type | Description |
33- | ---| ---| ---|
34- | ` product_search_schema ` | ` schemas ` | Unity Catalog schema that namespaces the index |
35- | ` product_search_endpoint ` | ` vector_search_endpoints ` | Managed ANN serving endpoint |
36- | ` product_index ` | ` vector_search_indexes ` | Direct Access index — schema defined in ` resources/index.yml ` |
37- | ` product_discovery_setup ` | ` jobs ` | Embeds product descriptions and upserts into the index |
38- | ` product_discovery_query ` | ` jobs ` | Embeds a query and returns ranked results |
32+ ```
33+ .
34+ ├── databricks.yml # Bundle name, variables, and the deploy target
35+ ├── data/
36+ │ └── products.json # Product catalog — synced to the workspace on deploy
37+ ├── resources/
38+ │ ├── schema.yml # Unity Catalog schema that namespaces the index
39+ │ ├── vector-search-endpoint.yml # Vector Search endpoint (managed ANN serving)
40+ │ ├── vector-search-index.yml # Direct Access index — schema defined inline
41+ │ ├── setup-job.yml # Job: embed product descriptions and upsert them
42+ │ └── query-job.yml # Job: embed a query and return ranked results
43+ └── src/
44+ ├── 01_upsert_products.py # Reads products.json, embeds, calls upsert_data
45+ └── 02_query_demo.py # Semantic search — runs as a job or interactively
46+ ```
3947
4048## Prerequisites
4149
4250- Databricks workspace with Unity Catalog enabled
43- - Databricks CLI that supports ` vector_search_endpoints ` / ` vector_search_indexes ` as bundle resources
51+ - Databricks CLI version 1.1.0 or above
4452- An existing Unity Catalog catalog (default: ` main ` )
4553
46- ## Quick start
54+ ## Usage
4755
48- 1 . ** Authenticate**
56+ 1 . Authenticate the CLI:
4957 ``` bash
5058 databricks auth login --host https://your-workspace.cloud.databricks.com
5159 ```
5260
53- 2 . ** Configure** ` databricks.yml ` — set the workspace host and any variable overrides
61+ 2 . Configure ` databricks.yml ` . Set the workspace host and any variable overrides.
5462
55- 3 . ** Deploy** — creates the schema, endpoint, index, jobs, and syncs ` data/products.json `
63+ 3 . Deploy the bundle. This creates the schema, endpoint, index, jobs, and syncs ` data/products.json ` .
5664 ``` bash
5765 databricks bundle deploy
5866 ```
5967 > Vector Search endpoint creation takes a few minutes to reach ONLINE status.
6068
61- 4 . ** Load the catalog** — embeds all product descriptions and upserts them into the index
69+ 4 . Load the catalog by running the bundle. This embeds all product descriptions and upserts them into the index.
6270 ``` bash
6371 databricks bundle run product_discovery_setup
6472 ```
6573
66- 5 . ** Search ** — pass any natural-language query
74+ 5 . Pass any natural-language query to search.
6775 ``` bash
6876 databricks bundle run product_discovery_query --params " query=footwear for slippery wet trails"
6977 ```
7078
71- 6 . ** Or open** ` src/02_query_demo.py ` in your workspace to run queries interactively
79+ 6 . Or open ` src/02_query_demo.py ` in your workspace to run queries interactively.
7280
7381## Configuration
7482
@@ -89,15 +97,15 @@ databricks bundle deploy \
8997| ` schema ` | ` product_search ` | Schema created by the bundle |
9098| ` endpoint_name ` | ` product-search-endpoint ` | Vector Search endpoint name (must be unique per workspace) |
9199| ` embedding_model ` | ` databricks-gte-large-en ` | Foundation model used for embeddings |
92- | ` embedding_dimension ` | ` 1024 ` | Vector dimension — must match ` embedding_dimension ` in ` resources/index.yml ` |
100+ | ` embedding_dimension ` | ` 1024 ` | Vector dimension — must match ` embedding_dimension ` in ` resources/vector-search- index.yml ` |
93101
94- > ** Note:** ` embedding_dimension ` in ` resources/index.yml ` is hardcoded to ` 1024 ` because
102+ > ** Note:** ` embedding_dimension ` in ` resources/vector-search- index.yml ` is hardcoded to ` 1024 ` because
95103> it is immutable after index creation. If you need a different dimension, change the value
96- > in ` index.yml ` before the first deploy.
104+ > in ` vector-search- index.yml` before the first deploy.
97105
98106## Index schema
99107
100- The index schema lives entirely in ` resources/index.yml ` :
108+ The index schema lives entirely in ` resources/vector-search- index.yml ` :
101109
102110``` yaml
103111direct_access_index_spec :
@@ -130,25 +138,7 @@ records enter the index via `upsert_data`. If you already have a pipeline writin
130138Delta table, a **Delta Sync** index is often simpler — you point the index at the source
131139table and it keeps itself up to date. Replace `index_type : DIRECT_ACCESS` and
132140`direct_access_index_spec` with `index_type : DELTA_SYNC` and `delta_sync_index_spec` in
133- ` resources/index.yml` , and remove the upsert job.
134-
135- # # Project structure
136-
137- ```
138- .
139- ├── databricks.yml
140- ├── data/
141- │ └── products.json # Product catalog — synced to workspace on deploy
142- ├── resources/
143- │ ├── schema.yml # Unity Catalog schema
144- │ ├── endpoint.yml # Vector Search endpoint
145- │ ├── index.yml # Direct Access index
146- │ ├── setup_job.yml # Embed + upsert job
147- │ └── query_demo.yml # Query job (--params "query=...")
148- └── src/
149- ├── 01_upsert_products.py # Reads products.json, embeds, calls upsert_data
150- └── 02_query_demo.py # Semantic search — runs as job or interactively
151- ```
141+ ` resources/vector-search-index.yml` , and remove the upsert job.
152142
153143# # Resources
154144
0 commit comments