Skip to content

Commit e6d00cb

Browse files
janniklasrosejanniklasrose
authored
Add Vector Search semantic product discovery example (#153)
## Summary Adds a Declarative Automation Bundle under `knowledge_base/vector_search_product_discovery/` that demonstrates semantic product search end-to-end with Databricks Vector Search: - `vector_search_endpoints` + `vector_search_indexes` declared as bundle resources; jobs reference them via `${resources.*.name}` so dev-mode prefixing flows through automatically - A `dev` (default, `mode: development`) and a `prod` target — a plain `bundle deploy` is isolated per user (per-user endpoint name; schema/jobs/index dev-prefixed), so several people can deploy into one workspace without colliding - Direct Access index (`engine: direct`); descriptions are embedded explicitly in `01_upsert_products.py` and the query notebook embeds the query before `similarity_search` — Direct Access indexes don't auto-embed (a Delta Sync feature) - A single `embedding_dimension` variable feeds both the index spec and the notebooks (immutable after index creation, so one knob prevents a silent mismatch) - The query job calls `dbutils.notebook.exit(...)` so ranked results come back from `databricks bundle run` / `jobs get-run-output` - `schema_json` uses the flat `{"col":"type"}` form required by the API ## Requirements Databricks CLI **v1.1.0+**, which ships `vector_search_endpoints` / `vector_search_indexes` as first-class DABs resources (was databricks/cli#5123). ## Test plan Verified with CLI v1.1.0 on a workspace (default `dev` target): - [x] `databricks bundle validate` — `dev` and `prod` - [x] `databricks bundle deploy` — endpoint ONLINE, index created - [x] `databricks bundle run product_discovery_setup` — products embedded + upserted - [x] `databricks bundle run product_discovery_query --params "query=something to keep my coffee hot all day"` — returns ranked results as JSON (e.g. the insulated water bottle surfaces with no keyword overlap) - [x] `databricks bundle destroy` — clean teardown This pull request and its description were written by Isaac.
1 parent cb02ade commit e6d00cb

10 files changed

Lines changed: 699 additions & 0 deletions

File tree

‎knowledge_base/vector_search_product_discovery/README.md‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
# Vector Search: Semantic Product Discovery
2+
3+
A Declarative Automation Bundle demonstrating semantic product search using
4+
[Databricks Vector Search](https://docs.databricks.com/en/generative-ai/vector-search.html).
5+
It automates the full setup — the Unity Catalog schema, the Vector Search endpoint and
6+
index, and the jobs that load and query the catalog — so a single `databricks bundle deploy`
7+
gives you a working semantic-search example to explore and adapt.
8+
9+
## How it works
10+
11+
Keyword search fails when shoppers use different words than what appears in product
12+
descriptions. A customer searching for "something to keep my coffee hot all day" won't
13+
match a product described as an "insulated stainless water bottle with double-wall vacuum
14+
insulation" even though it's the right answer. Semantic search using vector embeddings
15+
matches on meaning, not words.
16+
17+
Product descriptions are embedded at upsert time by the setup job using
18+
[`databricks-gte-large-en`](https://docs.databricks.com/en/machine-learning/foundation-models/supported-models.html).
19+
At query time the query is embedded with the same model and the index returns the nearest
20+
products in vector space.
21+
22+
```
23+
data/products.json (synced to workspace by bundle deploy)
24+
↓ embed descriptions → upsert_data()
25+
product_index (Direct Access Vector Search index)
26+
↓ embed query → similarity_search(query_vector=...)
27+
ranked results
28+
```
29+
30+
## Project structure
31+
32+
```
33+
.
34+
├── databricks.yml # Bundle name, variables, and the deploy target
35+
├── data/
36+
│ └── products.json # Product catalog — synced to the workspace on deploy
37+
├── resources/
38+
│ ├── schema.yml # Unity Catalog schema that namespaces the index
39+
│ ├── vector-search-endpoint.yml # Vector Search endpoint (managed ANN serving)
40+
│ ├── vector-search-index.yml # Direct Access index — schema defined inline
41+
│ ├── setup-job.yml # Job: embed product descriptions and upsert them
42+
│ └── query-job.yml # Job: embed a query and return ranked results
43+
└── src/
44+
├── 01_upsert_products.py # Reads products.json, embeds, calls upsert_data
45+
└── 02_query_demo.py # Semantic search — runs as a job or interactively
46+
```
47+
48+
## Prerequisites
49+
50+
- Databricks workspace with Unity Catalog enabled
51+
- Databricks CLI version 1.1.0 or above
52+
- An existing Unity Catalog catalog (default: `main`)
53+
54+
## Usage
55+
56+
1. Authenticate the CLI:
57+
```bash
58+
databricks auth login --host https://your-workspace.cloud.databricks.com
59+
```
60+
61+
2. Configure `databricks.yml`. Set the workspace host and any variable overrides.
62+
63+
3. Deploy the bundle. This creates the schema, endpoint, index, jobs, and syncs `data/products.json`.
64+
```bash
65+
databricks bundle deploy
66+
```
67+
This deploys the default `dev` target in development mode, so resources are namespaced
68+
per user — jobs and the schema get a `[dev you]` prefix and the endpoint is named after
69+
you — and several people can deploy into the same workspace without colliding. Use
70+
`databricks bundle deploy --target prod` for the shared production copy.
71+
72+
> Vector Search endpoint creation takes a few minutes to reach ONLINE status.
73+
74+
4. Load the catalog by running the bundle. This embeds all product descriptions and upserts them into the index.
75+
```bash
76+
databricks bundle run product_discovery_setup
77+
```
78+
79+
5. Pass any natural-language query to search.
80+
```bash
81+
databricks bundle run product_discovery_query --params "query=footwear for slippery wet trails"
82+
```
83+
84+
The job returns the ranked results as JSON — view them with
85+
`databricks jobs get-run-output <run-id>` or on the run page.
86+
87+
6. Or open `src/02_query_demo.py` in your workspace to run queries interactively.
88+
89+
## Configuration
90+
91+
Override variables at deploy time or run time:
92+
93+
```bash
94+
databricks bundle deploy \
95+
--var catalog=my_catalog \
96+
--var schema=product_search \
97+
--var endpoint_name=my-vs-endpoint \
98+
--var embedding_model=databricks-gte-large-en \
99+
--var embedding_dimension=1024
100+
```
101+
102+
| Variable | Default | Description |
103+
|---|---|---|
104+
| `catalog` | `main` | Existing Unity Catalog catalog |
105+
| `schema` | `product_search` | Schema created by the bundle |
106+
| `endpoint_name` | `product-search-endpoint` | Vector Search endpoint name. Shared in prod; the `dev` target overrides it per user. |
107+
| `embedding_model` | `databricks-gte-large-en` | Foundation model used for embeddings |
108+
| `embedding_dimension` | `1024` | Vector dimension. Drives both the index and the embedding requests; immutable after the index is created. |
109+
110+
> **Note:** `embedding_dimension` is immutable after the index is created. Set it (via the
111+
> `embedding_dimension` variable) before the first deploy — the index and the upsert/query
112+
> jobs all read from that one variable.
113+
114+
## Index schema
115+
116+
The index schema lives entirely in `resources/vector-search-index.yml`:
117+
118+
```yaml
119+
direct_access_index_spec:
120+
schema_json: >-
121+
{"product_id":"int","name":"string","category":"string","brand":"string",
122+
"price":"float","description":"string","description_vector":"array<float>"}
123+
embedding_vector_columns:
124+
- name: description_vector
125+
embedding_dimension: ${var.embedding_dimension}
126+
```
127+
128+
`schema_json` is a flat `{"column_name": "type"}` JSON string. `description_vector` stores
129+
the pre-computed embedding produced by `01_upsert_products.py`.
130+
131+
## Updating the product catalog
132+
133+
Edit `data/products.json`, then re-deploy and re-run setup:
134+
135+
```bash
136+
databricks bundle deploy
137+
databricks bundle run product_discovery_setup
138+
```
139+
140+
Upserts are idempotent on `product_id` — existing records are updated, new records added.
141+
142+
## Variant: Delta Sync index
143+
144+
This example uses a **Direct Access** index, which gives full control over when and how
145+
records enter the index via `upsert_data`. If you already have a pipeline writing to a
146+
Delta table, a **Delta Sync** index is often simpler — you point the index at the source
147+
table and it keeps itself up to date. Replace `index_type: DIRECT_ACCESS` and
148+
`direct_access_index_spec` with `index_type: DELTA_SYNC` and `delta_sync_index_spec` in
149+
`resources/vector-search-index.yml`, and remove the upsert job.
150+
151+
## Resources
152+
153+
- [Databricks Vector Search](https://docs.databricks.com/en/generative-ai/vector-search.html)
154+
- [Declarative Automation Bundles](https://docs.databricks.com/dev-tools/bundles/)
155+
- [Foundation Models — GTE Large](https://docs.databricks.com/en/machine-learning/foundation-models/supported-models.html)

‎knowledge_base/vector_search_product_discovery/data/products.json‎

Lines changed: 202 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,202 @@
1+
[
2+
{
3+
"product_id": 1,
4+
"name": "Alpine Thermal Jacket",
5+
"category": "Outdoor Clothing",
6+
"brand": "Highcairn",
7+
"price": 289.99,
8+
"description": "Insulated hardshell designed for alpine conditions. Features a windproof outer layer, sealed seams, and a 700-fill-power down inner. Packs into its own pocket. Ideal for mountaineering, ski touring, and above-treeline travel in sub-zero temperatures."
9+
},
10+
{
11+
"product_id": 2,
12+
"name": "Merino Wool Base Layer Top",
13+
"category": "Outdoor Clothing",
14+
"brand": "Wollund",
15+
"price": 89.99,
16+
"description": "Next-to-skin mid-weight top made from 100% New Zealand merino wool. Naturally temperature-regulating and odor-resistant. Flatlock seams prevent chafing on long days. Worn as a standalone layer or under a shell in cold weather."
17+
},
18+
{
19+
"product_id": 3,
20+
"name": "Softshell Fleece Jacket",
21+
"category": "Outdoor Clothing",
22+
"brand": "Glidewen",
23+
"price": 149.99,
24+
"description": "Four-way stretch softshell with a bonded fleece backer. Wind-resistant without being fully waterproof — ideal as a mid-layer or standalone jacket on dry, cool days. Two hand pockets and a chest zip pocket."
25+
},
26+
{
27+
"product_id": 4,
28+
"name": "Rain Jacket with Hood",
29+
"category": "Outdoor Clothing",
30+
"brand": "Stormvel",
31+
"price": 199.99,
32+
"description": "3-layer waterproof breathable shell rated at 20,000mm hydrostatic head. Helmet-compatible hood with a single-hand adjustment. Pit-zip vents for temperature control during high output activities. Packs to fist size."
33+
},
34+
{
35+
"product_id": 5,
36+
"name": "Waterproof Mid Hiking Boot",
37+
"category": "Footwear",
38+
"brand": "Treadwen",
39+
"price": 179.99,
40+
"description": "Full-grain leather upper with a waterproof membrane. A grippy rubber outsole bites into wet rock and loose trail. Mid-cut ankle collar supports the ankle on uneven terrain. Recommended for day hikes and multi-day trips with a loaded pack."
41+
},
42+
{
43+
"product_id": 6,
44+
"name": "Trail Running Shoe",
45+
"category": "Footwear",
46+
"brand": "Fellrun",
47+
"price": 139.99,
48+
"description": "Lightweight trail runner with a rock plate and aggressive lug pattern. 8mm drop and a wide toe box promote natural foot strike. Drainage ports shed water quickly on stream crossings. Built for technical singletrack and ultra-distance racing."
49+
},
50+
{
51+
"product_id": 7,
52+
"name": "Ultralight Backpacking Tent",
53+
"category": "Camping",
54+
"brand": "Tarnost",
55+
"price": 349.99,
56+
"description": "Two-person freestanding tent weighing 1.1 kg. Silnylon fly sheds rain and condensation. Interior mesh canopy maximizes airflow on warm nights. Sets up in under three minutes. Rated for three-season use; not designed for heavy snow loads."
57+
},
58+
{
59+
"product_id": 8,
60+
"name": "20°F Down Sleeping Bag",
61+
"category": "Camping",
62+
"brand": "Frostlin",
63+
"price": 279.99,
64+
"description": "Mummy-cut bag with 800-fill-power hydrophobic down. EN-tested lower limit of -7°C. Footbox baffle prevents cold spots at the toes. Corrosion-resistant zipper with anti-snag tape. Compresses to the size of a 1 L water bottle in the included stuff sack."
65+
},
66+
{
67+
"product_id": 9,
68+
"name": "Rechargeable Headlamp 350 lm",
69+
"category": "Camping",
70+
"brand": "Beamwick",
71+
"price": 49.99,
72+
"description": "USB-C rechargeable lamp with a 350-lumen flood beam and a 100-lumen red night-vision mode. IPX4 splash-resistant housing. Single button cycles through brightness levels. Runtime up to 40 hours on low. Tilt mechanism adjusts beam angle hands-free."
73+
},
74+
{
75+
"product_id": 10,
76+
"name": "Gravity Water Filter",
77+
"category": "Camping",
78+
"brand": "Streamwell",
79+
"price": 59.99,
80+
"description": "Hollow-fiber gravity filter removes bacteria, protozoa, and microplastics to 0.1 micron. No pumping required — hang the dirty reservoir and let gravity do the work. Filters 1.5 liters per minute. Includes clean and dirty reservoirs and a hydration hose adapter."
81+
},
82+
{
83+
"product_id": 11,
84+
"name": "Carbon Fiber Trekking Poles",
85+
"category": "Camping",
86+
"brand": "Polaract",
87+
"price": 119.99,
88+
"description": "100% carbon fiber shaft reduces arm fatigue on long days. Quick-lock mechanism adjusts from 100 to 135 cm in seconds. Cork grip wicks sweat and molds to hand shape over time. Tungsten carbide tips with interchangeable rubber feet for paved surfaces."
89+
},
90+
{
91+
"product_id": 12,
92+
"name": "Noise-Canceling Wireless Headphones",
93+
"category": "Electronics",
94+
"brand": "Aurivox",
95+
"price": 329.99,
96+
"description": "Over-ear headphones with hybrid active noise cancellation that adapts to ambient sound levels. 30-hour battery life. Multipoint pairing connects to two devices simultaneously. Foldable design with a hard carry case. Hi-Res Audio certified with a 4 Hz–40 kHz range."
97+
},
98+
{
99+
"product_id": 13,
100+
"name": "Wireless Mechanical Keyboard",
101+
"category": "Electronics",
102+
"brand": "Clackton",
103+
"price": 149.99,
104+
"description": "Tenkeyless layout with hot-swappable tactile switches. Bluetooth 5.0 pairs with up to three devices; a 2.4 GHz dongle provides sub-1ms latency for gaming. PBT keycaps resist shine. Per-key RGB lighting with 15 preset effects. 2000 mAh battery lasts two weeks on a single charge with lighting off."
105+
},
106+
{
107+
"product_id": 14,
108+
"name": "Portable Laptop Stand",
109+
"category": "Electronics",
110+
"brand": "Deskwen",
111+
"price": 49.99,
112+
"description": "Adjustable aluminum stand raises a laptop screen to eye level, reducing neck strain during long work sessions. Six height settings from 15 to 32 cm. Folds flat to 3 mm for bag transport. Supports laptops from 10 to 17 inches and up to 8 kg."
113+
},
114+
{
115+
"product_id": 15,
116+
"name": "Smart Air Purifier",
117+
"category": "Electronics",
118+
"brand": "Puralto",
119+
"price": 219.99,
120+
"description": "HEPA H13 filter captures 99.97% of particles down to 0.3 microns including pollen, dust mite debris, and pet dander. Activated carbon layer adsorbs VOCs and cooking odors. App-controlled with air quality sensor and auto mode. Covers rooms up to 50 m². Night mode drops fan noise to 22 dB."
121+
},
122+
{
123+
"product_id": 16,
124+
"name": "Voice-Controlled Smart Speaker",
125+
"category": "Electronics",
126+
"brand": "Voxhome",
127+
"price": 99.99,
128+
"description": "360-degree speaker with a woofer and two tweeters. Built-in voice assistant controls smart home devices, plays music, answers questions, and sets timers. Connects via Wi-Fi and Bluetooth. Multi-room audio links speakers across the home. Privacy mic mute button."
129+
},
130+
{
131+
"product_id": 17,
132+
"name": "Cast Iron Dutch Oven 5.5 qt",
133+
"category": "Kitchen",
134+
"brand": "Ferralto",
135+
"price": 89.99,
136+
"description": "Enameled cast iron with a tight-fitting lid that seals in moisture for braises, stews, and bread baking. Oven-safe to 260°C. Works on all cooktops including induction. Interior cream enamel shows browning clearly. Self-basting dimpled lid. Lifetime warranty against defects."
137+
},
138+
{
139+
"product_id": 18,
140+
"name": "Burr Coffee Grinder",
141+
"category": "Kitchen",
142+
"brand": "Roastel",
143+
"price": 79.99,
144+
"description": "40mm stainless steel conical burrs produce consistent grind size from coarse French press to fine espresso. 40g hopper capacity. 18 click-stop settings. Static-reducing grounds bin with a rubber seal. Quiet 120W motor. Removable upper burr for easy cleaning."
145+
},
146+
{
147+
"product_id": 19,
148+
"name": "10-Piece Stainless Knife Block Set",
149+
"category": "Kitchen",
150+
"brand": "Bladely",
151+
"price": 159.99,
152+
"description": "High-carbon German stainless blades forged from a single billet for full tang strength. Set includes 8-inch chef's, 8-inch bread, 7-inch santoku, 5-inch utility, 3.5-inch paring, six steak knives, shears, honing rod, and a beechwood block. Blades hand-sharpened to 15° per side."
153+
},
154+
{
155+
"product_id": 20,
156+
"name": "Pour-Over Coffee Dripper Set",
157+
"category": "Kitchen",
158+
"brand": "Pournal",
159+
"price": 44.99,
160+
"description": "Borosilicate glass dripper sits on a matching carafe. Spiral ribs promote even extraction by allowing air to escape uniformly. Includes 40 bleached paper filters and a stainless gooseneck pouring kettle. Produces a clean, bright cup that highlights single-origin floral and fruity notes."
161+
},
162+
{
163+
"product_id": 21,
164+
"name": "Extra-Thick Yoga Mat",
165+
"category": "Fitness",
166+
"brand": "Mattra",
167+
"price": 69.99,
168+
"description": "6mm natural rubber mat with a microfiber top layer that grips when wet. Non-slip bottom prevents sliding on hardwood and tile. Alignment lines guide stance width in warrior and standing poses. Rolled dimensions: 61 × 183 cm. Includes carrying strap. Free from latex, PVC, and phthalates."
169+
},
170+
{
171+
"product_id": 22,
172+
"name": "Vibrating Foam Roller",
173+
"category": "Fitness",
174+
"brand": "Rollwen",
175+
"price": 89.99,
176+
"description": "High-density EPP foam roller with four built-in vibration frequencies (20–40 Hz). Vibration penetrates deeper tissue than static rolling for myofascial release and delayed-onset muscle soreness. USB rechargeable; 2-hour runtime per charge. Hollow core stores the charging cable."
177+
},
178+
{
179+
"product_id": 23,
180+
"name": "Resistance Band Set",
181+
"category": "Fitness",
182+
"brand": "Bandwell",
183+
"price": 34.99,
184+
"description": "Five fabric-wrapped loop bands in progressive resistances from 5 to 40 lbs. Non-roll design stays in place during squats, hip thrusts, and lateral walks. Used for glute activation, mobility work, and upper-body accessory exercises. Includes a mesh carry bag and a printed exercise guide."
185+
},
186+
{
187+
"product_id": 24,
188+
"name": "Insulated Stainless Water Bottle 32 oz",
189+
"category": "Fitness",
190+
"brand": "Vesslo",
191+
"price": 39.99,
192+
"description": "Double-wall vacuum insulation keeps beverages cold 24 hours or hot 12 hours. 18/8 stainless steel; no plastic liner means no flavor transfer. Wide-mouth lid accepts ice cubes. Compatible with most car cup holders. Powder-coat finish resists dents and scratches."
193+
},
194+
{
195+
"product_id": 25,
196+
"name": "Compression Running Tights",
197+
"category": "Fitness",
198+
"brand": "Stridon",
199+
"price": 79.99,
200+
"description": "Four-way stretch fabric with graduated compression from ankle to waist improves circulation and reduces muscle oscillation during runs. UPF 50+ sun protection. Rear zip pocket fits a key or gel. Reflective piping increases visibility in low light. Available in lengths for inseams 28–34 inches."
201+
}
202+
]

0 commit comments

Comments
 (0)