Skip to content

Commit f6c24d4

Browse files
authored
Merge pull request #8 from oracle-devrel/merge/article-03-lab-into-main
Merge article/03 lab into main: module 03 (10K corpus, IVF filter-first proof, HVI) + module 02/03 reconciliation
2 parents 386532b + cdd1a3a commit f6c24d4

16 files changed

Lines changed: 1186 additions & 23 deletions

File tree

labs/converged-database-lab/.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,4 @@ validator/results.json
55
validator/.venv/
66
docker/.data/
77
*.pyc
8+
tasks/

labs/converged-database-lab/README.md

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# converged-database-lab
22

3-
![validate](https://github.com/oracle-devrel/oracle-umt-developer-hub/actions/workflows/converged-database-lab.yml/badge.svg)
3+
![validate](https://github.com/rhoulihan/converged-database-lab/actions/workflows/validate.yml/badge.svg)
44

55
Runnable proofs for the converged-database article series — every technical claim in the articles executes here, nightly, against a free container.
66

@@ -80,8 +80,10 @@ All modules run against one deterministically seeded domain. Seeds use an LCG (s
8080
| `idx_orders_customer` | B-tree | Customer order range scans |
8181
| `idx_items_product` | B-tree | Product-level aggregations |
8282
| `idx_tickets_customer` | B-tree | Per-customer ticket lookups |
83-
| `ticket_text_idx` | Oracle Text (SEARCH INDEX) | Full-text keyword search on ticket bodies |
84-
| `ticket_vec_idx` | Vector IVF (NEIGHBOR PARTITIONS, COSINE) | Approximate nearest-neighbor on embeddings |
83+
| `ticket_text_idx` | Oracle Text (SEARCH INDEX, SYNC ON COMMIT) | Transactional keyword search on ticket subjects (module 02's read-after-write proof; body text search belongs to the HVI) |
84+
| `ticket_vec_idx` | Vector IVF (NEIGHBOR PARTITIONS, COSINE) | Approximate nearest-neighbor on 8-dim demo embeddings |
85+
| `tickets_bodyvec_ivf` | Vector IVF (NEIGHBOR PARTITIONS, COSINE) | ANN on the real 384-dim `body_vec` embeddings (module 03) |
86+
| `tickets_hvi` | Hybrid Vector Index (Text + vector, one domain index) | Fused keyword + semantic search on ticket bodies (module 03) |
8587

8688
**Note on embeddings:** The 8-dim vectors are deterministic demo values generated by the LCG seed. All engine behaviors demonstrated (distance functions, vector indexes, transactions, cross-model joins) are dimension-independent. Module 03 documents the optional real-model flow using `DBMS_VECTOR.LOAD_ONNX_MODEL` with all-MiniLM-L12-v2.
8789

@@ -95,7 +97,7 @@ All modules run against one deterministically seeded domain. Seeds use an LCG (s
9597
|---|------|--------|--------|------------|-------|
9698
| 01 | `what-is-a-converged-database` | Live | 5 | 20 | [README](modules/01-what-is-a-converged-database/README.md) · [article](https://blogs.oracle.com/developers/what-is-a-converged-database-definition-five-tests-and-ai-use-cases) |
9799
| 02 | `converged-vs-multi-model` | Live | 3 | 18 | [README](modules/02-converged-vs-multi-model/README.md) · [article](https://blogs.oracle.com/developers/converged-database-vs-multi-model-database-whats-the-difference) |
98-
| 03 | `vector-vs-converged` | On branch, CI-green — article in draft | 4 | 19 | [README](https://github.com/oracle-devrel/oracle-umt-developer-hub/blob/article/03-vector-vs-converged/labs/converged-database-lab/modules/03-vector-vs-converged/README.md) · [branch](https://github.com/oracle-devrel/oracle-umt-developer-hub/tree/article/03-vector-vs-converged) · article at publish |
100+
| 03 | `vector-vs-converged` | On main, validated — article in draft | 4 | 23 | [README](modules/03-vector-vs-converged/README.md) · [branch](https://github.com/oracle-devrel/oracle-umt-developer-hub/tree/article/03-vector-vs-converged) · article at publish |
99101
| 04 | `ai-agents-enterprise-data` | On branch, CI-green — article in draft | 3 | 16 | [README](https://github.com/oracle-devrel/oracle-umt-developer-hub/blob/article/04-ai-agents-enterprise-data/labs/converged-database-lab/modules/04-ai-agents-enterprise-data/README.md) · [branch](https://github.com/oracle-devrel/oracle-umt-developer-hub/tree/article/04-ai-agents-enterprise-data) · article at publish |
100102
| 05 | `json-graph-vector-relational` | On branch, CI-green — article in draft | 4 | 26 | [README](https://github.com/oracle-devrel/oracle-umt-developer-hub/blob/article/05-json-graph-vector-relational/labs/converged-database-lab/modules/05-json-graph-vector-relational/README.md) · [branch](https://github.com/oracle-devrel/oracle-umt-developer-hub/tree/article/05-json-graph-vector-relational) · article at publish |
101103
| 06 | `secure-agent-access` | On branch, CI-green — article in draft | 4 | 7 | [README](https://github.com/oracle-devrel/oracle-umt-developer-hub/blob/article/06-secure-agent-access/labs/converged-database-lab/modules/06-secure-agent-access/README.md) · [branch](https://github.com/oracle-devrel/oracle-umt-developer-hub/tree/article/06-secure-agent-access) · article at publish |

labs/converged-database-lab/docker-compose.yml

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,10 @@ services:
1717
test: ['CMD-SHELL', '/opt/oracle/healthcheck.sh && curl -so /dev/null -w "%{http_code}" http://localhost:8181/ | grep -q "302\|200"']
1818
interval: 15s
1919
timeout: 10s
20-
retries: 40
20+
# 80 retries: first boot now embeds 10,000 ticket bodies in-database and
21+
# builds the HVI over them (~7-9 min of init on top of DB create), so the
22+
# container must not be declared unhealthy before init can finish.
23+
retries: 80
2124
start_period: 120s
2225
deploy:
2326
resources:

labs/converged-database-lab/docker/Dockerfile.oracle

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,34 @@ RUN chown -R oracle:oinstall /etc/ords
99

1010
# mongosh: used to verify and exercise the MongoDB API (port 27017) from inside
1111
# the container — the validator harness runs *.js scripts through it.
12-
RUN printf '[mongodb-org-8.0]\nname=MongoDB Repository\nbaseurl=https://repo.mongodb.org/yum/redhat/8/mongodb-org/8.0/x86_64/\ngpgcheck=1\nenabled=1\ngpgkey=https://pgp.mongodb.com/server-8.0.asc\n' > /etc/yum.repos.d/mongodb-org-8.0.repo \
12+
# Arch-aware repo URL: CI builds on x86_64; local builds on Apple-silicon (aarch64)
13+
# podman machines get MongoDB's aarch64 el8 packages instead of failing dependency
14+
# resolution against the x86_64 repo.
15+
RUN printf '[mongodb-org-8.0]\nname=MongoDB Repository\nbaseurl=https://repo.mongodb.org/yum/redhat/8/mongodb-org/8.0/%s/\ngpgcheck=1\nenabled=1\ngpgkey=https://pgp.mongodb.com/server-8.0.asc\n' "$(uname -m)" > /etc/yum.repos.d/mongodb-org-8.0.repo \
1316
&& microdnf install -y mongodb-mongosh \
1417
&& microdnf clean all
1518

19+
# Bake Oracle's prebuilt augmented all-MiniLM-L12-v2 ONNX model into the image so
20+
# every fresh build (and CI) loads identical in-DB embeddings deterministically —
21+
# no network dependency at init time. This is the official model published by
22+
# Oracle Machine Learning: the 26ai docs "ONNX Pipeline Models: Text Embedding"
23+
# page redirects (/pls/topic/lookup?...&id=oml_ai_models_object_storage) to the
24+
# OML-ai-models object-storage bucket whose index lists this exact URL. The zip
25+
# (~122.6 MB) unpacks to all_MiniLM_L12_v2.onnx (~133.3 MB, 384-dim output),
26+
# loaded by docker/init/08-vector-model.sql via DBMS_VECTOR.LOAD_ONNX_MODEL.
27+
# Placed AFTER the ORDS + mongosh layers so their (slow) cache layers are reused
28+
# when only the model layer changes. Fails the build loudly if the URL stops
29+
# resolving rather than baking a broken image.
30+
ARG MINILM_URL=https://adwc4pm.objectstorage.us-ashburn-1.oci.customer-oci.com/p/TtH6hL2y25EypZ0-rrczRZ1aXp7v1ONbRBfCiT-BDBN8WLKQ3lgyW6RxCfIFLdA6/n/adwc4pm/b/OML-ai-models/o/all_MiniLM_L12_v2_augmented.zip
31+
RUN microdnf install -y unzip \
32+
&& microdnf clean all \
33+
&& mkdir -p /opt/oracle/models \
34+
&& curl -fsSL -o /tmp/minilm.zip "$MINILM_URL" \
35+
&& unzip -o /tmp/minilm.zip -d /opt/oracle/models \
36+
&& test -s /opt/oracle/models/all_MiniLM_L12_v2.onnx \
37+
&& rm -f /tmp/minilm.zip \
38+
&& chown -R oracle:oinstall /opt/oracle/models
39+
1640
COPY init/ /container-entrypoint-initdb.d/
1741

1842
COPY scripts/entrypoint.sh /opt/oracle/scripts/entrypoint.sh

labs/converged-database-lab/docker/init/05-text-vector.sql

Lines changed: 17 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,23 @@
11
ALTER SESSION SET CONTAINER = FREEPDB1;
22
ALTER SESSION SET CURRENT_SCHEMA = lab_user;
33

4-
-- Oracle Text index on ticket bodies: keyword search in the same engine/transaction.
5-
-- SYNC (ON COMMIT): the index syncs inside the committing transaction, so a
6-
-- committed row is immediately findable via CONTAINS — transactional
7-
-- read-after-write search (module 02 proves this). The 26ai Free default for
8-
-- CREATE SEARCH INDEX is MAINTENANCE AUTO with deferred background sync
9-
-- (ctx_user_indexes showed IDX_SYNC_TYPE = MANUAL), under which a probe row was
10-
-- NOT visible to CONTAINS immediately after COMMIT.
11-
CREATE SEARCH INDEX ticket_text_idx ON support_tickets (body)
4+
-- Oracle Text index on ticket SUBJECTS: keyword search in the same
5+
-- engine/transaction. SYNC (ON COMMIT): the index syncs inside the committing
6+
-- transaction, so a committed row is immediately findable via CONTAINS —
7+
-- transactional read-after-write search (module 02 proves this). The 26ai Free
8+
-- default for CREATE SEARCH INDEX is MAINTENANCE AUTO with deferred background
9+
-- sync (ctx_user_indexes showed IDX_SYNC_TYPE = MANUAL), under which a probe
10+
-- row was NOT visible to CONTAINS immediately after COMMIT.
11+
--
12+
-- MERGE RECONCILIATION (modules 02 + 03 together on main): this index lives on
13+
-- SUBJECT here, not BODY. Module 03's Hybrid Vector Index owns body text search
14+
-- (an HVI's text component is the same indextype as a SEARCH index — two cannot
15+
-- share a column, ORA-29879), and the HVI's text component does NOT sync on
16+
-- commit (verified empirically 2026-07-24: a committed probe row was not
17+
-- CONTAINS-findable through the HVI). Module 02's read-after-write proof
18+
-- therefore probes the subject column on main; the article/02 BRANCH keeps the
19+
-- body form that the published article's snippets byte-match.
20+
CREATE SEARCH INDEX ticket_text_idx ON support_tickets (subject)
1221
PARAMETERS ('SYNC (ON COMMIT)');
1322

1423
-- Vector index. IVF (NEIGHBOR PARTITIONS) — works within Free-tier memory without

labs/converged-database-lab/docker/init/06-seed.sql

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,13 @@ ALTER SESSION SET CURRENT_SCHEMA = lab_user;
44
DECLARE
55
-- All item declarations must precede subprogram bodies (PLS-00103 otherwise).
66
seed NUMBER := 42;
7+
-- Ticket volume. 10,000 tickets is deliberately past the point where the
8+
-- cost-based optimizer chooses the IVF vector index over exact full-scan
9+
-- distance for module 03's filtered FETCH APPROX query (the crossover
10+
-- arrives by roughly 3,000 rows on the Free container with fresh stats) —
11+
-- so the captured plan shows the index + in-plan predicate evaluation, not
12+
-- a toy-size full scan.
13+
c_tickets CONSTANT PLS_INTEGER := 10000;
714
v_vec VARCHAR2(400);
815
v_norm NUMBER;
916
TYPE t_f IS TABLE OF NUMBER INDEX BY PLS_INTEGER;
@@ -18,13 +25,71 @@ DECLARE
1825
v_prod PLS_INTEGER;
1926
v_qty PLS_INTEGER;
2027
v_ts TIMESTAMP;
28+
v_cat PLS_INTEGER;
29+
v_body VARCHAR2(2000);
30+
-- Enriched ticket-body corpus for tickets 301..c_tickets: 6 intros x 8 issue
31+
-- categories x 4 details x 50 products x 5 closers ~= 48k distinct bodies, so
32+
-- the 384-dim embeddings form real semantic clusters instead of 4 duplicated
33+
-- points (matters for a meaningful IVF index and hybrid-search demo).
34+
TYPE t_v IS TABLE OF VARCHAR2(400);
35+
intro t_v := t_v('Customer reports an issue with their recent experience.',
36+
'Contacting support after several attempts to resolve this alone.',
37+
'Opening a ticket on behalf of the account holder.',
38+
'Follow-up on an earlier conversation with the support team.',
39+
'New request submitted through the help center.',
40+
'Escalated from chat: the problem is still not resolved.');
41+
refund t_v := t_v('The package arrived damaged and a refund is requested.',
42+
'The box was crushed in transit and the item inside is damaged beyond use.',
43+
'Item arrived with a cracked casing; customer asks for their money back.',
44+
'Shipment was left in the rain and the contents are damaged and unusable.');
45+
login t_v := t_v('Login fails with an authentication error after reset.',
46+
'Password reset completed but sign-in still rejects the new credentials.',
47+
'Two-factor prompt never arrives and access to the account is blocked.',
48+
'Account is locked out after the security update and recovery does not work.');
49+
shipdl t_v := t_v('The order has not arrived within the promised window.',
50+
'Tracking has shown no movement for a week and the delivery date passed.',
51+
'Carrier marked the parcel delivered but nothing has arrived.',
52+
'Estimated delivery keeps slipping and no update has been provided.');
53+
warr t_v := t_v('Customer asks whether the warranty covers accidental damage.',
54+
'Requesting warranty service for a unit that stopped powering on.',
55+
'Screen developed dead pixels within the coverage period.',
56+
'Asking how to start a warranty claim for a failed component.');
57+
billing t_v := t_v('The statement shows a duplicate charge for a single order.',
58+
'A promotional discount was not applied at checkout.',
59+
'The invoice total does not match the order confirmation email.',
60+
'Customer was billed after cancelling the subscription.');
61+
exch t_v := t_v('Requesting an exchange for a different size of the same product.',
62+
'The wrong color variant was delivered and a swap is requested.',
63+
'Received a different model than the one ordered.',
64+
'Asking to exchange a gift for store credit instead.');
65+
install t_v := t_v('The setup guide does not match the ports on the device.',
66+
'Firmware update fails at forty percent every attempt.',
67+
'The companion app cannot discover the device during pairing.',
68+
'Installation completes but the device reboots continuously.');
69+
perf t_v := t_v('The device becomes very slow after a few hours of use.',
70+
'Battery drains overnight even when the unit is idle.',
71+
'Audio cuts out intermittently during playback.',
72+
'The connection drops whenever a second device joins.');
73+
closing t_v := t_v('Customer asks for a response today.',
74+
'This is the second time the issue has been reported.',
75+
'The customer mentions they are a long-time subscriber.',
76+
'A callback number has been left on the account.',
77+
'No workaround has been found so far.');
2178
FUNCTION nxt RETURN NUMBER IS -- LCG: deterministic across runs
2279
BEGIN
2380
seed := MOD(seed * 1103515245 + 12345, 2147483648);
2481
RETURN seed / 2147483648;
2582
END;
2683
FUNCTION pick(n IN PLS_INTEGER) RETURN PLS_INTEGER IS
2784
BEGIN RETURN TRUNC(nxt * n) + 1; END;
85+
FUNCTION cat_detail(p_cat PLS_INTEGER) RETURN VARCHAR2 IS
86+
BEGIN
87+
RETURN CASE p_cat
88+
WHEN 1 THEN refund (pick(4)) WHEN 2 THEN login(pick(4))
89+
WHEN 3 THEN shipdl (pick(4)) WHEN 4 THEN warr (pick(4))
90+
WHEN 5 THEN billing(pick(4)) WHEN 6 THEN exch (pick(4))
91+
WHEN 7 THEN install(pick(4)) ELSE perf(pick(4)) END;
92+
END;
2893
BEGIN
2994
FOR i IN 1..200 LOOP
3095
INSERT INTO customers (email, full_name, segment)
@@ -95,6 +160,9 @@ BEGIN
95160
UPDATE orders SET total_amount = ROUND(v_total,2) WHERE order_id = i;
96161
END LOOP;
97162

163+
-- Tickets 1..300: the ORIGINAL 4-template corpus, byte-for-byte (module 01's
164+
-- ticket #1 proofs and module 03's read-after-write probe depend on ticket #1
165+
-- keeping its "Login fails with an authentication error after reset." body).
98166
FOR i IN 1..300 LOOP
99167
v_norm := 0;
100168
FOR d IN 1..8 LOOP f(d) := nxt*2 - 1; v_norm := v_norm + f(d)*f(d); END LOOP;
@@ -119,6 +187,38 @@ BEGIN
119187
CASE WHEN MOD(i,3)=0 THEN 'closed' WHEN MOD(i,7)=0 THEN 'pending' ELSE 'open' END,
120188
TO_VECTOR(v_vec, 8, FLOAT32));
121189
END LOOP;
190+
191+
-- Tickets 301..c_tickets: the enriched scaled corpus. The LCG is re-seeded to
192+
-- a fixed value here so this block is deterministic regardless of how the
193+
-- upstream seeding evolves — same bodies on every fresh build.
194+
seed := 342;
195+
FOR i IN 301..c_tickets LOOP
196+
v_cat := pick(8);
197+
v_cust := pick(200);
198+
v_body := intro(pick(6))||' '||cat_detail(v_cat)||
199+
' Relates to Product '||pick(50)||'. '||closing(pick(5));
200+
v_norm := 0;
201+
FOR d IN 1..8 LOOP f(d) := nxt*2 - 1; v_norm := v_norm + f(d)*f(d); END LOOP;
202+
v_norm := SQRT(v_norm);
203+
v_vec := '[';
204+
FOR d IN 1..8 LOOP
205+
v_vec := v_vec || TO_CHAR(ROUND(f(d)/v_norm, 6),'FM990.999999') || CASE WHEN d<8 THEN ',' END;
206+
END LOOP;
207+
v_vec := v_vec || ']';
208+
INSERT INTO support_tickets (customer_id, subject, body, status, embedding)
209+
VALUES (v_cust,
210+
CASE v_cat WHEN 1 THEN 'Refund request for damaged item'
211+
WHEN 2 THEN 'Cannot sign in after password reset'
212+
WHEN 3 THEN 'Shipping delay on recent order'
213+
WHEN 4 THEN 'Warranty question for product'
214+
WHEN 5 THEN 'Billing discrepancy on statement'
215+
WHEN 6 THEN 'Exchange request for recent order'
216+
WHEN 7 THEN 'Device setup and installation issue'
217+
ELSE 'Performance problem with product' END || ' #'||i,
218+
v_body,
219+
CASE WHEN MOD(i,3)=0 THEN 'closed' WHEN MOD(i,7)=0 THEN 'pending' ELSE 'open' END,
220+
TO_VECTOR(v_vec, 8, FLOAT32));
221+
END LOOP;
122222
COMMIT;
123223
END;
124224
/

0 commit comments

Comments
 (0)