Merge article/03 lab into main: module 03 (10K corpus, IVF filter-first proof, HVI) + module 02/03 reconciliation

labs/converged-database-lab/.gitignore

@@ -5,3 +5,4 @@ validator/results.json
 validator/.venv/
 docker/.data/
 *.pyc
+tasks/

labs/converged-database-lab/README.md

@@ -1,6 +1,6 @@
 # converged-database-lab
 
-![validate](https://github.com/oracle-devrel/oracle-umt-developer-hub/actions/workflows/converged-database-lab.yml/badge.svg)
+![validate](https://github.com/rhoulihan/converged-database-lab/actions/workflows/validate.yml/badge.svg)
 
 Runnable proofs for the converged-database article series — every technical claim in the articles executes here, nightly, against a free container.
 
@@ -80,8 +80,10 @@ All modules run against one deterministically seeded domain. Seeds use an LCG (s
 | `idx_orders_customer` | B-tree | Customer order range scans |
 | `idx_items_product` | B-tree | Product-level aggregations |
 | `idx_tickets_customer` | B-tree | Per-customer ticket lookups |
-| `ticket_text_idx` | Oracle Text (SEARCH INDEX) | Full-text keyword search on ticket bodies |
-| `ticket_vec_idx` | Vector IVF (NEIGHBOR PARTITIONS, COSINE) | Approximate nearest-neighbor on embeddings |
+| `ticket_text_idx` | Oracle Text (SEARCH INDEX, SYNC ON COMMIT) | Transactional keyword search on ticket subjects (module 02's read-after-write proof; body text search belongs to the HVI) |
+| `ticket_vec_idx` | Vector IVF (NEIGHBOR PARTITIONS, COSINE) | Approximate nearest-neighbor on 8-dim demo embeddings |
+| `tickets_bodyvec_ivf` | Vector IVF (NEIGHBOR PARTITIONS, COSINE) | ANN on the real 384-dim `body_vec` embeddings (module 03) |
+| `tickets_hvi` | Hybrid Vector Index (Text + vector, one domain index) | Fused keyword + semantic search on ticket bodies (module 03) |
 
 **Note on embeddings:** The 8-dim vectors are deterministic demo values generated by the LCG seed. All engine behaviors demonstrated (distance functions, vector indexes, transactions, cross-model joins) are dimension-independent. Module 03 documents the optional real-model flow using `DBMS_VECTOR.LOAD_ONNX_MODEL` with all-MiniLM-L12-v2.
 
@@ -95,7 +97,7 @@ All modules run against one deterministically seeded domain. Seeds use an LCG (s
 |---|------|--------|--------|------------|-------|
 | 01 | `what-is-a-converged-database` | Live | 5 | 20 | [README](modules/01-what-is-a-converged-database/README.md) · [article](https://blogs.oracle.com/developers/what-is-a-converged-database-definition-five-tests-and-ai-use-cases) |
 | 02 | `converged-vs-multi-model` | Live | 3 | 18 | [README](modules/02-converged-vs-multi-model/README.md) · [article](https://blogs.oracle.com/developers/converged-database-vs-multi-model-database-whats-the-difference) |
-| 03 | `vector-vs-converged` | On branch, CI-green — article in draft | 4 | 19 | [README](https://github.com/oracle-devrel/oracle-umt-developer-hub/blob/article/03-vector-vs-converged/labs/converged-database-lab/modules/03-vector-vs-converged/README.md) · [branch](https://github.com/oracle-devrel/oracle-umt-developer-hub/tree/article/03-vector-vs-converged) · article at publish |
+| 03 | `vector-vs-converged` | On main, validated — article in draft | 4 | 23 | [README](modules/03-vector-vs-converged/README.md) · [branch](https://github.com/oracle-devrel/oracle-umt-developer-hub/tree/article/03-vector-vs-converged) · article at publish |
 | 04 | `ai-agents-enterprise-data` | On branch, CI-green — article in draft | 3 | 16 | [README](https://github.com/oracle-devrel/oracle-umt-developer-hub/blob/article/04-ai-agents-enterprise-data/labs/converged-database-lab/modules/04-ai-agents-enterprise-data/README.md) · [branch](https://github.com/oracle-devrel/oracle-umt-developer-hub/tree/article/04-ai-agents-enterprise-data) · article at publish |
 | 05 | `json-graph-vector-relational` | On branch, CI-green — article in draft | 4 | 26 | [README](https://github.com/oracle-devrel/oracle-umt-developer-hub/blob/article/05-json-graph-vector-relational/labs/converged-database-lab/modules/05-json-graph-vector-relational/README.md) · [branch](https://github.com/oracle-devrel/oracle-umt-developer-hub/tree/article/05-json-graph-vector-relational) · article at publish |
 | 06 | `secure-agent-access` | On branch, CI-green — article in draft | 4 | 7 | [README](https://github.com/oracle-devrel/oracle-umt-developer-hub/blob/article/06-secure-agent-access/labs/converged-database-lab/modules/06-secure-agent-access/README.md) · [branch](https://github.com/oracle-devrel/oracle-umt-developer-hub/tree/article/06-secure-agent-access) · article at publish |

labs/converged-database-lab/docker-compose.yml

@@ -17,7 +17,10 @@ services:
       test: ['CMD-SHELL', '/opt/oracle/healthcheck.sh && curl -so /dev/null -w "%{http_code}" http://localhost:8181/ | grep -q "302\|200"']
       interval: 15s
       timeout: 10s
-      retries: 40
+      # 80 retries: first boot now embeds 10,000 ticket bodies in-database and
+      # builds the HVI over them (~7-9 min of init on top of DB create), so the
+      # container must not be declared unhealthy before init can finish.
+      retries: 80
       start_period: 120s
     deploy:
       resources:

labs/converged-database-lab/docker/Dockerfile.oracle

@@ -9,10 +9,34 @@ RUN chown -R oracle:oinstall /etc/ords
 
 # mongosh: used to verify and exercise the MongoDB API (port 27017) from inside
 # the container — the validator harness runs *.js scripts through it.
-RUN printf '[mongodb-org-8.0]\nname=MongoDB Repository\nbaseurl=https://repo.mongodb.org/yum/redhat/8/mongodb-org/8.0/x86_64/\ngpgcheck=1\nenabled=1\ngpgkey=https://pgp.mongodb.com/server-8.0.asc\n' > /etc/yum.repos.d/mongodb-org-8.0.repo \
+# Arch-aware repo URL: CI builds on x86_64; local builds on Apple-silicon (aarch64)
+# podman machines get MongoDB's aarch64 el8 packages instead of failing dependency
+# resolution against the x86_64 repo.
+RUN printf '[mongodb-org-8.0]\nname=MongoDB Repository\nbaseurl=https://repo.mongodb.org/yum/redhat/8/mongodb-org/8.0/%s/\ngpgcheck=1\nenabled=1\ngpgkey=https://pgp.mongodb.com/server-8.0.asc\n' "$(uname -m)" > /etc/yum.repos.d/mongodb-org-8.0.repo \
  && microdnf install -y mongodb-mongosh \
  && microdnf clean all
 
+# Bake Oracle's prebuilt augmented all-MiniLM-L12-v2 ONNX model into the image so
+# every fresh build (and CI) loads identical in-DB embeddings deterministically —
+# no network dependency at init time. This is the official model published by
+# Oracle Machine Learning: the 26ai docs "ONNX Pipeline Models: Text Embedding"
+# page redirects (/pls/topic/lookup?...&id=oml_ai_models_object_storage) to the
+# OML-ai-models object-storage bucket whose index lists this exact URL. The zip
+# (~122.6 MB) unpacks to all_MiniLM_L12_v2.onnx (~133.3 MB, 384-dim output),
+# loaded by docker/init/08-vector-model.sql via DBMS_VECTOR.LOAD_ONNX_MODEL.
+# Placed AFTER the ORDS + mongosh layers so their (slow) cache layers are reused
+# when only the model layer changes. Fails the build loudly if the URL stops
+# resolving rather than baking a broken image.
+ARG MINILM_URL=https://adwc4pm.objectstorage.us-ashburn-1.oci.customer-oci.com/p/TtH6hL2y25EypZ0-rrczRZ1aXp7v1ONbRBfCiT-BDBN8WLKQ3lgyW6RxCfIFLdA6/n/adwc4pm/b/OML-ai-models/o/all_MiniLM_L12_v2_augmented.zip
+RUN microdnf install -y unzip \
+ && microdnf clean all \
+ && mkdir -p /opt/oracle/models \
+ && curl -fsSL -o /tmp/minilm.zip "$MINILM_URL" \
+ && unzip -o /tmp/minilm.zip -d /opt/oracle/models \
+ && test -s /opt/oracle/models/all_MiniLM_L12_v2.onnx \
+ && rm -f /tmp/minilm.zip \
+ && chown -R oracle:oinstall /opt/oracle/models
+
 COPY init/ /container-entrypoint-initdb.d/
 
 COPY scripts/entrypoint.sh /opt/oracle/scripts/entrypoint.sh

labs/converged-database-lab/docker/init/05-text-vector.sql

@@ -1,14 +1,23 @@
 ALTER SESSION SET CONTAINER = FREEPDB1;
 ALTER SESSION SET CURRENT_SCHEMA = lab_user;
 
--- Oracle Text index on ticket bodies: keyword search in the same engine/transaction.
--- SYNC (ON COMMIT): the index syncs inside the committing transaction, so a
--- committed row is immediately findable via CONTAINS — transactional
--- read-after-write search (module 02 proves this). The 26ai Free default for
--- CREATE SEARCH INDEX is MAINTENANCE AUTO with deferred background sync
--- (ctx_user_indexes showed IDX_SYNC_TYPE = MANUAL), under which a probe row was
--- NOT visible to CONTAINS immediately after COMMIT.
-CREATE SEARCH INDEX ticket_text_idx ON support_tickets (body)
+-- Oracle Text index on ticket SUBJECTS: keyword search in the same
+-- engine/transaction. SYNC (ON COMMIT): the index syncs inside the committing
+-- transaction, so a committed row is immediately findable via CONTAINS —
+-- transactional read-after-write search (module 02 proves this). The 26ai Free
+-- default for CREATE SEARCH INDEX is MAINTENANCE AUTO with deferred background
+-- sync (ctx_user_indexes showed IDX_SYNC_TYPE = MANUAL), under which a probe
+-- row was NOT visible to CONTAINS immediately after COMMIT.
+--
+-- MERGE RECONCILIATION (modules 02 + 03 together on main): this index lives on
+-- SUBJECT here, not BODY. Module 03's Hybrid Vector Index owns body text search
+-- (an HVI's text component is the same indextype as a SEARCH index — two cannot
+-- share a column, ORA-29879), and the HVI's text component does NOT sync on
+-- commit (verified empirically 2026-07-24: a committed probe row was not
+-- CONTAINS-findable through the HVI). Module 02's read-after-write proof
+-- therefore probes the subject column on main; the article/02 BRANCH keeps the
+-- body form that the published article's snippets byte-match.
+CREATE SEARCH INDEX ticket_text_idx ON support_tickets (subject)
   PARAMETERS ('SYNC (ON COMMIT)');
 
 -- Vector index. IVF (NEIGHBOR PARTITIONS) — works within Free-tier memory without

labs/converged-database-lab/docker/init/06-seed.sql

@@ -4,6 +4,13 @@ ALTER SESSION SET CURRENT_SCHEMA = lab_user;
 DECLARE
   -- All item declarations must precede subprogram bodies (PLS-00103 otherwise).
   seed    NUMBER := 42;
+  -- Ticket volume. 10,000 tickets is deliberately past the point where the
+  -- cost-based optimizer chooses the IVF vector index over exact full-scan
+  -- distance for module 03's filtered FETCH APPROX query (the crossover
+  -- arrives by roughly 3,000 rows on the Free container with fresh stats) —
+  -- so the captured plan shows the index + in-plan predicate evaluation, not
+  -- a toy-size full scan.
+  c_tickets CONSTANT PLS_INTEGER := 10000;
   v_vec   VARCHAR2(400);
   v_norm  NUMBER;
   TYPE t_f IS TABLE OF NUMBER INDEX BY PLS_INTEGER;
@@ -18,13 +25,71 @@ DECLARE
   v_prod  PLS_INTEGER;
   v_qty   PLS_INTEGER;
   v_ts    TIMESTAMP;
+  v_cat   PLS_INTEGER;
+  v_body  VARCHAR2(2000);
+  -- Enriched ticket-body corpus for tickets 301..c_tickets: 6 intros x 8 issue
+  -- categories x 4 details x 50 products x 5 closers ~= 48k distinct bodies, so
+  -- the 384-dim embeddings form real semantic clusters instead of 4 duplicated
+  -- points (matters for a meaningful IVF index and hybrid-search demo).
+  TYPE t_v IS TABLE OF VARCHAR2(400);
+  intro   t_v := t_v('Customer reports an issue with their recent experience.',
+                     'Contacting support after several attempts to resolve this alone.',
+                     'Opening a ticket on behalf of the account holder.',
+                     'Follow-up on an earlier conversation with the support team.',
+                     'New request submitted through the help center.',
+                     'Escalated from chat: the problem is still not resolved.');
+  refund  t_v := t_v('The package arrived damaged and a refund is requested.',
+                     'The box was crushed in transit and the item inside is damaged beyond use.',
+                     'Item arrived with a cracked casing; customer asks for their money back.',
+                     'Shipment was left in the rain and the contents are damaged and unusable.');
+  login   t_v := t_v('Login fails with an authentication error after reset.',
+                     'Password reset completed but sign-in still rejects the new credentials.',
+                     'Two-factor prompt never arrives and access to the account is blocked.',
+                     'Account is locked out after the security update and recovery does not work.');
+  shipdl  t_v := t_v('The order has not arrived within the promised window.',
+                     'Tracking has shown no movement for a week and the delivery date passed.',
+                     'Carrier marked the parcel delivered but nothing has arrived.',
+                     'Estimated delivery keeps slipping and no update has been provided.');
+  warr    t_v := t_v('Customer asks whether the warranty covers accidental damage.',
+                     'Requesting warranty service for a unit that stopped powering on.',
+                     'Screen developed dead pixels within the coverage period.',
+                     'Asking how to start a warranty claim for a failed component.');
+  billing t_v := t_v('The statement shows a duplicate charge for a single order.',
+                     'A promotional discount was not applied at checkout.',
+                     'The invoice total does not match the order confirmation email.',
+                     'Customer was billed after cancelling the subscription.');
+  exch    t_v := t_v('Requesting an exchange for a different size of the same product.',
+                     'The wrong color variant was delivered and a swap is requested.',
+                     'Received a different model than the one ordered.',
+                     'Asking to exchange a gift for store credit instead.');
+  install t_v := t_v('The setup guide does not match the ports on the device.',
+                     'Firmware update fails at forty percent every attempt.',
+                     'The companion app cannot discover the device during pairing.',
+                     'Installation completes but the device reboots continuously.');
+  perf    t_v := t_v('The device becomes very slow after a few hours of use.',
+                     'Battery drains overnight even when the unit is idle.',
+                     'Audio cuts out intermittently during playback.',
+                     'The connection drops whenever a second device joins.');
+  closing t_v := t_v('Customer asks for a response today.',
+                     'This is the second time the issue has been reported.',
+                     'The customer mentions they are a long-time subscriber.',
+                     'A callback number has been left on the account.',
+                     'No workaround has been found so far.');
   FUNCTION nxt RETURN NUMBER IS  -- LCG: deterministic across runs
   BEGIN
     seed := MOD(seed * 1103515245 + 12345, 2147483648);
     RETURN seed / 2147483648;
   END;
   FUNCTION pick(n IN PLS_INTEGER) RETURN PLS_INTEGER IS
   BEGIN RETURN TRUNC(nxt * n) + 1; END;
+  FUNCTION cat_detail(p_cat PLS_INTEGER) RETURN VARCHAR2 IS
+  BEGIN
+    RETURN CASE p_cat
+      WHEN 1 THEN refund (pick(4)) WHEN 2 THEN login(pick(4))
+      WHEN 3 THEN shipdl (pick(4)) WHEN 4 THEN warr (pick(4))
+      WHEN 5 THEN billing(pick(4)) WHEN 6 THEN exch (pick(4))
+      WHEN 7 THEN install(pick(4)) ELSE perf(pick(4)) END;
+  END;
 BEGIN
   FOR i IN 1..200 LOOP
     INSERT INTO customers (email, full_name, segment)
@@ -95,6 +160,9 @@ BEGIN
     UPDATE orders SET total_amount = ROUND(v_total,2) WHERE order_id = i;
   END LOOP;
 
+  -- Tickets 1..300: the ORIGINAL 4-template corpus, byte-for-byte (module 01's
+  -- ticket #1 proofs and module 03's read-after-write probe depend on ticket #1
+  -- keeping its "Login fails with an authentication error after reset." body).
   FOR i IN 1..300 LOOP
     v_norm := 0;
     FOR d IN 1..8 LOOP f(d) := nxt*2 - 1; v_norm := v_norm + f(d)*f(d); END LOOP;
@@ -119,6 +187,38 @@ BEGIN
             CASE WHEN MOD(i,3)=0 THEN 'closed' WHEN MOD(i,7)=0 THEN 'pending' ELSE 'open' END,
             TO_VECTOR(v_vec, 8, FLOAT32));
   END LOOP;
+
+  -- Tickets 301..c_tickets: the enriched scaled corpus. The LCG is re-seeded to
+  -- a fixed value here so this block is deterministic regardless of how the
+  -- upstream seeding evolves — same bodies on every fresh build.
+  seed := 342;
+  FOR i IN 301..c_tickets LOOP
+    v_cat  := pick(8);
+    v_cust := pick(200);
+    v_body := intro(pick(6))||' '||cat_detail(v_cat)||
+              ' Relates to Product '||pick(50)||'. '||closing(pick(5));
+    v_norm := 0;
+    FOR d IN 1..8 LOOP f(d) := nxt*2 - 1; v_norm := v_norm + f(d)*f(d); END LOOP;
+    v_norm := SQRT(v_norm);
+    v_vec := '[';
+    FOR d IN 1..8 LOOP
+      v_vec := v_vec || TO_CHAR(ROUND(f(d)/v_norm, 6),'FM990.999999') || CASE WHEN d<8 THEN ',' END;
+    END LOOP;
+    v_vec := v_vec || ']';
+    INSERT INTO support_tickets (customer_id, subject, body, status, embedding)
+    VALUES (v_cust,
+            CASE v_cat WHEN 1 THEN 'Refund request for damaged item'
+                       WHEN 2 THEN 'Cannot sign in after password reset'
+                       WHEN 3 THEN 'Shipping delay on recent order'
+                       WHEN 4 THEN 'Warranty question for product'
+                       WHEN 5 THEN 'Billing discrepancy on statement'
+                       WHEN 6 THEN 'Exchange request for recent order'
+                       WHEN 7 THEN 'Device setup and installation issue'
+                       ELSE 'Performance problem with product' END || ' #'||i,
+            v_body,
+            CASE WHEN MOD(i,3)=0 THEN 'closed' WHEN MOD(i,7)=0 THEN 'pending' ELSE 'open' END,
+            TO_VECTOR(v_vec, 8, FLOAT32));
+  END LOOP;
   COMMIT;
 END;
 /