Skip to content

feat: native Elasticsearch vector search support#27111

Open
joaopamaral wants to merge 44 commits intoopen-metadata:mainfrom
Automattic:feat/es-elasticsearch-vector-search-upstream
Open

feat: native Elasticsearch vector search support#27111
joaopamaral wants to merge 44 commits intoopen-metadata:mainfrom
Automattic:feat/es-elasticsearch-vector-search-upstream

Conversation

@joaopamaral
Copy link
Copy Markdown

@joaopamaral joaopamaral commented Apr 7, 2026

Summary

Adds native Elasticsearch 9.x vector search support, mirroring the existing OpenSearch implementation. OpenMetadata deployments backed by Elasticsearch 9.x can now use the same semantic/vector search features as OpenSearch deployments.

Architecture: ES vs OpenSearch Vector Search

                    ┌─────────────────────────────────────────────────────────────┐
                    │                   VectorIndexService (interface)             │
                    │  search() · updateEntityEmbedding() · getExistingFingerprint │
                    │  generateEmbeddingFields()                                   │
                    └───────────────┬───────────────────────┬───────────────────┘
                                    │                       │
               ┌────────────────────▼──────┐   ┌───────────▼──────────────────┐
               │  OpenSearchVectorService  │   │  ElasticSearchVectorService   │
               │  (existing, unchanged)    │   │  (new)                        │
               ├───────────────────────────┤   ├──────────────────────────────┤
               │ Transport: generic client  │   │ Transport: Rest5Client        │
               │ Query: query.knn (nested)  │   │ Query: top-level knn object   │
               │ Index field: knn_vector    │   │ Index field: dense_vector     │
               │                           │   │                               │
               │ Hybrid pipeline: ✅        │   │ Hybrid pipeline: ✗ (ES        │
               │  ensureHybridSearchPipeline│   │  doesn't support RRF pipeline │
               │  (OpenSearch RRF)          │   │  via search pipeline API)     │
               └───────────────────────────┘   └──────────────────────────────┘
                          │                                  │
               ┌──────────▼──────────┐           ┌──────────▼──────────┐
               │   OpenSearch index  │           │ Elasticsearch index  │
               │                     │           │                      │
               │ embedding:          │           │ embedding:           │
               │   type: knn_vector  │           │   type: dense_vector │
               │   dimension: N      │           │   dims: N            │
               │   method:           │           │   similarity: cosine │
               │     name: hnsw      │           │   (ES 9.x BBQ-HNSW)  │
               │     engine: lucene  │           │                      │
               │     space: cosine   │           │                      │
               └─────────────────────┘           └──────────────────────┘

Query format difference

OpenSearch (query.knn — nested under "query")   Elasticsearch (top-level knn)
─────────────────────────────────────────────   ─────────────────────────────
{                                               {
  "query": {                                      "knn": {
    "knn": {                                        "field": "embedding",
      "embedding": {                                "query_vector": [...],
        "vector": [...],                            "k": 10,
        "k": 10,                                    "num_candidates": 100,
        "filter": { ... }                           "filter": { ... }
      }                                           }
    }                                           }
  }
}

Ingestion flow

Entity created/updated
        │
        ▼
VectorEmbeddingHandler (lifecycle event)
        │
        ▼
VectorDocBuilder.buildEmbeddingFields()
  · textToLLMContext (concatenated text)
  · fingerprint (SHA-256, change detection)
  · embedding (float[] from EmbeddingClient)
  · parentId, chunkIndex, chunkCount
        │
        ├──── OpenSearch ──▶ OpenSearchVectorService.partialUpdateEntity()
        │                      POST /{index}/_update/{id}  (generic client)
        │
        └──── Elasticsearch ▶ ElasticSearchVectorService.partialUpdateEntity()
                               POST /{index}/_update/{id}  (Rest5Client)

Changes

New

  • ElasticSearchVectorService: ES implementation of VectorIndexService. Uses Rest5Client for all HTTP calls (same approach as the rest of the ES client layer). Mirrors OpenSearchVectorService for pagination, fingerprint deduplication, and embedding lifecycle.
  • vector_search_index_es_native.json (en/jp/ru/zh): ES-native index mappings using dense_vector / dims / cosine similarity (ES 9.x format).
  • SemanticSearchQueryBuilder (ES package): mirrors the OpenSearch equivalent.
  • VectorSearchQueryBuilder.buildNativeESQuery(): emits the ES 9.x top-level knn format with configurable num_candidates.

Modified

  • ElasticSearchIndexManager.extractMappingsJson(): extracts the mappings sub-object before putMapping — ES rejects the full index JSON (with settings/aliases) at the mappings endpoint.
  • EsUtils.enrichIndexMappingForElasticsearch(): injects dense_vector field into ES index mappings when vector search is enabled.
  • SearchRepository: wires ElasticSearchVectorService for the ELASTICSEARCH search type; applies enrichIndexMappingForElasticsearch to index templates (OpenSearch path unchanged).
  • ElasticSearchBulkSink: calls vectorIndexService.updateEntityEmbedding() on write, same as OpenSearchBulkSink.
  • VectorSearchQueryBuilder: reformatVectorIndexWithDimension() handles both "dims" (ES) and "dimension" (OpenSearch).

Bug fixes (post-review)

  • appendKnnQuery closing braces: }}}}}}} — the extra brace produced malformed JSON, causing OpenSearch to return 400 on every KNN request (silently yielding empty results).
  • executeGenericRequest HTTP status check: now throws on 4xx/5xx instead of silently returning {}.
  • extractRestClient type guard: replaces unchecked cast with instanceof pattern match; throws IllegalArgumentException with the actual transport class name on mismatch.
  • knnNumCandidatesMultiplier: wired from naturalLanguageSearch.knnNumCandidatesMultiplier config; defaults to 2×.
  • getFingerprint endpoint: gated to admin users.
  • Index template enrichment: createOrUpdateIndexTemplates() and createOrUpdateIndexTemplate() now call enrichIndexMappingForElasticsearch() so templates include dense_vector when vector search is enabled.

Review-feedback fixes (commit 9854029a)

  • SystemRepository.getEmbeddingsValidation(): removed the ES short-circuit. Elasticsearch now flows through the same checks (embedding generation + hybrid pipeline) as OpenSearch instead of always failing with the legacy "not supported" message.
  • EsUtils.addDenseVectorSettings(): detects _meta.embedding_dimension drift vs. the active embedding client and emits a WARN log, mirroring OsUtils.addKnnVectorSettings. Helps surface stale indexes after a model switch (e.g. DJL 384 → OpenAI 1536).
  • ElasticSearchBulkSink.isVectorEmbeddingEnabledForEntity: also gates on searchRepository.getIndexMapping(entityType) != null to skip entities whose mapping isn't loaded.
  • ElasticSearchBulkSink.fetchExistingFingerprints: takes a ReindexContext and routes to the staged index when one exists. Without this the fingerprint dedup reads the canonical (old) index during a staged reindex and re-processes everything it was supposed to skip.
  • New unit tests for each of the above (in EsUtilsTest, ElasticSearchBulkSinkBehaviorTest, SystemRepositoryEmbeddingsValidationTest).

Test plan

  • mvn test -pl openmetadata-service -Dtest=VectorSearchQueryBuilderTest,ElasticSearchIndexManagerTest,ElasticSearchVectorServiceTest,SearchRepositoryBehaviorTest,EsUtilsTest,ElasticSearchBulkSinkBehaviorTest,SystemRepositoryEmbeddingsValidationTest
  • mvn test -pl openmetadata-integration-tests -Dtest=VectorEmbeddingIntegrationIT (uses Testcontainers OpenSearch 3.4.0)
  • Local end-to-end on ES 9.3.0 + DJL all-MiniLM-L6-v2: top-1 accuracy 3/3 across distinct semantic queries (revenue from salescustomer_purchases 0.76; login authenticationuser_logins 0.71; temperature humidityweather_data 0.69). Index has dense_vector{dims:384, similarity:cosine, BBQ-HNSW}. See docs/elasticsearch-semantic-search-local-test.md.
  • No regression on OpenSearch 3.4.0: same test re-run with SEARCH_TYPE=opensearch. OpenSearchVectorService initializes, hybrid-rrf pipeline registered, knn_vector mapping created, semantic queries return identical scores to ES (same embedding model → deterministic cosine ranking).
  • End-to-end re-verified after the review-feedback fixes on both ES 9.3.0 and OpenSearch 3.4.0 — identical scores to the original run, no regression.

References

🤖 Generated with Claude Code


Summary by Gitar

  • Vector search configuration:
    • Refactored VectorSearchQueryBuilder to clean up clamping logic for num_candidates to ensure robustness against integer overflow.
  • ElasticSearch integration:
    • Removed redundant interface constants in ElasticSearchVectorService to streamline the service implementation.

This will update automatically on new commits.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 7, 2026

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@joaopamaral
Copy link
Copy Markdown
Author

joaopamaral commented Apr 7, 2026

Initial results look good, but I've run a test only with ES 9.x and version 1.12.4 (not the one from main). I also need to double-check if OpenSearch is affected by this change. Also need to review some AI-resolved conflicts from version 1.12.4 with main.

@harshach
Copy link
Copy Markdown
Collaborator

harshach commented Apr 7, 2026

Thanks @joaopamaral this is great!!. Can you make it ready for review? and also address comments here #27111 (comment)

@joaopamaral
Copy link
Copy Markdown
Author

Sure @harshach! I'll work on the bot review first before making it ready for review! 👍

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 8, 2026

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

2 similar comments
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 8, 2026

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@github-actions
Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@github-actions
Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

5 similar comments
@github-actions
Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@github-actions
Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@github-actions
Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@github-actions
Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@github-actions
Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@joaopamaral joaopamaral marked this pull request as ready for review April 10, 2026 19:41
Copilot AI review requested due to automatic review settings April 10, 2026 19:41
@joaopamaral
Copy link
Copy Markdown
Author

Hi @harshach, ’ve addressed the bot review, but I still need to re-review the code after rebasing/merging with main and rerun the tests against a real server. So far, I’ve tested this PR with version 1.12.4 and ES 9.3.1. I still need to validate that everything continues to work correctly with OpenSearch and ES 8.x.

I won’t be able to run tests for the next couple of days, but feel free to proceed with any testing on your side in the meantime.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds native Elasticsearch (8.x/9.x) vector search support to OpenMetadata, aiming to provide semantic/vector search capabilities on Elasticsearch deployments comparable to the existing OpenSearch implementation.

Changes:

  • Added a new ElasticSearchVectorService plus wiring in SearchRepository / ElasticSearchBulkSink to initialize and use it when Elasticsearch is the configured backend.
  • Introduced ES-native vector index mapping templates (vector_search_index_es_native.json) and extended query-building to emit Elasticsearch’s top-level knn query format.
  • Added/updated tests around the ES-native query format and Elasticsearch vector service behavior.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 10 comments.

Show a summary per file
File Description
openmetadata-spec/src/main/resources/json/schema/search/searchRequest.json Adds semanticSearch flag to the search request schema.
openmetadata-spec/src/main/resources/elasticsearch/en/vector_search_index_es_native.json New ES-native vector index template (en).
openmetadata-spec/src/main/resources/elasticsearch/jp/vector_search_index_es_native.json New ES-native vector index template (jp).
openmetadata-spec/src/main/resources/elasticsearch/ru/vector_search_index_es_native.json New ES-native vector index template (ru).
openmetadata-spec/src/main/resources/elasticsearch/zh/vector_search_index_es_native.json New ES-native vector index template (zh).
openmetadata-service/src/main/java/org/openmetadata/service/search/vector/VectorSearchQueryBuilder.java Adds buildNativeESQuery and refactors filter emission for vector search queries.
openmetadata-service/src/test/java/org/openmetadata/service/search/vector/VectorSearchQueryBuilderTest.java Adds coverage for ES-native top-level knn query structure and filter behavior.
openmetadata-service/src/main/java/org/openmetadata/service/search/vector/VectorIndexService.java Extends vector service interface and adds an alias helper.
openmetadata-service/src/main/java/org/openmetadata/service/search/vector/OpenSearchVectorService.java Adjusts to use the new interface default alias method and annotates overrides.
openmetadata-service/src/main/java/org/openmetadata/service/search/vector/ElasticSearchVectorService.java New Elasticsearch vector service implementation using Rest5Client for generic requests.
openmetadata-service/src/test/java/org/openmetadata/service/search/vector/ElasticSearchVectorServiceTest.java New tests for ES vector service result parsing, grouping, and dimension patching.
openmetadata-service/src/main/java/org/openmetadata/service/search/SearchRepository.java Initializes ES vector service when Elasticsearch backend is configured; mapping selection tweaks for ES-native template.
openmetadata-service/src/main/java/org/openmetadata/service/search/RecreateWithEmbeddings.java Attempts to include a vector “entity” key in recreate flow when vector search is enabled.
openmetadata-service/src/main/java/org/openmetadata/service/search/elasticsearch/SemanticSearchQueryBuilder.java New builder for semantic/hybrid query composition on Elasticsearch.
openmetadata-service/src/main/java/org/openmetadata/service/search/elasticsearch/ElasticSearchIndexManager.java Extracts mappings sub-object before calling putMapping.
openmetadata-service/src/test/java/org/openmetadata/service/search/elasticsearch/ElasticSearchIndexManagerTest.java Adds a test asserting updateIndex handles full index JSON by extracting mappings.
openmetadata-service/src/main/java/org/openmetadata/service/resources/search/VectorSearchResource.java Switches to repository-provided VectorIndexService and adds a fingerprint endpoint.
openmetadata-service/src/main/java/org/openmetadata/service/apps/bundles/searchIndex/ElasticSearchBulkSink.java Adds async vector-embedding task execution + migration path for ES indexing jobs.
openmetadata-service/src/test/java/org/openmetadata/service/apps/bundles/searchIndex/ElasticSearchBulkSinkSimpleTest.java Adds minimal coverage for vector-embedding helpers on the ES sink.
openmetadata-mcp/src/main/java/org/openmetadata/mcp/tools/SemanticSearchTool.java Uses repository VectorIndexService rather than OpenSearch-only implementation.

@joaopamaral
Copy link
Copy Markdown
Author

Also need to review all after this refactor #26000 😢

@harshach harshach added the safe to test Add this label to run secure Github workflows on PRs label Apr 22, 2026
@harshach
Copy link
Copy Markdown
Collaborator

@joaopamaral thanks for your work on this, can you check the co-pilot comments and address the merge conflict here please

@github-actions
Copy link
Copy Markdown
Contributor

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

@github-actions
Copy link
Copy Markdown
Contributor

⚠️ TypeScript Types Need Update

The generated TypeScript types are out of sync with the JSON schema changes.

Since this is a pull request from a forked repository, the types cannot be automatically committed.
Please generate and commit the types manually:

cd openmetadata-ui/src/main/resources/ui
./json2ts-generate-all.sh -l true
git add src/generated/
git commit -m "Update generated TypeScript types"
git push

After pushing the changes, this check will pass automatically.

… interface constant

EsUtils.addDenseVectorSettings:
- Also check existing properties.embedding.dims, not just
  _meta.embedding_dimension. Either source can carry an immutable
  dimension; rewriting either via putMapping is rejected by ES.
- Both checks delegate to a shared assertDimensionMatches helper that
  throws IllegalStateException with a clear "drop+reindex" message
  before any mapping rewrite.
- Preserve existing _meta fields when upserting embedding_model /
  embedding_dimension; previous putObject call clobbered siblings.

ElasticSearchVectorService:
- Drop the language field and its constructor / init parameters. It was
  stored and normalized but never read anywhere; OpenSearchVectorService
  has no equivalent.

VectorIndexService:
- Remove unused VECTOR_INDEX_KEY constant — never referenced.

Tests updated for new ctor / init signatures + spotless formatting.

Addresses PR review comments r3170730416, r3170730465, r3182117105,
r3182117165, r3182241460, r3182241528.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 22 out of 24 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

openmetadata-service/src/main/java/org/openmetadata/service/search/SearchRepository.java:393

  • createOrUpdateIndexTemplates() catches all exceptions from mapping enrichment (including IllegalStateException from EsUtils.enrichIndexMappingForElasticsearch on embedding dimension drift) and only logs e.getMessage() at WARN. This effectively downgrades a critical misconfiguration to a soft failure and also drops the stack trace, making the root cause harder to diagnose. Consider either (a) logging the full exception (pass e to the logger) and/or (b) rethrowing/propagating dimension-mismatch failures so startup/reindex clearly fails and operators are forced to reindex as intended.
  public void createOrUpdateIndexTemplates() {
    LOG.info("Creating/updating index templates for all entities...");
    int success = 0;
    int failed = 0;
    for (Map.Entry<String, IndexMapping> entry : entityIndexMap.entrySet()) {
      try {
        IndexMapping indexMapping = entry.getValue();
        String indexName = indexMapping.getIndexName(clusterAlias);
        String templateName = "om_" + indexName;
        String indexPattern = indexName + "*";
        String mappingContent = enrichForElasticsearch(readIndexMapping(indexMapping));
        if (mappingContent != null) {
          searchClient.createOrUpdateIndexTemplate(templateName, indexPattern, mappingContent);
          success++;
        } else {
          failed++;
          LOG.warn("No mapping content found for entity type: {}", entry.getKey());
        }
      } catch (Exception e) {
        failed++;
        LOG.warn("Failed to create index template for {}: {}", entry.getKey(), e.getMessage());
      }

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 4, 2026

Jest test Coverage

UI tests summary

Lines Statements Branches Functions
Coverage: 62%
62.45% (62922/100755) 42.77% (33911/79285) 45.77% (10035/21924)

joaopamaral and others added 2 commits May 4, 2026 15:19
EsUtils.enrichIndexMappingForElasticsearch:
- Document the two failure modes in Javadoc:
  * IllegalArgumentException on null/empty input
  * IllegalStateException on embedding dimension drift
  Callers/operators were previously left to discover these the hard way.

SearchRepository.createOrUpdateIndexTemplates:
- Rethrow IllegalStateException (dimension mismatch) instead of swallowing
  it as a per-entity WARN. A broken vector setup must fail loudly so the
  operator runs an explicit reindex.
- For other Exceptions, pass the throwable to the logger so the stack
  trace is preserved (was logging only e.getMessage()).

Addresses PR review comments r3182610888 and reviewer follow-up on
createOrUpdateIndexTemplates exception handling.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 22 out of 24 changed files in this pull request and generated no new comments.

@joaopamaral
Copy link
Copy Markdown
Author

Copilot reviewed 22 out of 24 changed files in this pull request and generated no new comments.

@lautel I think Copilot is happy now 😄

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 4, 2026

Jest test Coverage

UI tests summary

Lines Statements Branches Functions
Coverage: 62%
62.45% (62922/100755) 42.77% (33911/79285) 45.77% (10035/21924)

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 4, 2026

Jest test Coverage

UI tests summary

Lines Statements Branches Functions
Coverage: 62%
62.45% (62922/100755) 42.77% (33911/79285) 45.77% (10035/21924)

Copy link
Copy Markdown
Contributor

@lautel lautel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's just one clean up to do and the PR should be good to get it in 🚀 Please check my comment to remove the dead code in ElasticSearchVectorService. Also please solve the merge conflict, and don't forget to run mvn spotless:apply before committing to apply the right formatting! Thanks!

…with OpenSearch

Reviewer flagged the manual `if (statusCode >= 400)` branch as unreachable,
asserting Rest5Client throws ResponseException on any non-2xx. Empirically
verified with a probe test that this is only true for 5xx — Rest5Client's
internal isCorrectServerResponse returns `code < 500`, so 4xx responses
are returned normally and would silently slip through if the manual check
is removed.

Final shape:
- 4xx: manual check throws IOException with same message format used by
  OpenSearchVectorService.
- 5xx: catch ResponseException, extract status + body, rethrow
  RuntimeException with the same "Elasticsearch request failed with
  status N: body" wording — symmetric to the OS path.

Comment in code documents the asymmetric Rest5Client behavior so future
readers don't repeat the same misread.

Addresses PR review comment in pullrequestreview-4226084412.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 22 out of 24 changed files in this pull request and generated 3 comments.

lautel
lautel previously approved these changes May 5, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 5, 2026

Jest test Coverage

UI tests summary

Lines Statements Branches Functions
Coverage: 62%
62.45% (62922/100755) 42.77% (33911/79285) 45.77% (10035/21924)

joaopamaral and others added 2 commits May 5, 2026 10:57
Resolve conflict in ElasticSearchBulkSinkSimpleTest.java: keep the new
testIsVectorEmbeddingEnabledForEntity test from this branch alongside the
new semaphoreTimeoutRecordsPermanentFailureWithoutIncrementingActiveRequests
test + helper methods landed on main.

Also include num_candidates int-overflow fix from copilot review:
- VectorSearchQueryBuilder.buildNativeESQuery: compute k * multiplier in
  long, clamp to Integer.MAX_VALUE so a configurable multiplier can
  never produce a negative or wrapped num_candidates.
- VectorSearchQueryBuilderTest: pin the clamp behavior.

Addresses copilot review pullrequestreview-4228238013 (overflow item).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 5, 2026

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 22 out of 24 changed files in this pull request and generated no new comments.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 5, 2026

Jest test Coverage

UI tests summary

Lines Statements Branches Functions
Coverage: 62%
62.42% (62939/100817) 42.75% (33918/79324) 45.75% (10035/21934)

joaopamaral and others added 3 commits May 5, 2026 11:35
…tity

ElasticSearchVectorService.collectSearchHits:
- When _source has no parentId we fall back to the document _id for
  grouping, but the fallback was never written into the result map.
  SemanticSearchTool.cleanHit pulls parentId via copyIfPresent — a
  missing key silently drops it from MCP responses. Now persist the
  fallback into hitMap so consumers always see a populated parentId.

ElasticSearchVectorService.executeGenericRequest:
- response.getEntity() can be null (some ES endpoints return no body
  on 4xx). The previous unconditional getEntity().getContent() NPE'd
  and masked the real HTTP status. Extract a readEntityBody helper
  that returns "" for a null/unreadable entity and use it on both
  the 4xx success-but-error path and the 5xx ResponseException path.
- Throw RuntimeException directly on 4xx (was IOException then
  wrapped) so the status + body message survives the outer catch.
- Add a "rethrow RuntimeException as-is" branch so a meaningful
  status-bearing message isn't double-wrapped into the generic
  "Elasticsearch generic request failed" string.

Tests:
- testParentIdFallbackToDocIdIsWrittenIntoResultMap: pins fallback
  write-back.
- testExecuteGenericRequestHandlesNullEntityWithout4xxNPE: pins null
  entity tolerance + status surfacing.

Addresses copilot review pullrequestreview-4228238013.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 5, 2026

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

@gitar-bot
Copy link
Copy Markdown

gitar-bot Bot commented May 5, 2026

Code Review 👍 Approved with suggestions 6 resolved / 7 findings

Native Elasticsearch vector search integration implemented with robust handling for index mappings and client transport safety. Address the minor fanout logic discrepancy where null targets currently return only staged indices.

💡 Edge Case: getWriteFanoutTargets(null) returns only staged indices

📄 openmetadata-service/src/main/java/org/openmetadata/service/search/SearchRepository.java:630-632

When aliasOrIndex is null, getWriteFanoutTargets returns new ArrayList<>(activeStagedIndices.values()) — i.e. only the staged indices without any base/alias target. If a caller passes null expecting an empty or no-op result, it will instead fan writes to all staged indices. This is unlikely to be hit in practice since callers resolve index names from mappings, but the null-handling semantics are surprising compared to getWriteIndexName(null) which returns null.

Suggested fix
if (aliasOrIndex == null) {
  return Collections.emptyList();
}
✅ 6 resolved
Bug: Test calls build() with 4 args but method requires 6 — won't compile

📄 openmetadata-service/src/test/java/org/openmetadata/service/search/vector/VectorSearchQueryBuilderTest.java:864 📄 openmetadata-service/src/main/java/org/openmetadata/service/search/vector/VectorSearchQueryBuilder.java:19-25
The test testNativeESQueryAndOpenSearchQueryProduceSameFilters at line 864 calls VectorSearchQueryBuilder.build(vector, 10, 100, filters) with 4 arguments, but the build() method signature requires 6 parameters: (float[] vector, int size, int from, int k, Map<String, List<String>> filters, double threshold). This will fail to compile. The intent appears to be comparing OpenSearch and ES native queries with the same filters.

Edge Case: loadIndexMapping dimension replacement is brittle — exact string match

📄 openmetadata-service/src/main/java/org/openmetadata/service/search/vector/ElasticSearchVectorService.java:523-527 📄 openmetadata-service/src/main/java/org/openmetadata/service/search/SearchRepository.java:3060-3068
In ElasticSearchVectorService.loadIndexMapping(), the dimension placeholder is replaced via exact string matching: template.replace(""dims": 512", ""dims": " + dimension). This requires the JSON template to have exactly one space after the colon. If the template is reformatted (e.g., minified, or extra spaces), the replacement silently fails. The code does have a post-check that throws if no replacement happened, but the same brittleness exists in SearchRepository.reformatVectorIndexWithDimension() fallback (lines 3062-3068) which does multiple .replace() calls for both with-space and without-space variants — showing awareness of the problem but an incomplete fix.

Edge Case: init() assigns instance before registerVectorEmbeddingHandler completes

📄 openmetadata-service/src/main/java/org/openmetadata/service/search/vector/ElasticSearchVectorService.java:65-79
In ElasticSearchVectorService.init() (line 70), instance is assigned the new service object, and then instance.registerVectorEmbeddingHandler() is called on line 71. Since getInstance() is not synchronized, a concurrent caller could observe the instance in a partially-initialized state (before the handler is registered). The volatile keyword ensures the reference is visible but doesn't guarantee that registerVectorEmbeddingHandler() has completed.

This mirrors the existing OpenSearch pattern, but the ES version is new code where it can be fixed.

Bug: ES search pagination is broken vs OpenSearch implementation

📄 openmetadata-service/src/main/java/org/openmetadata/service/search/vector/ElasticSearchVectorService.java:95-109 📄 openmetadata-service/src/main/java/org/openmetadata/service/search/vector/OpenSearchVectorService.java:196-210
The ElasticSearchVectorService.search() passes from directly to the ES kNN query, which skips raw hits (chunks). However, the API contract expects from to skip parent entities (grouped results). The OpenSearch implementation correctly handles this: it fetches from + size + 1 parents via a loop, then skips from parents in application code.

With the current ES implementation, requesting from=5, size=10 skips 5 raw chunks (not 5 parents), leading to incorrect pagination results. Additionally, the ES version uses the 2-arg VectorSearchResponse constructor, so totalHits and hasMore are always null — callers lose pagination metadata.

Quality: Unsafe downcast defeats purpose of VectorIndexService interface

📄 openmetadata-service/src/main/java/org/openmetadata/service/apps/bundles/searchIndex/ElasticSearchBulkSink.java:778
ElasticSearchBulkSink casts vectorService to (ElasticSearchVectorService) to call copyExistingVectorDocuments(). This couples the sink back to the concrete class, undermining the new interface abstraction. If vectorService is ever a different implementation, this throws ClassCastException at runtime.

...and 1 more resolved from earlier reviews

🤖 Prompt for agents
Code Review: Native Elasticsearch vector search integration implemented with robust handling for index mappings and client transport safety. Address the minor fanout logic discrepancy where null targets currently return only staged indices.

1. 💡 Edge Case: getWriteFanoutTargets(null) returns only staged indices
   Files: openmetadata-service/src/main/java/org/openmetadata/service/search/SearchRepository.java:630-632

   When `aliasOrIndex` is `null`, `getWriteFanoutTargets` returns `new ArrayList<>(activeStagedIndices.values())` — i.e. only the staged indices without any base/alias target. If a caller passes `null` expecting an empty or no-op result, it will instead fan writes to all staged indices. This is unlikely to be hit in practice since callers resolve index names from mappings, but the null-handling semantics are surprising compared to `getWriteIndexName(null)` which returns `null`.

   Suggested fix:
   if (aliasOrIndex == null) {
     return Collections.emptyList();
   }

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Gitar

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 22 out of 24 changed files in this pull request and generated no new comments.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 5, 2026

Jest test Coverage

UI tests summary

Lines Statements Branches Functions
Coverage: 62%
62.42% (62939/100817) 42.75% (33918/79324) 45.75% (10035/21934)

@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud Bot commented May 5, 2026

@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud Bot commented May 5, 2026

@joaopamaral
Copy link
Copy Markdown
Author

hey @lautel, the failing tests in CI don't seem related to this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

safe to test Add this label to run secure Github workflows on PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants