fix(engine): fix ES indexing cursor skipping expectations with same timestamp (#6490)#6491
Closed
Seb-MIGUEL wants to merge 6 commits into
Closed
fix(engine): fix ES indexing cursor skipping expectations with same timestamp (#6490)#6491Seb-MIGUEL wants to merge 6 commits into
Seb-MIGUEL wants to merge 6 commits into
Conversation
…6474) Add onError handler to connector logo <img> in ConnectorTitle.tsx. When a logo fails to load (404 or network error), display the HelpCenterOutlined placeholder icon instead of a broken image. Also reset error state when the logo URL changes.
Add ensureCatalogLogo() hook in IntegrationFactory, called when the catalog entry already exists in the database. This ensures logos lost from MinIO (e.g. after bucket reset or storage migration) are restored on the next application startup. Each factory overrides ensureCatalogLogo() with its logo upload call. insertCatalogEntry() delegates to ensureCatalogLogo() to avoid duplication.
…Logo - Revert frontend ConnectorTitle.tsx changes (backend fix is sufficient) - Extract getLogoFilename() in each factory to avoid duplicating the formatted string - Make ensureCatalogLogo() best-effort in initialise() (try-catch with log.warn) to prevent MinIO outages from blocking application startup
…ith same timestamp When multiple InjectExpectation entities are saved in the same batch (e.g. during expiration of many expectations), Hibernate sets the same @UpdateTimestamp millisecond for all of them. The existing polling query uses a strict > comparison on updated_at, so when the indexing batch size (500) cuts through a set of items sharing the same timestamp, the remaining items are permanently skipped. Fix: introduce a compound keyset cursor (updated_at, inject_expectation_id). - Add indexing_last_id column to indexing_status (V5_35 migration) - Add findForIndexingAfter query with compound cursor in repository - Add fetch(from, lastId, limit) default method to Handler interface; InjectExpectationHandler overrides it with the compound-cursor query - OpenSearchService and ElasticService track lastId per indexing cycle: when a full batch is returned, lastId is persisted so the next cycle continues from the exact item boundary; otherwise lastId is cleared This guarantees all expectations are eventually indexed regardless of how many share the same updated_at timestamp, fixing the inconsistency between the dashboard detection percentage (ES) and the inject detail page (DB).
Contributor
There was a problem hiding this comment.
Pull request overview
This PR aims to prevent Elasticsearch/OpenSearch indexing from permanently skipping InjectExpectation updates when more than one batch shares the same updated_at timestamp, by introducing a compound keyset cursor (updated_at, id) and persisting the secondary cursor (lastId) in indexing_status.
Changes:
- Add
indexing_last_idtoindexing_statusand expose it viaIndexingStatusto support compound cursor persistence. - Thread
lastIdthroughElasticService/OpenSearchServicebulk processing and use the newHandler.fetch(from, lastId, limit)API. - Add a new compound-cursor native query for inject expectation indexing and update
InjectExpectationHandlerto use it.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| openaev-model/src/main/java/io/openaev/service/OpenSearchService.java | Passes lastId into handler fetch and persists it in indexing status for full batches. |
| openaev-model/src/main/java/io/openaev/service/ElasticService.java | Mirrors OpenSearch changes: compound cursor threading + persistence. |
| openaev-model/src/main/java/io/openaev/engine/model/injectexpectation/InjectExpectationHandler.java | Adds compound-cursor fetch implementation for inject expectations (currently contains a blocking duplicate class declaration). |
| openaev-model/src/main/java/io/openaev/engine/Handler.java | Adds backward-compatible default fetch(from, lastId, limit) method for handlers. |
| openaev-model/src/main/java/io/openaev/database/repository/InjectExpectationRepository.java | Introduces findForIndexingAfter(from, lastId, limit) with compound cursor ordering. |
| openaev-model/src/main/java/io/openaev/database/model/IndexingStatus.java | Adds lastId field mapped to indexing_last_id. |
| openaev-api/src/main/java/io/openaev/migration/V5_35__Add_indexing_last_id.java | Flyway migration adding indexing_last_id column to indexing_status. |
| openaev-api/src/main/java/io/openaev/integration/IntegrationFactory.java | Adds ensureCatalogLogo() hook and runs it best-effort when catalog entry already exists. |
| openaev-api/src/main/java/io/openaev/integration/impl/injectors/ovh/OvhInjectorIntegrationFactory.java | Refactors logo upload into ensureCatalogLogo() + shared filename helper. |
| openaev-api/src/main/java/io/openaev/integration/impl/injectors/opencti/OpenCTIInjectorIntegrationFactory.java | Refactors logo upload into ensureCatalogLogo() + shared filename helper. |
| openaev-api/src/main/java/io/openaev/integration/impl/executors/tanium/TaniumExecutorIntegrationFactory.java | Refactors logo upload into ensureCatalogLogo() + shared filename helper. |
| openaev-api/src/main/java/io/openaev/integration/impl/executors/sentinelone/SentinelOneExecutorIntegrationFactory.java | Refactors logo upload into ensureCatalogLogo() + shared filename helper. |
| openaev-api/src/main/java/io/openaev/integration/impl/executors/paloaltocortex/PaloAltoCortexExecutorIntegrationFactory.java | Refactors logo upload into ensureCatalogLogo() + shared filename helper. |
| openaev-api/src/main/java/io/openaev/integration/impl/executors/crowdstrike/CrowdStrikeExecutorIntegrationFactory.java | Refactors logo upload into ensureCatalogLogo() + shared filename helper. |
| openaev-api/src/main/java/io/openaev/integration/impl/executors/caldera/CalderaExecutorIntegrationFactory.java | Refactors logo upload into ensureCatalogLogo() + shared filename helper. |
| openaev-api/src/test/java/io/openaev/integration/local_fixtures/regular/TestIntegrationFactory.java | Updates test factory to use ensureCatalogLogo() and shared logo filename helper. |
| openaev-api/src/test/java/io/openaev/integration/local_fixtures/factory_throws/TestIntegrationFactoryInitThrows.java | Same refactor for the throwing factory test fixture. |
| .github/skills/review-code/SKILL.md | Updates review skill documentation with additional anti-pattern checks. |
Comment on lines
+19
to
+22
| @Service | ||
| @RequiredArgsConstructor | ||
| public class InjectExpectationHandler implements Handler<EsInjectExpectation> { | ||
|
|
Comment on lines
+392
to
+395
| SELECT ie.inject_expectation_id FROM injects_expectations ie | ||
| JOIN injects i ON i.inject_id = ie.inject_id | ||
| WHERE i.inject_updated_at > :from | ||
| UNION |
Comment on lines
+396
to
+399
| SELECT ie.inject_expectation_id FROM injects_expectations ie | ||
| JOIN injects i ON i.inject_id = ie.inject_id | ||
| JOIN injectors_contracts ic ON ic.injector_contract_id = i.inject_injector_contract | ||
| WHERE ic.injector_contract_updated_at > :from |
Comment on lines
+28
to
+33
| /** | ||
| * Ensures the catalog logo exists in MinIO. Called on every startup (best-effort), even when the | ||
| * catalog entry already exists in the database. Override in subclasses that upload a logo during | ||
| * {@link #insertCatalogEntry()}. | ||
| * | ||
| * <p>Default implementation is a no-op for factories that do not manage a catalog logo (e.g. |
Contributor
Author
|
Closing: la branche incluait les commits de PR #6475 (InjectorFactory). Remplacé par une PR propre depuis main. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The custom dashboard shows 100% detected for some injects. Clicking opens a drawer with
inject_expectation_status = SUCCESSfrom Elasticsearch. Clicking an inject navigates to the inject detail page which shows 100% not detected — inconsistent data between ES and DB.Fixes #6490
Root Cause
findForIndexinguses a strict>comparison onupdated_at:When
bulkComputeTechnicalExpectationsexpires hundreds of expectations in onesaveAll()call, Hibernate's@UpdateTimestampassigns the same millisecond to all of them. If there are >500 with the same timestamp, the indexing cursor advances toT, thenupdated_at > Tpermanently skips the rest.Fix: Compound keyset cursor
Add a secondary
inject_expectation_idcursor so the query can resume within a timestamp:Changes
V5_35__Add_indexing_last_id.javaindexing_last_id TEXT NULLtoindexing_statusIndexingStatus.javalastIdfield for compound cursorInjectExpectationRepository.javafindForIndexingAfter(from, lastId, limit)queryHandler.javafetch(from, lastId, limit)method (backward-compatible)InjectExpectationHandler.javaOpenSearchService.javalastIdinbulkProcessing, pass to handlerElasticService.javaCursor logic
lastId = lastItem.id, keep samelastIndexinglastId = null, advancelastIndexingas before