Skip to content

fix(engine): fix ES indexing cursor skipping expectations with same timestamp (#6490)#6491

Closed
Seb-MIGUEL wants to merge 6 commits into
mainfrom
fix/es-expectation-indexing-compound-cursor
Closed

fix(engine): fix ES indexing cursor skipping expectations with same timestamp (#6490)#6491
Seb-MIGUEL wants to merge 6 commits into
mainfrom
fix/es-expectation-indexing-compound-cursor

Conversation

@Seb-MIGUEL

Copy link
Copy Markdown
Contributor

Problem

The custom dashboard shows 100% detected for some injects. Clicking opens a drawer with inject_expectation_status = SUCCESS from Elasticsearch. Clicking an inject navigates to the inject detail page which shows 100% not detected — inconsistent data between ES and DB.

Fixes #6490

Root Cause

findForIndexing uses a strict > comparison on updated_at:

WHERE inject_expectation_updated_at > :from
ORDER BY inject_expectation_updated_at ASC
LIMIT 500

When bulkComputeTechnicalExpectations expires hundreds of expectations in one saveAll() call, Hibernate's @UpdateTimestamp assigns the same millisecond to all of them. If there are >500 with the same timestamp, the indexing cursor advances to T, then updated_at > T permanently skips the rest.

Fix: Compound keyset cursor

Add a secondary inject_expectation_id cursor so the query can resume within a timestamp:

WHERE (inject_expectation_updated_at > :from)
   OR (inject_expectation_updated_at = :from AND inject_expectation_id > :lastId)
ORDER BY inject_expectation_updated_at ASC, inject_expectation_id ASC

Changes

File Change
V5_35__Add_indexing_last_id.java Migration: add indexing_last_id TEXT NULL to indexing_status
IndexingStatus.java Add lastId field for compound cursor
InjectExpectationRepository.java Add findForIndexingAfter(from, lastId, limit) query
Handler.java Add default fetch(from, lastId, limit) method (backward-compatible)
InjectExpectationHandler.java Override compound-cursor fetch + extract shared mapping helper
OpenSearchService.java Track lastId in bulkProcessing, pass to handler
ElasticService.java Same as OpenSearch

Cursor logic

  • Full batch returned (size == 500): save lastId = lastItem.id, keep same lastIndexing
  • Partial batch (size < 500): clear lastId = null, advance lastIndexing as before

…6474)

Add onError handler to connector logo <img> in ConnectorTitle.tsx.
When a logo fails to load (404 or network error), display the
HelpCenterOutlined placeholder icon instead of a broken image.
Also reset error state when the logo URL changes.
Add ensureCatalogLogo() hook in IntegrationFactory, called when the
catalog entry already exists in the database. This ensures logos lost
from MinIO (e.g. after bucket reset or storage migration) are restored
on the next application startup.

Each factory overrides ensureCatalogLogo() with its logo upload call.
insertCatalogEntry() delegates to ensureCatalogLogo() to avoid
duplication.
…Logo

- Revert frontend ConnectorTitle.tsx changes (backend fix is sufficient)
- Extract getLogoFilename() in each factory to avoid duplicating the formatted string
- Make ensureCatalogLogo() best-effort in initialise() (try-catch with log.warn)
  to prevent MinIO outages from blocking application startup
…ith same timestamp

When multiple InjectExpectation entities are saved in the same batch
(e.g. during expiration of many expectations), Hibernate sets the same
@UpdateTimestamp millisecond for all of them. The existing polling query
uses a strict > comparison on updated_at, so when the indexing batch
size (500) cuts through a set of items sharing the same timestamp, the
remaining items are permanently skipped.

Fix: introduce a compound keyset cursor (updated_at, inject_expectation_id).
- Add indexing_last_id column to indexing_status (V5_35 migration)
- Add findForIndexingAfter query with compound cursor in repository
- Add fetch(from, lastId, limit) default method to Handler interface;
  InjectExpectationHandler overrides it with the compound-cursor query
- OpenSearchService and ElasticService track lastId per indexing cycle:
  when a full batch is returned, lastId is persisted so the next cycle
  continues from the exact item boundary; otherwise lastId is cleared

This guarantees all expectations are eventually indexed regardless of
how many share the same updated_at timestamp, fixing the inconsistency
between the dashboard detection percentage (ES) and the inject detail
page (DB).
Copilot AI review requested due to automatic review settings June 30, 2026 10:16
@Filigran-Automation Filigran-Automation changed the title [backend] fix(engine): fix ES indexing cursor skipping expectations with same timestamp (#6490) fix(engine): fix ES indexing cursor skipping expectations with same timestamp (#6490) Jun 30, 2026
@Filigran-Automation Filigran-Automation added the filigran team Item from the Filigran team. label Jun 30, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to prevent Elasticsearch/OpenSearch indexing from permanently skipping InjectExpectation updates when more than one batch shares the same updated_at timestamp, by introducing a compound keyset cursor (updated_at, id) and persisting the secondary cursor (lastId) in indexing_status.

Changes:

  • Add indexing_last_id to indexing_status and expose it via IndexingStatus to support compound cursor persistence.
  • Thread lastId through ElasticService/OpenSearchService bulk processing and use the new Handler.fetch(from, lastId, limit) API.
  • Add a new compound-cursor native query for inject expectation indexing and update InjectExpectationHandler to use it.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
openaev-model/src/main/java/io/openaev/service/OpenSearchService.java Passes lastId into handler fetch and persists it in indexing status for full batches.
openaev-model/src/main/java/io/openaev/service/ElasticService.java Mirrors OpenSearch changes: compound cursor threading + persistence.
openaev-model/src/main/java/io/openaev/engine/model/injectexpectation/InjectExpectationHandler.java Adds compound-cursor fetch implementation for inject expectations (currently contains a blocking duplicate class declaration).
openaev-model/src/main/java/io/openaev/engine/Handler.java Adds backward-compatible default fetch(from, lastId, limit) method for handlers.
openaev-model/src/main/java/io/openaev/database/repository/InjectExpectationRepository.java Introduces findForIndexingAfter(from, lastId, limit) with compound cursor ordering.
openaev-model/src/main/java/io/openaev/database/model/IndexingStatus.java Adds lastId field mapped to indexing_last_id.
openaev-api/src/main/java/io/openaev/migration/V5_35__Add_indexing_last_id.java Flyway migration adding indexing_last_id column to indexing_status.
openaev-api/src/main/java/io/openaev/integration/IntegrationFactory.java Adds ensureCatalogLogo() hook and runs it best-effort when catalog entry already exists.
openaev-api/src/main/java/io/openaev/integration/impl/injectors/ovh/OvhInjectorIntegrationFactory.java Refactors logo upload into ensureCatalogLogo() + shared filename helper.
openaev-api/src/main/java/io/openaev/integration/impl/injectors/opencti/OpenCTIInjectorIntegrationFactory.java Refactors logo upload into ensureCatalogLogo() + shared filename helper.
openaev-api/src/main/java/io/openaev/integration/impl/executors/tanium/TaniumExecutorIntegrationFactory.java Refactors logo upload into ensureCatalogLogo() + shared filename helper.
openaev-api/src/main/java/io/openaev/integration/impl/executors/sentinelone/SentinelOneExecutorIntegrationFactory.java Refactors logo upload into ensureCatalogLogo() + shared filename helper.
openaev-api/src/main/java/io/openaev/integration/impl/executors/paloaltocortex/PaloAltoCortexExecutorIntegrationFactory.java Refactors logo upload into ensureCatalogLogo() + shared filename helper.
openaev-api/src/main/java/io/openaev/integration/impl/executors/crowdstrike/CrowdStrikeExecutorIntegrationFactory.java Refactors logo upload into ensureCatalogLogo() + shared filename helper.
openaev-api/src/main/java/io/openaev/integration/impl/executors/caldera/CalderaExecutorIntegrationFactory.java Refactors logo upload into ensureCatalogLogo() + shared filename helper.
openaev-api/src/test/java/io/openaev/integration/local_fixtures/regular/TestIntegrationFactory.java Updates test factory to use ensureCatalogLogo() and shared logo filename helper.
openaev-api/src/test/java/io/openaev/integration/local_fixtures/factory_throws/TestIntegrationFactoryInitThrows.java Same refactor for the throwing factory test fixture.
.github/skills/review-code/SKILL.md Updates review skill documentation with additional anti-pattern checks.

Comment on lines +19 to +22
@Service
@RequiredArgsConstructor
public class InjectExpectationHandler implements Handler<EsInjectExpectation> {

Comment on lines +392 to +395
SELECT ie.inject_expectation_id FROM injects_expectations ie
JOIN injects i ON i.inject_id = ie.inject_id
WHERE i.inject_updated_at > :from
UNION
Comment on lines +396 to +399
SELECT ie.inject_expectation_id FROM injects_expectations ie
JOIN injects i ON i.inject_id = ie.inject_id
JOIN injectors_contracts ic ON ic.injector_contract_id = i.inject_injector_contract
WHERE ic.injector_contract_updated_at > :from
Comment on lines +28 to +33
/**
* Ensures the catalog logo exists in MinIO. Called on every startup (best-effort), even when the
* catalog entry already exists in the database. Override in subclasses that upload a logo during
* {@link #insertCatalogEntry()}.
*
* <p>Default implementation is a no-op for factories that do not manage a catalog logo (e.g.
@Seb-MIGUEL

Copy link
Copy Markdown
Contributor Author

Closing: la branche incluait les commits de PR #6475 (InjectorFactory). Remplacé par une PR propre depuis main.

@Seb-MIGUEL Seb-MIGUEL closed this Jun 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

filigran team Item from the Filigran team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(dashboard): inconsistency between dashboard detection % and inject detail page

3 participants