Skip to content

Delay 5 minutes in DimensionalTimeSliceCrawler for partition creation of live event#6104

Merged
graytaylor0 merged 1 commit into
opensearch-project:mainfrom
wjyao0316:delay
Nov 13, 2025
Merged

Delay 5 minutes in DimensionalTimeSliceCrawler for partition creation of live event#6104
graytaylor0 merged 1 commit into
opensearch-project:mainfrom
wjyao0316:delay

Conversation

@wjyao0316

Copy link
Copy Markdown
Contributor

Description

This commit add 5 minutes delay to partition creation on live event in DimensionalTimeSliceCrawler.

In general, newly generated events become queryable after 30 ~ 120 second. Delay 5 minutes give enough time for the newly generated events to become queryable to largely reduce the possibility of losing events due to eventual consistency in vender API side.

Issues Resolved

Resolves #[N/A]

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

I also manually run the test locally and verify the partitions covering all time range.

Logs I added

2025-09-25T10:33:15,046 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Creating worker partitions from 1758801600 to 1758805200
2025-09-25T10:33:15,048 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Creating worker partitions from 1758805200 to 1758808800
2025-09-25T10:33:15,048 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Creating worker partitions from 1758808800 to 1758812400
2025-09-25T10:33:15,049 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Creating worker partitions from 1758812400 to 1758816000
2025-09-25T10:33:15,050 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Creating worker partitions from 1758816000 to 1758819600
2025-09-25T10:33:15,051 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Creating worker partitions from 1758819600 to 1758821295
2025-09-25T10:34:15,056 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Creating worker partitions from 1758821295 to 1758821355
2025-09-25T10:35:15,068 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Creating worker partitions from 1758821355 to 1758821415

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

… on live event

This commit add 5 minutes delay to partition creation on live event in
DimensionalTimeSliceCrawler.

In general, newly generated events become queryable after 30 ~ 120 second.
Delay 5 minutes give enough time for the newly generated events to
become queryable to largely reduce the possibility of losing events due
to eventual consistency in vender API side.

Signed-off-by: Wenjie Yao <wjyao@amazon.com>
@graytaylor0 graytaylor0 merged commit d2184d8 into opensearch-project:main Nov 13, 2025
45 of 47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants