Skip to content

Move submission of exectuor service to end of opensearch sink initialization#5922

Merged
graytaylor0 merged 1 commit into
opensearch-project:mainfrom
graytaylor0:S3schedulefix
Sep 17, 2025
Merged

Move submission of exectuor service to end of opensearch sink initialization#5922
graytaylor0 merged 1 commit into
opensearch-project:mainfrom
graytaylor0:S3schedulefix

Conversation

@graytaylor0
Copy link
Copy Markdown
Member

@graytaylor0 graytaylor0 commented Jul 29, 2025

Description

If using the query_lookup and there is an exception initializing the sink, retries will result in a RejectedExecutionException.

WARN  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink with a retryable exception. 
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7341b399[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@49b256e5[Wrapped task = org.opensearch.dataprepper.plugins.sink.opensearch.index.ExistingDocumentQueryManager@11827a57]] rejected from java.util.concurrent.ThreadPoolExecutor@344ae185[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
	at java.base/java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:748)
	at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.doInitializeInternal(OpenSearchSink.java:254)
	at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.doInitialize(OpenSearchSink.java:2

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

san81
san81 previously approved these changes Jul 30, 2025
pluginMetrics, connectionConfiguration, clientFunction);

if (queryExecutorService != null) {
if (queryExecutorService != null && existingDocumentQueryManager == null) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bug fix like this should have a unit test that reproduces the issue along with the fix.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added unit test coverage and confirmed without the change the test will fail

@graytaylor0 graytaylor0 changed the title Don't resubmit executor service for query manager on retries Move submission of exectuor service to end of opensearch sink initialization Sep 15, 2025
…nsearch sink

Signed-off-by: Taylor Gray <tylgry@amazon.com>
@graytaylor0 graytaylor0 merged commit 306ef63 into opensearch-project:main Sep 17, 2025
67 of 68 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants