Skip to content

Log uncaught errors in S3 scan worker to prevent silent thread death#6890

Merged
oeyh merged 1 commit into
opensearch-project:mainfrom
oeyh:fix/s3-scan-uncaught-error-logging
May 29, 2026
Merged

Log uncaught errors in S3 scan worker to prevent silent thread death#6890
oeyh merged 1 commit into
opensearch-project:mainfrom
oeyh:fix/s3-scan-uncaught-error-logging

Conversation

@oeyh

@oeyh oeyh commented May 29, 2026

Copy link
Copy Markdown
Collaborator

Description

When the S3 scan worker thread encounters an uncaught Error (e.g., NoSuchMethodError due to dependency version mismatch), the thread dies silently because:

  1. ScanObjectWorker.run() only catches Exception, not Error
  2. ExecutorService.submit() swallows all Throwable into the Future without logging

This results in the S3 scan pipeline silently stopping — no errors in logs, no stderr output — making it extremely difficult to diagnose. RDS source pipelines were affected by this because they transform to S3 scan pipelines.

Changes

Wrap the scan worker execution in a try/catch for Throwable to ensure any uncaught errors are logged before the thread exits.

Testing

  • Existing unit tests pass
  • Manually verified with a dependency mismatch scenario (sdk-core-2.44 + retries-spi-2.39) that the NoSuchMethodError is now logged
2026-05-29T16:55:30,890 [s3-source-scan-1] ERROR org.opensearch.dataprepper.plugins.source.s3.S3ScanService - S3 scan worker terminated due to uncaught error
java.lang.NoSuchMethodError: 'software.amazon.awssdk.retries.api.RefreshRetryTokenRequest$Builder software.amazon.awssdk.retries.api.RefreshRetryTokenRequest$Builder.isLongPolling(boolean)'
... ...
	at org.opensearch.dataprepper.plugins.sourcecoordinator.dynamodb.DynamoDbSourceCoordinationStore.tryAcquireAvailablePartition(DynamoDbSourceCoordinationStore.java:114)
	at org.opensearch.dataprepper.core.sourcecoordination.LeaseBasedSourceCoordinator.getNextPartitionInternal(LeaseBasedSourceCoordinator.java:173)
	at org.opensearch.dataprepper.core.sourcecoordination.LeaseBasedSourceCoordinator.getNextPartition(LeaseBasedSourceCoordinator.java:167)
	at org.opensearch.dataprepper.plugins.source.s3.ScanObjectWorker.startProcessingObject(ScanObjectWorker.java:214)
	at org.opensearch.dataprepper.plugins.source.s3.ScanObjectWorker.run(ScanObjectWorker.java:181)
	at org.opensearch.dataprepper.plugins.source.s3.S3ScanService.lambda$start$1(S3ScanService.java:114)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Issues Resolved

N/A

Check List

  • New functionality includes testing
  • New functionality has been documented
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@oeyh oeyh merged commit a3fd3b1 into opensearch-project:main May 29, 2026
72 checks passed
kkondaka pushed a commit to kkondaka/kk-data-prepper-f2 that referenced this pull request Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants