Standardize Exception handling in souce plugins#6255
Merged
san81 merged 1 commit intoNov 13, 2025
Conversation
vecheka
requested review from
KarstenSchnitter,
chenqi0805,
dinujoh,
dlvenable,
engechas,
graytaylor0,
kkondaka,
oeyh,
san81,
sb2k16 and
srikanthjg
as code owners
wjyao0316
reviewed
Nov 11, 2025
| import org.opensearch.dataprepper.metrics.PluginMetrics; | ||
| import org.opensearch.dataprepper.model.record.Record; | ||
| import org.opensearch.dataprepper.plugins.source.microsoft_office365.exception.Office365Exception; | ||
| import org.opensearch.dataprepper.plugins.source.source_crawler.exception.CrawlerException; |
Contributor
There was a problem hiding this comment.
If there is a new revision. Suggest to change to SaasCrawlerException
Contributor
Author
There was a problem hiding this comment.
I think I like the suggested name as well. Will update to address your other comments here as well. Thank you!
| requestErrorsCounter.increment(); | ||
| throw e; | ||
| if (e instanceof CrawlerException) { | ||
| throw (CrawlerException) e; |
Contributor
There was a problem hiding this comment.
do we need to cast it to CrawlerException? I think it is safe to just throw e
| break; | ||
| } | ||
| } | ||
| } catch (CrawlerException e) { |
Contributor
There was a problem hiding this comment.
No blocker. If there is a newer revision, we can remove line 97-99 to avoid confusion.
Contributor
Author
There was a problem hiding this comment.
Makes sense. Can see why it could cause confusion until we actually implement the retry strategy for non-retryable exception.
wjyao0316
approved these changes
Nov 11, 2025
vecheka
force-pushed
the
feature/standardize-exceptions
branch
2 times, most recently
from
7226b33 to
68eee67
Compare
vecheka
commented
Nov 11, 2025
| logType, startTime, endTime, e); | ||
| requestErrorsCounter.increment(); | ||
| throw e; | ||
| if (e instanceof SaaSCrawlerException) { |
wjyao0316
approved these changes
Nov 11, 2025
Signed-off-by: Vecheka Chhourn <vecheka@amazon.com>
vecheka
force-pushed
the
feature/standardize-exceptions
branch
from
68eee67 to
c3f0349
Compare
wjyao0316
approved these changes
Nov 12, 2025
graytaylor0
approved these changes
Nov 13, 2025
san81
approved these changes
Nov 13, 2025
This was referenced Nov 13, 2025
Closed
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This is going to be a two part change to address exceptions handling for retrayble vs non-retryable exceptions.
Previous cancelled PR for more context: #6247
How
We are adding a new generic exception class
CrawlerExceptionto be shared by all connectors. This class is similar to previous API specific exception class (e.gOffice365Exception).CrawlerException will have two criterias:
We will utilize CrawlerException, and throw this up all the way to
WorkerSchedulewhere in the followup PR:sourceCoordinator.saveProgressStateForPartition(workerPartition, DURATION_TO_DELAY_RETRY). If it continues to fail up to 30 days (using partitionCreationTime field to confirm), we will give up the worker partition.Is this change backward compatible?
Yes. We ensure to keep the catch block on generic "Exception" so all other connectors will still use the default behaviour of backoff retry every 5ms for all exception types.
Testing
Unit tests, ran the below successfully:
Local pipeline run succeeded:
Issues Resolved
N/A
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.