feat: Add configurable lease interval for crawler source#6432
Merged
graytaylor0 merged 1 commit intoJan 27, 2026
Merged
Conversation
This change adds support for configurable lease interval in the crawler source plugin, allowing users to customize the leader scheduler's lease interval instead of using a hardcoded value. Changes: - Added getLeaseInterval() method to CrawlerSourceConfig interface with default value of 1 minute - Modified CrawlerSourcePlugin to use the configurable lease interval from the source configuration Signed-off-by: Alexander Christensen <alchrisk@amazon.com>
chrisale000
requested review from
KarstenSchnitter,
chenqi0805,
dinujoh,
dlvenable,
engechas,
graytaylor0,
kkondaka,
oeyh,
san81,
sb2k16 and
srikanthjg
as code owners
wjyao0316
approved these changes
Jan 26, 2026
vecheka
approved these changes
Jan 26, 2026
san81
approved these changes
Jan 26, 2026
graytaylor0
reviewed
Jan 26, 2026
| * @return Duration for lease interval | ||
| */ | ||
| default Duration getLeaseInterval() { | ||
| return Duration.ofMinutes(1); |
Member
There was a problem hiding this comment.
This default is a little low. The default is normally 10 minutes at this time I think.
graytaylor0
approved these changes
Jan 27, 2026
san81
pushed a commit
to san81/data-prepper
that referenced
this pull request
Jan 27, 2026
…project#6432) This change adds support for configurable lease interval in the crawler source plugin, allowing users to customize the leader scheduler's lease interval instead of using a hardcoded value. Changes: - Added getLeaseInterval() method to CrawlerSourceConfig interface with default value of 1 minute - Modified CrawlerSourcePlugin to use the configurable lease interval from the source configuration Signed-off-by: Alexander Christensen <alchrisk@amazon.com>
simonelbaz
pushed a commit
to simonelbaz/data-prepper
that referenced
this pull request
Jan 31, 2026
…project#6432) This change adds support for configurable lease interval in the crawler source plugin, allowing users to customize the leader scheduler's lease interval instead of using a hardcoded value. Changes: - Added getLeaseInterval() method to CrawlerSourceConfig interface with default value of 1 minute - Modified CrawlerSourcePlugin to use the configurable lease interval from the source configuration Signed-off-by: Alexander Christensen <alchrisk@amazon.com> Signed-off-by: Simon ELBAZ <elbazsimon9@gmail.com>
simonelbaz
pushed a commit
to simonelbaz/data-prepper
that referenced
this pull request
Jan 31, 2026
…project#6432) This change adds support for configurable lease interval in the crawler source plugin, allowing users to customize the leader scheduler's lease interval instead of using a hardcoded value. Changes: - Added getLeaseInterval() method to CrawlerSourceConfig interface with default value of 1 minute - Modified CrawlerSourcePlugin to use the configurable lease interval from the source configuration Signed-off-by: Alexander Christensen <alchrisk@amazon.com>
simonelbaz
pushed a commit
to simonelbaz/data-prepper
that referenced
this pull request
Jan 31, 2026
…project#6432) This change adds support for configurable lease interval in the crawler source plugin, allowing users to customize the leader scheduler's lease interval instead of using a hardcoded value. Changes: - Added getLeaseInterval() method to CrawlerSourceConfig interface with default value of 1 minute - Modified CrawlerSourcePlugin to use the configurable lease interval from the source configuration Signed-off-by: Alexander Christensen <alchrisk@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds support for configurable lease interval in the crawler source plugin, allowing users to customize the leader scheduler's lease interval instead of using a hardcoded value.
Changes
getLeaseInterval()method toCrawlerSourceConfiginterface with default value of 1 minuteCrawlerSourcePluginto use the configurable lease interval from the source configurationTesting
Signed-off-by: Alexander Christensen alchrisk@amazon.com