FEAT: encryption extension integration with kafka#5625
Merged
chenqi0805 merged 117 commits intoMay 30, 2025
Merged
Conversation
Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
…nsearch-project#5272) Signed-off-by: RashmiRam <ras.xena@gmail.com> Signed-off-by: George Chen <qchea@amazon.com>
…t#5344) * Handling end to end acknowledgement Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> * introduced boolean to control end to end Acknowledgment Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> * acknowledgments on case Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> --------- Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> Signed-off-by: George Chen <qchea@amazon.com>
* add batch size field for jira source Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> * remove unused config fields Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> * add interface function to simplify batchSize code Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> * default batch size comments Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> --------- Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> Signed-off-by: Maxwell Brown <55033421+Galactus22625@users.noreply.github.com> Signed-off-by: George Chen <qchea@amazon.com>
…ject#5358) Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>
…project#5310) * First working version Signed-off-by: Hai Yan <oeyh@amazon.com> * More progress and update existing unit tests Signed-off-by: Hai Yan <oeyh@amazon.com> * Add unit tests Signed-off-by: Hai Yan <oeyh@amazon.com> * Remove and rename classes Signed-off-by: Hai Yan <oeyh@amazon.com> * Remove test code Signed-off-by: Hai Yan <oeyh@amazon.com> * Address review comments Signed-off-by: Hai Yan <oeyh@amazon.com> * Address minor issues Signed-off-by: Hai Yan <oeyh@amazon.com> * Group MySQL and Postgres stream states Signed-off-by: Hai Yan <oeyh@amazon.com> * Address more comments Signed-off-by: Hai Yan <oeyh@amazon.com> * Fix Java21 build Signed-off-by: Hai Yan <oeyh@amazon.com> --------- Signed-off-by: Hai Yan <oeyh@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>
* Handling end to end acknowledgement Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> * checking pointing leader state for every one minute Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> * corresponding test cases fix Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> --------- Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> Signed-off-by: George Chen <qchea@amazon.com>
…roject#5362) Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: Hai Yan <oeyh@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>
…arch-project#5320) * lambda processor should retry for certain class of exceptions Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Address Comment on complete codec Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Add retryCondidition to lambda Client Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Address comments Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Address comments and add UT and IT Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Address comment on completeCodec Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> --------- Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: Hai Yan <oeyh@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>
…partition.assignment.strategy, close consumer on shutdown (opensearch-project#5373) Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>
Improve Jira logging Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>
* injectable plugin metrics Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> * removed an unused parameter Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> * fixing a flaky test Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> --------- Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>
…pensearch-project#5361) * initial refactoring Signed-off-by: Jeremy Michael <jsusanto@amazon.com> * refactored sqs-source to use sqs-common Signed-off-by: Jeremy Michael <jsusanto@amazon.com> * refactored SqsWorker to use the common library Signed-off-by: Jeremy Michael <jsusanto@amazon.com> * minor changes Signed-off-by: Jeremy Michael <jsusanto@amazon.com> * another small fix Signed-off-by: Jeremy Michael <jsusanto@amazon.com> * added unit tests for sqs-common Signed-off-by: Jeremy Michael <jsusanto@amazon.com> * updated tests Signed-off-by: Jeremy Michael <jsusanto@amazon.com> --------- Signed-off-by: Jeremy Michael <jsusanto@amazon.com> Co-authored-by: Jeremy Michael <jsusanto@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>
* schema revisions, add json aliases Signed-off-by: Katherine Shen <katshen@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>
…pensearch-project#5375) * Initial commit Signed-off-by: Hai Yan <oeyh@amazon.com> * Update unit tests Signed-off-by: Hai Yan <oeyh@amazon.com> * Add more metrics Signed-off-by: Hai Yan <oeyh@amazon.com> * Add more tests Signed-off-by: Hai Yan <oeyh@amazon.com> * Address review comments Signed-off-by: Hai Yan <oeyh@amazon.com> * Address review comments Signed-off-by: Hai Yan <oeyh@amazon.com> --------- Signed-off-by: Hai Yan <oeyh@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>
* Add cloudwatch logs sink Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> * Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> --------- Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>
…bles. (opensearch-project#5417) Signed-off-by: David Venable <dlv@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>
…m from the CODEOWNERS, so this keeps these in sync. (opensearch-project#5419) Signed-off-by: David Venable <dlv@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>
…DelayTimer Metric for Auto-Scaling (opensearch-project#5409) Signed-off-by: Jeremy Michael <jsusanto@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: Hai Yan <oeyh@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>
… messages that have been received many times (opensearch-project#5408) Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>
…opensearch-project#5420) Signed-off-by: David Venable <dlv@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>
* Zero Buffer Implementation and Tests Signed-off-by: Mohammed Aghil Puthiyottil <57040494+MohammedAghil@users.noreply.github.com> * Moved ZeroBuffer Implementation into data-prepper-core and addressed comments Signed-off-by: Mohammed Aghil Puthiyottil <57040494+MohammedAghil@users.noreply.github.com> * Modified ZeroBufferTests to use MockitoExtension and addressed comments Signed-off-by: Mohammed Aghil Puthiyottil <57040494+MohammedAghil@users.noreply.github.com> --------- Signed-off-by: Mohammed Aghil Puthiyottil <57040494+MohammedAghil@users.noreply.github.com> Signed-off-by: George Chen <qchea@amazon.com>
* Fix merge conflict Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Address concurrency/synchronization comment Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Fix InMemoryBufferSynchronized and Add IT Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Address timeout threshold comment Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Add IT for timeout threshold Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Fix checkstyle Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> --------- Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
…on-3 Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
…ion-integration-with-kafka Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
…ion-integration-with-kafka Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
…ion-integration-with-kafka Signed-off-by: George Chen <qchea@amazon.com>
…afka Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
oeyh
approved these changes
May 30, 2025
shenkw1
approved these changes
May 30, 2025
jeffreyAaron
pushed a commit
to jeffreyAaron/data-prepper
that referenced
this pull request
Jun 13, 2025
…#5625) * FEAT: encryption extension integration with kafka buffer Signed-off-by: George Chen <qchea@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR integrates the encryption plugin with the kafka buffer by
Manually tested with
Note:
The kafka buffer integration test failure is on the existing kafka topic kms config unrelated to encryption extension. There is a separate effort to fix the integ test.
Issues Resolved
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.