Skip to content

FEAT: encryption extension integration with kafka#5625

Merged
chenqi0805 merged 117 commits into
opensearch-project:mainfrom
chenqi0805:feat/encryption-extension-integration-with-kafka
May 30, 2025
Merged

FEAT: encryption extension integration with kafka#5625
chenqi0805 merged 117 commits into
opensearch-project:mainfrom
chenqi0805:feat/encryption-extension-integration-with-kafka

Conversation

@chenqi0805
Copy link
Copy Markdown
Collaborator

@chenqi0805 chenqi0805 commented Apr 18, 2025

Description

This PR integrates the encryption plugin with the kafka buffer by

  • introduce encryption_id which references the encryption config ID in the extension config under topic configuration
  • introduce BufferMessageEncryptionDeserializer and BufferMessageEncryptionSerializer as a wrapper around inner dataSerializer with encryption engine.
  • avoids clash with existing kms config under kafka topic by adding mutual exclusive validation check between encryption_key and encryption_id.

Manually tested with

version: "2"
extension:
  encryption:
    default:
      kms:
        encryption_key_directory: s3://my-encryption-1/key
        key_id: 0f1e4812-b5a3-454a-afc6-fbce9a251263
        encryption_context:
          aws:osis:arn: "arn:aws:osis:us-west-2:253613578708:pipeline/test-persistent-buffer-1"
        region: us-east-1
      #  #sts_role_arn: arn:aws:iam::253613578708:role/FullAccess
waf-access-log-pipeline:
  source:
    http:
      path: "/${pipelineName}/logs"
      ssl: false
      port: 21890
  buffer:
    kafka:
      bootstrap_servers:
        - "localhost:9092"
      encryption:
        type: none
      topics:
        - group_id: "BufferGroup-sbw44hdc2cqybr6ghmbkrdwzue"
          name: "BufferTopic-sbw44hdc2cqybr6ghmbkrdwzue"
          encryption_id: default
          #create_topic: true
          #encryption_key: AQIDAHi4L/vFhsXe6BbZnQC2Y/2j9I9hZG3I4bTZi1rXOep0DQG7bFN0uQMPI/4XP0JOcnQPAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM/IflndDoqKPzeR5vAgEQgDsVdh4c7ghXJg30RdudEXk+e2CB5GAFNqPHwxyl+yLOe5lPWT3oXuKEirJ1AJ1J5yPYIc/qZ1pWfcilUw
          #kms:
          #  key_id: 0f1e4812-b5a3-454a-afc6-fbce9a251263
  sink:
    - stdout:

Note:
The kafka buffer integration test failure is on the existing kafka topic kms config unrelated to encryption extension. There is a separate effort to fix the integ test.

Issues Resolved

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

chenqi0805 and others added 30 commits April 2, 2025 14:19
Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
…nsearch-project#5272)

Signed-off-by: RashmiRam <ras.xena@gmail.com>
Signed-off-by: George Chen <qchea@amazon.com>
…t#5344)

* Handling end to end acknowledgement

Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com>

* introduced boolean to control end to end Acknowledgment

Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com>

* acknowledgments on case

Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com>

---------

Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com>
Signed-off-by: George Chen <qchea@amazon.com>
* add batch size field for jira source

Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com>

* remove unused config fields

Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com>

* add interface function to simplify batchSize code

Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com>

* default batch size comments

Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com>

---------

Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com>
Signed-off-by: Maxwell Brown <55033421+Galactus22625@users.noreply.github.com>
Signed-off-by: George Chen <qchea@amazon.com>
…ject#5358)

Signed-off-by: Taylor Gray <tylgry@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
…project#5310)

* First working version

Signed-off-by: Hai Yan <oeyh@amazon.com>

* More progress and update existing unit tests

Signed-off-by: Hai Yan <oeyh@amazon.com>

* Add unit tests

Signed-off-by: Hai Yan <oeyh@amazon.com>

* Remove and rename classes

Signed-off-by: Hai Yan <oeyh@amazon.com>

* Remove test code

Signed-off-by: Hai Yan <oeyh@amazon.com>

* Address review comments

Signed-off-by: Hai Yan <oeyh@amazon.com>

* Address minor issues

Signed-off-by: Hai Yan <oeyh@amazon.com>

* Group MySQL and Postgres stream states

Signed-off-by: Hai Yan <oeyh@amazon.com>

* Address more comments

Signed-off-by: Hai Yan <oeyh@amazon.com>

* Fix Java21 build

Signed-off-by: Hai Yan <oeyh@amazon.com>

---------

Signed-off-by: Hai Yan <oeyh@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
* Handling end to end acknowledgement

Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com>

* checking pointing leader state for every one minute

Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com>

* corresponding test cases fix

Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com>

---------

Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com>
Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com>
Signed-off-by: George Chen <qchea@amazon.com>
…roject#5362)

Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com>
Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: Hai Yan <oeyh@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: Taylor Gray <tylgry@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
…arch-project#5320)

* lambda processor should retry for certain class of exceptions

Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>

* Address Comment on complete codec

Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>

* Add retryCondidition to lambda Client

Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>

* Address comments

Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>

* Address comments and add UT and IT

Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>

* Address comment on completeCodec

Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>

---------

Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: Hai Yan <oeyh@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
…partition.assignment.strategy, close consumer on shutdown (opensearch-project#5373)

Signed-off-by: Taylor Gray <tylgry@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
Improve Jira logging

Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
* injectable plugin metrics

Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com>

* removed an unused parameter

Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com>

* fixing a flaky test

Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com>

---------

Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com>
Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
…pensearch-project#5361)

* initial refactoring

Signed-off-by: Jeremy Michael <jsusanto@amazon.com>

* refactored sqs-source to use sqs-common

Signed-off-by: Jeremy Michael <jsusanto@amazon.com>

* refactored SqsWorker to use the common library

Signed-off-by: Jeremy Michael <jsusanto@amazon.com>

* minor changes

Signed-off-by: Jeremy Michael <jsusanto@amazon.com>

* another small fix

Signed-off-by: Jeremy Michael <jsusanto@amazon.com>

* added unit tests for sqs-common

Signed-off-by: Jeremy Michael <jsusanto@amazon.com>

* updated tests

Signed-off-by: Jeremy Michael <jsusanto@amazon.com>

---------

Signed-off-by: Jeremy Michael <jsusanto@amazon.com>
Co-authored-by: Jeremy Michael <jsusanto@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
* schema revisions, add json aliases

Signed-off-by: Katherine Shen <katshen@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
…pensearch-project#5375)

* Initial commit

Signed-off-by: Hai Yan <oeyh@amazon.com>

* Update unit tests

Signed-off-by: Hai Yan <oeyh@amazon.com>

* Add more metrics

Signed-off-by: Hai Yan <oeyh@amazon.com>

* Add more tests

Signed-off-by: Hai Yan <oeyh@amazon.com>

* Address review comments

Signed-off-by: Hai Yan <oeyh@amazon.com>

* Address review comments

Signed-off-by: Hai Yan <oeyh@amazon.com>

---------

Signed-off-by: Hai Yan <oeyh@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
* Add cloudwatch logs sink

Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

* Addressed review comments

Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>

---------

Signed-off-by: Krishna Kondaka <krishkdk@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
…bles. (opensearch-project#5417)

Signed-off-by: David Venable <dlv@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
…m from the CODEOWNERS, so this keeps these in sync. (opensearch-project#5419)

Signed-off-by: David Venable <dlv@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
…DelayTimer Metric for Auto-Scaling (opensearch-project#5409)

Signed-off-by: Jeremy Michael <jsusanto@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: Hai Yan <oeyh@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
… messages that have been received many times (opensearch-project#5408)

Signed-off-by: Taylor Gray <tylgry@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
…opensearch-project#5420)

Signed-off-by: David Venable <dlv@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
* Zero Buffer Implementation and Tests

Signed-off-by: Mohammed Aghil Puthiyottil <57040494+MohammedAghil@users.noreply.github.com>

* Moved ZeroBuffer Implementation into data-prepper-core and addressed comments

Signed-off-by: Mohammed Aghil Puthiyottil <57040494+MohammedAghil@users.noreply.github.com>

* Modified ZeroBufferTests to use MockitoExtension and addressed comments

Signed-off-by: Mohammed Aghil Puthiyottil <57040494+MohammedAghil@users.noreply.github.com>

---------

Signed-off-by: Mohammed Aghil Puthiyottil <57040494+MohammedAghil@users.noreply.github.com>
Signed-off-by: George Chen <qchea@amazon.com>
* Fix merge conflict

Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>

* Address concurrency/synchronization comment

Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>

* Fix InMemoryBufferSynchronized and Add IT

Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>

* Address timeout threshold comment

Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>

* Add IT for timeout threshold

Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>

* Fix checkstyle

Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>

---------

Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
chenqi0805 added 14 commits May 19, 2025 11:47
Signed-off-by: George Chen <qchea@amazon.com>
…on-3

Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
…ion-integration-with-kafka

Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
…ion-integration-with-kafka

Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
…ion-integration-with-kafka

Signed-off-by: George Chen <qchea@amazon.com>
…afka

Signed-off-by: George Chen <qchea@amazon.com>
Signed-off-by: George Chen <qchea@amazon.com>
@chenqi0805 chenqi0805 merged commit 540f3a1 into opensearch-project:main May 30, 2025
41 of 50 checks passed
@chenqi0805 chenqi0805 deleted the feat/encryption-extension-integration-with-kafka branch May 30, 2025 21:21
jeffreyAaron pushed a commit to jeffreyAaron/data-prepper that referenced this pull request Jun 13, 2025
…#5625)

* FEAT: encryption extension integration with kafka buffer

Signed-off-by: George Chen <qchea@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.