ENH: encryption extension by chenqi0805 · Pull Request #5581 · opensearch-project/data-prepper

chenqi0805 · 2025-04-02T19:27:11Z

Description

This PR

implements encryption pipeline extension
supports kms encryption engine in the extension
supports encryption key rotation by
- periodically polling new key stored in either s3 folder path or local file directory.
- allowing user to trigger encrypted data key rotation by calling /encryption/rotate API on Data Prepper server

Issues Resolved

Resolves #5335
Resolves #5336

Check List

New functionality includes testing.
New functionality has a documentation issue. Please link to it in this PR.
- New functionality has javadoc added
Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: George Chen <qchea@amazon.com>

…nsearch-project#5272) Signed-off-by: RashmiRam <ras.xena@gmail.com> Signed-off-by: George Chen <qchea@amazon.com>

…t#5344) * Handling end to end acknowledgement Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> * introduced boolean to control end to end Acknowledgment Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> * acknowledgments on case Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> --------- Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> Signed-off-by: George Chen <qchea@amazon.com>

* add batch size field for jira source Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> * remove unused config fields Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> * add interface function to simplify batchSize code Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> * default batch size comments Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> --------- Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> Signed-off-by: Maxwell Brown <55033421+Galactus22625@users.noreply.github.com> Signed-off-by: George Chen <qchea@amazon.com>

…ject#5358) Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

…project#5310) * First working version Signed-off-by: Hai Yan <oeyh@amazon.com> * More progress and update existing unit tests Signed-off-by: Hai Yan <oeyh@amazon.com> * Add unit tests Signed-off-by: Hai Yan <oeyh@amazon.com> * Remove and rename classes Signed-off-by: Hai Yan <oeyh@amazon.com> * Remove test code Signed-off-by: Hai Yan <oeyh@amazon.com> * Address review comments Signed-off-by: Hai Yan <oeyh@amazon.com> * Address minor issues Signed-off-by: Hai Yan <oeyh@amazon.com> * Group MySQL and Postgres stream states Signed-off-by: Hai Yan <oeyh@amazon.com> * Address more comments Signed-off-by: Hai Yan <oeyh@amazon.com> * Fix Java21 build Signed-off-by: Hai Yan <oeyh@amazon.com> --------- Signed-off-by: Hai Yan <oeyh@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

* Handling end to end acknowledgement Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> * checking pointing leader state for every one minute Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> * corresponding test cases fix Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> --------- Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> Signed-off-by: George Chen <qchea@amazon.com>

Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> Signed-off-by: George Chen <qchea@amazon.com>

…roject#5362) Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> Signed-off-by: George Chen <qchea@amazon.com>

Signed-off-by: Hai Yan <oeyh@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

…arch-project#5320) * lambda processor should retry for certain class of exceptions Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Address Comment on complete codec Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Add retryCondidition to lambda Client Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Address comments Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Address comments and add UT and IT Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Address comment on completeCodec Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> --------- Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

Signed-off-by: Hai Yan <oeyh@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

…partition.assignment.strategy, close consumer on shutdown (opensearch-project#5373) Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

Improve Jira logging Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

* injectable plugin metrics Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> * removed an unused parameter Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> * fixing a flaky test Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> --------- Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> Signed-off-by: George Chen <qchea@amazon.com>

Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

…pensearch-project#5361) * initial refactoring Signed-off-by: Jeremy Michael <jsusanto@amazon.com> * refactored sqs-source to use sqs-common Signed-off-by: Jeremy Michael <jsusanto@amazon.com> * refactored SqsWorker to use the common library Signed-off-by: Jeremy Michael <jsusanto@amazon.com> * minor changes Signed-off-by: Jeremy Michael <jsusanto@amazon.com> * another small fix Signed-off-by: Jeremy Michael <jsusanto@amazon.com> * added unit tests for sqs-common Signed-off-by: Jeremy Michael <jsusanto@amazon.com> * updated tests Signed-off-by: Jeremy Michael <jsusanto@amazon.com> --------- Signed-off-by: Jeremy Michael <jsusanto@amazon.com> Co-authored-by: Jeremy Michael <jsusanto@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

* schema revisions, add json aliases Signed-off-by: Katherine Shen <katshen@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

…pensearch-project#5375) * Initial commit Signed-off-by: Hai Yan <oeyh@amazon.com> * Update unit tests Signed-off-by: Hai Yan <oeyh@amazon.com> * Add more metrics Signed-off-by: Hai Yan <oeyh@amazon.com> * Add more tests Signed-off-by: Hai Yan <oeyh@amazon.com> * Address review comments Signed-off-by: Hai Yan <oeyh@amazon.com> * Address review comments Signed-off-by: Hai Yan <oeyh@amazon.com> --------- Signed-off-by: Hai Yan <oeyh@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

* Add cloudwatch logs sink Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> * Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> --------- Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

…bles. (opensearch-project#5417) Signed-off-by: David Venable <dlv@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

…m from the CODEOWNERS, so this keeps these in sync. (opensearch-project#5419) Signed-off-by: David Venable <dlv@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

…DelayTimer Metric for Auto-Scaling (opensearch-project#5409) Signed-off-by: Jeremy Michael <jsusanto@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

Signed-off-by: Hai Yan <oeyh@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

… messages that have been received many times (opensearch-project#5408) Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

…opensearch-project#5420) Signed-off-by: David Venable <dlv@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

* Zero Buffer Implementation and Tests Signed-off-by: Mohammed Aghil Puthiyottil <57040494+MohammedAghil@users.noreply.github.com> * Moved ZeroBuffer Implementation into data-prepper-core and addressed comments Signed-off-by: Mohammed Aghil Puthiyottil <57040494+MohammedAghil@users.noreply.github.com> * Modified ZeroBufferTests to use MockitoExtension and addressed comments Signed-off-by: Mohammed Aghil Puthiyottil <57040494+MohammedAghil@users.noreply.github.com> --------- Signed-off-by: Mohammed Aghil Puthiyottil <57040494+MohammedAghil@users.noreply.github.com> Signed-off-by: George Chen <qchea@amazon.com>

* Fix merge conflict Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Address concurrency/synchronization comment Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Fix InMemoryBufferSynchronized and Add IT Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Address timeout threshold comment Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Add IT for timeout threshold Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> * Fix checkstyle Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> --------- Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

dlvenable · 2025-04-03T16:07:11Z

+    implementation libs.commons.io
+    implementation libs.caffeine
+    implementation 'org.hibernate.validator:hibernate-validator:8.0.2.Final'
+    testImplementation(platform("org.junit:junit-bom:5.10.0"))


You don't need these two lines.

dlvenable · 2025-04-03T16:07:26Z

+    testImplementation("org.junit.jupiter:junit-jupiter")
+}
+
+test {


You don't need this.

dlvenable · 2025-04-03T16:07:35Z

+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+plugins {


You don't need this block.

dlvenable · 2025-04-03T16:08:22Z

+
+package org.opensearch.dataprepper.model.encryption;
+
+import com.sun.net.httpserver.HttpHandler;


Let's find a way to keep this out of data-prepper-api.

A simple solution would be to create a coupling between data-prepper-core and extension-plugin. I recommend this over exposing this in data-prepper-api. Longer-term we can have a better approach for extending the server.

dlvenable · 2025-04-03T16:13:32Z

+        this.awsCredentialsConfig = awsCredentialsConfig;
+    }
+
+    public AwsCredentialsProvider getOrDefault() {


This code looks similar to what we already support in the aws-plugin.

This class was refactored out from kafka-plugins and I did not see similar code in aws-plugin. The plan is to replace those classes in kafka-plugins with the counterparts here.

Or do you think it is better to refactor it into aws-plugin and let encryption-plugin depend on aws-plugin?

Similar methods are provided in AwsCredentialsSupplier interface, and detailed actions through CredentialsProviderFactory.

Kafka plugins also use AwsCredentialsSupplier in AwsContext

I refactored AwsContext into aws-plugin-api so that it will be reused by other plugins

@dlvenable Can you also take a look? I'm good with other changes. Approving.

Signed-off-by: George Chen <qchea@amazon.com>

oeyh

Some comments on minor issues below.

One question: when data key rotates, how does Data Prepper track which data key is used on the encrypted data so it knows which data key to use for decryption?

oeyh · 2025-05-13T01:31:30Z

+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+
+class EncryptionContext {


Is there a reason this class is not public?

This class does not need to be visible to other plugins.

Signed-off-by: George Chen <qchea@amazon.com>

…on-3 Signed-off-by: George Chen <qchea@amazon.com>

Signed-off-by: George Chen <qchea@amazon.com>

dlvenable

Thank you @chenqi0805 !

dlvenable · 2025-05-27T20:19:38Z

+
+import java.util.UUID;
+
+public class AwsContext {


This class seems decoupled from the rest of the AWS plugin.

Maybe the AwsPlugin should provide this in a new provider.

public void apply(final ExtensionPoints extensionPoints) { var extensionsProvider = new AwsExtensionProvider(defaultAwsCredentialsSupplier); extensionPoints.addExtensionProvider(new AwsExtensionProvider(defaultAwsCredentialsSupplier)); // new extensionPoints.addExtensionProvider(new AwsContext(extensionsProvider)); }

Now, it might look like this:

public interface AwsContext { AwsCredentialsProvider getOrDefault() Region getRegionOrDefault(); }

Discussed offline. As an intermediate step to unblock the PR, I made AwsContext an interface and AwsContextImpl a temporary implementation. Once this issue: #2825 gets resolved, AwsContext bean will be provided by aws-plugin.

Signed-off-by: George Chen <qchea@amazon.com>

dlvenable

Thanks @chenqi0805 !

ADD: encryption extension Signed-off-by: George Chen <qchea@amazon.com> Signed-off-by: Jeffrey Aaron Jeyasingh <jeffreyaaron06@gmail.com>

chenqi0805 and others added 30 commits April 2, 2025 14:19

INIT: encryption extension

337f006

Signed-off-by: George Chen <qchea@amazon.com>

ADD: rotation http handler and data key supplier

4e7f651

Signed-off-by: George Chen <qchea@amazon.com>

Fixed kinesis retrieval config argument passing to KCL scheduler (ope…

2d9b978

…nsearch-project#5272) Signed-off-by: RashmiRam <ras.xena@gmail.com> Signed-off-by: George Chen <qchea@amazon.com>

Implement delete_s3_objects_on_read for S3-SQS source (opensearch-pro…

01750e3

…ject#5358) Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

Fixing a flaky test (opensearch-project#5365)

0a8b62e

Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> Signed-off-by: George Chen <qchea@amazon.com>

disabling caffine cache instead of setting the max to 1 (opensearch-p…

bb6dcec

…roject#5362) Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> Signed-off-by: George Chen <qchea@amazon.com>

Disable flaky tests fow now (opensearch-project#5366)

33d4462

Signed-off-by: Hai Yan <oeyh@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

Add debug logs for Kafka consumer (opensearch-project#5369)

7861cd6

Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

Fix postgres stream (opensearch-project#5367)

3734197

Signed-off-by: Hai Yan <oeyh@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

Change Kafka Buffer defaults for fetch.max.wait.ms, fetch.min.bytes, …

df07aee

…partition.assignment.strategy, close consumer on shutdown (opensearch-project#5373) Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

Improve Jira logging (opensearch-project#5351)

ea2f770

Improve Jira logging Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

Fixed javadoc errors (opensearch-project#5379)

dec63be

Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

schema revisions, add json aliases (opensearch-project#5349)

993dc23

* schema revisions, add json aliases Signed-off-by: Katherine Shen <katshen@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

Updates the Dockerfile to use non-legacy syntax for environment varia…

6e86afb

…bles. (opensearch-project#5417) Signed-off-by: David Venable <dlv@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

Moves asifsmohammed to the Emeritus section. We previously removed hi…

023fd5e

…m from the CODEOWNERS, so this keeps these in sync. (opensearch-project#5419) Signed-off-by: David Venable <dlv@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

SQS Source: Add on_error Config, Multi-Region Support, and sqsMessage…

6030f63

…DelayTimer Metric for Auto-Scaling (opensearch-project#5409) Signed-off-by: Jeremy Michael <jsusanto@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

Support export from postgres (opensearch-project#5414)

8b11fc4

Signed-off-by: Hai Yan <oeyh@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

Add max_receive_count configuration option in S3-SQS source to delete…

c98792f

… messages that have been received many times (opensearch-project#5408) Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

Set the JVM file encoding to UTF-8. Resolves opensearch-project#5238. (…

b6f140d

…opensearch-project#5420) Signed-off-by: David Venable <dlv@amazon.com> Signed-off-by: George Chen <qchea@amazon.com>

chenqi0805 requested review from KarstenSchnitter, dinujoh, dlvenable, graytaylor0, kkondaka, oeyh and srikanthjg as code owners April 3, 2025 15:27

dlvenable requested changes Apr 3, 2025

View reviewed changes

MAINT: refactoring to address some comments

64956f1

Signed-off-by: George Chen <qchea@amazon.com>

chenqi0805 requested a review from dlvenable April 15, 2025 15:32

oeyh reviewed May 13, 2025

View reviewed changes

chenqi0805 added 3 commits May 19, 2025 11:47

REF: AwsContext

ce1a485

Signed-off-by: George Chen <qchea@amazon.com>

Merge branch 'tmp/encryption-extension-3' into enh/encryption-extensi…

94b06e5

…on-3 Signed-off-by: George Chen <qchea@amazon.com>

MAINT: renaming

e9c1775

Signed-off-by: George Chen <qchea@amazon.com>

oeyh previously approved these changes May 20, 2025

View reviewed changes

dlvenable requested changes May 27, 2025

View reviewed changes

chenqi0805 added 3 commits May 28, 2025 11:14

MAINT: refactoring some classes and interfaces

f1018a1

Signed-off-by: George Chen <qchea@amazon.com>

MAINT: missing interface

710222a

Signed-off-by: George Chen <qchea@amazon.com>

MAINT: public class

0daa645

Signed-off-by: George Chen <qchea@amazon.com>

chenqi0805 dismissed oeyh’s stale review via 0daa645 May 28, 2025 19:59

chenqi0805 added 3 commits May 29, 2025 13:08

STY: doc

6e76b21

Signed-off-by: George Chen <qchea@amazon.com>

REF: AwsContext and AwsContextImpl

283ced9

Signed-off-by: George Chen <qchea@amazon.com>

MAINT: documentation

f1d2752

Signed-off-by: George Chen <qchea@amazon.com>

chenqi0805 requested a review from dlvenable May 29, 2025 20:39

dlvenable approved these changes May 29, 2025

View reviewed changes

chenqi0805 requested a review from oeyh May 29, 2025 20:54

oeyh approved these changes May 29, 2025

View reviewed changes

chenqi0805 merged commit d568fa4 into opensearch-project:main May 29, 2025
69 of 74 checks passed

chenqi0805 deleted the enh/encryption-extension-3 branch May 29, 2025 21:11


		package org.opensearch.dataprepper.model.encryption;

		import com.sun.net.httpserver.HttpHandler;

Conversation

chenqi0805 commented Apr 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues Resolved

Check List

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

oeyh May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

oeyh left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dlvenable left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dlvenable left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

18 participants

chenqi0805 commented Apr 2, 2025 •

edited

Loading

oeyh May 12, 2025 •

edited

Loading

oeyh left a comment •

edited

Loading