ENH: encryption extension#5581

Merged

chenqi0805 merged 110 commits into

opensearch-project:mainfrom

chenqi0805:enh/encryption-extension-3

May 29, 2025

Commits on Apr 2, 2025

INIT: encryption extension
chenqi0805
committed
ADD: rotation http handler and data key supplier
chenqi0805
committed
Fixed kinesis retrieval config argument passing to KCL scheduler (#5272)

RashmiRam
authored and
chenqi0805
committed
Handling end to end acknowledgement in Jira Source (#5344)

san81
authored and
chenqi0805
committed
add batch size field for jira source (#5348)

Galactus22625
authored and
chenqi0805
committed
Implement delete_s3_objects_on_read for S3-SQS source (#5358)

graytaylor0
authored and
chenqi0805
committed
Initial PR for stream support for Postgres in Rds source (#5310)

oeyh
authored and
chenqi0805
committed
Check pointing Leader Scheduler State (#5352)

san81
authored and
chenqi0805
committed
Fixing a flaky test (#5365)

san81
authored and
chenqi0805
committed
disabling caffine cache instead of setting the max to 1 (#5362)

san81
authored and
chenqi0805
committed
Disable flaky tests fow now (#5366)

oeyh
authored and
chenqi0805
committed
Add debug logs for Kafka consumer (#5369)

graytaylor0
authored and
chenqi0805
committed
lambda processor should retry for certain class of exceptions (#5320)

srikanthjg
authored and
chenqi0805
committed
Fix postgres stream (#5367)

oeyh
authored and
chenqi0805
committed
Change Kafka Buffer defaults for fetch.max.wait.ms, fetch.min.bytes, partition.assignment.strategy, close consumer on shutdown (#5373)

graytaylor0
authored and
chenqi0805
committed
Improve Jira logging (#5351)

Galactus22625
authored and
chenqi0805
committed
Plugin metrics injection (#5372)

san81
authored and
chenqi0805
committed
Fixed javadoc errors (#5379)

kkondaka
authored and
chenqi0805
committed
Implementation of sqs-common plugin and refactored sqs and s3 source (#5361)

authored and
chenqi0805
committed
schema revisions, add json aliases (#5349)

shenkw1
authored and
chenqi0805
committed
Add e2e ack, checkpointing and metrics to Postgres stream processing (#5375)

oeyh
authored and
chenqi0805
committed
Add cloudwatch logs sink (#5406)

kkondaka
authored and
chenqi0805
committed
Updates the Dockerfile to use non-legacy syntax for environment variables. (#5417)

dlvenable
authored and
chenqi0805
committed
Moves asifsmohammed to the Emeritus section. We previously removed him from the CODEOWNERS, so this keeps these in sync. (#5419)

dlvenable
authored and
chenqi0805
committed
SQS Source: Add on_error Config, Multi-Region Support, and sqsMessageDelayTimer Metric for Auto-Scaling (#5409)

jmsusanto
authored and
chenqi0805
committed
Support export from postgres (#5414)

oeyh
authored and
chenqi0805
committed
Add max_receive_count configuration option in S3-SQS source to delete messages that have been received many times (#5408)

graytaylor0
authored and
chenqi0805
committed
Set the JVM file encoding to UTF-8. Resolves #5238. (#5420)

dlvenable
authored and
chenqi0805
committed
Zero Buffer Implementation and Tests (#5416)

MohammedAghil
authored and
chenqi0805
committed
Add stateful buffer for lambda sink (#5354)

srikanthjg
authored and
chenqi0805
committed
Confluence as source plugin (#5404)

san81
authored and
chenqi0805
committed
Add OTEL protobuf decoder (#5421)

kkondaka
authored and
chenqi0805
committed
Postgres data type mapping in stream processing (#5410)

authored and
chenqi0805
committed
migrate Blocking Buffer and StdInSource configurations off of plugin setting (#5317)

Galactus22625
authored and
chenqi0805
committed
Add support to increase acknowledgment expiry and use in progress check for Dynamo source (#5428)

graytaylor0
authored and
chenqi0805
committed
Add NOISY marker to opensearch sink failure logs (#5436)

srikanthjg
authored and
chenqi0805
committed
Postgres schema name fix (#5432)

oeyh
authored and
chenqi0805
committed
Deal with Control Characters in DynamoDB (#5437)

Galactus22625
authored and
chenqi0805
committed
Upgrade opensearch-java client to latest 2.20.0 version. (#5426)

Rishikesh1159
authored and
chenqi0805
committed
Confluence pagination handling (#5440)

san81
authored and
chenqi0805
committed
Data type mapping for postgres export (#5438)

divbok
authored and
chenqi0805
committed
Jira reuse atlassian commons module (#5441)

san81
authored and
chenqi0805
committed
Thread.sleep replaced with await() (#5445)

san81
authored and
chenqi0805
committed
Update aws sdk to latest version (#5450)

oeyh
authored and
chenqi0805
committed
Confluence oauth2 pagination fix (#5451)

san81
authored and
chenqi0805
committed
additional validations based on the validation service checks instead of plugin service check (#5454)

san81
authored and
chenqi0805
committed
MAINT: more test cases
chenqi0805
committed
Migrate OpenSearchSink off of plugin Settings (#5273)

Galactus22625
authored and
chenqi0805
committed
Modify the exception handling to be more descriptive when stream is not available. (#5457)

authored and
chenqi0805
committed
Few improvements to Rds Source (#5453)

oeyh
authored and
chenqi0805
committed
ENH: error message in pipeline and plugin deserialization (#5459)
chenqi0805
committed
Adding enum and range datatype support for postgres (#5458)

divbok
authored and
chenqi0805
committed
ENH: improve regex validaiton message (#5447)
chenqi0805
committed
Extracting Confluence page history (#5456)

san81
authored and
chenqi0805
committed
Update otel proto buf specification (#5434)

authored and
chenqi0805
committed
S3-Source Bug Fix: Handle null object size for non S3-Create Events (#5449)

jmsusanto
authored and
chenqi0805
committed
Only add s3 object to be deleted if optional is present for sqs message (#5469)

graytaylor0
authored and
chenqi0805
committed
Handle Mongo Binary data type when subtype is UUID_STANDARD (value 4). (#5472)

dinujoh
authored and
chenqi0805
committed
Add support for Scope metrics (#5471)

kkondaka
authored and
chenqi0805
committed
Fix secrets rotation handling in Rds source (#5465)

oeyh
authored and
chenqi0805
committed
Fix lambda processor batching (#5474)

srikanthjg
authored and
chenqi0805
committed
Fix/opensearch action validation (#5496)
chenqi0805
committed
Add @SingleThread annotation to dissect processor (#5463)

graytaylor0
authored and
chenqi0805
committed
Add table filters in Rds source configuration (#5494)

oeyh
authored and
chenqi0805
committed
Give up partition when there is no Ack (#5499)

dinujoh
authored and
chenqi0805
committed
FIX: sts headers override in AWS secret extension (#5506)
chenqi0805
committed
Confluece Search API timezone fix (#5500)

san81
authored and
chenqi0805
committed
MAINT: more test cases
chenqi0805
committed
ENH: add duration validation
chenqi0805
committed
Adding array data types for rds (#5497)

divbok
authored and
chenqi0805
committed
Pipeline Runner Implementation and Tests (#5435)

MohammedAghil
authored and
chenqi0805
committed
typo fix to correct the timezone issue (#5511)

san81
authored and
chenqi0805
committed
Jira metrics fix (#5513)

san81
authored and
chenqi0805
committed
Fixes for metrics in Rds Source (#5514)

oeyh
authored and
chenqi0805
committed
Enh/support default credentials in dynamic transformation (#5516)
chenqi0805
committed
Add OTEL standard classes for Logs, Traces and Metrics (#5508)

kkondaka
authored and
chenqi0805
committed
one more additional metric added (#5520)

san81
authored and
chenqi0805
committed
Add option to process only metadata of objects in S3 scan mode (#5470)

kkondaka
authored and
chenqi0805
committed
Fix lambda plugin threshold and flaky test (#5523)

srikanthjg
authored and
chenqi0805
committed
FIX: invalid Jakarta annotation on enum type in ActionConfiguration (#5528)
chenqi0805
committed
FIX: invalid Jakarta annotation on enum type in ActionConfiguration (#5528)
chenqi0805
committed
Add bucket_owner option to S3 DLQ (#5519)

graytaylor0
authored and
chenqi0805
committed
Allow setting non-standard AWS Glue Registry endpoints for testing (#5502)

authored and
chenqi0805
committed
FIX: data type of type in actions (#5533)
chenqi0805
committed
Add support for generating Standard(OTEL) conformant events in DataPrepper (#5524)

kkondaka
authored and
chenqi0805
committed
Fix for Issue 5531 (#5532)

kkondaka
authored and
chenqi0805
committed
ENH: use default credentials for aws glue registry in kafka source (#5537)
chenqi0805
committed
Add circuit breaker support for lambda processor (#5540)

srikanthjg
authored and
chenqi0805
committed
Fixes an issue where @SingleThread-annotated processors are called multiple times. The recent addition of the PipelineRunner was getting the full list of processors from the Pipeline. This includes…

dlvenable
authored and
chenqi0805
committed
Improves the AWS Lambda processor's memory usage by creating fewer byte arrays when creating the request for the Lambda function. This uses the "unsafe" methods on SdkBytes. However, these are fine…

dlvenable
authored and
chenqi0805
committed
Improves the AWS Lambda processor's memory usage by creating fewer byte arrays when handling the response from the Lambda function. This uses the existing InputStream from SdkBytes. It also convert…

dlvenable
authored and
chenqi0805
committed
Fixes a few dependency versions. Several new projects have redeclared their dependencies which may lead to inconsistent dependencies in testing than runtime. This updates these to use the dependenc…

dlvenable
authored and
chenqi0805
committed
Build an uber-jar for the Data Prepper performance-test project that can be deployed with all the necessary Gatling dependencies and run stand-alone. (#5552)

dlvenable
authored and
chenqi0805
committed
Add null check for writer (#5550)

oeyh
authored and
chenqi0805
committed
Implement querying for existing documents to handle duplicates on condition or bulk request/operation failures (#5525)

graytaylor0
authored and
chenqi0805
committed
comparing an hour difference range for all the timezones (#5553)

san81
authored and
chenqi0805
committed
MAINT: resolve conflict
chenqi0805
committed
FIX: allow null encryption handler
chenqi0805
committed
MAINT: remove unused imports
chenqi0805
committed
FIX: larger than 0 duration
chenqi0805
committed

Commits

Commits on Apr 2, 2025

Commits on Apr 14, 2025

Commits on May 19, 2025

Commits on May 28, 2025

Commits on May 29, 2025