data-prepper.change-log-2.8.0.md

• Adding 'single_word_only' option to obfuscate processor (#4476) (#4550)

  opensearch-trigger-bot[bot] - Thu, 16 May 2024 09:37:36 -0700

  EAD -> refs/heads/2.8, tag: refs/tags/2.8.0, refs/remotes/origin/2.8 Adding 'single_word_only' option to obfuscate processor Signed-off-by: Utkarsh Agarwal <utkarsh07379@gmail.com> (cherry picked from commit 6d48efba0d71ae0e0674b65ef33ce492b8b215ee) Co-authored-by: Utkarsh Agarwal <126544832+Utkarsh-Aga@users.noreply.github.com>

• Generated THIRD-PARTY file for cedaf87 (#4548)

  opensearch-trigger-bot[bot] - Wed, 15 May 2024 15:36:08 -0700

  Signed-off-by: GitHub <noreply@github.com> Co-authored-by: dlvenable <dlvenable@users.noreply.github.com>

• Updates werkzeug to 3.0.3 in examples to fix CVE-2024-34069. Resolves #4515 (#4546) (#4547)

  opensearch-trigger-bot[bot] - Wed, 15 May 2024 15:26:50 -0700

  Signed-off-by: David Venable <dlv@amazon.com> (cherry picked from commit 906e8255f3440faa90bc2cde4cbd6ffa76ba004c) Co-authored-by: David Venable <dlv@amazon.com>

• Change version for 2.8 release to be 2.8.0 (#4540)

  Krishna Kondaka - Wed, 15 May 2024 10:55:21 -0700

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Logging update and config validation (#4541) (#4542)

  opensearch-trigger-bot[bot] - Wed, 15 May 2024 10:54:55 -0700

  • Logging improvements for export and stream processing for DocumentDB source Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Add validation for DocumentDB Collection Config Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  ---

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com> (cherry picked from commit 94fa30d74d5857a19110eb04e959d990b755b48e) Co-authored-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Add aggregate metrics (#4531) (#4537)

  opensearch-trigger-bot[bot] - Tue, 14 May 2024 17:43:09 -0500

  Signed-off-by: Hai Yan <oeyh@amazon.com> (cherry picked from commit 0495fae1ae0c6eb36a45e587de89932892e5cbbb) Co-authored-by: Hai Yan <8153134+oeyh@users.noreply.github.com>

• Adds an ndjson input codec. This reads JSON objects for ND-JSON and more lenient formats that do not have the newline. (#4533) (#4536)

  opensearch-trigger-bot[bot] - Tue, 14 May 2024 14:17:04 -0700

  Signed-off-by: David Venable <dlv@amazon.com> (cherry picked from commit 59e9fd6227117d3b309975256e87b1ab324d6e30) Co-authored-by: David Venable <dlv@amazon.com>

• Address route and subpipeline for pipeline tranformation (#4528) (#4535)

  opensearch-trigger-bot[bot] - Tue, 14 May 2024 12:15:31 -0700

  Address route and subpipeline for pipeline tranformation Signed-off-by: srigovs <srigovs@amazon.com> (cherry picked from commit e35b4eaf26af79d67834c0f183a846f501be4c16) Co-authored-by: Srikanth Govindarajan <srigovs@amazon.com>

• Fix an issue that exception messages are masked (#4416)

  Hai Yan - Mon, 13 May 2024 16:25:42 -0500

  • Show exception stacktrace Signed-off-by: Hai Yan <oeyh@amazon.com>

  • Tweak log messages Signed-off-by: Hai Yan <oeyh@amazon.com>

  ---

  Signed-off-by: Hai Yan <oeyh@amazon.com>

• Fix aggregate processor local mode (#4529)

  Krishna Kondaka - Mon, 13 May 2024 12:30:19 -0700

  • Fix aggregate processor local mode Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Added comments and removed commented out code in test case Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed code for failing tests Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Key value processor enhancements (#4521)

  Krishna Kondaka - Mon, 13 May 2024 08:53:15 -0700

  • Added auto mode Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Added tests Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Added tests Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Testing changes Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Testing changes Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Added new config and tests Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Checkpoint records at an interval for TPS case when AckSet is enabled (#4526)

  Dinu John - Sat, 11 May 2024 08:27:29 -0500

  • Checkpoint records at an interval for TPS case when AckSet is enabled Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Fix regular checkpoint internal with ack received Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  ---

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Fix write json basic test (#4527)

  Krishna Kondaka - Fri, 10 May 2024 14:13:33 -0700

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Complete AcknowledgementSet for checkpoint thread & fix BsonTimeStamp field conversion (#4525)

  Dinu John - Fri, 10 May 2024 10:10:45 -0500

  • Complete AcknowledgementSet for checkpoint thread when Ack is enabled Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Use time in seconds for BsonTimeStamp field when converting to record Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  ---

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Write stream events that timeout to write to internal buffer in separate thread (#4524)

  Dinu John - Thu, 9 May 2024 19:51:23 -0500

  • Write stream events that timeout to write to internal buffer in separate thread Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Fix Log message level and minor PR comments Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Handle exception in Stream Acknowledgement Manager. Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Extend the lease by default time Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  ---

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Modify typeof operator grammar and add support for ArrayList (#4523)

  Krishna Kondaka - Thu, 9 May 2024 13:54:36 -0700

  • Modify typeof operator grammar and add support for ArrayList Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed checkstyle error Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Added test case for more code coverage Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Clear system property to disable s3 scan when stream worker exits (#4522)

  Taylor Gray - Thu, 9 May 2024 14:38:58 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Add logging change to documentdb source and s3 source with folder partitions (#4519)

  Taylor Gray - Thu, 9 May 2024 10:22:38 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Bump parquet version to 1.14.0. (#4520)

  Adi Suresh - Thu, 9 May 2024 09:50:48 -0500

  1.14.0 has critical bug fixes for parquet files written in OCSF 1.1 schema. Signed-off-by: Adi Suresh <adsuresh@amazon.com>

• Write json processor (#4514)

  Krishna Kondaka - Wed, 8 May 2024 20:48:47 -0700

  • Write Json processor Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Write Json processor - added tests Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Removed the DataPrepperMarker Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Add bytesProcessed metric for stream (#4518)

  Hai Yan - Wed, 8 May 2024 21:37:23 -0500

  Signed-off-by: Hai Yan <oeyh@amazon.com>

• Add bucket owner validation support to s3 sink (#4504)

  Taylor Gray - Wed, 8 May 2024 13:28:35 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Exclude updateDescription field from Mongo/DocDB change stream (#4516)

  Dinu John - Wed, 8 May 2024 10:42:38 -0500

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Fix document version for DocDB/Mongo Stream events (#4513)

  Dinu John - Tue, 7 May 2024 13:55:37 -0500

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• MAINT: inject external origination timestamp (#4507)

  Qi Chen - Tue, 7 May 2024 13:46:05 -0500

  • MAINT: inject external origination timestamp

  • MAINT: set External origination time in event handle Signed-off-by: George Chen <qchea@amazon.com>

• Add metadata for primary key and primary key type and validations for Mongo/DocDb source (#4512)

  Dinu John - Tue, 7 May 2024 12:03:33 -0500

  • Add metadata for primary key and primary key type and validations for Mongo/DocDb source Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  ---

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Update docdb metrics (#4508)

  Hai Yan - Tue, 7 May 2024 10:30:40 -0500

  • Update docdb metrics Signed-off-by: Hai Yan <oeyh@amazon.com>

  • Address comments Signed-off-by: Hai Yan <oeyh@amazon.com>

  ---

  Signed-off-by: Hai Yan <oeyh@amazon.com>

• Fix depth field in template (#4509)

  Srikanth Govindarajan - Tue, 7 May 2024 06:53:51 -0500

  Signed-off-by: srigovs <srigovs@amazon.com>

• Add BsonType of primary key to metadata (#4506)

  Dinu John - Mon, 6 May 2024 18:19:11 -0500

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Add support to export/full load MongoDB/DocumentDB collection with _id field of different data type (#4503)

  Dinu John - Mon, 6 May 2024 17:24:20 -0500

  • Add support to export/full load MongoDB/DocumentDB collection with _id field of different data type Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Add internal type for sorting Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Fix 'static' modifier out of order Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  ---

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Add 'typeof' operator to DataPrepper expressions (#4500)

  Krishna Kondaka - Mon, 6 May 2024 15:12:44 -0700

  • Rebased to latest Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Added test case Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Removed unrelated changes Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Added new tests Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed checkStyle errors Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed checkStyle errors Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Add new S3 source option and modify docdb template (#4492)

  Srikanth Govindarajan - Mon, 6 May 2024 11:18:21 -0500

  Signed-off-by: srigovs <srigovs@amazon.com>

• Move stopping of s3 scan from docdb leader scheduler to stream scheduler (#4498)

  Taylor Gray - Mon, 6 May 2024 11:01:52 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Initial work to support core data types in Data Prepper (#4496)

  David Venable - Fri, 3 May 2024 12:59:49 -0700

  Adds an enum to represent core data types in Data Prepper. This is initially created to help connect the convert_entry_type processor with upcoming work for evaluating type information as part of #4478. Signed-off-by: David Venable <dlv@amazon.com>

• Support default mapping for Mongo/DocumentDB data types (#4499)

  Dinu John - Fri, 3 May 2024 13:55:00 -0500

  • Support default mapping for Mongo/DocumentDB data types Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Fix test indentation Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Use ParameterizedTest for Mongo/DocDB dataType mapping Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  ---

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Support Mongo/DocDB stream Delete and other Operation types (#4497)

  Dinu John - Thu, 2 May 2024 17:35:04 -0500

  • Support Mongo/DocDB stream Delete and other Operation types Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Add reset checkpoint method to mongo/docdb partition checkpoint class Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  ---

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• HTTP data chunking support for kafka buffer (#4475)

  Krishna Kondaka - Thu, 2 May 2024 14:51:46 -0700

  • HTTP data chunking support for kafka buffer Signed-off-by: Kondaka <krishkdk@amazon.com>

  • Addressed comments Signed-off-by: Kondaka <krishkdk@amazon.com>

  • Addressed comments and added tests Signed-off-by: Kondaka <krishkdk@amazon.com>

  • Fixed checkstyle errors Signed-off-by: Kondaka <krishkdk@amazon.com>

  • Addressed review comments Signed-off-by: Kondaka <krishkdk@amazon.com>

  • Fixed checkstyle error Signed-off-by: Kondaka <krishkdk@amazon.com>

  ---

  Signed-off-by: Kondaka <krishkdk@amazon.com>

• Add system property to allow pausing and resuming s3 scan worker thread processing, optionally set this property to pause on documentdb leader (#4495)

  Taylor Gray - Thu, 2 May 2024 14:36:51 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• FIX: null certificate value should be valid in opensearch connection (#4494)

  Qi Chen - Thu, 2 May 2024 10:22:24 -0500

  • FIX: null certificate value should be valid Signed-off-by: George Chen <qchea@amazon.com>

• Fix JacksonEvent to propagate ExternalOriginalTime if its set at the time of construction (#4489)

  Krishna Kondaka - Wed, 1 May 2024 17:06:53 -0700

  • Fix JacksonEvent to propagate ExternalOriginalTime if its set at the time of construction Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Added test Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed code coverage failure by adding more tests Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Move MongoDB/DocDB stream checkpoint to separate thread (#4477)

  Dinu John - Tue, 30 Apr 2024 19:16:55 -0500

  • Move MongoDB/DocDB stream checkpoint to separate thread Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Checkpoint DocDB/MongoDB stream processing periodically Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  ---

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Fix transient test failure for subpipelines (#4479)

  Srikanth Govindarajan - Tue, 30 Apr 2024 13:37:53 -0700

  Signed-off-by: srigovs <srigovs@amazon.com>

• Fix for S3PartitionCreatorScheduler ConcurrentModification Exception (#4473)

  Dinu John - Mon, 29 Apr 2024 12:41:24 -0500

  • Fix for S3PartitionCreatorScheduler ConcurrentModification Exception Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Remove warn log message Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  ---

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Publish data-prepper-expression and data-prepper-logstash-configuration to Maven. (#4474)

  David Venable - Mon, 29 Apr 2024 10:37:28 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Fix DocumentDB source S3PathPrefix null or empty (#4472)

  Dinu John - Mon, 29 Apr 2024 11:26:40 -0500

  • Fix DocumentDB source S3PathPrefix null or empty Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  ---

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Modify documentdb template (#4469)

  Srikanth Govindarajan - Fri, 26 Apr 2024 13:19:50 -0500

  Signed-off-by: srigovs <srigovs@amazon.com>

• Changing logging level for config transformation and fixing rule (#4466)

  Srikanth Govindarajan - Fri, 26 Apr 2024 12:50:08 -0500

  • Changing logging level for config transformation and fixing rule Signed-off-by: srigovs <srigovs@amazon.com>

  • Adding absolute path for template and rule Signed-off-by: srigovs <srigovs@amazon.com>

  • Fix template indentation Signed-off-by: srigovs <srigovs@amazon.com>

  • Fix template indentation Signed-off-by: srigovs <srigovs@amazon.com>

  • Adding event json to codec in template Signed-off-by: srigovs <srigovs@amazon.com>

  • Handled PipelinesDataFlow mapping and addressed array type when replacing Node Signed-off-by: srigovs <srigovs@amazon.com>

  • Add relative path as file stream Signed-off-by: srigovs <srigovs@amazon.com>

  • Fix environment variable Signed-off-by: srigovs <srigovs@amazon.com>

  ---

  Signed-off-by: srigovs <srigovs@amazon.com>

• Let data-prepper-core know if docdb has acknowledgments enabled (#4467)

  Taylor Gray - Fri, 26 Apr 2024 10:39:09 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Add folder-based partitioning for s3 scan source (#4455)

  Taylor Gray - Thu, 25 Apr 2024 23:45:55 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Pipeline Configuration Transformation (#4446)

  Srikanth Govindarajan - Wed, 24 Apr 2024 20:13:36 -0700

  • Adding templates Signed-off-by: srigovs <srigovs@amazon.com>

  • Added Dynamic yaml transformer Signed-off-by: srigovs <srigovs@amazon.com>

  • Added Rule evaluator Signed-off-by: srigovs <srigovs@amazon.com>

  • Added rule evaluator Signed-off-by: srigovs <srigovs@amazon.com>

  • Added json walk Signed-off-by: srigovs <srigovs@amazon.com>

  • Add transformation logic Signed-off-by: srigovs <srigovs@amazon.com>

  • Add dynamic rule Signed-off-by: srigovs <srigovs@amazon.com>

  • Almost working Signed-off-by: srigovs <srigovs@amazon.com>

  • Adding multiple pipelines part1 Signed-off-by: srigovs <srigovs@amazon.com>

  • Adding multiple pipelines part2-incomplete Signed-off-by: srigovs <srigovs@amazon.com>

  • Works e2e for 1 pipeline Signed-off-by: srigovs <srigovs@amazon.com>

  • Added multi pipeline template and pipelinemodel support, works for docDB with big template Signed-off-by: srigovs <srigovs@amazon.com>

  • added tests for models and fixed beans, one more fix needed for bean Signed-off-by: srigovs <srigovs@amazon.com>

  • Fixed IT and beans Signed-off-by: srigovs <srigovs@amazon.com>

  • Update bean to have only pipelineDataModel and not parser Signed-off-by: srigovs <srigovs@amazon.com>

  • Add banner Signed-off-by: srigovs <srigovs@amazon.com>

  • Code cleanup and add comments Signed-off-by: srigovs <srigovs@amazon.com>

  • Support user pipeline configuration dynamic transformation based on

  templates and rules Signed-off-by: srigovs <srigovs@amazon.com>

  • Address comments Signed-off-by: srigovs <srigovs@amazon.com>

  • Added Function Call support in templates Signed-off-by: srigovs <srigovs@amazon.com>

  • Added Function Call support in templates Signed-off-by: srigovs <srigovs@amazon.com>

  • Modify documentDB template. Signed-off-by: srigovs <srigovs@amazon.com>

  • Code clean up Signed-off-by: srigovs <srigovs@amazon.com>

  • Code clean up Signed-off-by: srigovs <srigovs@amazon.com>

  ---

  Signed-off-by: srigovs <srigovs@amazon.com>

• MAINT: use authentication-for-basic-credentials (#4435)

  Qi Chen - Wed, 24 Apr 2024 14:39:25 -0500

  Signed-off-by: George Chen <qchea@amazon.com>

• Adding Support for 'BigDecimal' for integer, long, double, string type. (#4395)

  Utkarsh-Aga - Wed, 24 Apr 2024 11:00:27 -0700

  Adding Support for 'BigDecimal' for the following 'Target type' - integer, long, double, string Signed-off-by: Utkarsh Agarwal <utkarsh07379@gmail.com> Co-authored-by: Utkarsh Agarwal <utkarsh07379@gmail.com>

• MAINT: use authentication block to replace username and password in opensearch sink (#4438)

  Qi Chen - Wed, 24 Apr 2024 12:35:26 -0500

  Signed-off-by: George Chen <qchea@amazon.com>

• Revert DocumentDB Source String array host to String host (#4457)

  Dinu John - Wed, 24 Apr 2024 12:02:01 -0500

  • Revert DocumentDB Source String array host to String host Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• MAINT: allow latest schema version if not specified in confluent schema (#4453)

  Qi Chen - Wed, 24 Apr 2024 11:13:54 -0500

  Signed-off-by: George Chen <qchea@amazon.com>

• Update DocumentDB source pipeline config parameters (#4451)

  Dinu John - Wed, 24 Apr 2024 11:11:46 -0500

  • Update DocumentDB source pipeline config parameters Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  ---

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Added support for multiple workers in S3 Scan Source (#4439)

  Krishna Kondaka - Tue, 23 Apr 2024 13:56:14 -0700

  • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed failing integration tests Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed failing check style Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• MAINT: deprecate certificate_content with certificate_key (#4434)

  Qi Chen - Tue, 23 Apr 2024 10:11:55 -0500

  • MAINT: deprecate certificate_content with certificate_key Signed-off-by: George Chen <qchea@amazon.com>

• Support for Event Json input and output codecs (#4436)

  Krishna Kondaka - Mon, 22 Apr 2024 16:22:17 -0700

  • Event Json input and output codecs Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Modified test case to check for event metadata attributes Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Modified the coverage to 0.9 Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixes for failing coverage tests Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed test coverage Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Added more tests Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Added more tests for coverage Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed code coverage failure Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• ENH: docdb credential auto refreshment (#4399)

  Qi Chen - Mon, 22 Apr 2024 16:22:29 -0500

  • ENH: docdb credential refreshment Signed-off-by: George Chen <qchea@amazon.com>

• Use positive numbers in S3DlqWriterTest for consistent test success. (#4443)

  David Venable - Mon, 22 Apr 2024 08:27:15 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Maint/renaming kafka source plugin setting (#4429)

  Qi Chen - Fri, 19 Apr 2024 09:29:17 -0500

  • MAINT: schema registry setting renaming Signed-off-by: George Chen <qchea@amazon.com>

• MAINT: allow either plain or plaintext under SASL (#4433)

  Qi Chen - Fri, 19 Apr 2024 09:28:20 -0500

  Signed-off-by: George Chen <qchea@amazon.com>

• MAINT: deprecate pipeline_configurations with extension (#4428)

  Qi Chen - Fri, 19 Apr 2024 09:27:16 -0500

  • MAINT: deprecate-pipeline_configurations with extension Signed-off-by: George Chen <qchea@amazon.com>

• Updates Ameria to 1.28.2. To support this change, also updates gRPC to 1.63.0, and Netty to 4.1.108. Fixes unit tests that were expecting a failure. Armeria fixed a bug so the old expectation was no longer valid. (#4440)

  David Venable - Thu, 18 Apr 2024 16:32:28 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Make s3 partition size configurable and add unit test for S3 partition creator classes (#4437)

  Dinu John - Thu, 18 Apr 2024 13:19:44 -0500

  • Make s3 partition size configurable and add unit test for S3 partition creator classes Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Rename export partition size to export batch size Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  ---

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Change s3 sink client to async client (#4425)

  Taylor Gray - Thu, 18 Apr 2024 11:09:26 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Remove creating S3 prefix path partition upfront (#4432)

  Dinu John - Wed, 17 Apr 2024 14:59:30 -0500

  This will be done when S3 sink writes data with path prefix. Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Gradle 8.7 (#4417)

  David Venable - Wed, 17 Apr 2024 11:02:05 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Rename DocDB/MongoDB config parameters (#4426)

  Dinu John - Tue, 16 Apr 2024 18:15:34 -0500

  • Rename DocDB/MongoDB config parameters Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Update hostname to host in DocDB/MongDb config parameter Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  ---

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Encode MongoDB/DocumentDB username and password while constructing connection string (#4423)

  Dinu John - Tue, 16 Apr 2024 13:30:35 -0500

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Moves the Maven publish Gradle configuration into a Gradle convention plugin. This splits the build logic for publication and allows different projects to determine whether they publish to Maven rather than having to track this with conditionals in the root Gradle project. (#4421)

  David Venable - Tue, 16 Apr 2024 10:57:25 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Fix acknowledgements in DynamoDB (#4419)

  Krishna Kondaka - Mon, 15 Apr 2024 16:45:58 -0700

  • Fix acknowledgements in DynamoDB Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Added tests Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Fix bug in s3 sink dynamic bucket and catch invalid bucket message (#4413)

  Taylor Gray - Thu, 11 Apr 2024 18:48:14 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• AWS DocumentDB Stream/Export improvements with S3 partition (#4409)

  Dinu John - Thu, 11 Apr 2024 17:02:12 -0500

  • AWS DocumentDB Stream/Export improvements with S3 partition Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Update S3 Folder partition with one global state Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  ---

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Add support for dynamic bucket and default bucket in S3 sink (#4402)

  Taylor Gray - Thu, 11 Apr 2024 16:51:19 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Do not write empty lists of DlqObject to the DLQ. Just ignore these requests in the S3DlqWriter class. (#4403)

  David Venable - Thu, 11 Apr 2024 12:14:20 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Validate the AWS account Id in the S3 source when configuring either the default_bucket_owner or the bucket_owners map. Implemented this by adding a new bean validation annotation @AwsAccountId in the aws-plugin-api. Resolves #4398 (#4400)

  David Venable - Thu, 11 Apr 2024 12:14:13 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Fix bug where using upsert or update without routing parameter caused NoSuchElementException (#4397)

  Taylor Gray - Thu, 11 Apr 2024 13:51:18 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Create new codec for each s3 group in s3 sink (#4410)

  Taylor Gray - Thu, 11 Apr 2024 11:18:34 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Create unit test report as html (#4384)

  Dinu John - Wed, 10 Apr 2024 15:48:21 -0500

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Modified GRPC exception handler return BAD_REQUEST for certain internal errors (#4387)

  Krishna Kondaka - Tue, 9 Apr 2024 15:16:58 -0700

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Simple script to help generate the release notes file as a starting point (#4322)

  David Venable - Tue, 9 Apr 2024 12:43:46 -0700

  This PR adds a small script that I've used to create release notes recently. It can help us create release notes going forward. Signed-off-by: David Venable <dlv@amazon.com>

• MAINT: float up exception (#4391)

  Qi Chen - Tue, 9 Apr 2024 14:24:25 -0500

  Signed-off-by: George Chen <qchea@amazon.com>

• Updates the S3/file codecs to use the EventFactory over JacksonLog/JacksonEvent builder(). (#4355)

  David Venable - Tue, 9 Apr 2024 11:47:14 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Add developer guide/integration test section (#4330)

  Jürgen Walter - Fri, 5 Apr 2024 14:08:33 -0700

  • Add developer guide/integration test section Aligned with Kafka example

  https://github.com/opensearch-project/data-prepper/blob/51ee0df595aaad1b921c888394bf3e110ffc74e9/data-prepper-plugins/kafka-plugins/README.md#developer-guide

  Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

  • Use latest OpenSearch for testing Version 2.12.0 onwards requires to use non-default admin password

  https://opensearch.org/blog/replacing-default-admin-credentials/ Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

  ---

  Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

• Corrects the release date for 2.7.0 (#4349)

  David Venable - Fri, 5 Apr 2024 08:42:11 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Cache geolocation data within any given batch of Data Prepper events to avoid extra calls to the MaxMind GeoIP code. (#4357)

  David Venable - Fri, 5 Apr 2024 07:24:16 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Adds KarstenSchnitter as a maintainer to the Data Prepper project (#4389)

  David Venable - Fri, 5 Apr 2024 07:23:06 -0700

  Adds KarstenSchnitter as a maintainer to the Data Prepper project. Signed-off-by: David Venable <dlv@amazon.com>

• Remove unexpected event handle message (#4388)

  Krishna Kondaka - Thu, 4 Apr 2024 15:09:03 -0700

  • Remove unexpected event handle message Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed failing tests Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Add aggregate_threshold with maximum_size to s3 sink (#4385)

  Taylor Gray - Thu, 4 Apr 2024 16:54:05 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Add server connections metric to http and otel sources (#4393)

  Hai Yan - Thu, 4 Apr 2024 10:53:26 -0500

  • Add metric to http source Signed-off-by: Hai Yan <oeyh@amazon.com>

  • Add to Otel logs source Signed-off-by: Hai Yan <oeyh@amazon.com>

  • Add to Otel metrics source Signed-off-by: Hai Yan <oeyh@amazon.com>

  • Add to Otel trace source Signed-off-by: Hai Yan <oeyh@amazon.com>

  • Optimize imports Signed-off-by: Hai Yan <oeyh@amazon.com>

  ---

  Signed-off-by: Hai Yan <oeyh@amazon.com>

• Added 'long' as a target type for 'convert_entry_type' processor (#4359)

  Utkarsh-Aga - Thu, 4 Apr 2024 08:14:59 -0700

  Added 'long' as a target type for 'convert_entry_type' processor Signed-off-by: Utkarsh Agarwal <utkarsh07379@gmail.com>

  ---

  Signed-off-by: Utkarsh Agarwal <utkarsh07379@gmail.com> Co-authored-by: Utkarsh Agarwal <utkarsh07379@gmail.com>

• Log the User-Agent when Data Prepper shuts down from the POST /shutdown request. (#4390)

  David Venable - Thu, 4 Apr 2024 07:12:33 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• ENH: automatic credential refresh in kafka source (#4258)

  Qi Chen - Wed, 3 Apr 2024 14:49:20 -0500

  • ADD: basic credentials and dynamic refreshment Signed-off-by: George Chen <qchea@amazon.com>

• Fix flaky PipelineConfigurationFileReaderTest (#4386)

  Taylor Gray - Wed, 3 Apr 2024 12:16:16 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Do not require field_split_characters is not empty for key_value processor (#4358)

  Taylor Gray - Wed, 3 Apr 2024 12:15:52 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Update Stream Ack Manager unit test and code refactor (#4383)

  Dinu John - Tue, 2 Apr 2024 15:17:16 -0500

  • Update Stream Ack Manager unit test Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Move CheckpointStatus to stream sub package and make it package protected Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  ---

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Add AcknowledgementSet support to DocumentDB/MongoDB streams (#4379)

  Dinu John - Tue, 2 Apr 2024 13:20:59 -0500

  • Add AcknowledgementSet support to DocumentDB/MongoDB streams Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Update StreamAcknowledgementManagerTest Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Add BackgroundThreadFactory that adds thread name prefix for debugging Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Update unit test Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  ---

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Add creation and aggregation of dynamic S3 groups based on events (#4346)

  Taylor Gray - Mon, 1 Apr 2024 17:43:12 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Aggregate Processor: local mode should work when there is no when condition (#4380)

  Krishna Kondaka - Mon, 1 Apr 2024 12:22:09 -0700

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Fix count aggregation exemplar data (#4341)

  Krishna Kondaka - Mon, 1 Apr 2024 12:12:39 -0700

  • Fix count aggregation exemplar data Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed failing tests Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-15df884d.us-west-2.amazon.com>

  • Fixed failing tests Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-15df884d.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-15df884d.us-west-2.amazon.com>

  Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-15df884d.us-west-2.amazon.com>

• Bump org.jetbrains.kotlin:kotlin-stdlib-common (#4363)

  dependabot[bot] - Mon, 1 Apr 2024 09:29:19 -0700

  Bumps org.jetbrains.kotlin:kotlin-stdlib-common from 1.9.22 to 1.9.23.

  • Release notes
  • Changelog
  • Commits

  ---

  updated-dependencies:

  • dependency-name: org.jetbrains.kotlin:kotlin-stdlib-common dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

• Bump software.amazon.awssdk:auth in /performance-test (#4362)

  dependabot[bot] - Mon, 1 Apr 2024 09:25:52 -0700

  Bumps software.amazon.awssdk:auth from 2.25.0 to 2.25.21.

  ---

  updated-dependencies:

  • dependency-name: software.amazon.awssdk:auth dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

• Adds integration tests for the geoip processor (#4353)

  David Venable - Thu, 28 Mar 2024 12:06:31 -0700

  Adds integration tests for the geoip processor. Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com> Signed-off-by: David Venable <dlv@amazon.com> Co-authored-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

• ENH: automatic credential refresher opensearch sink (#4283)

  Qi Chen - Thu, 28 Mar 2024 13:49:44 -0500

  ENH: automatic credential refresher opensearch sink Signed-off-by: George Chen <qchea@amazon.com>

• Integration test to verify that the core HTTP server starts. (#4255)

  David Venable - Thu, 28 Mar 2024 09:19:08 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Fixes the S3 source tests when running together (#4280)

  David Venable - Thu, 28 Mar 2024 09:18:33 -0700

  When running all the S3 source tests together, there are empty objects. This adds a mocked counter to avoid a NPE. Signed-off-by: David Venable <dlv@amazon.com>

• Add support for MongoDB/DocumentDB stream processing (#4338)

  Dinu John - Thu, 28 Mar 2024 11:05:36 -0500

  • Add support for MongoDB/DocumentDB stream processing Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Add test for DataStreamPartitionCheckpoint Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Fix ExportScheduler unit test Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Add ingestion type meta data and minor fixes Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Update unit test Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  ---

  Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

• Use the org.lz4:lz4-java project instead of the old net.jpountz.lz4 project. (#4347)

  David Venable - Wed, 27 Mar 2024 16:27:28 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Adds new metrics to help debug errors connecting to OpenSearch. Resolves #4343, #4344 (#4348)

  David Venable - Wed, 27 Mar 2024 15:51:00 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Fix the release build by only applying the Maven publish task to either projects with a name starting with data-prepper or any projects that are children of the data-prepper-plugins project. (#4337)

  David Venable - Tue, 26 Mar 2024 15:28:44 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Handle errors from OpenSearch by checking status field as well as error (#4335)

  David Venable - Tue, 26 Mar 2024 14:27:44 -0700

  Handle errors from OpenSearch by checking both the status field and the error body for each bulk response item. Signed-off-by: David Venable <dlv@amazon.com>

• Removes the release flag from the release GitHub Action. It is no longer needed and it conflicts with the release task. (#4332)

  David Venable - Tue, 26 Mar 2024 12:14:51 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Revert HTTP data chunking changes for kafka buffer done in PR 4266 (#4329)

  Krishna Kondaka - Tue, 26 Mar 2024 11:14:08 -0700

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Adds the release notes for Data Prepper 2.7.0. (#4321)

  David Venable - Tue, 26 Mar 2024 10:44:24 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Fix Router performance issue (#4327)

  Krishna Kondaka - Mon, 25 Mar 2024 16:17:02 -0700

  • Fix Router performance issue Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Truncate Processor: Add support to truncate all fields in an event (#4317)

  Krishna Kondaka - Mon, 25 Mar 2024 11:09:01 -0700

  Truncate Processor: Add support to truncate all fields in an event Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Split up the Maven groupIds for core, plugins, and test to match package names. (#4324)

  David Venable - Mon, 25 Mar 2024 10:44:36 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Updates the next Data Prepper version to 2.8. (#4320)

  David Venable - Fri, 22 Mar 2024 14:05:51 -0700

  Signed-off-by: David Venable <dlv@amazon.com>