data-prepper.change-log-2.4.0.md

• Fix for kafka source issue #3264 (aws glue excetion handling) (#3265) (#3266)

  opensearch-trigger-bot[bot] - Sun, 27 Aug 2023 15:37:31 -0500

  EAD -> refs/heads/2.4, tag: refs/tags/2.4.0, refs/remotes/upstream/2.4 (cherry picked from commit d908c40ad946625ef9b183466a9c55ce8033dbb8) Co-authored-by: Hardeep Singh <mzhrde@amazon.com>

• Fix for kafka source issue #3247 (offset commit stops on deserialization error) (#3260) (#3262)

  opensearch-trigger-bot[bot] - Fri, 25 Aug 2023 17:18:11 -0700

  Signed-off-by: Hardeep Singh <mzhrde@amazon.com> (cherry picked from commit 8114ab40e75c0471bf47b9e78e35f3d4fd11e6a2) Co-authored-by: Hardeep Singh <mzhrde@amazon.com>

• Disallow the combination of a user-defined schema and include/exclude keys (#3254) (#3261)

  opensearch-trigger-bot[bot] - Fri, 25 Aug 2023 16:10:12 -0700

  Disallow the combination of a user-defined schema and include/exclude keys in the Parquet/Avro sink codecs. Resolves #3253. Signed-off-by: David Venable <dlv@amazon.com> (cherry picked from commit 01981a0d07169d359bbf63a1bd03b2a107335239) Co-authored-by: David Venable <dlv@amazon.com>

• Fixes build broken by RELEASING.md spotless check. (#3258) (#3259)

  opensearch-trigger-bot[bot] - Fri, 25 Aug 2023 15:41:53 -0700

  Signed-off-by: David Venable <dlv@amazon.com> (cherry picked from commit 1bcf9f6f5a437802722212cbaebff13253c0f3d4) Co-authored-by: David Venable <dlv@amazon.com>

• Adds a RELEASING.md file to the root of the project (#3251) (#3257)

  opensearch-trigger-bot[bot] - Fri, 25 Aug 2023 14:19:11 -0700

  Adds a RELEASING.md file to the root of the project. This has updated instructions for the new release workflow. Resolves #3108. Signed-off-by: David Venable <dlv@amazon.com> (cherry picked from commit 1eec63f169a2cba3d32a5cab2cd5349c49a570c6) Co-authored-by: David Venable <dlv@amazon.com>

• Fixes a bug with the S3 parquet codec which was not calculating size correctly. Require the parquet codec only with in_memory which is how it is buffering data. Some debugging help. (#3249) (#3256)

  opensearch-trigger-bot[bot] - Fri, 25 Aug 2023 14:18:07 -0700

  Signed-off-by: David Venable <dlv@amazon.com> (cherry picked from commit 7914e93b345cec3969e828ca082d246c7c067b37) Co-authored-by: David Venable <dlv@amazon.com>

• Improve the S3 sink integration tests combinations. The tests are now more consistent and avoid some redundant tests, thus also running faster. Sets up to have fewer combinations while testing all codecs. (#3199) (#3248)

  opensearch-trigger-bot[bot] - Fri, 25 Aug 2023 10:17:34 -0700

  Signed-off-by: David Venable <dlv@amazon.com> (cherry picked from commit f9f37299ce1f39e090752c8864255203df1ed798) Co-authored-by: David Venable <dlv@amazon.com>

• Behavioral change to Avro codecs and schema handling (#3238) (#3245)

  opensearch-trigger-bot[bot] - Thu, 24 Aug 2023 14:47:33 -0700

  Change the behavior of Avro-based codecs. When a schema is defined, rely on the schema rather than the incoming event. If the schema is auto-generated, then the incoming event data must continue to match. Fix Avro arrays which were only supporting arrays of strings previously. Signed-off-by: David Venable <dlv@amazon.com> (cherry picked from commit f17e8338225aefab9d0241d1b540fdd2face2d81) Co-authored-by: David Venable <dlv@amazon.com>

• Batch the errors writing to the S3 sink to reduce the number of errors reported. (#3242) (#3244)

  opensearch-trigger-bot[bot] - Thu, 24 Aug 2023 14:47:05 -0700

  Signed-off-by: David Venable <dlv@amazon.com> (cherry picked from commit 2cca4b187c706fd3ab079b422c96c53bfb8b9b40) Co-authored-by: David Venable <dlv@amazon.com>

• Catch when no object exists and mark as completed in s3 scan (#3241) (#3243)

  opensearch-trigger-bot[bot] - Thu, 24 Aug 2023 14:19:40 -0700

  Signed-off-by: Taylor Gray <tylgry@amazon.com> (cherry picked from commit c2fc9495c09ba936994f58ff8dbbd94330a03582) Co-authored-by: Taylor Gray <tylgry@amazon.com>

• Fix for kafka source not committing offsets issue #3231 (#3232) (#3239)

  opensearch-trigger-bot[bot] - Thu, 24 Aug 2023 13:30:58 -0700

  Signed-off-by: Hardeep Singh <mzhrde@amazon.com> (cherry picked from commit 89e8f3945607499a292d59c1e9fc1ccbbf5b2218) Co-authored-by: Hardeep Singh <mzhrde@amazon.com>

• Removes @cmanning09 from the CODEOWNERS file to allow the release build to proceed. (#3225) (#3227)

  opensearch-trigger-bot[bot] - Wed, 23 Aug 2023 15:48:59 -0700

  Signed-off-by: David Venable <dlv@amazon.com> (cherry picked from commit a4df0bd80e2d47dd8b5392bccb39587307bc544b) Co-authored-by: David Venable <dlv@amazon.com>

• Improvements in the release.yml GitHub Action: Better conditional to fail the promote if the build fails, increased the timeout, added the issues write permissions, string literal correction. (#3224) (#3226)

  opensearch-trigger-bot[bot] - Wed, 23 Aug 2023 14:27:09 -0700

  Signed-off-by: David Venable <dlv@amazon.com> (cherry picked from commit 2cd9e693365d41a79d0d9850a963a2823dfef0e3) Co-authored-by: David Venable <dlv@amazon.com>

• Add 2.4 release notes (#3220) (#3222)

  opensearch-trigger-bot[bot] - Wed, 23 Aug 2023 09:16:00 -0500

  Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com> (cherry picked from commit 990497a91c5df9d45cc71b1ed96f33ec0cd7f6e4) Co-authored-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

• Kafka source fixes: commit offsets, consumer group mutations, consumer shutdown (#3207) (#3218)

  opensearch-trigger-bot[bot] - Tue, 22 Aug 2023 15:27:14 -0700

  Removed acknowledgments_timeout config from kafka source Signed-off-by: Hardeep Singh <mzhrde@amazon.com> (cherry picked from commit b5443634a4704cec4c33ec386747e12270aed073) Co-authored-by: Hardeep Singh <mzhrde@amazon.com>

• Updates documentation for the Avro codec and S3 sink. Resolves #3162. (#3211) (#3219)

  opensearch-trigger-bot[bot] - Tue, 22 Aug 2023 15:18:17 -0700

  Signed-off-by: David Venable <dlv@amazon.com> (cherry picked from commit 3d67468ab1ec7008378410fd14bb2e8d6742f9a2) Co-authored-by: David Venable <dlv@amazon.com>

• Catch exceptions when writing to the output codec and drop the event. (#3210) (#3216)

  opensearch-trigger-bot[bot] - Tue, 22 Aug 2023 15:17:57 -0700

  Catch exceptions when writing to the output codec and drop the event. Correctly release failed events in the S3 sink. Signed-off-by: David Venable <dlv@amazon.com> (cherry picked from commit c1cbb22fc5a80003d881a66bd028772909b33bba) Co-authored-by: David Venable <dlv@amazon.com>

• Set release version to 2.4 (#3214)

  Asif Sohail Mohammed - Tue, 22 Aug 2023 15:16:28 -0700

  Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

• Removes some experimental projects from the 2.4 release and branch. (#3217)

  David Venable - Tue, 22 Aug 2023 15:15:21 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Generated THIRD-PARTY file for fecb842 (#3212) (#3213)

  opensearch-trigger-bot[bot] - Tue, 22 Aug 2023 14:20:56 -0700

  Signed-off-by: GitHub <noreply@github.com> Co-authored-by: dlvenable <dlvenable@users.noreply.github.com> (cherry picked from commit 733e7bc8381d1fe02c30b6c75c4cc2dd8984c21c) Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>

• Normalize the include/exclude keys in the JacksonEvent implementation in order to fix a problem where the ndjson codec was not correctly including/excluding keys. (#3209)

  David Venable - Tue, 22 Aug 2023 13:47:29 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Fix bug where enum for s3 select was using NotBlank annotation (#3208)

  Taylor Gray - Tue, 22 Aug 2023 14:04:46 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Adds an auto_schema flag to require a user to be explicit in using auto-schema generation. (#3206)

  David Venable - Tue, 22 Aug 2023 10:25:03 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Add jenkins release workflow and refactor GHA workflow (#3204)

  Sayali Gaikawad - Tue, 22 Aug 2023 09:31:00 -0700

  Signed-off-by: Sayali Gaikawad <gaiksaya@amazon.com>

• Removes code which isn't used for the Avro and Parquet codecs. This will keep untested and errant code paths out of the project. Resolves #3201. (#3205)

  David Venable - Tue, 22 Aug 2023 07:40:32 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Updates to the S3 sink to speed up the unit test time. There are a few major changes - use the Duration class instead of a nebulous long to have millisecond options and clarity; inject the retry sleep time so that the tests can sleep for shorter time; using mocking where possible to avoid unnecessary sleeps. (#3203)

  David Venable - Tue, 22 Aug 2023 07:05:30 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Fix: IllegalArgument Exception in String converter (#3172)

  Asif Sohail Mohammed - Mon, 21 Aug 2023 12:06:13 -0700

  • Fix: IllegalArgument Exception in String converter Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Added tags_on_failure Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Addressed feedback Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  ---

  Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

• Corrects the output codecs for Avro/Parquet to use the include/exclude keys. Also adds a shouldNotIncludeKey method to OutputCodecContext. (#3197)

  David Venable - Mon, 21 Aug 2023 08:56:19 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Correctly add compression extensions to the generated S3 sink keys. If compression is internal, does not utilize. Resolves #3158. (#3196)

  David Venable - Mon, 21 Aug 2023 08:47:04 -0700

  Signed-off-by: David Venable <dlv@amazon.com>

• Fix uncaught exception causing pipeline shutdown (#3189)

  Hai Yan - Mon, 21 Aug 2023 10:15:19 -0500

  • Catch ClassCastException in JacksonOtelLog.toJsonString()
  • Add overwrite option to parse-json processor

  ---

  Signed-off-by: Hai Yan <oeyh@amazon.com>

• Updates the Data Prepper tar.gz artifact to include JDK 17.0.8_7 which is the current latest version available. (#3136)

  David Venable - Fri, 18 Aug 2023 18:17:15 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• Updated scan start_time ,end_time and range combinations (#3188)

  Asif Sohail Mohammed - Fri, 18 Aug 2023 16:56:54 -0500

  Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

• Corrects the JSON output codec to write Events as provided rather than convert to string. Also fixes the include/exclude keys. Adds a boolean check in OutputCodecContext so that this can be used by other codecs. (#3195)

  David Venable - Fri, 18 Aug 2023 16:44:40 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• Suppport null values in Avro and Parquet output codecs when the schema is null. Auto-generate schemas that are nullable so that null values can be included in these schemas. Resolves part of #3158. (#3194)

  David Venable - Fri, 18 Aug 2023 16:15:56 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• Added validations in include and exclude keys (#3181)

  Asif Sohail Mohammed - Fri, 18 Aug 2023 13:46:27 -0500

  Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

• FIX: set default value for enable_compression (#3190)

  Qi Chen - Fri, 18 Aug 2023 10:47:14 -0500

  Signed-off-by: George Chen <qchea@amazon.com>

• Automatically promote the releases in the release workflow once the release issue has been approved by two maintainers. Resolves #2122. (#3149)

  David Venable - Fri, 18 Aug 2023 08:30:16 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• Continue calling S3SinkService::output even if records is empty to flush stale batches (#3187)

  Chase - Fri, 18 Aug 2023 08:29:49 -0500

  Continue calling S3SinkService::output even if records is empty to flush stale batches (#3187)

  ---

  Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

• merged with latest (#3182)

  kkondaka - Thu, 17 Aug 2023 16:23:06 -0700

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Correct Parquet support for the S3 sink and a new multipart buffer type (#3186)

  David Venable - Thu, 17 Aug 2023 16:35:25 -0500

  Correct the ParquetOutputCodec and moved into the S3 sink project for now. It has a few corrections including support for compression and avoiding multiple S3 copies. This PR also adds a new buffer type to the S3 sink - Multipart uploads. This PR also includes a number of refactorings to the project and the integration tests. Signed-off-by: David Venable <dlv@amazon.com>

• Update Kafka source/sink to use ByteCount (#3183)

  kkondaka - Thu, 17 Aug 2023 14:24:30 -0700

  • Retry without seeking incase of AWSSchemaRegistryException Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Updated to link usage and config to DataPrepper documentation Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Update Kafka source/sink to use ByteCount Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Add Exemplars to metrics generated in aggregate processor (#3165)

  kkondaka - Thu, 17 Aug 2023 09:43:58 -0700

  • Add Exemplars to metrics generated in aggregate processor Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Updated documentation Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Add limit to cardinality key, and metric for cardinality overflow (#3173)

  Jonah Calvo - Thu, 17 Aug 2023 08:40:39 -0700

  • Add limit to cardinality key, and metric for cardinality overflow Signed-off-by: Jonah Calvo <caljonah@amazon.com>

  • Change cardinality overflow warning from once to every five minutes Signed-off-by: Jonah Calvo <caljonah@amazon.com>

  ---

  Signed-off-by: Jonah Calvo <caljonah@amazon.com>

• Adds a new integration test to the S3 sink which can test different scenarios. This currently is testing against ndjson since this codec generally works. (#3179)

  David Venable - Wed, 16 Aug 2023 14:50:36 -0500

  efs/heads/3158-avro-schema Signed-off-by: David Venable <dlv@amazon.com>

• Only log that the scan is complete one time for s3 scan (#3168)

  Taylor Gray - Wed, 16 Aug 2023 13:33:10 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Fix NPE on s3 source stopping without sqs, stop s3 scan worker thread on stopping of the s3 source (#3178)

  Taylor Gray - Wed, 16 Aug 2023 13:09:41 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Support other scan time configs= combinations (#3151)

  Asif Sohail Mohammed - Wed, 16 Aug 2023 12:53:02 -0500

  • Added support for additional time comibinations in s3 scan Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  ---

  Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

• Deprecate document_id_field and add support for document_id with formatting (#3153)

  Taylor Gray - Wed, 16 Aug 2023 12:27:29 -0500

  Deprecate document_id_field and add support for document_id with formatting Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Minor fixes to Kafka Source (#3174)

  kkondaka - Wed, 16 Aug 2023 10:23:15 -0700

  • Minor fixes to Kafka Source Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Removed unused configs Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Reword circuit breaker configuration log message (#3175)

  Taylor Gray - Wed, 16 Aug 2023 12:18:00 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Fix timestamp format (#3171)

  Chase - Wed, 16 Aug 2023 11:52:53 -0500

  Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

• Fix S3 sink writing to closed stream exception (#3170)

  Chase - Wed, 16 Aug 2023 11:22:07 -0500

  Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

• Snappy as a compression option in the S3 sink: adds new option and engine, adds missing unit test for the CompressionOption class, make other compression engine classes package private. (#3155)

  David Venable - Wed, 16 Aug 2023 10:39:35 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• Add catching and logging of exceptions for s3 scan worker (#3159)

  Taylor Gray - Tue, 15 Aug 2023 15:36:04 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Support s3:// prefix (#3156)

  Asif Sohail Mohammed - Tue, 15 Aug 2023 10:39:01 -0500

  Support s3:// prefix Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

• Fixes two flaky unit tests that have failed recently (#3150)

  David Venable - Mon, 14 Aug 2023 16:02:18 -0500

  Fixes two unit tests that have failed and are probably flaky. The ParseTreeEvaluatorListenerTest appears to be using negative values sometimes, which seems to be unsupported. The OTelLogsSourceTest test failed as well, but it appears this code may not always be executed because it was mixing Hamcrest and Mockito. Signed-off-by: David Venable <dlv@amazon.com>

• Fix flaky integration test by wrapping a list in a new list to avoid a ConcurrentModificationException. Resolves #3139. (#3152)

  David Venable - Mon, 14 Aug 2023 16:02:01 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• Fix flakey test caused by RCF variance. Update metric for RCF Instances (#3145)

  Jonah Calvo - Mon, 14 Aug 2023 13:10:21 -0500

  • Fix flakey test caused by RCF variance Signed-off-by: Jonah Calvo <caljonah@amazon.com>

  • Change metric name and type. Update test readability Signed-off-by: Jonah Calvo <caljonah@amazon.com>

  • Fix unit test to account for metric change Signed-off-by: Jonah Calvo <caljonah@amazon.com>

  ---

  Signed-off-by: Jonah Calvo <caljonah@amazon.com>

• Fix Null pointer exception when schema registry not specified (#3147)

  kkondaka - Fri, 11 Aug 2023 18:35:17 -0700

  • Fix Null pointer exception when schema registry not specified Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fix failing test cases Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Glue registry fixes. Fixed a bug in getMSKBootstrapServers (#3142)

  kkondaka - Fri, 11 Aug 2023 10:28:08 -0700

  • Glue registry fixes. Fixed a bug in getMSKBootstrapServers Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Changed default auto commit reset to earliest Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Add default value to cardinality keys (#3144)

  Jonah Calvo - Fri, 11 Aug 2023 11:14:43 -0500

  Signed-off-by: Jonah Calvo <caljonah@amazon.com>

• Adds S3 sink compression. Resolves #3130. (#3138)

  David Venable - Fri, 11 Aug 2023 10:58:22 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• Add support for scheduled scan to s3 scan (#3140)

  Taylor Gray - Fri, 11 Aug 2023 10:54:42 -0500

  Add support for scheduled scan to s3 scan Signed-off-by: Taylor Gray <tylgry@amazon.com>

• The rss-source is defined twice in the settings.gradle and this removes the extra one. (#3134)

  David Venable - Thu, 10 Aug 2023 20:02:37 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• Adding cardinality key support for AD processor (#3073)

  Jonah Calvo - Thu, 10 Aug 2023 13:33:20 -0700

  • Adding cardinality key support for AD processor Signed-off-by: Jonah Calvo <caljonah@amazon.com>

  • Refactor hash function to common package. Add metrics for RCF instances. Implement optional verbose mode for RCF Signed-off-by: Jonah Calvo <caljonah@amazon.com>

  ---

  Signed-off-by: Jonah Calvo <caljonah@amazon.com>

• S3 single scan improvements (#3124)

  Asif Sohail Mohammed - Thu, 10 Aug 2023 15:30:28 -0500

  • S3 single scan improvements Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  ---

  Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

• Correct the behavior of the JSON output codec to write a JSON object first. Adds a configurable keyName for the array. (#3132)

  David Venable - Thu, 10 Aug 2023 15:13:27 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• Add metrics to Kafka Source (#3118)

  kkondaka - Thu, 10 Aug 2023 11:38:43 -0700

  • Add metrics to Kafka Source Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Removed debug print statement Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed failing test case Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Added total committed metric and fixed tests Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed number of committed records stat. Also fixed bug when acknowledgements enabled Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comments. Fixed acknowledgements related bug Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed to use counters for records/bytes consumed metrics Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Removed unused code Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Added a metric for keeping track of number of consumers without any partitions assigned Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Added unit test for KafkaTopicMetrics Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Update the developer_guide.md with updated instructions for user documentation. Updated the PR template to include documentation, and updated the Issues Resolved section to encourage use of "Resolves #". (#2772)

  David Venable - Thu, 10 Aug 2023 11:51:16 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• Improved Avro error reporting related to schemas (#3110)

  David Venable - Thu, 10 Aug 2023 11:22:33 -0500

  Check the Avro schema when starting Data Prepper and throw a clearer exception when an Avro field is missing from the schema. Some code clean-up. Signed-off-by: David Venable <dlv@amazon.com>

• Updates to the Avro codec README.md to include an example of using an Avro schema for VPC Flow Logs. Updates the YAML to make the string easier to handle. (#3111)

  David Venable - Thu, 10 Aug 2023 10:21:05 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• MAINT: add integ test coverage for ODFE 0.10.0 (#3131)

  Qi Chen - Wed, 9 Aug 2023 19:29:13 -0500

  • MAINT: add integ test coverage for ODFE 0.10.0 Signed-off-by: George Chen <qchea@amazon.com>

• Adds Apache commons-lang3 to the Gradle version catalog and updates it to version 3.13.0. (#3120)

  David Venable - Wed, 9 Aug 2023 17:23:16 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• Fixed issue with CloudWatchLogMetrics not updating counters when no event handles are present (#3114)

  Marcos Gonzalez Mayedo - Wed, 9 Aug 2023 16:20:10 -0500

  Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> Co-authored-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

• Add ttl to all dynamo source coordination store items on creation, not just when they are COMPLETED (#3121)

  Taylor Gray - Wed, 9 Aug 2023 13:26:55 -0500

  efs/heads/csv-validation-up-front Add ttl to all dynamo source coordination store items on creation, not just when they are COMPLETED Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Add include_keys and exclude_keys to S3 sink (#3122)

  Aiden Dai - Wed, 9 Aug 2023 13:15:39 -0500

  • Add validation and update document Signed-off-by: Aiden Dai <daixb@amazon.com>

  • Add OutputCodecContext for output codecs. Signed-off-by: Aiden Dai <daixb@amazon.com>

  • Add OutputCodecContext for output codecs. Signed-off-by: Aiden Dai <daixb@amazon.com>

  ---

  Signed-off-by: Aiden Dai <daixb@amazon.com>

• Default keys (#3075)

  Katherine Shen - Tue, 8 Aug 2023 12:19:36 -0500

  • readme and config Signed-off-by: Kat Shen <katshen@amazon.com>

  • skeleton logic written Signed-off-by: Kat Shen <katshen@amazon.com>

  • default keys impl and tests Signed-off-by: Kat Shen <katshen@amazon.com>

  • finish tests Signed-off-by: Kat Shen <katshen@amazon.com>

  • rerun checks Signed-off-by: Kat Shen <katshen@amazon.com>

  • change impl to have parity with logstash Signed-off-by: Kat Shen <katshen@amazon.com>

  • add clarifying example to readme, fix edge cases, add tests Signed-off-by: Kat Shen <katshen@amazon.com>

  • rename test for clarity Signed-off-by: Kat Shen <katshen@amazon.com>

  • change logging statements from string.format() Signed-off-by: Kat Shen <katshen@amazon.com>

  • fix default key check error Signed-off-by: Kat Shen <katshen@amazon.com>

  • change default config name to default_values, fix to have parity with logstash, enhance relevant tests Signed-off-by: Kat Shen <katshen@amazon.com>

  • rerun checks Signed-off-by: Kat Shen <katshen@amazon.com>

  • fix nits Signed-off-by: Kat Shen <katshen@amazon.com>

  • remove extraneous test Signed-off-by: Kat Shen <katshen@amazon.com>

  • clean up illegal argument statements, parameterize tests Signed-off-by: Kat Shen <katshen@amazon.com>

  ---

  Signed-off-by: Kat Shen <katshen@amazon.com> Co-authored-by: Kat Shen <katshen@amazon.com>

• S3 scan enhancements (#3049)

  Asif Sohail Mohammed - Mon, 7 Aug 2023 16:50:20 -0500

  • S3 scan enhancements Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  ---

  Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

• -Support for kafka-sink (#2999)

  rajeshLovesToCode - Mon, 7 Aug 2023 09:07:18 -0700

  • -Support for kafka-sink Signed-off-by: rajeshLovesToCode <rajesh.dharamdasani3021@gmail.com>

  • -Support for kafka-sink Signed-off-by: rajeshLovesToCode <rajesh.dharamdasani3021@gmail.com>

• Centralize exception handling and fix behavior for RequestTimeoutException (#3063)

  Chase - Fri, 4 Aug 2023 16:05:54 -0500

  • Centralize exception handling and fix behavior for RequestTimeoutException Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Fix existing tests Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Add unit tests for exception handlers Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Add copyright headers Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Add better default messages Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  ---

  Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

• Fix code to set max poll interval and fetch min bytes config (#3115)

  kkondaka - Fri, 4 Aug 2023 12:50:56 -0700

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Sns Sink Plugin with junit test cases (#2995)

  Uday Chintala - Fri, 4 Aug 2023 12:58:24 -0500

  Sns Sink Plugin with junit test cases

  ---

  Signed-off-by: Uday Kumar Chintala <udaych20@gmail.com> Signed-off-by: Uday Chintala <udaych20@gmail.com>

• Prometheus Sink Boiler plate code for issue #1744. (#3078)

  mallikagogoi7 - Fri, 4 Aug 2023 10:10:49 -0500

  • Prometheus Sink boiler plate code for issue #1744. Signed-off-by: mallikagogoi7 <mallikagogoi7@gmail.com>

  • Prometheus Sink Fix for issue #1744. Signed-off-by: mallikagogoi7 <mallikagogoi7@gmail.com>

  • Prometheus Sink review comment rsolved for issue #1744. Signed-off-by: mallikagogoi7 <mallikagogoi7@gmail.com>

• ENH: support custom index template for ES6 in opensearch sink (#3061)

  Qi Chen - Thu, 3 Aug 2023 11:59:06 -0500

  Signed-off-by: George Chen <qchea@amazon.com>

• GitHub-Issue#2778: Added README for CloudWatch Logs Sink (#3101)

  Marcos Gonzalez Mayedo - Wed, 2 Aug 2023 17:19:06 -0500

  • Adding README Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added README to sink Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added PluginFunctionality to README Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Update data-prepper-plugins/cloudwatch-logs/README.md Co-authored-by: Mark Kuhn <kuhnmar@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <95880281+MaGonzalMayedo@users.noreply.github.com>

  • Update data-prepper-plugins/cloudwatch-logs/README.md Co-authored-by: Mark Kuhn <kuhnmar@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <95880281+MaGonzalMayedo@users.noreply.github.com>

  • Added fixes to configuration in README and example id Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added extra info in plugin functionality Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  ---

  Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <95880281+MaGonzalMayedo@users.noreply.github.com>

  Co-authored-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> Co-authored-by: Mark Kuhn <kuhnmar@amazon.com>

• Bump com.opencsv:opencsv from 5.7.1 to 5.8 (#3097)

  dependabot[bot] - Wed, 2 Aug 2023 17:13:11 -0500

  Bumps com.opencsv:opencsv from 5.7.1 to 5.8.

  ---

  updated-dependencies:

  • dependency-name: com.opencsv:opencsv dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

• GitHub-Issue#2778: Refactored Config to include Duration and ByteCount types (#3099)

  Marcos Gonzalez Mayedo - Wed, 2 Aug 2023 16:43:33 -0500

  • Converted data types in the configuration to Data-Prepper types Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added Duration to backOffTime Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Attempting to fix unused imports Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  ---

  Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> Co-authored-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

• Updated GitHub Actions to use "Data Prepper" in the job titles for consistency and aligning with the project name. (#3104)

  David Venable - Wed, 2 Aug 2023 13:59:04 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• Fix consumer synchronization. Fix consumer to use user-specified groupId (#3100)

  kkondaka - Wed, 2 Aug 2023 10:43:37 -0700

  • Fix consumer synchronization. Fix consumer to use user-specified groupId Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fix check style error Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed to retry if consume records encounters an exception Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Bump org.xerial.snappy:snappy-java in /data-prepper-plugins/common (#3095)

  dependabot[bot] - Wed, 2 Aug 2023 09:00:13 -0500

  Bumps org.xerial.snappy:snappy-java from 1.1.10.1 to 1.1.10.3.

  • Release notes

  Commits

  ---

  updated-dependencies:

  • dependency-name: org.xerial.snappy:snappy-java dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

• GitHub-Issue#2778: Added CloudWatchLogsSink (#3084)

  Marcos Gonzalez Mayedo - Tue, 1 Aug 2023 15:58:09 -0500

  GitHub-Issue#2778: Refactoring config files for CloudWatchLogs Sink (#4)

  ---

  Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <95880281+MaGonzalMayedo@users.noreply.github.com> Co-authored-by: Taylor Gray <tylgry@amazon.com> Co-authored-by: Marcos <alemayed@amazon.com>

• Fix Negative acknowledgement handling and other minor issues (#3082)

  kkondaka - Tue, 1 Aug 2023 10:10:22 -0700

  • Fix Negative acknowledgement handling and other minor issues Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed check style errors Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Cleanup of unused files and config Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• HttpSink Plugin Functionality for #874. (#3036)

  mallikagogoi7 - Tue, 1 Aug 2023 08:57:06 -0700

  • HttpSink Plugin Functionality for #874. Signed-off-by: mallikagogoi7 <mallikagogoi7@gmail.com>

  • Fixed review comments for #874. Signed-off-by: mallikagogoi7 <mallikagogoi7@gmail.com>

  • Fixes for #874. Signed-off-by: mallikagogoi7 <mallikagogoi7@gmail.com>

• Added Translate Processor README.md file (#3033)

  Vishal Boinapalli - Mon, 31 Jul 2023 16:19:08 -0700

  Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

• GitHub-Issue#2778: Added CouldWatchLogsService, Tests and RetransmissionException (#3023)

  Marcos Gonzalez Mayedo - Mon, 31 Jul 2023 13:03:33 -0500

  • Elasticsearch client implementation with pit and no context search (#2910) Create Elasticsearch client, implement search and pit apis for ElasticsearchAccessor Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • GitHub-Issue#2778: Refactoring config files for CloudWatchLogs Sink (#4) Added Config Files for CloudWatchLogs Sink. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added fixes from comments to code (including pathing and nomenclature syntax)

  Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Refactoring config (#5) Added default params for back_off and log_send_interval alongside test cases for ThresholdConfig. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Fixed deleted AwsConfig file Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Removed the s3 dependency from build.gradle, replaced the AwsAuth.. with AwsConfig. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added modifiable back_off_timer, added threshold test for back_off_timer and params to AwsConfig Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added fixes to gradle file, added tests to AwsConfig, and used Reflective mapping to tests CwlSink Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added default value test to ThresholdConfig and renamed getter for maxRequestSize Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Removed unnecessary imports Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added cloudwatch-logs to settings.gradle Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added a quick fix to the back_off_time range Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added Buffer classes, ClientFactory similar to S3, and ThresholdCheck Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Removed unnecessary default method from ClientFactory Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added comments in Buffer Interface, change some default values to suit the plugin use case more Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Removed unused imports Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Changed the unused imports, made parameters final in the ThresholdCheck Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Made changes to the tests and the method signatures in ThresholdCheck, made fixes to gradle file to include catalog Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Removed unused methods/comments Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added CloudWatchLogsService, CloudWatchLogsServiceTest and RetransmissionLimitException Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Fixed retransmission logging fixed value Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Fixed unused imports Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Fixed making ThresholdCheck public Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added fixes to ThresholdCheck and CloudWatchLogsService to decouple methods Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Fixed syntax start import in CloudWatchLogsServiceTest Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Extracted LogPusher and SinkStopWatch classes for code cleanup. Addded fixes to variables and retry logic for InterruptExceptions Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Changed method uses in CloudWatchLogsService and removed logging the batch size in LogPusher Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added Multithreaded CloudWatchLogsDispatcher for handling various async calls to perform PLE's and added tests Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added fixesto test and defaulted the parameters in the config to CloudWatchLogs limits, customer can change this in config file Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added exponential backofftime Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Fixed unused imports Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Fixed up deepcopy of arraylist for service workers in CloudWatchLogsService, and fixed Log calling methods Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added CloudWatchLogsDispatcher builder pattern, fixed tests for Service and Dispatcher and modified backOffTimeBase Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Removed unused imports Signed-off-by:Marcos Gonzalez Mayedo <alemayed@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added resetBuffer method, removed unnecessary RetransmissionException, and added logString pass in parameter for staging log events. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Started making changes to the tests to implement the new class structure (performance enhancement) Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Refactored the CloudWatchLogsDispatcher into two classes with the addition of Uploader, introduced simple multithread tests for CloudWatchLogsService Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Fixed issues with locking in try block and added final multithreaded tests to the CloudWatchLogsService class Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added CloudWatchLogsMetricsTest, changed upper back off time bound and scale, and refactoring changes for better code syntax (renaming, refactoring methods for conciseness, etc...) Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added changes to javadoc Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Update data-prepper-plugins/cloudwatch-logs/src/main/java/org/opensearch/dataprepper/plugins/sink/client/CloudWatchLogsDispatcher.java

  Co-authored-by: Mark Kuhn <kuhnmar@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <95880281+MaGonzalMayedo@users.noreply.github.com>

  • Fixed comment on CloudWatchLogsDispatcher Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  ---

  Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <95880281+MaGonzalMayedo@users.noreply.github.com> Co-authored-by: Taylor Gray <tylgry@amazon.com> Co-authored-by: Marcos <alemayed@amazon.com>

  Co-authored-by: Mark Kuhn <kuhnmar@amazon.com>

• Config changes and support for adding different modes to put kafka key in the event (#3076)

  kkondaka - Sun, 30 Jul 2023 22:54:47 -0700

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Add include_keys and exclude_keys to sink (#2989)

  Aiden Dai - Fri, 28 Jul 2023 16:43:20 -0700

  Signed-off-by: Aiden Dai <daixb@amazon.com>

• Removed IterateOn otpion (#3050)

  Vishal Boinapalli - Fri, 28 Jul 2023 09:33:08 -0700

  Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

• Changed non-exact mathching logic (#3046)

  Vishal Boinapalli - Fri, 28 Jul 2023 09:31:02 -0700

  Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

• Translate Processor : Added support for S3 file mappings (#3054)

  Vishal Boinapalli - Fri, 28 Jul 2023 09:30:43 -0700

  • Added support for S3 file Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

  • Changed logic for retrieving mappings from S3 file Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

  ---

  Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

• Moves the S3 sink and HTTP sink into their own packages. This fixes an issue where there are class conflicts at runtime. (#3067)

  David Venable - Thu, 27 Jul 2023 19:37:18 -0500

• Exclude keys (#3055)

  Katherine Shen - Thu, 27 Jul 2023 11:35:48 -0500

  • Add exclude keys Signed-off-by: Kat Shen <katshen@amazon.com>

  ---

  Signed-off-by: Kat Shen <katshen@amazon.com> Co-authored-by: Kat Shen <katshen@amazon.com>

• Parquet Sink Codec (#2928)

  umayr-codes - Thu, 27 Jul 2023 08:35:31 -0700

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  ---

  Co-authored-by: umairofficial <umairhusain1010@gmail.com>

• Adds new configurations to the S3 source to better define bucket ownership. Resolves #2012. (#3012)

  David Venable - Wed, 26 Jul 2023 14:41:29 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• -Support for Sink Codecs (#2986)

  umayr-codes - Wed, 26 Jul 2023 11:52:00 -0700

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  • -Support for Sink Codecs Signed-off-by: umairofficial <umairhusain1010@gmail.com>

  ---

  Co-authored-by: umairofficial <umairhusain1010@gmail.com>

• Add support for Glue registry (#3056)

  kkondaka - Tue, 25 Jul 2023 15:32:30 -0700

  • Add Support for Glue registry Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed MskGlueRegistryMultiTypeIT test Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed integration test failures Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Modified change Encryption:PLAINTEXT to Encryption:NONE Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed failing tests Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Moved serdeFormat to TopicConfig Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Remove * imports from MskGlue test Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Improves some of the timeouts for the peer forwarder tests to reduce testing time. (#3020)

  David Venable - Tue, 25 Jul 2023 12:49:46 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• Improves some of the test timing in Data Prepper core tests which are showing somewhat high test times. (#3021)

  David Venable - Tue, 25 Jul 2023 12:49:32 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• Work to reduce test time by reducing some repeated tests, using Awaitility, and reducing delays (#3019)

  David Venable - Tue, 25 Jul 2023 12:49:18 -0500

  Work to reduce test time by reducing some repeated tests, using Awaitility, and reducing some wait times. Signed-off-by: David Venable <dlv@amazon.com>

• FIX: missing request index (#3058)

  Qi Chen - Tue, 25 Jul 2023 12:03:25 -0500

  Signed-off-by: George Chen <qchea@amazon.com>

• Fix bug where exception is thrown when csv source key does not exist or is null (#3053)

  Taylor Gray - Tue, 25 Jul 2023 12:01:25 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• ENH: support es 6 in sink (#3045)

  Qi Chen - Mon, 24 Jul 2023 15:52:17 -0500

  • ENH: support es 6 for bulk API Signed-off-by: George Chen <qchea@amazon.com>

• Kafka source Confluent schema registry connectivity and OAuth implementation (#3037)

  Ajeesh Gopalakrishnakurup - Mon, 24 Jul 2023 11:27:48 -0700

  • Schema registry connectivity with the oauth configurations Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

  • Junit fixes Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

  • Defect fixes Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

  • Updated the review comments for the PR3037 Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

  ---

  Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

• Remove brackets feature option (#3035)

  Katherine Shen - Mon, 24 Jul 2023 13:17:14 -0500

  *add remove brackets feature option Signed-off-by: Kat Shen <katshen@amazon.com>

  ---

  Signed-off-by: Kat Shen <katshen@amazon.com> Co-authored-by: Kat Shen <katshen@amazon.com>

• Add Support for Auth/NoAuth with/without Encryption in Kafka with integration tests (#3042)

  kkondaka - Thu, 20 Jul 2023 09:20:15 -0700

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Bump word-wrap from 1.2.3 to 1.2.4 in /release/staging-resources-cdk (#3044)

  dependabot[bot] - Thu, 20 Jul 2023 10:26:17 -0500

  Bumps word-wrap from 1.2.3 to 1.2.4.

  • Release notes
  • Commits

  ---

  updated-dependencies:

  • dependency-name: word-wrap dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

• Translate Processor - Added file_path functionality for local files (#3034)

  Vishal Boinapalli - Wed, 19 Jul 2023 14:20:24 -0700

  • Changed target_type option name to type Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

  • Added file_path functionality for local file Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

  • Fixed file parsing issue and changed error msgs Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

  • Added IOException to log, made testcase change for mappings validation Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

  ---

  Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

• Consolidate logic related to extracting data from a BulkOperation (#3041)

  Chase - Wed, 19 Jul 2023 13:15:41 -0500

  Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

• Kafka Source - Cleanup and Enhancements for MSK (#3029)

  kkondaka - Tue, 18 Jul 2023 15:54:21 -0700

  • Kafka Source - Cleanup and Enhancements for MSK Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed checkstyle error Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Fix race condition in data prepper sources using e2e acknowledgements (#3039)

  kkondaka - Tue, 18 Jul 2023 15:53:41 -0700

  • Fix race condition in data prepper sources using e2e acknowledgements Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed checkStyle error Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Add support for using expressions with formatString in JacksonEvent, use for index in OpenSearch sink (#3032)

  Taylor Gray - Tue, 18 Jul 2023 12:10:24 -0500

  Add support for using expressions with formatString in JacksonEvent, use for index in OpenSearch sink Signed-off-by: Taylor Gray <tylgry@amazon.com>

  ---

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• -Support for Sink Codecs (#3030)

  Omkar - Mon, 17 Jul 2023 13:28:58 -0700

  • -Support for Sink Codecs Signed-off-by: omkarmmore95 <omkar.m.more95@gmail.com>

  • -Support for Sink Codecs Signed-off-by: omkarmmore95 <omkar.m.more95@gmail.com>

  • -Support for Sink Codecs Signed-off-by: omkarmmore95 <omkar.m.more95@gmail.com>

• Duplicate values (#3026)

  Katherine Shen - Mon, 17 Jul 2023 09:35:57 -0700

  • implement transform_key feature Signed-off-by: Kat Shen <katshen@amazon.com>

  • fix unit tests Signed-off-by: Kat Shen <katshen@amazon.com>

  • fix unit tests Signed-off-by: Kat Shen <katshen@amazon.com>

  • remove bin files Signed-off-by: Kat Shen <katshen@amazon.com>

  • add static final variable for string comparison Signed-off-by: Kat Shen <katshen@amazon.com>

  • add whitespace description to readme, add configs Signed-off-by: Kat Shen <katshen@amazon.com>

  • writing whitespace impl Signed-off-by: Kat Shen <katshen@amazon.com>

  • add whitespace impl Signed-off-by: Kat Shen <katshen@amazon.com>

  • fix impl, writing tests Signed-off-by: Kat Shen <katshen@amazon.com>

  • write whitespace test Signed-off-by: Kat Shen <katshen@amazon.com>

  • fix formatting, whitespace() -> trimWhitespace() Signed-off-by: Kat Shen <katshen@amazon.com>

  • edit readme, add config Signed-off-by: Kat Shen <katshen@amazon.com>

  • update logic to valid values set Signed-off-by: Kat Shen <katshen@amazon.com>

  • correct return value Signed-off-by: Kat Shen <katshen@amazon.com>

  • update variables to static Signed-off-by: Kat Shen <katshen@amazon.com>

  • correct convention for private variables Signed-off-by: Kat Shen <katshen@amazon.com>

  • impl allow duplicate values, writing tests Signed-off-by: Kat Shen <katshen@amazon.com>

  • allow duplicate values impl + tests Signed-off-by: Kat Shen <katshen@amazon.com>

  • modify regex portion to final variables, remove some whitespace Signed-off-by: Kat Shen <katshen@amazon.com>

  • rerun checks Signed-off-by: Kat Shen <katshen@amazon.com>

  • rename methods/variables for more clarity, change default bool value to be false Signed-off-by: Kat Shen <katshen@amazon.com>

  • rerun checks Signed-off-by: Kat Shen <katshen@amazon.com>

  • change logic to reflect skip_duplicate_values Signed-off-by: Kat Shen <katshen@amazon.com>

  • modify tests according to changed logic Signed-off-by: Kat Shen <katshen@amazon.com>

  • remove include keys content (accidentally included it oops) Signed-off-by: Kat Shen <katshen@amazon.com>

  ---

  Signed-off-by: Kat Shen <katshen@amazon.com> Signed-off-by: Katherine Shen <40495707+shenkw1@users.noreply.github.com> Co-authored-by: Kat Shen <katshen@amazon.com>

• Connection code of HttpSink Plugin for #874. (#2987)

  mallikagogoi7 - Mon, 17 Jul 2023 09:49:32 -0500

  Connection code of HttpSink Plugin. Signed-off-by: mallikagogoi7 <mallikagogoi7@gmail.com>

• GitHub-issue#253 : Implemented GeoIP processor integration test (#2927)

  venkataraopasyavula - Fri, 14 Jul 2023 14:54:33 -0700

  • GitHub-issue#253 : Implemented GeoIP processor integration test

  Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

  • GitHub-issue#253 : Implemented GeoIP processor integration test

  Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

  • GitHub-issue#253 : Implemented GeoIP processor integration test

  Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

  • GitHub-issue#253 : Implemented GeoIP processor integration test

  Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

  • GitHub-issue#253 : Implemented GeoIP processor integration test

  Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

• Add support for Data Prepper expressions in the document_id_field of the OpenSearch sink, add opensearch prefix to opensearch source metadata keys (#3025)

  Taylor Gray - Fri, 14 Jul 2023 10:51:25 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Translate Plugin: Simplified Config. (#3022)

  Vishal Boinapalli - Thu, 13 Jul 2023 22:18:06 -0700

  • Translate Plugin: Simplified Config. Added functionality for multiple sources and multiple targets Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

  • Moved helper methods out of config file Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

  ---

  Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

• Adds the Data Prepper 2.3.2 change log. (#3024)

  David Venable - Thu, 13 Jul 2023 12:02:45 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• Updated Kafka security configuration (#2994)

  kkondaka - Thu, 13 Jul 2023 08:20:50 -0700

  • Add Kafka Security Configurations Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Modified kafka security config. Added new fields to AwsConfig Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Modified AwsConfig to have msk option that can take multiple options Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• GitHub-Issue#2778: Added CloudWatchLogs Buffer, ThresholdCheck, and ClientFactory utilities. (#2982)

  Marcos Gonzalez Mayedo - Wed, 12 Jul 2023 16:35:47 -0500

  Added CloudWatchLogs Buffer, ThresholdCheck, and ClientFactory utilities.

  ---

  Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <95880281+MaGonzalMayedo@users.noreply.github.com>

  Co-authored-by: Marcos <alemayed@amazon.com>

• Whitespace (#3004)

  Katherine Shen - Wed, 12 Jul 2023 13:28:59 -0500

  implement transform_key feature Signed-off-by: Kat Shen <katshen@amazon.com>

  ---

  Signed-off-by: Kat Shen <katshen@amazon.com> Co-authored-by: Kat Shen <katshen@amazon.com>

• Release notes for Data Prepper 2.3.2 (#3016)

  David Venable - Wed, 12 Jul 2023 13:13:11 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• GitHub-issue#253 : Implemented GeoIP processor functionality (#2925)

  venkataraopasyavula - Wed, 12 Jul 2023 08:58:22 -0700

  • GitHub-issue#253 : Implemented GeoIP processor functionality Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

  • GitHub-issue#253 : Implemented GeoIP processor functionality Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

  • GitHub-issue#253 : Implemented GeoIP processor functionality Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

  • GitHub-issue#253 : Implemented GeoIP processor functionality Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

  • GitHub-issue#253 : Implemented GeoIP processor functionality Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

  • GitHub-issue#253 : Implemented GeoIP processor functionality. Addressed Code review comments Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

  • GitHub-issue#253 : Implemented GeoIP processor functionality. Addressed Code review comments Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

  • GitHub-issue#253 : Fixed the test-case-failed issue. Signed-off-by: Deepak Sahu <deepak.sahu562@gmail.com>

  • GitHub-issue#253 : Implemented GeoIP processor functionality. Addressed Code review comments Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

  • GitHub-issue#253 : Implemented GeoIP processor functionality. Addressed Code review comments Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

  • GitHub-issue#253 : Implemented GeoIP processor functionality. Addressed Code review comments Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

  • GitHub-issue#253 : Implemented GeoIP processor functionality. Addressed Code review comments Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

  • GitHub-issue#253 : Implemented GeoIP processor functionality. Addressed Code review comments Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

  ---

  Signed-off-by: Deepak Sahu <deepak.sahu562@gmail.com> Co-authored-by: Deepak Sahu <deepak.sahu562@gmail.com>

• Fix bucket ownership validation. Resolves #3005 (#3009)

  David Venable - Tue, 11 Jul 2023 21:30:00 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• Introduce option to measure bulk sizes with or without compression (#2985)

  Chase - Tue, 11 Jul 2023 18:17:33 -0500

  • Initial bulk estimation improvements Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Add setting to enable/disable estimation with compression Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Make max local compressions configurable Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Add documentation for new settings Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Remove debug comment Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Remove another debug log Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  ---

  Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

• Fix race condition in SqsWorker when acknowledgements are enabled (#3001)

  kkondaka - Tue, 11 Jul 2023 15:09:26 -0700

  • Fix race condition in SqsWorker when acknowledgements are enabled Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Modified to do the synchronization in the acknowledgement set framework Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed failing tests Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Removed unused variable Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comment and fixed failing tests Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed failing tests Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed checkStyle failure Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Retry s3 reads on socket exceptions. (#2992)

  Adi Suresh - Tue, 11 Jul 2023 12:25:20 -0500

  • Retry s3 reads on socket exceptions. S3 will reset the conenction on their end frequently. To not lose data, data prepper should retry all socket exceptions by attempting to re-open the stream. Signed-off-by: Adi Suresh <adsuresh@amazon.com>

  • Bubble up parquet exceptions. Signed-off-by: Adi Suresh <adsuresh@amazon.com>

  ---

  Signed-off-by: Adi Suresh <adsuresh@amazon.com>

• Fix SqsWorker error messages (#2991)

  kkondaka - Fri, 7 Jul 2023 10:25:47 -0700

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Translate Plugin - Target Type implementation (#2979)

  Vishal Boinapalli - Thu, 6 Jul 2023 16:29:02 -0700

  • Translate Plugin - Target Type implementation Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

  • addressed review comments Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

  ---

  Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

• Implement transform_key feature (#2977)

  Katherine Shen - Thu, 6 Jul 2023 11:37:55 -0500

  implement transform_key feature Signed-off-by: Kat Shen <katshen@amazon.com>

• Fix S3 errors around end of file behavior. (#2983)

  Adi Suresh - Wed, 5 Jul 2023 16:20:12 -0500

  Signed-off-by: Adi Suresh <adsuresh@amazon.com>

• Fix Stdout and File sink (#2978)

  kkondaka - Wed, 5 Jul 2023 10:15:15 -0700

  • Fix Stdout and File sink Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed javadoc warnings and errors Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Boiler plate code of HttpSink for #874. (#2916)

  mallikagogoi7 - Wed, 5 Jul 2023 09:27:57 -0700

  • Boiler plate code of HttpSink for #874. Signed-off-by: mallikagogoi7 <mallikagogoi7@gmail.com>

  • Added copyright on classes of HttpSink for #874. Signed-off-by: mallikagogoi7 <mallikagogoi7@gmail.com>

  • Moved Accumulator package to common for #874. Signed-off-by: mallikagogoi7 <mallikagogoi7@gmail.com>

  • Test cases added for accumulator related classes for #874. Signed-off-by: mallikagogoi7 <mallikagogoi7@gmail.com>

  • Added HttpSink related methos in accumulator for #874. Signed-off-by: mallikagogoi7 <mallikagogoi7@gmail.com>

  • Removed plugin specific methods from common for #874. Signed-off-by: mallikagogoi7 <mallikagogoi7@gmail.com>

• GitHub-Issue#2778: Added CloudWatchLogs Sink Config Files (#2922)

  Marcos Gonzalez Mayedo - Wed, 5 Jul 2023 09:12:03 -0700

  • Elasticsearch client implementation with pit and no context search (#2910) Create Elasticsearch client, implement search and pit apis for ElasticsearchAccessor Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • GitHub-Issue#2778: Refactoring config files for CloudWatchLogs Sink (#4) Added Config Files for CloudWatchLogs Sink. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added fixes from comments to code (including pathing and nomenclature syntax)

  Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Refactoring config (#5) Added default params for back_off and log_send_interval alongside test cases for ThresholdConfig. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Fixed deleted AwsConfig file Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Removed the s3 dependency from build.gradle, replaced the AwsAuth.. with AwsConfig. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added modifiable back_off_timer, added threshold test for back_off_timer and params to AwsConfig Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added fixes to gradle file, added tests to AwsConfig, and used Reflective mapping to tests CwlSink Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added default value test to ThresholdConfig and renamed getter for maxRequestSize Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Removed unnecessary imports Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added cloudwatch-logs to settings.gradle Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  • Added a quick fix to the back_off_time range Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

  ---

  Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <95880281+MaGonzalMayedo@users.noreply.github.com> Co-authored-by: Taylor Gray <tylgry@amazon.com> Co-authored-by: Marcos <alemayed@amazon.com>

• Bump org.apache.commons:commons-compress in /data-prepper-plugins/common (#2960)

  dependabot[bot] - Wed, 5 Jul 2023 10:59:41 -0500

  Bumps org.apache.commons:commons-compress from 1.21 to 1.23.0.

  ---

  updated-dependencies:

  • dependency-name: org.apache.commons:commons-compress dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

• Bump commons-io:commons-io from 2.11.0 to 2.13.0 in /data-prepper-api (#2900)

  dependabot[bot] - Wed, 5 Jul 2023 10:57:43 -0500

  Bumps commons-io:commons-io from 2.11.0 to 2.13.0.

  ---

  updated-dependencies:

  • dependency-name: commons-io:commons-io dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

• Bump com.fasterxml.jackson.datatype:jackson-datatype-jsr310 (#2796)

  dependabot[bot] - Wed, 5 Jul 2023 10:55:11 -0500

  Bumps com.fasterxml.jackson.datatype:jackson-datatype-jsr310 from 2.14.2 to 2.15.2.

  ---

  updated-dependencies:

  • dependency-name: com.fasterxml.jackson.datatype:jackson-datatype-jsr310 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

• -Support for Sink Codecs (#2881)

  Omkar - Wed, 5 Jul 2023 10:53:44 -0500

  -Support for Sink Codecs Signed-off-by: omkarmmore95 <omkar.m.more95@gmail.com>

• Added Kafka config to support acknowledgments and MSK arn (#2976)

  kkondaka - Tue, 4 Jul 2023 21:25:09 -0700

  • Added Kafka config to support acknowledgments and MSK arn Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Modified to use data-prepper-core in testImplementation Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed failing test Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Define multiple keys for type conversion (#2934)

  wanghd89 - Tue, 4 Jul 2023 21:42:07 -0500

  • feat: add include_key options to KeyValueProcessor Signed-off-by: Haidong <whaidong@amazon.com>

  ---

  Signed-off-by: Haidong <whaidong@amazon.com> Co-authored-by: Haidong <whaidong@amazon.com>

• Translate Plugin - Added functionality for iterate_on, default, exact options (#2953)

  Vishal Boinapalli - Mon, 3 Jul 2023 14:32:32 -0500

  Added IterateOn functionality, default, exact and testcases for translate processor Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

• Kafka Source code refactoring (#2951)

  kkondaka - Fri, 30 Jun 2023 13:01:29 -0700

  • Kafka Source code refactoring Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixes for failing build/tests Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Addressed review comments. Cleaned up code Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Add disable_authentication flag to the opensearch source (#2942)

  Taylor Gray - Thu, 29 Jun 2023 11:37:34 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Fix CVE-2023-35165, CVE-2023-34455, CVE-2023-34453, CVE-2023-34454, C… (#2948)

  Asif Sohail Mohammed - Thu, 29 Jun 2023 09:59:03 -0500

  • Fix CVE-2023-35165, CVE-2023-34455, CVE-2023-34453, CVE-2023-34454, CVE-2023-2976 Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Updated snappy version in build.gradle files Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  ---

  Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

• Update simple_pipelines.md (#2947)

  Katherine Shen - Thu, 29 Jun 2023 09:29:49 -0500

  Removed a space between the second ${PWD} and the rest of the line Signed-off-by: Katherine Shen <40495707+shenkw1@users.noreply.github.com>

• Adding Translate Processor functionality and config files (#2913)

  Vishal Boinapalli - Wed, 28 Jun 2023 13:45:02 -0700

  • Adding MapValues Processor functionality and config file Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

  • Changed logic for splitting of Keys, Added config file for Regex option Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

  • Added testcases, made changes addressing previous review comments, Changed the processor name from map_values to translate Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

  • Added regexConfigTests, made code structure changes, added check for patterns under regex Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

  • Removed * imports Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

  ---

  Signed-off-by: Vishal Boinapalli <vishalboinapalli3@gmail.com>

• Sqs Source Integration Tests (#2874)

  Uday Chintala - Wed, 28 Jun 2023 10:02:25 -0500

  Sqs Source Integration Tests Signed-off-by: Uday Kumar Chintala <udaych20@gmail.com>

• Remove validation that made keys starting or ending with . - or _ invalid, catch all exceptions in the parse json processor (#2945)

  Taylor Gray - Tue, 27 Jun 2023 16:25:19 -0500

  Remove validation that made keys starting or ending with . - or _ invalid, catch all exceptions in the parse json processor Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Kafka source integration test (#2891)

  Ajeesh Gopalakrishnakurup - Tue, 27 Jun 2023 10:11:52 -0700

  • Integration testcases Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

  • Fix for the Integration testcases Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

  • Fix for the white source issue Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

  • Fixes for the merge conflicts Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

  ---

  Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

• Add basic opensearch source documentation for config (#2940)

  Taylor Gray - Tue, 27 Jun 2023 12:02:36 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Add support for writing tags along with events to Sink (#2850)

  kkondaka - Tue, 27 Jun 2023 08:42:27 -0700

  • Updated to pass SinkContext to Sink constructors as suggested in the previous comments Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed check style errors and renamed RoutedPluginSetting to SinkContextPluginSetting Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed s3-sink integration test Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Added javadoc for SinkContext Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Temporarily reduce coverage minimum (#2937)

  kkondaka - Mon, 26 Jun 2023 16:50:03 -0700

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Implement scroll apis for ElasticSearch Accessor (#2930)

  Taylor Gray - Mon, 26 Jun 2023 14:58:36 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Sqs Source initial changes (#2786)

  Uday Chintala - Mon, 26 Jun 2023 13:12:28 -0500

  • Sqs Source implementation Signed-off-by: Uday Kumar Chintala <udaych20@gmail.com>

  ---

  Signed-off-by: Uday Kumar Chintala <udaych20@gmail.com> Signed-off-by: Uday Chintala <udaych20@gmail.com>

• Fix DLQ writer writing empty list (#2931)

  kkondaka - Mon, 26 Jun 2023 09:48:31 -0700

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• S3 Scan time range improvements (#2883)

  Hai Yan - Mon, 26 Jun 2023 11:11:40 -0500

  • When no time range set, default to scan all objects; allow setting time range for specific bucket Signed-off-by: Hai Yan <oeyh@amazon.com>

  ---

  Signed-off-by: Hai Yan <oeyh@amazon.com>

• Implement searching with scroll contexts for OpenSearch (#2923)

  Taylor Gray - Fri, 23 Jun 2023 15:26:52 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Fix addTags API in EventMetadata (#2926)

  kkondaka - Fri, 23 Jun 2023 10:07:09 -0700

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• GitHub-issue#253 : Implemented GeoIP Plugin Configuration Code JUnit test cases (#2909)

  venkataraopasyavula - Thu, 22 Jun 2023 09:11:48 -0700

  Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

• Elasticsearch client implementation with pit and no context search (#2910)

  Taylor Gray - Wed, 21 Jun 2023 13:59:00 -0500

  Create Elasticsearch client, implement search and pit apis for ElasticsearchAccessor Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Added Readme.md (#2714)

  Ajeesh Gopalakrishnakurup - Wed, 21 Jun 2023 11:49:55 -0700

  • Added Readme.md Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

  • Updated Readme.md Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

  • Updated Readme.md Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

  ---

  Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

• OpenSearch Sink Optimizations (#2908)

  Chase - Wed, 21 Jun 2023 10:12:26 -0500

  • Fix size estimation for compression. Maintain requests across iterations for better packing. Limit bulk response size Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Add unit tests, slight refactors Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Add null handling Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Increase gradle heap Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Set flush timeout in IT Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Set flush timeout to 0 in ITs Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Add documentation for flush_timeout and fix OpenSearchSinkITs Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Add default to documentation Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  • Set flush_timeout to 5s in e2e tests to fall within timeouts Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

  ---

  Signed-off-by: Chase Engelbrecht <engechas@amazon.com>

• Updated the release date (#2911)

  Asif Sohail Mohammed - Tue, 20 Jun 2023 13:45:17 -0500

  Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

• Kafka source retry (#2708)

  Ajeesh Gopalakrishnakurup - Tue, 20 Jun 2023 09:50:26 -0700

  • Code rebase for the PR2708 Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

  • Fixes the code rebase issue for the PR2708 Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

  • Fixes the code rebase issue and code refactoring Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

  • Fixes the build issue Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

  • Refactored the consumer code for plaintext,json and avro Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

  • Updated the review comments for the PR2708 Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

  ---

  Signed-off-by: Ajeesh Gopalakrishnakurup <ajeesh.akd@gmail.com>

• Configuration PR for HttpSink for #874. (#2890)

  mallikagogoi7 - Mon, 19 Jun 2023 11:40:49 -0700

  Signed-off-by: mallikagogoi7 <mallikagogoi7@gmail.com>

• Added 2.3.1 release notes (#2871)

  Asif Sohail Mohammed - Mon, 19 Jun 2023 12:44:40 -0500

  • Added 2.3.1 release notes Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Updated release notes Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  ---

  Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

• Added 2.3.1 change log (#2872)

  Asif Sohail Mohammed - Mon, 19 Jun 2023 12:44:25 -0500

  • Added 2.3.1 change log Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Updated change log Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  ---

  Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

• GitHub-issue#253 : Implemented GeoIP Plugin boiler plate Code Structure (#2840)

  venkataraopasyavula - Mon, 19 Jun 2023 10:01:15 -0700

  Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

• Change the title to be consistent with configuration (#2899)

  kkondaka - Mon, 19 Jun 2023 09:13:06 -0700

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Do not suppress logs when there are exception in s3 source. (#2896)

  Adi Suresh - Sat, 17 Jun 2023 10:04:23 -0500

  Signed-off-by: Adi Suresh <adsuresh@amazon.com>

• Change log for index name format failure in opensearch sink (#2894)

  Taylor Gray - Fri, 16 Jun 2023 16:09:38 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Updated s3 sink metrics (#2888)

  Asif Sohail Mohammed - Fri, 16 Jun 2023 08:48:44 -0500

• Implement NoSearchContextWorker to search with search_after and not use pit or scroll, allow override with search_context_type parameter (#2873)

  Taylor Gray - Thu, 15 Jun 2023 16:14:33 -0500

  • Implement NoSearchContextWorker to search with search_after and not use pit or scroll, allow override with search_context_type parameter Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Fix silent dropping of data when index format has null keys, write to dlq if configured (#2885)

  Taylor Gray - Thu, 15 Jun 2023 15:12:30 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• GitHub-issue#253 : Implemented GeoIP Plugin Configuration Code (#2811)

  venkataraopasyavula - Thu, 15 Jun 2023 12:31:04 -0700

  • GitHub-issue#253 : Implemented GeoIP Plugin Configuration Code

  Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

  • GitHub-issue#253 : Incorporated code review comments of GeoIP Plugin Configuration Code Signed-off-by: venkataraopasyavula <venkataraopasyavula@gmail.com>

• Update README.md for S3 sink (#2878)

  Travis Benedict - Thu, 15 Jun 2023 10:39:53 -0500

  Signed-off-by: Travis Benedict <benedtra@amazon.com>

• Add exception when gzip input stream not have magic header. (#2879)

  Adi Suresh - Thu, 15 Jun 2023 09:28:17 -0500

  Signed-off-by: Adi Suresh <adsuresh@amazon.com>

• FIX: concurrentModification (#2876)

  Qi Chen - Wed, 14 Jun 2023 21:24:07 -0500

  Signed-off-by: George Chen <qchea@amazon.com>

• S3 EventBridge and security lake support (#2861)

  Asif Sohail Mohammed - Tue, 13 Jun 2023 17:53:18 -0500

  • EventBridge initial working draft Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

• Add STS external ID to all STS configurations. (#2862)

  Adi Suresh - Tue, 13 Jun 2023 16:08:30 -0500

  STS external ID is required by some AWS services when making an STS AssumeRole call. Signed-off-by: Adi Suresh <adsuresh@amazon.com>

• Save searchAfter to state so pagination can continue where it left off when using PIT on opensearch source (#2856)

  Taylor Gray - Mon, 12 Jun 2023 15:56:57 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Consolidate BufferAccumulator to buffer-api module (#2857)

  Taylor Gray - Mon, 12 Jun 2023 13:29:56 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Add end-to-end acknowledgement support to Stdout and File Sinks (#2860)

  kkondaka - Mon, 12 Jun 2023 11:18:18 -0700

  • Add end-to-end acknowledgement support to Stdout and File Sinks Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Fixed code to check for object is an instance of event Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  ---

  Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

• Sqs Source configuration code changes for #2679 (#2801)

  Uday Chintala - Mon, 12 Jun 2023 12:04:53 -0500

  Sqs Source configuration code changes for #2679 Signed-off-by: Uday Kumar Chintala <udaych20@gmail.com>

• Os source buffer backoff retry (#2849)

  Taylor Gray - Mon, 12 Jun 2023 09:40:10 -0500

  Use buffer accumulator in opensearch source to backoff and retry Signed-off-by: Taylor Gray <tylgry@amazon.com>

• -Support for Sink Codecs (#2845)

  Omkar - Fri, 9 Jun 2023 13:40:05 -0500

  Support for Sink Codecs Signed-off-by: omkarmmore95 <omkar.m.more95@gmail.com>

• Implement basic search with point in time and search after (#2847)

  Taylor Gray - Fri, 9 Jun 2023 10:06:50 -0500

  Implement basic search with point in time and search after Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Fix CVE in maven-artifact by excluding that dependency (#2848)

  Taylor Gray - Thu, 8 Jun 2023 15:52:39 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• -Support for Sink Codecs (#2842)

  Omkar - Thu, 8 Jun 2023 14:47:08 -0500

  Signed-off-by: omkarmmore95 <omkar.m.more95@gmail.com>

• Create and delete point in time for processing an index (#2839)

  Taylor Gray - Thu, 8 Jun 2023 11:32:22 -0500

  Signed-off-by: Taylor Gray <tylgry@amazon.com>

• GitHub-issue#2822: Define null characters in convert processor (#2844)

  saydar31 - Thu, 8 Jun 2023 11:28:37 -0500

  GitHub-issue#2822: Define null characters in convert processor Signed-off-by: Aidar Shaidullin <ajdarshaydullin@gmail.com> Signed-off-by: saydar31 <ajdarshaydullin@gmail.com>

  ---

  Signed-off-by: saydar31 <ajdarshaydullin@gmail.com> Co-authored-by: saydar31 <ajdarshaydullin@gmail.com>

• Implement opensearch index partition creation supplier and PitWorker without processing indices (#2821)

  Taylor Gray - Tue, 6 Jun 2023 17:59:59 -0500

  Implement opensearch index partition creation supplier and PitWorker without processing indices Signed-off-by: Taylor Gray <tylgry@amazon.com>

• Added change log for 2.3 (#2836)

  Asif Sohail Mohammed - Tue, 6 Jun 2023 12:32:55 -0500

  • Added change log for 2.3 Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  • Generated change log in 2.3 branch Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

  ---

  Signed-off-by: Asif Sohail Mohammed <nsifmoh@amazon.com>

• Adds release notes for Data Prepper 2.3.0. (#2833)

  David Venable - Tue, 6 Jun 2023 10:59:57 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• Updates main to Data Prepper 2.4. (#2832)

  David Venable - Tue, 6 Jun 2023 10:44:47 -0500

  Signed-off-by: David Venable <dlv@amazon.com>

• Fix bug where s3 stream was closing too early. (#2830)

  Adi Suresh - Mon, 5 Jun 2023 20:21:40 -0500

  Signed-off-by: Adi Suresh <adsuresh@amazon.com>

• Generated THIRD-PARTY file for 3a70e73 (#2828)

  opensearch-trigger-bot[bot] - Mon, 5 Jun 2023 17:06:11 -0500

  Signed-off-by: GitHub <noreply@github.com> Co-authored-by: dlvenable <dlvenable@users.noreply.github.com>