Skip to content

data-prepper.change-log-2.10.0.md

Latest commit

 

History

History
1055 lines (608 loc) · 37 KB

data-prepper.change-log-2.10.0.md

File metadata and controls

1055 lines (608 loc) · 37 KB

  • Add retry to make sure source is not shutdown when exceptions are thrown on the main thread (#5029)

    Souvik Bose - Thu, 10 Oct 2024 19:47:40 -0700

    EAD -> refs/heads/add-change-log-2.10, refs/heads/main

    • Add retry to make sure source is not shutdown when exceptions are thrown on the main thread Signed-off-by: Souvik Bose <souvbose@amazon.com>

    • Address review comments Signed-off-by: Souvik Bose <souvbose@amazon.com>

    Signed-off-by: Souvik Bose <souvbose@amazon.com> Co-authored-by: Souvik Bose <souvbose@amazon.com>

  • New OpenSearch API source implementation (#5024)

    Souvik Bose - Thu, 10 Oct 2024 19:45:54 -0700

    Signed-off-by: Souvik Bose <souvik04in@gmail.com> Signed-off-by: Souvik Bose <souvbose@amazon.com> Co-authored-by: Souvik Bose <souvbose@amazon.com>

  • Adding support for compression config per stream. (#5046)

    Souvik Bose - Thu, 10 Oct 2024 19:20:41 -0700

    Signed-off-by: Souvik Bose <souvbose@amazon.com> Co-authored-by: Souvik Bose <souvbose@amazon.com>

  • Rename convert_entry_type to convert_type and keep the original name as a deprecated name. (#5047)

    David Venable - Thu, 10 Oct 2024 21:08:33 -0500

    Signed-off-by: David Venable <dlv@amazon.com>

  • Adding Kinesis README (#5048)

    Souvik Bose - Thu, 10 Oct 2024 21:08:17 -0500

    Signed-off-by: Souvik Bose <souvbose@amazon.com> Co-authored-by: Souvik Bose <souvbose@amazon.com>

  • Adding ttl_delete parameter to metadata for DynamoDB (#4982)

    Lee - Thu, 10 Oct 2024 15:11:27 -0500

    Signed-off-by: Lee Hannigan <lhnng@amazon.com>

  • Fix service map tests (#5043)

    Hai Yan - Thu, 10 Oct 2024 12:28:07 -0500

  • Support secret rotation in RDS source (#5016)

    Hai Yan - Thu, 10 Oct 2024 12:04:07 -0500

    Signed-off-by: Hai Yan <oeyh@amazon.com>

  • FIX: mutate string processor configs (#5042)

    Qi Chen - Thu, 10 Oct 2024 09:37:26 -0700

    Signed-off-by: George Chen <qchea@amazon.com>

  • Add NotEmpty annotation to grok match parameter (#5033)

    Taylor Gray - Wed, 9 Oct 2024 15:03:34 -0500

    Signed-off-by: Taylor Gray <tylgry@amazon.com>

  • ENH: expose dbPath to be configurable (#5037)

    Qi Chen - Wed, 9 Oct 2024 13:54:25 -0500

    • ENH: expose dbPath to be configurable Signed-off-by: George Chen <qchea@amazon.com>

  • Adding a metadata attribute for sequence number of record (#5036)

    Souvik Bose - Wed, 9 Oct 2024 11:26:30 -0700

    • Adding a metadata attribute for sequence number of record Signed-off-by: Souvik Bose <souvbose@amazon.com>

    • Add test to check for the metadata Signed-off-by: Souvik Bose <souvbose@amazon.com>

    Signed-off-by: Souvik Bose <souvbose@amazon.com> Co-authored-by: Souvik Bose <souvbose@amazon.com>

  • Documentation improvements for the aggregate processor. (#5035)

    David Venable - Wed, 9 Oct 2024 09:28:54 -0700

    Adds property and class description to configurations. Corrects property order. Adds configuration classes with documentation for put_all and remove_duplicates which now allows for including these. Corrects use of enums and using @JsonValue to have usable documentation on these enums. Signed-off-by: David Venable <dlv@amazon.com>

  • Fix the build break for kinesis source (#5034)

    Souvik Bose - Tue, 8 Oct 2024 14:18:12 -0700

    efs/heads/base64-codec Signed-off-by: Souvik Bose <souvbose@amazon.com> Co-authored-by: Souvik Bose <souvbose@amazon.com>

  • Documentation improvements for a number of Data Prepper processors: Re-ordering properties; adding missing documentation; missed enums. (#5026)

    David Venable - Tue, 8 Oct 2024 11:22:50 -0700

    Signed-off-by: David Venable <dlv@amazon.com>

  • Documentation improvements for a number of Data Prepper processors. (#5023)

    David Venable - Mon, 7 Oct 2024 13:22:45 -0700

    Documentation improvements for a number of Data Prepper processors. Updates some enumerations used in processors to support the @JsonValue. Corrects @JsonClassDescription to use HTML rather than Markdown. Signed-off-by: David Venable <dlv@amazon.com>

  • Documentation improvements for a number of Data Prepper processors. (#5025)

    David Venable - Mon, 7 Oct 2024 13:17:55 -0700

    Adds missing enumerations in the key-value processor to support better documentation. Corrects @JsonClassDescription to use HTML rather than Markdown. This set of changes is for key_value, flatten, translate, parse_json, parse_xml, parse_ion, and csv. Also, this adds documentation to the csv input codec. Signed-off-by: David Venable <dlv@amazon.com>

  • Add integration test for Json codec in S3 sink (#4411)

    Taylor Gray - Mon, 7 Oct 2024 11:19:00 -0500

    Signed-off-by: Taylor Gray <tylgry@amazon.com>

  • Make acknowledgment_timeout configurable for s3 scan source, configure timeout to 10 minutes for MongoDB L2 transform (#4988)

    Taylor Gray - Mon, 7 Oct 2024 11:18:13 -0500

    Signed-off-by: Taylor Gray <tylgry@amazon.com>

  • Caching implementation of EventKeyFactory (#4843)

    David Venable - Mon, 7 Oct 2024 09:01:54 -0700

    Create a caching implementation of EventKeyFactory that will cache a configurable number of EventKeys. Refactored the approach to loading the EventKeyFactory in the application context for a couple of reasons: 1. Allow for skipping the CachingEventKeyFactory if not needed; 2. Help it run better in the data-prepper-test-event project. Signed-off-by: David Venable <dlv@amazon.com>

  • Update the test logging to include the failed assertion to help investigations, especially during CI builds on GitHub Actions. (#4987)

    David Venable - Mon, 7 Oct 2024 07:24:46 -0700

    Signed-off-by: David Venable <dlv@amazon.com>

  • Providing an option for the plugins to use Spring DI (#5012)

    Santhosh Gandhe - Fri, 4 Oct 2024 11:40:57 -0700

    providing an option for the plugins to use Spring DI Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> Update data-prepper-api/src/main/java/org/opensearch/dataprepper/model/annotations/DataPrepperPlugin.java

    modified the comment line based on the suggession Co-authored-by: David Venable <dlv@amazon.com> Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> Integration test to validate the DI context enabling in plugins

    Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com>

    Co-authored-by: David Venable <dlv@amazon.com>

  • Remove references to deprecated processor names otel_trace_raw and service_map_stateful (#5017)

    Ling Hengqian - Thu, 3 Oct 2024 13:32:21 -0700

    Signed-off-by: linghengqian <linghengqian@outlook.com>

  • Updates to JSON schema and Data Prepper documentation. (#5019)

    David Venable - Thu, 3 Oct 2024 11:00:32 -0700

    Support @JsonValue for determining enumeration values in the JSON Schema. Provide a JSON schema type of string for EventKey objects. Documentation wording improvements to the mutate event and mutate string processors. Signed-off-by: David Venable <dlv@amazon.com>

  • Add OpenTelemetry Metrics example (#5014)

    Karsten Schnitter - Thu, 3 Oct 2024 19:24:09 +0200

    Add OpenTelemetry Metrics example Provides a small Docker Compose setup to show metrics ingestion with Data Prepper. This example is similar to the traces and logs examples. It also shows the metrics Data Prepper emits on its metrics endpoint. Signed-off-by: Karsten Schnitter <k.schnitter@sap.com>

  • ENH: support plugin loading in conifg (#4974)

    Qi Chen - Wed, 2 Oct 2024 16:00:58 -0500

    Signed-off-by: George Chen <qchea@amazon.com>

  • Updates JsonPropertyDescription annotations to use HTML rather than Markdown. (#4985)

    David Venable - Wed, 2 Oct 2024 09:52:26 -0700

    Signed-off-by: David Venable <dlv@amazon.com>

  • Adding a new API to get the current transformed pipelines as a JSON (#4980)

    Souvik Bose - Tue, 1 Oct 2024 11:34:43 -0700

    • Adding a new API to get the current transformed pipelines as a JSON Signed-off-by: Souvik Bose <souvbose@amazon.com>

    • Rename the api and address comments Signed-off-by: Souvik Bose <souvbose@amazon.com>

    Signed-off-by: Souvik Bose <souvbose@amazon.com> Co-authored-by: Souvik Bose <souvbose@amazon.com>

  • Add: trace peerforwarder config (#4992)

    Qi Chen - Tue, 1 Oct 2024 10:12:09 -0500

    Signed-off-by: George Chen <qchea@amazon.com>

  • FIX: missing required fields annotation (#4990)

    Qi Chen - Tue, 1 Oct 2024 09:55:58 -0500

    Signed-off-by: George Chen <qchea@amazon.com>

  • MAINT: update required properties and property descriptions in some processors (#4989)

    Katherine Shen - Mon, 30 Sep 2024 14:58:12 -0500

    • update rename key config with missing descriptions

    • update convert entry type required fields

    • update delay property description

    • putting convert entry type processor on hold due to the mutually exclusive properties Signed-off-by: Katherine Shen <katshen@amazon.com>

    Signed-off-by: Katherine Shen <katshen@amazon.com>

  • Additional logging when shutting down the pipeline. (#4986)

    David Venable - Mon, 30 Sep 2024 08:59:14 -0700

    Signed-off-by: David Venable <dlv@amazon.com>

  • Support convert entry type on arrays (#4925)

    Krishna Kondaka - Fri, 27 Sep 2024 12:12:43 -0700

    Signed-off-by: Kondaka <krishkdk@amazon.com>

  • Support BigDecimal data type in expressions (#4930)

    Krishna Kondaka - Fri, 27 Sep 2024 09:04:09 -0700

    Signed-off-by: Kondaka <krishkdk@amazon.com>

  • OpenSearchSink - Enhance logs to include index name and last exception information (#4841)

    Krishna Kondaka - Fri, 27 Sep 2024 09:03:47 -0700

    Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • Add origination time to buffer event and populate the partition key (#4971)

    Souvik Bose - Thu, 26 Sep 2024 12:32:09 -0500

    Signed-off-by: Souvik Bose <souvbose@amazon.com> Co-authored-by: Souvik Bose <souvbose@amazon.com>

  • Add integration test for Kinesis source (#4967)

    Souvik Bose - Tue, 24 Sep 2024 16:25:02 -0500

    Signed-off-by: Souvik Bose <souvbose@amazon.com>

  • Add Lambda Synchronous processor support (#4700)

    Srikanth Govindarajan - Tue, 24 Sep 2024 11:01:45 -0700

    Add Lambda Processor Synchronous Mode support Make LambdaClientFactory common to sink and processor Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>

  • Variable drain timeouts when shutting down over HTTP shutdown. (#4970)

    David Venable - Mon, 23 Sep 2024 16:03:21 -0700

    Variable drain timeouts when shutting down over HTTP shutdown. Adds two new parameters to the shutdown API. The first is bufferReadTimeout which controls the amount of time to wait for the buffer to be empty. The second is bufferDrainTimeout which controls the overall wait time for the process worker threads to complete. To support Data Prepper durations in HTTP query parameters, I extracted the parsing logic for durations out of DataPrepperDurationDeserializer and into a new DataPrepperDurationParser class. Resolves #4966. Signed-off-by: David Venable <dlv@amazon.com>

  • Add mongodb as alternate source name for documentdb source (#4969)

    Dinu John - Sun, 22 Sep 2024 13:39:13 -0500

    Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Add alternate name for Plugin (#4968)

    Dinu John - Sat, 21 Sep 2024 16:24:34 -0700

    Add alternate name for Plugin Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Refactoring chunking in the HTTP source to improve the performance. (#4950)

    David Venable - Fri, 20 Sep 2024 14:00:50 -0700

    The HTTP source was parsing the entire message and then serializing from strings. This created a bit of memory churn and probably duplicate processing. The new approach is to chunk the message from the start which allows us to stream the reading and perform copies of data. This also has a JMH benchmark added which shows that this new approach doubles the number of operations per second. Signed-off-by: David Venable <dlv@amazon.com>

  • Update rds template with new s3 sink client options (#4963)

    Hai Yan - Fri, 20 Sep 2024 12:39:15 -0500

    Signed-off-by: Hai Yan <oeyh@amazon.com>

  • Fix a bug with event listener in RDS source (#4962)

    Hai Yan - Fri, 20 Sep 2024 09:41:45 -0700

    Unregister eventListener on stop Signed-off-by: Hai Yan <oeyh@amazon.com>

  • Add s3 sink client options (#4959)

    Hai Yan - Thu, 19 Sep 2024 22:28:03 -0500

    Signed-off-by: Hai Yan <oeyh@amazon.com>

  • Support AWS Kinesis Data Streams as a Source (#4836)

    Souvik Bose - Thu, 19 Sep 2024 15:23:02 -0700

    Support AWS Kinesis Data Streams as a Source Signed-off-by: Souvik Bose <souvbose@amazon.com>

  • Add acknowledgment progress check for s3 source with folder partitions (#4957)

    Taylor Gray - Thu, 19 Sep 2024 10:06:46 -0500

    Signed-off-by: Taylor Gray <tylgry@amazon.com>

  • Shorten S3 prefix to meet the requirement of RDS export API (#4955)

    Hai Yan - Wed, 18 Sep 2024 14:22:37 -0500

    Signed-off-by: Hai Yan <oeyh@amazon.com>

  • Improve logging for exceptions (#4942)

    Srikanth Govindarajan - Wed, 18 Sep 2024 09:31:49 -0700

    Improve logging for exceptions Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>

  • Add ReplaceStringProcessor for simple string substitution that doesn't involve regex (#4954)

    Dinu John - Wed, 18 Sep 2024 10:18:08 -0500

    Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Kafka Plugin: support for SASL/SCRAM mechanisms (#4912)

    franky-m - Tue, 17 Sep 2024 12:44:09 -0700

    Kafka Plugin SCRAM Support Signed-off-by: Franky Meier <franky.meier.1@gmx.de> Signed-off-by: Franky Meier <franky.meier.1@gmx.de> Signed-off-by: George Chen <qchea@amazon.com> Co-authored-by: Qi Chen <qchea@amazon.com>

  • Adds config transformation template for RDS source (#4946)

    Hai Yan - Tue, 17 Sep 2024 14:26:40 -0500

    Signed-off-by: Hai Yan <oeyh@amazon.com>

  • Decrease the sleep when pausing the Kafka consumer to 1 second when the circuit breaker is in use. (#4947)

    David Venable - Fri, 13 Sep 2024 13:30:13 -0700

    Signed-off-by: David Venable <dlv@amazon.com>

  • Release events for Noop Sink (#4944)

    Srikanth Govindarajan - Thu, 12 Sep 2024 16:31:16 -0500

    Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>

  • Move CsvMapper and Schema creation to constructor (#4941)

    Dinu John - Thu, 12 Sep 2024 10:10:10 -0700

    Move CsvMapper and Schema creation to constructor Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>

  • Fixes Jaeger Hotrod demo failure on OpenSearch 2.16.0 (#4921)

    Ling Hengqian - Wed, 11 Sep 2024 19:21:59 -0700

    Signed-off-by: linghengqian <linghengqian@outlook.com>

  • Updating the WhiteSource/Mend configuration to match the version found in opensearch-system-templates. (#4933)

    David Venable - Wed, 11 Sep 2024 13:48:12 -0700

    https://github.com/opensearch-project/opensearch-system-templates/blob/e3b4fc6/.whitesource

    Signed-off-by: David Venable <dlv@amazon.com>

  • adding registry to image value in docker-compose.yaml (#2980)

    qhung - Tue, 10 Sep 2024 13:06:24 -0700

    Podman requires the registry url in order to pull out the image Signed-off-by: qhung <11665161+quanghungb@users.noreply.github.com>

  • Support start_time or range options for the first scan of scheduled s3 scan (#4929)

    Taylor Gray - Tue, 10 Sep 2024 13:53:16 -0500

    Signed-off-by: Taylor Gray <tylgry@amazon.com>

  • Add progress check callback to update partition ownership in S3 scan source (#4918)

    Taylor Gray - Tue, 10 Sep 2024 11:14:00 -0500

    Signed-off-by: Taylor Gray <tylgry@amazon.com>

  • ENH: add shutdown into extension plugin (#4924)

    Qi Chen - Tue, 10 Sep 2024 10:35:51 -0500

    • ENH: add shutdown into extension plugin Signed-off-by: George Chen <qchea@amazon.com>

  • Updated README.md according to previous changes of docker files. (#4845)

    Jayesh Parmar - Mon, 9 Sep 2024 22:07:40 +0200

    Signed-off-by: jayeshjeh <jay.parmar.11169@gmail.com>

  • Updates to the code for HTTP chunking. (#4919)

    David Venable - Mon, 9 Sep 2024 09:27:44 -0700

    This refactors the code by placing all logic for serializing the data into the Codec itself. In so doing, it allows for improved testing such as symmetric testing. It also decouples the serialization format from the HTTP server. This also uses the Jackson library for the serialization which yields more accurate JSON. Signed-off-by: David Venable <dlv@amazon.com>

  • MAINT: add data prepper plugin schema module into build resource (#4920)

    Qi Chen - Fri, 6 Sep 2024 13:24:03 -0500

    Signed-off-by: George Chen <qchea@amazon.com>

  • Move event processing to separate threads and add event processing timer in RDS source (#4914)

    Hai Yan - Thu, 5 Sep 2024 10:58:59 -0500

    • Move event processing to separate threads and add event processing timer Signed-off-by: Hai Yan <oeyh@amazon.com>

    • Address comments Signed-off-by: Hai Yan <oeyh@amazon.com>

    Signed-off-by: Hai Yan <oeyh@amazon.com>

  • Corrects the S3SinkStack for AWS testing (#4913)

    David Venable - Wed, 4 Sep 2024 14:30:28 -0700

    Corrects the S3SinkStack for AWS testing. The S3SinkStack was not in use and didn't quite work. This corrects it so that we can use it to automate the tests for the S3 sink in GitHub. Signed-off-by: David Venable <dlv@amazon.com>

  • Use awaitility to read data in Kafka buffer tests to fix flakiness (#4703)

    David Venable - Wed, 4 Sep 2024 14:18:09 -0700

    Use awaitility to read data in KafkaBufferIT to promote stability and speed of execution. Contributes toward #4168 Signed-off-by: David Venable <dlv@amazon.com>

  • Adds a .gitignore for Python virtual environments. (#4881)

    David Venable - Wed, 4 Sep 2024 12:24:52 -0700

    Some projects such as the trace-analytics-sample-app and the current smoke tests use Python. Using Python virtual environments lets developers use these without affecting their local Python environment. Ignore .venv directories which Python virtual environments use. Signed-off-by: David Venable <dlv@amazon.com>

  • Updating the instructions for releasing a new version of Data Prepper. (#4878)

    David Venable - Wed, 4 Sep 2024 12:24:25 -0700

    This adds instructions for the release setup so that maintainers can have a consistent set of instructions to follow. Specifically, it adds steps for setting up the branch, updating version numbers, creating the release notes and change log. Additionally, this fixes a broken fragment link to the Backporting section of the Developer Guide. Signed-off-by: David Venable <dlv@amazon.com>

  • ENH: respect JsonProperty defaultValue in JsonSchemaConverter (#4889)

    Qi Chen - Tue, 3 Sep 2024 10:38:08 -0500

    • ENH: respect JsonProperty defaultValue in JsonSchemaConverter Signed-off-by: George Chen <qchea@amazon.com>

  • Minor updates and bug fixes for RDS source (#4887)

    Hai Yan - Thu, 29 Aug 2024 13:20:47 -0500

    • Minor updates and bug fixes to prepare for performance testing Signed-off-by: Hai Yan <oeyh@amazon.com>

    • Address comments Signed-off-by: Hai Yan <oeyh@amazon.com>

    Signed-off-by: Hai Yan <oeyh@amazon.com>

  • Adds a script to help generate the Thank You text for the release blogs. (#4884)

    David Venable - Thu, 29 Aug 2024 08:44:17 -0700

    Adds a script to help generate the Thank You text for the release blogs. Use en dashes in the Thank You text to meet the standards of the project-website. When there is no name for a GitHub user, don't include a None. Signed-off-by: David Venable <dlv@amazon.com> Co-authored-by: Hai Yan <oeyh@amazon.com>

  • Creates the release notes for Data Prepper 2.9.0 (#4879)

    David Venable - Wed, 28 Aug 2024 08:20:02 -0700

    Signed-off-by: David Venable <dlv@amazon.com>

  • ENH: Plugin errors consolidator (#4863)

    Qi Chen - Tue, 27 Aug 2024 11:16:55 -0500

  • REF: data prepper plugin schema (#4872)

    Qi Chen - Tue, 27 Aug 2024 09:57:53 -0500

    Signed-off-by: George Chen <qchea@amazon.com>

  • Add e2e acknowledgment and checkpointing to RDS source (#4819)

    Hai Yan - Mon, 26 Aug 2024 17:37:22 -0500

    Signed-off-by: Hai Yan <oeyh@amazon.com>

  • Update Spring to 5.3.39 to fix CVE-2024-38808. Require commons-configuration2 2.11.0 to fix CVE-2024-29131 and CVE-2024-29133. Hadoop pulls this dependency in. (#4874)

    David Venable - Mon, 26 Aug 2024 10:34:06 -0700

    Signed-off-by: David Venable <dlv@amazon.com>

  • Updates Bouncy Castle to 1.78.1. Update any projects that attempt to use Bouncy Castle jdk15on dependencies with the jdk18on dependency instead. This will prevent any of the older jdk15on dependencies from getting into our classpath. In particular, this was coming from hadoop-common. (#4871)

    David Venable - Mon, 26 Aug 2024 08:30:35 -0700

    Signed-off-by: David Venable <dlv@amazon.com>

  • Require dnsjava 3.6.1 to resolve CVE-2024-25638. This is a transitive dependency from Hadoop. (#4868)

    David Venable - Fri, 23 Aug 2024 14:49:17 -0700

    Signed-off-by: David Venable <dlv@amazon.com>

  • Removes the trace-analytics-sample-app from the examples that are provided in the release. (#4867)

    David Venable - Fri, 23 Aug 2024 14:49:10 -0700

    The motivation for removing this is that the samples pull in dependencies which often trigger CVE reports. It is not likely customers are trying to run this example from a Data Prepper deployment, especially since the example is not made to run from the installed version, but runs from Docker and runs using the latest 2.x version. Signed-off-by: David Venable <dlv@amazon.com>

  • Fix visibility timeout errors (#4812) (#4831)

    Daniel Li - Fri, 23 Aug 2024 11:59:12 -0700

    Fix visibility timeout errors (#4812) Signed-off-by: Daniel Li <lhouqua@amazon.com>

  • Change version in DataPrepper Version class to 2.10 (#4852)

    Krishna Kondaka - Fri, 23 Aug 2024 11:35:50 -0700

    Signed-off-by: Kondaka <krishkdk@amazon.com>

  • Adds integration tests for pipeline connectors. (#4834)

    David Venable - Fri, 23 Aug 2024 07:58:01 -0700

    This commit adds integration testing for the pipeline connector sink/source which connects two pipelines. There are two tests. The first tests against a single connection with a single sink. The second test also includes a second sink to verify that pipeline connections work with additional sinks. This commit also includes fixes for CoreHttpServerIT. When running the new pipeline connector tests, the CoreHttpServerIT tests started failing. I found some places where shutdowns were not occurring and fixed those. And I added some additional logging to help debug. The root problem turned out to be that the ExecutorService used in the DataPrepperServer was a static field. The CoreHttpServerIT was working because it was the first test that JUnit chose. With the new tests, it is being chosen later and by that point, the static ExecutorService was shutdown. The fix is simply to avoid using a static ExecutorService. Signed-off-by: David Venable <dlv@amazon.com>

  • Mask s3 object key in logs (#4861)

    Hai Yan - Thu, 22 Aug 2024 17:12:30 -0500

    Signed-off-by: Hai Yan <oeyh@amazon.com>

  • Changelog for release 2.9 (#4855)

    Krishna Kondaka - Thu, 22 Aug 2024 12:58:22 -0700

    Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com>

  • ENH: collect plugin config and loading errors during data prepper launch (#4816)

    Qi Chen - Wed, 21 Aug 2024 17:41:09 -0500

    • ENH: collect plugin errors within core application context Signed-off-by: George Chen <qchea@amazon.com>

  • ENH: respect json order and class description on processors (#4857)

    Qi Chen - Wed, 21 Aug 2024 14:32:05 -0500

    • ENH: json order and class description on processors Signed-off-by: George Chen <qchea@amazon.com>

  • Address missing processor JsonPropertyDescriptions and validations (#4837)

    Taylor Gray - Wed, 21 Aug 2024 10:03:27 -0500

    Signed-off-by: Taylor Gray <tylgry@amazon.com>

  • Catch error that could cause LeaderScheduler thread to crash (#4850)

    Taylor Gray - Tue, 20 Aug 2024 19:43:10 -0500

    Signed-off-by: Taylor Gray <tylgry@amazon.com>

  • Change main branch version to 2.10-SNAPSHOT (#4851)

    Krishna Kondaka - Tue, 20 Aug 2024 13:46:09 -0700

    Change main branch version to 2.10-SNAPSHOT Signed-off-by: Kondaka <krishkdk@amazon.com>

    Signed-off-by: Kondaka <krishkdk@amazon.com>

  • Add support for AWS security lake sink as a bucket selector mode in S3 sink (#4846)

    Krishna Kondaka - Tue, 20 Aug 2024 13:19:42 -0700

    Signed-off-by: Kondaka <krishkdk@amazon.com>