feat(cloudwatch-logs): Add Entity config support to sink by bagmarnikhil · Pull Request #6864 · opensearch-project/data-prepper

bagmarnikhil · 2026-05-15T23:14:30Z

Add an optional 'entity' configuration block on the CloudWatch Logs sink that attaches CloudWatch Entity metadata (key_attributes + attributes) to every PutLogEvents request, enabling entity-based correlation in CloudWatch.

When 'entity' is omitted the sink behaves identically to before. When configured, an SDK Entity is built and passed to the dispatcher via the builder; the dispatcher conditionally sets it on each PLE request.

If CloudWatch rejects the entity, the request is still considered successful (events are released), a WARN log records the RejectedEntityInfo error type, and a new 'cloudWatchLogsEntityRejected' counter is incremented as the primary alarmable signal.

Validation is intentionally minimal — @notempty on key_attributes only. AWS-owned limits (max entries, allowed key names, length caps) are enforced server-side and surfaced via the rejection metric, avoiding client-side drift when the Entity API contract changes.

Tests follow TDD: each test was written before the code that makes it pass. New unit tests cover EntityConfig defaults, deserialization, and @notempty validation; @Valid cascade from CloudWatchLogsSinkConfig; the new counter; dispatcher entity-on-request, no-entity, and rejection paths; and sink wiring of EntityConfig -> Entity -> dispatcher. TestSinkOperationWithEntity integration test exercises the SDK end-to-end.

Acceptance criteria from the spec are met: backward compatibility, entity attached when configured, non-fatal rejection with metric and log, all existing tests green, new tests cover new code paths, license headers on new files.

Refs: data-prepper-plugins/cloudwatch-logs/specs/entity-config.md

Description

[Describe what this change achieves]

Issues Resolved

Resolves #6860

Check List

New functionality includes testing.
New functionality has a documentation issue. Please link to it in this PR.
- New functionality has javadoc added
Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2026-05-19T15:30:00Z

✅ License Header Check Passed

All newly added files have proper license headers. Great work! 🎉

@notempty

Add an optional 'entity' configuration block on the CloudWatch Logs sink that attaches CloudWatch Entity metadata (key_attributes + attributes) to every PutLogEvents request, enabling entity-based correlation in CloudWatch. When 'entity' is omitted the sink behaves identically to before. When configured, an SDK Entity is built and passed to the dispatcher via the builder; the dispatcher conditionally sets it on each PLE request. If CloudWatch rejects the entity, the request is still considered successful (events are released), a WARN log records the RejectedEntityInfo error type, and a new 'cloudWatchLogsEntityRejected' counter is incremented as the primary alarmable signal. Validation is intentionally minimal — @notempty on key_attributes only. AWS-owned limits (max entries, allowed key names, length caps) are enforced server-side and surfaced via the rejection metric, avoiding client-side drift when the Entity API contract changes. Tests follow TDD: each test was written before the code that makes it pass. New unit tests cover EntityConfig defaults, deserialization, and @notempty validation; @Valid cascade from CloudWatchLogsSinkConfig; the new counter; dispatcher entity-on-request, no-entity, and rejection paths; and sink wiring of EntityConfig -> Entity -> dispatcher. TestSinkOperationWithEntity integration test exercises the SDK end-to-end. Acceptance criteria from the spec are met: backward compatibility, entity attached when configured, non-fatal rejection with metric and log, all existing tests green, new tests cover new code paths, license headers on new files. Refs: data-prepper-plugins/cloudwatch-logs/specs/entity-config.md Signed-off-by: Nikhil Bagmar <nikhilbagmar73@gmail.com>

Signed-off-by: Nikhil Bagmar <nikhilbagmar73@gmail.com>

@test

- Remove duplicate @test annotations from auto-merge - Add Hamcrest imports for entity assertion matchers - Update EXPECTED_DISPATCHER_ARITY to 11 (entity field added) Signed-off-by: Nikhil Bagmar <nikhilbagmar73@gmail.com>

…rejection check - Default keyAttributes and attributes to Collections.emptyMap() - Return Collections.unmodifiableMap() from both getters - Move entity rejection metric before releaseEventHandles - Add unmodifiable map and createResources exception tests Signed-off-by: Nikhil Bagmar <nikhilbagmar73@gmail.com>

dlvenable

Thanks @bagmarnikhil for this contribution!

srikanthpadakanti · 2026-05-26T16:44:00Z

+    @NotEmpty
+    private Map<String, String> keyAttributes = Collections.emptyMap();
+
+    @JsonProperty("attributes")


attributes has no @NotNull. Explicit attributes: null in YAML deserializes to null and unmodifiableMap(null) throws NPE; either add @NotNull or null-guard the getter.

srikanthpadakanti · 2026-05-26T16:45:49Z

                    dlqObjects = getDlqObjectsFromResponse(putLogEventsResponse);
                }
+                if (putLogEventsResponse != null && putLogEventsResponse.rejectedEntityInfo() != null) {
+                    cloudWatchLogsMetrics.increaseEntityRejectedCounter(1);


A misconfigured entity will return rejectedEntityInfo on every successful PutLogEvents. This WARN will fire per batch under load; consider NOISY marker or rate-limited logging like the SDK exception path.

srikanthpadakanti · 2026-05-26T16:48:04Z

+public class EntityConfig {
+
+    @JsonProperty("key_attributes")
+    @NotEmpty


@notempty only rejects empty maps. Values can still be empty/blank strings and produce a server rejection on every request; consider @SiZe(max=…) and @notblank on values to fail fast

Thanks for the suggestion. I intentionally kept validation minimal here — the Entity API has a complex set of constraints (key names restricted to 5 allowed values, min 2 / max 4 entries for keyAttributes, value lengths 1–512, max 10 attributes, etc.).

Replicating a subset client-side (e.g. @notblank on values) creates a false sense of completeness while still drifting if AWS changes the contract. The rejection metric (cloudWatchLogsEntityRejected) + WARN log are designed to be the primary signal for misconfiguration.

This avoids coupling to server-side limits that may evolve.

srikanthpadakanti · 2026-05-26T16:49:31Z

    private int retryCount;
    private boolean createLogGroup;
    private boolean createLogStream;
+    private Entity entity;


Inner Uploader declares its private final fields but the outer dispatcher leaves entity (and the older fields above) non-final. Flag if you want the new field to follow the inner class' immutability convention.

The existing fields in the outer dispatcher (retryCount, createLogGroup, createLogStream, etc.) are all non-final. Making only entity final would be inconsistent with the rest of the class.

Happy to make it final if we want to refactor all the outer fields to final in a follow-up, but keeping it consistent with the existing style for now.

srikanthpadakanti · 2026-05-26T16:51:08Z

+        assertThat(entityConfig.getAttributes(), aMapWithSize(0));
+    }
+
+    @Test


GIVEN_entity_config_with_empty_key_attributes… and GIVEN_entity_config_with_default_key_attributes… exercise the same constraint (empty map default vs explicitly-set empty map). Collapse into one parameterized test

bagmarnikhil requested review from KarstenSchnitter, Zhangxunmt, dinujoh, divbok, dlvenable, graytaylor0, kkondaka, oeyh, san81, sb2k16 and srikanthjg as code owners May 15, 2026 23:14

bagmarnikhil mentioned this pull request May 19, 2026

Create log group and stream in CloudWatch Logs sink #6863

Merged

1 task

bagmarnikhil added 3 commits May 21, 2026 14:09

Add full license headers to EntityConfig and EntityConfigTest

36a3c34

Signed-off-by: Nikhil Bagmar <nikhilbagmar73@gmail.com>

fix: resolve rebase conflicts with create_log_group_and_stream

60dc039

- Remove duplicate @test annotations from auto-merge - Add Hamcrest imports for entity assertion matchers - Update EXPECTED_DISPATCHER_ARITY to 11 (entity field added) Signed-off-by: Nikhil Bagmar <nikhilbagmar73@gmail.com>

bagmarnikhil force-pushed the feature/cwl-sink-entity-config branch from f18af61 to 60dc039 Compare May 21, 2026 21:26

bagmarnikhil requested a review from srikanthpadakanti as a code owner May 21, 2026 21:26

Himtanaya approved these changes May 21, 2026

View reviewed changes

srikanthpadakanti requested changes May 21, 2026

View reviewed changes

dlvenable approved these changes May 26, 2026

View reviewed changes

srikanthpadakanti requested changes May 26, 2026

View reviewed changes

Himtanaya approved these changes May 26, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cloudwatch-logs): Add Entity config support to sink#6864

feat(cloudwatch-logs): Add Entity config support to sink#6864
bagmarnikhil wants to merge 4 commits into
opensearch-project:mainfrom
bagmarnikhil:feature/cwl-sink-entity-config

bagmarnikhil commented May 15, 2026 •

edited by dlvenable

Loading

Uh oh!

github-actions Bot commented May 19, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dlvenable left a comment

Uh oh!

srikanthpadakanti May 26, 2026

Uh oh!

srikanthpadakanti May 26, 2026

Uh oh!

srikanthpadakanti May 26, 2026

Uh oh!

bagmarnikhil May 26, 2026

Uh oh!

srikanthpadakanti May 26, 2026

Uh oh!

bagmarnikhil May 26, 2026

Uh oh!

srikanthpadakanti May 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

bagmarnikhil commented May 15, 2026 • edited by dlvenable Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues Resolved

Check List

Uh oh!

github-actions Bot commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ License Header Check Passed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dlvenable left a comment

Choose a reason for hiding this comment

Uh oh!

srikanthpadakanti May 26, 2026

Choose a reason for hiding this comment

Uh oh!

srikanthpadakanti May 26, 2026

Choose a reason for hiding this comment

Uh oh!

srikanthpadakanti May 26, 2026

Choose a reason for hiding this comment

Uh oh!

bagmarnikhil May 26, 2026

Choose a reason for hiding this comment

Uh oh!

srikanthpadakanti May 26, 2026

Choose a reason for hiding this comment

Uh oh!

bagmarnikhil May 26, 2026

Choose a reason for hiding this comment

Uh oh!

srikanthpadakanti May 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bagmarnikhil commented May 15, 2026 •

edited by dlvenable

Loading

github-actions Bot commented May 19, 2026 •

edited

Loading