Skip to content

Add codec to http source#5694

Merged
oeyh merged 2 commits into
opensearch-project:mainfrom
oeyh:http-source-codec
May 21, 2025
Merged

Add codec to http source#5694
oeyh merged 2 commits into
opensearch-project:mainfrom
oeyh:http-source-codec

Conversation

@oeyh

@oeyh oeyh commented May 14, 2025

Copy link
Copy Markdown
Collaborator

Description

Add codec to http source so it can use the functionality introduced in #5054 to parse CloudWatch Logs data.
The config would look like this:

    http:
      codec:
        json:
          key_name: "logEvents"
          include_keys: [ 'owner', 'logGroup', 'logStream' ]
          include_keys_metadata: [ 'owner', 'logGroup', 'logStream' ]

Issues Resolved

N/A

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Hai Yan <oeyh@amazon.com>
@oeyh oeyh force-pushed the http-source-codec branch from 76c0da0 to 3efd668 Compare May 20, 2025 17:25
httpRequestExceptionHandler = new HttpRequestExceptionHandler(pluginMetrics);
final PluginModel codecConfiguration = sourceConfig.getCodec();
if (codecConfiguration == null) {
codec = null;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we initialize the json codec here itself.

@oeyh oeyh May 20, 2025

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I follow here. Can you clarify a bit more?

Do you mean you want it to be a JsonInputCodec and disallow other type of input codecs? If so, we can add a validation on the sourceConfig to prevent the use of other codecs that don't fit here.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sync'ed up and resolved offline.

Signed-off-by: Hai Yan <oeyh@amazon.com>
@oeyh oeyh force-pushed the http-source-codec branch from 3efd668 to 719c8bd Compare May 20, 2025 21:33
@oeyh oeyh merged commit 1173763 into opensearch-project:main May 21, 2025
38 of 47 checks passed
@dlvenable

Copy link
Copy Markdown
Member

@oeyh , The current implementation is pretty much the same as the json codec. What codec did you need here?

@oeyh

oeyh commented May 22, 2025

Copy link
Copy Markdown
Collaborator Author

@oeyh , The current implementation is pretty much the same as the json codec. What codec did you need here?

@dlvenable I needed the functionality introduced in #5054 in JsonInputCodec to parse CloudWatch Logs data

alparish pushed a commit to alparish/data-prepper that referenced this pull request May 22, 2025
* Add codec to http source

Signed-off-by: Hai Yan <oeyh@amazon.com>

* Add a test with sample data

Signed-off-by: Hai Yan <oeyh@amazon.com>

---------

Signed-off-by: Hai Yan <oeyh@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants