Skip to content

Added throttle metric to S3 input stream#6245

Merged
dlvenable merged 1 commit into
opensearch-project:mainfrom
mzurita-amz:s3-throttle-metric
Nov 7, 2025
Merged

Added throttle metric to S3 input stream#6245
dlvenable merged 1 commit into
opensearch-project:mainfrom
mzurita-amz:s3-throttle-metric

Conversation

@mzurita-amz

Copy link
Copy Markdown
Contributor

Description

  • Added throttle metric to S3 input stream
  • Added integration tests to EMF records generated by S3

Issues Resolved

N/A

Check List

  • New functionality includes testing.
  • [-] New functionality has a documentation issue. Please link to it in this PR.
    • [-] New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

assertThat("Succeeded metric should be zero", emf.get("test-pipeline.s3-source.s3ObjectsSucceeded.count").asDouble(), equalTo(0.0));
}

private void spyEMFOutput() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use another Metric registry to avoid relying on stdout? This seems too brittle and intrusive.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure thing. Using Prometheus scrape now.

@mzurita-amz mzurita-amz force-pushed the s3-throttle-metric branch 3 times, most recently from a9fa2f5 to dc144ac Compare November 5, 2025 15:21
Signed-off-by: Manuel Mangas Zurita <mzurita@amazon.com>

@dlvenable dlvenable left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mzurita-amz .

This is currently a draft. Would you like to open it? That will run the tests as well.

@dlvenable dlvenable merged commit 4b4febc into opensearch-project:main Nov 7, 2025
46 of 47 checks passed
@mzurita-amz mzurita-amz deleted the s3-throttle-metric branch November 13, 2025 22:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants