Add Prometheus scrape/pull source to prometheus plugin [Adding in the same Prometheus Remote write source] by srikanthpadakanti · Pull Request #6743 · opensearch-project/data-prepper

srikanthpadakanti · 2026-04-08T20:24:42Z

Description

Add Prometheus scrape/pull source to the existing prometheus plugin. The source periodically HTTP GETs target /metrics endpoints, parses the Prometheus text exposition format, and converts metrics into Data Prepper events (Gauge, Sum, Histogram, Summary). Supports static target URLs, configurable scrape interval/timeout, HTTP Basic Auth, Bearer Token, and TLS certificates. Both scrape and remote write modes can run simultaneously under the same plugin.

Three usage modes:

 Mode 1: Remote write only (existing behavior, unchanged)
  source:
    prometheus:
      port: 9090

Mode 2: Scrape only
  source:
    prometheus:
      scrape:
        targets:
          - url: "http://host:9090/metrics"

Mode 3: Both simultaneously
  source:
    prometheus:
      port: 9090
      scrape:
        targets:
          - url: "http://host:9090/metrics"

Issues Resolved

#1997
Resolves #1997

Check List

[ X ] New functionality includes testing.
New functionality has a documentation issue. Please link to it in this PR.
- [ X ] New functionality has javadoc added
[ X ] Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

dlvenable

Thanks @srikanthpadakanti .

At a high-level would a scraper be expected to get data from the same Prometheus nodes pushing? I'm wondering if it makes sense to have a separate source or not.

dlvenable · 2026-04-10T15:41:33Z

+    @JsonProperty("http_basic")
+    private HttpBasicConfig httpBasic;
+
+    @JsonProperty("bearer_token")


Maybe this should be an oauth2 group with only bearer_token for consistency.

scrape: oauth2: bearer_token:

Bearer token here is a static token passed as an HTTP Authorization header, not an OAuth2 flow. Nesting it under oauth2 would imply token endpoint and refresh support that we do not have. Kept it flat under authentication to match the actual behavior.

Happy to add an oauth2 group later when we support the full flow with token exchange.

dlvenable · 2026-04-10T15:46:32Z

+        final HttpHeadersBuilder builder = HttpHeaders.builder()
+                .add(HttpHeaderNames.ACCEPT, ACCEPT_HEADER_VALUE);
+
+        if (config.getAuthentication() != null) {


It would probably be ideal to have small classes and a common interface for updating the request based on the authentication. This could allow more complicated auth schemes like SigV4 in the future.

I had some similar comments on an http sink PR that is in progress currently. #6747 (comment). Though my comments went the other way. Have the code more amenable to basic auth instead of SigV4. 😄

Extracted into a ScrapeRequestAuthenticator interface with BasicAuthenticator and BearerTokenAuthenticator as separate classes. Factory method handles selection. Adding SigV4 or other schemes in the future is just a new implementation without touching the scraper.

dlvenable · 2026-04-10T15:53:35Z

+    @JsonProperty("flatten_labels")
+    private boolean flattenLabels = false;
+
+    @JsonProperty("insecure")


If any of the URLs in targets has a non-HTTPS scheme we should throw a validation error unless insecure is true.

You can add a JSR @AssertTrue to handle this in this class.

Done.
Added @AssertTrue validation.

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

srikanthpadakanti · 2026-04-10T17:44:18Z

Thanks @srikanthpadakanti .

At a high-level would a scraper be expected to get data from the same Prometheus nodes pushing? I'm wondering if it makes sense to have a separate source or not.

A scraper would not typically target the same nodes pushing via remote write. They serve different use cases: remote write is Prometheus servers configured to push, while scrape pulls from any endpoint exposing the Prometheus text exposition format, including application instances that are not Prometheus servers themselves.

The reason they share a plugin is the precedent from kkondaka's feedback on PR #6627 where the recommendation was to keep it as a single prometheus source with config options for different modes, similar to how s3 source handles s3-sqs and s3-scan under one plugin. Both modes can also run simultaneously which simplifies pipeline configuration when users need both ingestion paths.

That said, if you prefer a separate source I am open to splitting it out.

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

dlvenable · 2026-04-20T16:21:46Z

Thanks @srikanthpadakanti .
At a high-level would a scraper be expected to get data from the same Prometheus nodes pushing? I'm wondering if it makes sense to have a separate source or not.

A scraper would not typically target the same nodes pushing via remote write. They serve different use cases: remote write is Prometheus servers configured to push, while scrape pulls from any endpoint exposing the Prometheus text exposition format, including application instances that are not Prometheus servers themselves.

The reason they share a plugin is the precedent from kkondaka's feedback on PR #6627 where the recommendation was to keep it as a single prometheus source with config options for different modes, similar to how s3 source handles s3-sqs and s3-scan under one plugin. Both modes can also run simultaneously which simplifies pipeline configuration when users need both ingestion paths.

That said, if you prefer a separate source I am open to splitting it out.

@srikanthpadakanti , @kkondaka ,

I think this should be a different source and that the S3 analogy doesn't quite fit here.

The S3 source is entirely pull-based. Whether it is scan or SQS or EventBridge, it is pulling the data from S3. In no case does it open a port. The scan and the SQS are just two different ways to discover the content. The S3 source pulls the data from S3 the exact same way and uses the same code path regardless of the discovery. Additionally we had also once wanted the ability to scan and then start using SQS for a CDC-like behavior, similar to what we do for DynamoDB/RDS.

In this case, there is a big difference between opening a port and going out to an external system to make requests. The source configuration will have a security difference depending on the configuration and may not be clear.

I recommend that we split this into a prometheus_pull source (maybe there is a better name). @kkondaka , please chime in.

kkondaka · 2026-04-20T19:13:32Z

@dlvenable Whether it is pull or push is not something users/customers know about or care about, right? Why make it difficult for the users?

dlvenable · 2026-04-20T19:27:20Z

@kkondaka ,

Whether it is pull or push is not something users/customers know about or care about, right? Why make it difficult for the users?

I think this is not at all the case. Users must know whether it is push or pull because that is how they get the data there right? If it is pull-based, they must enable a scrape URI and provide that. If it is push-based they must configure their remote-write protocol.

I think this simplifies it for users because these are in reality two different sources with disjoint configurations. The documentation will need to say for just about every field whether it is remote-write or pull/scape. It seems more straightforward as a user to start with one model and see the configurations only for that model.

In what way would two sources make the the configuration more complicated for users?

dlvenable · 2026-04-20T19:44:35Z

@kkondaka , I suggest that we accept this PR and discuss further on the source.

kkondaka · 2026-04-20T19:59:09Z

+public class ScrapeAuthenticationConfig {
+
+    @Valid
+    @JsonProperty("http_basic")


Looks like these auth mechanisms are common for all http. We should move them to a common location.

acknowledge, offer to do it as a follow-up

I have created a new issue for the followup - #6766

…ource Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

srikanthpadakanti · 2026-04-20T20:53:17Z

@kkondaka , I suggest that we accept this PR and discuss further on the source.

Makes sense. Opened an issue to track moving these to a shared module. Will do that in a follow-up PR so other HTTP plugins can reuse them.

dlvenable

Thank you @srikanthpadakanti for this contribution!

srikanthpadakanti requested review from KarstenSchnitter, dinujoh, dlvenable, engechas, graytaylor0, kkondaka, oeyh, san81, sb2k16 and srikanthjg as code owners April 8, 2026 20:24

srikanthpadakanti force-pushed the prometheus-scrape-source branch from f327b15 to d118aa1 Compare April 8, 2026 21:29

dlvenable requested changes Apr 10, 2026

View reviewed changes

Add Prometheus scrape/pull source to prometheus plugin

713b2d2

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

srikanthpadakanti force-pushed the prometheus-scrape-source branch from d118aa1 to 713b2d2 Compare April 10, 2026 17:29

Address review feedback for scrape source

d59016d

Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

srikanthpadakanti requested a review from dlvenable April 16, 2026 18:33

kkondaka reviewed Apr 20, 2026

View reviewed changes

kkondaka approved these changes Apr 20, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/main' into prometheus-scrape-s…

40a4601

…ource Signed-off-by: Srikanth Padakanti <srikanth_padakanti@apple.com>

srikanthpadakanti requested a review from kkondaka April 20, 2026 21:04

This was referenced Apr 20, 2026

Discuss: Separate prometheus_scrape source vs combined prometheus source #6766

Closed

Move common HTTP auth mechanisms (Basic Auth, Bearer Token) to a shared module #6767

Closed

kkondaka approved these changes Apr 20, 2026

View reviewed changes

dlvenable approved these changes Apr 21, 2026

View reviewed changes

dlvenable merged commit 4990f26 into opensearch-project:main Apr 21, 2026
70 of 72 checks passed

srikanthpadakanti mentioned this pull request Apr 22, 2026

Extract shared HTTP client auth into http-client-common module #6776

Merged

1 task

dlvenable mentioned this pull request May 19, 2026

Adds Srikanth Padakanti (srikanthpadakanti) as a maintainer. #6870

Merged

4 tasks

Conversation

srikanthpadakanti commented Apr 8, 2026

Description

Issues Resolved

Check List

Uh oh!

dlvenable left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

srikanthpadakanti commented Apr 10, 2026

Uh oh!

dlvenable commented Apr 20, 2026

Uh oh!

kkondaka commented Apr 20, 2026

Uh oh!

dlvenable commented Apr 20, 2026

Uh oh!

dlvenable commented Apr 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

srikanthpadakanti commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dlvenable left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

srikanthpadakanti commented Apr 20, 2026 •

edited

Loading