Expand OTLP Sink to support any compatible endpoint and support otlphttp protocol

[tillwulfram]

Is your feature request related to a problem? Please describe.
Currently, the OTLP sink (introduced in 2.12 #5663) only supports AWS X-Ray as an endpoint. The initial proposal mentioned general OTLP endpoint support, which would greatly benefit users who also want to send there transformed data to additional OTLP-compatible backends which makes data-prepper more versatile.

Additionally, while the OTLP source already supports both gRPC and HTTP protocols, the sink currently only supports gRPC. Many observability backends prefer or only support OTLP/HTTP, making Data Prepper less versatile for these scenarios.

Describe the solution you'd like

1. General OTLP endpoint support: Extend the OTLP sink to support any OTLP protobuf-compatible endpoint (not just AWS X-Ray), with authentication options including:

   • Basic Auth (default)
   • mTLS
   • Optional: AWS SigV4 for AWS services

2. OTLP/HTTP protocol support: Add support for the OTLP/HTTP protocol in addition to the existing gRPC support. This could be implemented either as:

   • Option A (Preferred): A configuration flag within the existing OTLP sink (e.g., protocol: grpc or protocol: http), similar to how the unified OTLP source handles both protocols
   • Option B: A separate otlp_http sink plugin if the implementation differences warrant it. I can also open another issue if this is preferred.

Example configuration:

sink:
  - otlp:
      protocol: http  # or grpc 
      # Option 1: Base endpoint (preferred for HTTP)
      endpoint: "https://traces.example.com:4318"
      
      # Option 2: Signal-specific Endpoints (optional override)
      traces_endpoint: "https://traces.example.com:4318/custom/traces"
      metrics_endpoint: "https://traces.example.com:4318/custom/metrics"
      logs_endpoint: "https://traces.example.com:4318/custom/logs"
      insecure: true
      username: admin
      password: admin
      compression: gzip
      dlq_file: "/custom/path"

Endpoint and Path Handling

Data Prepper should handle endpoint configuration and path construction automatically based on the selected protocol:

For gRPC protocol (protocol: grpc):

• User only needs to specify endpoint with host:port format (e.g., collector:4317)
• The gRPC client automatically constructs the correct service paths:
  • /opentelemetry.proto.collector.trace.v1.TraceService/Export for traces
  • /opentelemetry.proto.collector.metrics.v1.MetricsService/Export for metrics
  • /opentelemetry.proto.collector.logs.v1.LogsService/Export for logs
• No manual path configuration needed by the user
• Example: endpoint: "traces.example.com:4317"

For HTTP protocol (protocol: http):

• User specifies a base endpoint URL (e.g., https://traces.example.com:4318)
• Data Prepper automatically appends the OTLP-standard signal-specific paths:
  • /v1/traces for trace data
  • /v1/metrics for metric data
  • /v1/logs for log data
• Example configuration:

  endpoint: "https://traces.example.com:4318"

Results in:

• https://traces.example.com:4318/v1/traces for traces
• https://traces.example.com:4318/v1/metrics for metrics
• https://traces.example.com:4318/v1/logs for logs

Optional custom endpoints for HTTP:
For non-standard deployments, users can override with signal-specific endpoints:

traces_endpoint: "https://custom.example.com/custom/path/traces"
metrics_endpoint: "https://custom.example.com/custom/path/metrics"
logs_endpoint: "https://custom.example.com/custom/path/logs"

This approach follows OpenTelemetry SDK conventions and ensures compatibility with standard OTLP receivers while keeping configuration simple for users.

Describe alternatives you've considered

• Using the HTTP sink (which currently is unfortunately not released) with manual JSON formatting, but this requires complex configuration and doesn't leverage OTLP's protobuf efficiency

Additional context

• The OTLP source ([RFC] OTel telemetry unified source #5596) already supports both gRPC and HTTP protocols successfully
• Issue Support OpenTelemetry OTLP/HTTP as addition to OTLP/gRPC #4983 discusses OTLP/HTTP support for sources, showing community interest in this protocol
• This would make Data Prepper more versatile as a transformation layer in modern observability pipelines

[dlvenable]

@tillwulfram , Are you interested in working on this feature? The otlp sink was originally contributed by the community. I don't think the maintainers are currently working on this sink so we may need support from the community to get this feature in.

[dlvenable]

@tillwulfram , Regarding gRPC versus HTTP, I also prefer option A you mentioned - have a single sink.

Option A (Preferred): A configuration flag within the existing OTLP sink (e.g., protocol: grpc or protocol: http), similar to how the unified OTLP source handles both protocols

[KarstenSchnitter]

@dlvenable we are currently working on the support for otlphttp. We already contributed it for the OTelTraceSource with #5322. Support for the OTelLogs is almost done in #6250. We plan on going to metrics and the combined source next.

[tillwulfram]

Sorry, @dlvenable , I didn't saw your first comment. Unfortunately, due to my lack of Java knowledge, I am currently unable to contribute to this important topic. But I am currently working and learning on those skills, and would like to contribute code to the project in the future, in addition to feature requests, etc.

@KarstenSchnitter thanks for the huge upgrades coming for the source!
It would be great if you could also implement the functionality with otlphttp for all signals to the OTLP sink similar to what you're currently implemented/implementing in the OTLP source, and also add basic auth to the sink etc. to maximize the compatibility.

[dlvenable]

Some of the support for more signals is being implemented in PR #6488.

[dlvenable]

@tillwulfram , Do the additions in #6768 give you what you need?

[tillwulfram]

Hey @dlvenable thanks for checking in. I've tested the pr #6768 locally but it doesn't cover everything currently.
The biggest blocker would be that the otlp sink Currently still requires an aws block to build the pipeline. Also it does not support a non aws endpoint bc of the required SigV4 signing. This workaround in the docker setup did the job for non aws endpoint but I don't think this should be the goal:

    environment:
      - AWS_ACCESS_KEY_ID=fake
      - AWS_SECRET_ACCESS_KEY=fake

Also basicauth is not covered, it could potentially be covered by the headers, but the Authorization header is blocked due to PROTECTED_HEADERS.
Nice to have would also be an tls setting for ca or insecure mode