[feature request] Support Service Discovery configuration standards for OTLP exporter endpoint

[julealgon]

Package

OpenTelemetry.Exporter.OpenTelemetryProtocol

Is your feature request related to a problem?

We would like to centrally configure the OTLP endpoints, both gRPC as well as http, then have each application use the endpoint that makes sense for it based on the current protocol configuration.

Since 1.12, .NET Framework apps use http by default, whereas modern .NET uses gRPC. This breaks our centralized configuration today as it is not possible to define the 2 endpoints globally as the only setting that is supported today is OTEL_EXPORTER_OTLP_ENDPOINT.

What is the expected behavior?

I'd propose supporting Service Discovery for OTLP endpoints so that we could define them like so:

"Services": {
  "otlp-receiver": {
    "grpc": ["<our_server>:4317"],
    "http": ["<our_server>:4318"]
  }
}

Then, the exporter would use

• grpc://otlp-receiver when configured with gRPC protocol
• https+http://otlp-receiver when configured with http protocols

Which alternative solutions or features have you considered?

We don't really have any alternative today other than overriding our global endpoint config in many places so that all of our NET Framework apps get the http endpoint. This will require this override to be added to dozens of places today.

Alternatively, we could rely on custom labels in Azure AppConfig to provide one or the other, but this would similarly require changes to all of our .NET Framework applications to provide such label.

We will likely end up implementing this Service Discovery approach by ourselves as an extension today.

Additional context

I'm not exactly sure how this would interact/conflict with the upcoming "declarative config".

I've also opened this other issue which is related and would provide a different means to solve this at a more basic level:

• OTEL_EXPORTER_OTLP_ENDPOINT should have per-protocol alternatives opentelemetry-specification#5088

The otlp-receiver identifier in the proposal is of course just a suggestion. It could be anything else, or be parameterized if needed.

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[martincostello]

I think first-class support for this would be driven through the specification (so open-telemetry/opentelemetry-specification#5088) rather than something we would independently invent here.

As you've also mentioned, this sounds like a use case for Declarative Configuration (DC) (which we don't have yet for .NET anyway), but I don't think that would still solve your "I want one config that gets chosen dynamically" scenario without you shipping different YAML files based on whether it's .NET or .NET Framework which overrides the host to use there.

I think for the time being the easiest thing to do would be to create a custom IPostConfigureOptions<OtlpExporterOptions> implementation that does the picking-the-value-based-on-the-protocol you have, then register that into the applications' service collection.

For clarity, by "Support Service Discovery configuration standards", do you just mean what the .NET libraries/Aspire do in this area, or do you mean an actual web standard that's documented somewhere?

[martincostello]

Since 1.12, .NET Framework apps use http by default, whereas modern .NET uses gRPC

What is preventing you from just using http/protobuf for both? Presumably the both export telemetry to the same back-end?

Rather than relying on the default, just could just configure both to use OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf.

[julealgon]

I think first-class support for this would be driven through the specification (so open-telemetry/opentelemetry-specification#5088) rather than something we would independently invent here.

I'm not 100% sure that's going to be the case since .NET's configuration mechanism is something that is unique to .NET. Similar to how it accepts IConfiguration-based settings today on top of the spec's env vars, this could be a custom extension that is .NET-specific.

I think for the time being the easiest thing to do would be to create a custom IPostConfigureOptions<OtlpExporterOptions> implementation that does the picking-the-value-based-on-the-protocol you have, then register that into the applications' service collection.

Right. This is exactly what I was going to attempt creating for now. I'll likely need to manually call ServiceEndpointResolver but a native implementation could just propagate service discovery support to the underlying HttpClient used by the exporter: much cleaner and simpler.

EDIT: Actually, maybe it will work if I just propagate the service discovery string directly as the endpoint address. I'll try that. If that works, it would not require manually calling ServiceEndpointResolver.

For clarity, by "Support Service Discovery configuration standards", do you just mean what the .NET libraries/Aspire do in this area, or do you mean an actual web standard that's documented somewhere?

I was mostly talking about the configuration structure convention there, the "Services" node, with the named service inside and the multiple endpoints resolvable via the ServiceEndpointResolver with the schema-based abstraction used everywhere else (including Aspire).

The Microsoft.Extensions.ServiceDiscovery package started as an Aspire-related effort but was generalized later to be used independently. We ended up adopting it even before we introduced Aspire to our solution.

Since 1.12, .NET Framework apps use http by default, whereas modern .NET uses gRPC

What is preventing you from just using http/protobuf for both? Presumably the both export telemetry to the same back-end?

Rather than relying on the default, just could just configure both to use OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf.

I did think about defaulting everything to http/protobuf to streamline the configuration, but our collector instances are already very stressed in terms of resources. Switching from gRPC would cause extra strain on those servers that we currently don't have headroom for.