feat(target allocator): export self-telemetry via OTLP

[odubajDT]

TA binary side of the OTLP self-telemetry feature (no CRD changes):

• OTLP metric export from the TA config file (telemetry.metrics.otlp)
• Prometheus bridge so SD/Go/process metrics reach OTLP alongside /metrics
  (SDK metrics on a dedicated registry to avoid double counting; /metrics = union)
• service.name resource, graceful meter-provider shutdown
• ${env:VAR} expansion in header/endpoint values; gRPC accepts URL or host:port
• config validation + unit and end-to-end export tests

Based on #5068

[codecov]

Codecov Report

❌ Patch coverage is 78.57143% with 36 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.30%. Comparing base (5431489) to head (1dfb2fb).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
cmd/otel-allocator/telemetry.go	79.25%	23 Missing and 5 partials ⚠️
cmd/otel-allocator/main.go	0.00%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5294      +/-   ##
==========================================
+ Coverage   63.10%   63.30%   +0.19%     
==========================================
  Files         215      216       +1     
  Lines       15291    15453     +162     
==========================================
+ Hits         9650     9783     +133     
- Misses       4961     4985      +24     
- Partials      680      685       +5     

Flag	Coverage Δ
integration	69.05% <78.57%> (+0.94%)	⬆️
unittests	63.30% <78.57%> (+0.19%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[swiatekm]

Conceptually, the changes look good and make sense to me. The main issue now is how do make them configurable for the user.

The end result I'd ultimately like to achieve is that the otel declarative config is the source of truth for all telemetry of the otel allocator, and the existing prometheus endpoint more of a legacy option for backwards compatibility.

There's three questions I'm trying to answer right now:

1. Can we use the declarative config and customize the resulting sdk with the prometheus bridge and exporter?
2. If so, is it better to embed the declarative config or keep a separate config file.
3. If the above is not feasible, too complex, or we don't want to do it in this PR, then can we make our config compatible with the declarative spec?

[odubajDT]

Thanks! Moved all the telemetry setup out of main.go into telemetry.go

On the three questions, having dug into otelconf:

1. Can we use the declarative config and customize the resulting SDK with the Prometheus bridge + exporter?

The OTLP exporter side is fully covered by otelconf - NewSDK(WithOpenTelemetryConfiguration(...)) builds the meter provider with the periodic OTLP reader straight from the config.

The Prometheus bridge is the catch. otelconf builds the readers internally and doesn't expose a way to attach a sdkmetric.Producer to them. WithMeterProviderOptions lets you append SDK options, but adding a separate reader carrying the bridge producer would double-collect the SDK metrics, since every reader on a meter provider collects all of its metrics. So we can't cleanly graft the bridge onto an otelconf-built SDK now.

2. Embed vs. separate config file?

For the plain TA config I don't think a separate file is needed; embedding the declarative config in the existing config file is consistent with how the collector does it, and keeps everything in one place.

3. If (1) isn't clean, can we make our config compatible with the declarative spec?

Yes, and I think that might be the sweet spot for this PR - adopt otelconf's config types as the schema for the telemetry block and parse through otelconf (so its validation + env substitution run), but keep building the meter provider ourselves so the Prometheus bridge and the /metrics endpoint keep working. That makes the config forward-compatible with the declarative spec now, and leaves a clean migration to full NewSDK later.

The genuinely clean end state for declarative config is to stop registering the metrics on the Prometheus default registry and register them on the OTel SDK instead - then the bridge disappears and NewSDK covers everything, with the Prometheus endpoint reduced to the legacy pull reader. That's a larger change, so I'd suggest it as a follow-up rather than blocking this PR.

So my proposal - keep this PR as the working feature, and land it as-is with our current config and do the otelconf-schema swap next

Let me know what you think!

[swiatekm]

@odubajDT sorry for the wait. I'm ok with keeping the mechanics of this PR as they are, and making the config compatible with the declarative config. It'd be nice to have a smoke test where we load a test yaml file into declarative config to verify that this works.