Run otel collector on gateway VMSS#4876
Conversation
- Use otelcollector key with IP 10.88.0.9 (was gateway_otel_collector at 10.88.0.3) - Remove cluster_mdsd from static IPs (uses host network for localhost:2020) - Update cluster-mdsd service to use --network=host instead of podman network
|
Skipping CI for Draft Pull Request. |
| otlp/cluster-mdsd: | ||
| endpoint: localhost:2020 | ||
| tls: | ||
| insecure: true |
There was a problem hiding this comment.
This communication is between the otel and mdsd containers running directly on the VMSS instance, over localhost. We can enable TLS for this communication but it will require additional configuration of all components.
| @@ -64,8 +64,10 @@ const ( | |||
| PortalServerClientSecretName = "portal-client" | |||
There was a problem hiding this comment.
want to add these to devconfig as well? We don't really maintain it, but might be useful.
There was a problem hiding this comment.
The full-service env will likely require an alternative implementation of this (e.g. forwarding logs directly into an Azure Monitor workspace). Since that env is currently non-functional I'll leave this for a follow-up action.
| protocols: | ||
| grpc: | ||
| endpoint: 0.0.0.0:4317 | ||
| middlewares: |
There was a problem hiding this comment.
The receivers.otlp.protocols.grpc block includes middlewares and auth.authenticator fields that belong on the service pipeline or extension config, not the receiver; this will likely fail collector config validation.
…ermdsd deployment
tsatam
requested review from
cadenmarchese,
hawkowl,
hlipsig,
mociarain,
mrWinston,
rogbas,
tiguelu and
yjst2012
as code owners
tsatam
requested review from
alcasim,
kevinobriendotca,
kimorris27,
sankur-codes,
tuxerrante,
ventifus and
wanghaoran1988
as code owners
There was a problem hiding this comment.
Pull request overview
This PR updates the Gateway VMSS deployment pipeline to run an OpenTelemetry (OTel) Collector on gateway instances, ingest cluster logs over OTLP, and forward them to a colocated “cluster-mdsd” (GIG bridge mode) for Geneva/MDSD forwarding.
Changes:
- Adds new gateway deployment/config parameters for cluster MDSD (account/namespace/config version) and the gateway OTel collector image.
- Extends gateway VMSS bootstrap to pull/run new containers (
gateway-otel-collector,cluster-mdsd), open required ports, and refresh TLS credentials from Key Vault. - Updates gateway ARM resources to expose OTLP (4317) via the internal load balancer and add an OTel health probe.
Reviewed changes
Copilot reviewed 8 out of 10 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/env/env.go | Adds constants for the gateway OTel TLS secret name and key vault suffix. |
| pkg/deploy/generator/templates_gateway.go | Registers new gateway template parameters needed to deploy the OTel collector + cluster-mdsd. |
| pkg/deploy/generator/scripts/util-services.sh | Adds systemd+podman service definitions for gateway-otel-collector and cluster-mdsd, plus cert refresh/watch logic for gateway-otel. |
| pkg/deploy/generator/scripts/gatewayVMSS.sh | Extends gateway VMSS bootstrap to pull/run the new containers, open ports, and provide OTel + cluster-mdsd config. |
| pkg/deploy/generator/resources_gateway.go | Adds LB rule for OTLP 4317 and an HTTP probe for the OTel collector health endpoint. |
| pkg/deploy/devconfig.go | Extends dev config to include the new otelClusterMdsdConfigVersion value. |
| pkg/deploy/config.go | Adds required deploy config fields for gateway OTel collector image and cluster-mdsd config versioning. |
| pkg/deploy/assets/gateway-production.json | Updates the generated gateway production ARM template to include new params and LB resources. |
| pkg/deploy/assets/gateway-production-parameters.json | Updates the parameters file with the new required parameters. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Which issue this PR addresses:
Fixes no Jira
What this PR does / why we need it:
Updates the Gateway VMSS deployment to run an instance of the otel-collector, set up to ingest logs from ARO clusters and forward to MDSD.
Test plan for issue:
Is there any documentation that needs to be updated for this PR?
Yes, will be in a future change
How do you know this will function as expected in production?
Above testing and validation