Skip to content

Run otel collector on gateway VMSS #4876

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hlipsig merged 17 commits into from
Jun 11, 2026
Merged

Run otel collector on gateway VMSS #4876

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
233158e
config: add GatewayOtelCollectorImage field
tsatam Jun 5, 2026
2932124
env: add Gateway OTel KeyVault constants
tsatam Jun 5, 2026
58c36e1
gateway: add OTel Collector health check and load balancer config
tsatam Jun 5, 2026
102c628
gateway: add OTel Collector and cluster MDSD parameters
tsatam Jun 5, 2026
477beb9
gateway: add OTel Collector and MDSD to bootstrap script
tsatam Jun 5, 2026
077d7f3
gateway: add Gateway OTel Collector service configuration
tsatam Jun 5, 2026
e9a06ec
gateway: add Cluster MDSD service in GIG Bridge Mode
tsatam Jun 5, 2026
2cd5d33
gateway: update static IP allocation and network config
tsatam Jun 5, 2026
7631334
gateway: implement TLS certificate management for OTel Collector
tsatam Jun 5, 2026
e77f657
gateway: use MSI token auth for cluster MDSD instead of KeyVault cert
tsatam Jun 5, 2026
6e1edb0
Update generated files
tsatam Jun 8, 2026
a6a1770
Correct casing on clusterMdsd params
tsatam Jun 10, 2026
63b0bbc
Prefix clustermdsd image with ACR prefix
tsatam Jun 10, 2026
c0f580e
Merge all clustermdsd config into single conf file
tsatam Jun 10, 2026
08717f0
Add special-case handling to cert refresh script to handle otel tls cert
tsatam Jun 10, 2026
a318d7a
Use dedicated otelClusterMdsdConfigVersion property for gateway clust…
tsatam Jun 10, 2026
b6e0fe5
Address Copilot feedback
tsatam Jun 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions pkg/deploy/assets/gateway-production-parameters.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,12 @@
"azureSecPackVSATenantId": {
"value": ""
},
"clusterMdsdAccount": {
"value": ""
},
"clusterMdsdNamespace": {
"value": ""
},
"databaseAccountName": {
"value": ""
},
Expand All @@ -35,6 +41,9 @@
"gatewayMdsdConfigVersion": {
"value": ""
},
"gatewayOtelCollectorImage": {
"value": ""
},
"gatewayServicePrincipalId": {
"value": ""
},
Expand Down Expand Up @@ -70,6 +79,9 @@
"uaecentral"
]
},
"otelClusterMdsdConfigVersion": {
"value": ""
},
"rpImage": {
"value": ""
},
Expand Down
41 changes: 40 additions & 1 deletion pkg/deploy/assets/gateway-production.json
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion pkg/deploy/assets/rp-production.json
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions pkg/deploy/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,11 +55,13 @@ type Configuration struct {
ClusterMDSDConfigVersion *string `json:"clusterMdsdConfigVersion,omitempty" value:"required"`
ClusterMDSDNamespace *string `json:"clusterMdsdNamespace,omitempty" value:"required"`
ClusterParentDomainName *string `json:"clusterParentDomainName,omitempty" value:"required"`
OtelClusterMDSDConfigVersion *string `json:"otelClusterMdsdConfigVersion,omitempty" value:"required"`
DatabaseAccountName *string `json:"databaseAccountName,omitempty" value:"required"`
CosmosDB *CosmosDBConfiguration `json:"cosmosDB,omitempty"`
DisableCosmosDBFirewall *bool `json:"disableCosmosDBFirewall,omitempty"`
ExtraCosmosDBIPs []string `json:"extraCosmosDBIPs,omitempty"`
FluentbitImage *string `json:"fluentbitImage,omitempty" value:"required"`
GatewayOtelCollectorImage *string `json:"gatewayOtelCollectorImage,omitempty" value:"required"`
FPClientID *string `json:"fpClientId,omitempty" value:"required"`
FPTenantID *string `json:"fpTenantId,omitempty" value:"required"`
FPServerCertCommonName *string `json:"fpServerCertCommonName,omitempty"`
Expand Down
15 changes: 10 additions & 5 deletions pkg/deploy/devconfig.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ func DevConfig(_env env.Core) (*Config, error) {
},
Configuration: &Configuration{
ACRResourceID: pointerutils.ToPtr("/subscriptions/" + _env.SubscriptionID() + "/resourceGroups/" + azureUniquePrefix + "-global/providers/Microsoft.ContainerRegistry/registries/" + azureUniquePrefix + "aro"),
ACRReplicaDisabled: pointerutils.ToPtr(true),
AdminAPICABundle: pointerutils.ToPtr(string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: ca}))),
AdminAPIClientCertCommonName: &clientCert.Subject.CommonName,
ARMAPICABundle: pointerutils.ToPtr(string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: ca}))),
Expand All @@ -85,17 +86,19 @@ func DevConfig(_env env.Core) (*Config, error) {
ClusterMDSDAccount: pointerutils.ToPtr(version.DevClusterGenevaLoggingAccount),
ClusterMDSDConfigVersion: pointerutils.ToPtr(version.DevClusterGenevaLoggingConfigVersion),
ClusterMDSDNamespace: pointerutils.ToPtr(version.DevClusterGenevaLoggingNamespace),
OtelClusterMDSDConfigVersion: pointerutils.ToPtr(version.DevClusterGenevaLoggingConfigVersion),
ClusterParentDomainName: pointerutils.ToPtr(azureUniquePrefix + "-clusters." + os.Getenv("PARENT_DOMAIN_NAME")),
CosmosDB: &CosmosDBConfiguration{
StandardProvisionedThroughput: 1000,
PortalProvisionedThroughput: 400,
GatewayProvisionedThroughput: 400,
},
DisableCosmosDBFirewall: pointerutils.ToPtr(true),
FluentbitImage: pointerutils.ToPtr(version.FluentbitImage(azureUniquePrefix + "aro." + _env.Environment().ContainerRegistryDNSSuffix)),
FPClientID: pointerutils.ToPtr(os.Getenv("AZURE_FP_CLIENT_ID")),
FPServicePrincipalID: pointerutils.ToPtr(os.Getenv("AZURE_FP_SERVICE_PRINCIPAL_ID")),
FPTenantID: pointerutils.ToPtr(os.Getenv("AZURE_TENANT_ID")),
DisableCosmosDBFirewall: pointerutils.ToPtr(true),
FluentbitImage: pointerutils.ToPtr(version.FluentbitImage(azureUniquePrefix + "aro." + _env.Environment().ContainerRegistryDNSSuffix)),
GatewayOtelCollectorImage: pointerutils.ToPtr(""),
FPClientID: pointerutils.ToPtr(os.Getenv("AZURE_FP_CLIENT_ID")),
FPServicePrincipalID: pointerutils.ToPtr(os.Getenv("AZURE_FP_SERVICE_PRINCIPAL_ID")),
FPTenantID: pointerutils.ToPtr(os.Getenv("AZURE_TENANT_ID")),
GatewayDomains: []string{
"eastus-shared.ppe.warm.ingest.monitor.core.windows.net",
"gcs.ppe.monitoring.core.windows.net",
Expand All @@ -110,6 +113,7 @@ func DevConfig(_env env.Core) (*Config, error) {
"test1.diagnostics.monitoring.core.windows.net",
},
GatewayMDSDConfigVersion: pointerutils.ToPtr(version.DevGatewayGenevaLoggingConfigVersion),
GatewayVMSize: pointerutils.ToPtr("Standard_D2s_v3"),
GatewayVMSSCapacity: pointerutils.ToPtr(1),
GlobalResourceGroupLocation: pointerutils.ToPtr(_env.Location()),
GlobalResourceGroupName: pointerutils.ToPtr(azureUniquePrefix + "-global"),
Expand Down Expand Up @@ -158,6 +162,7 @@ func DevConfig(_env env.Core) (*Config, error) {
TokenContributorRoleID: pointerutils.ToPtr("48983534-3d06-4dcb-a566-08a694eb1279"),
TokenContributorRoleName: pointerutils.ToPtr("ARO v4 ContainerRegistry Token Contributor"),
VMSize: pointerutils.ToPtr("Standard_D2s_v3"),
Environment: pointerutils.ToPtr("development"),

// TODO: Replace with Live Service Configuration in KeyVault
InstallViaHive: pointerutils.ToPtr(os.Getenv("ARO_INSTALL_VIA_HIVE")),
Expand Down
41 changes: 41 additions & 0 deletions pkg/deploy/generator/resources_gateway.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,24 @@ func (g *generator) gatewayLB() *arm.Resource {
},
Name: pointerutils.ToPtr("gateway-lbrule-http"),
},
{
Properties: &armnetwork.LoadBalancingRulePropertiesFormat{
FrontendIPConfiguration: &armnetwork.SubResource{
ID: pointerutils.ToPtr("[resourceId('Microsoft.Network/loadBalancers/frontendIPConfigurations', 'gateway-lb-internal', 'gateway-frontend')]"),
},
BackendAddressPool: &armnetwork.SubResource{
ID: pointerutils.ToPtr("[resourceId('Microsoft.Network/loadBalancers/backendAddressPools', 'gateway-lb-internal', 'gateway-backend')]"),
},
Probe: &armnetwork.SubResource{
ID: pointerutils.ToPtr("[resourceId('Microsoft.Network/loadBalancers/probes', 'gateway-lb-internal', 'gateway-probe-otel')]"),
},
Protocol: pointerutils.ToPtr(armnetwork.TransportProtocolTCP),
LoadDistribution: pointerutils.ToPtr(armnetwork.LoadDistributionDefault),
FrontendPort: pointerutils.ToPtr(int32(4317)),
BackendPort: pointerutils.ToPtr(int32(4317)),
},
Name: pointerutils.ToPtr("gateway-lbrule-otel"),
},
},
Probes: []*armnetwork.Probe{
{
Expand All @@ -142,6 +160,15 @@ func (g *generator) gatewayLB() *arm.Resource {
},
Name: pointerutils.ToPtr("gateway-probe"),
},
{
Properties: &armnetwork.ProbePropertiesFormat{
Protocol: pointerutils.ToPtr(armnetwork.ProbeProtocolHTTP),
Port: pointerutils.ToPtr(int32(13133)),
NumberOfProbes: pointerutils.ToPtr(int32(2)),
RequestPath: pointerutils.ToPtr("/"),
Comment thread
tsatam marked this conversation as resolved.
},
Name: pointerutils.ToPtr("gateway-probe-otel"),
},
},
},
Name: pointerutils.ToPtr("gateway-lb-internal"),
Expand Down Expand Up @@ -196,13 +223,17 @@ func (g *generator) gatewayVMSS() *arm.Resource {
"azureCloudName",
"azureSecPackQualysUrl",
"azureSecPackVSATenantId",
"clusterMdsdAccount",
"clusterMdsdNamespace",
"databaseAccountName",
"otelClusterMdsdConfigVersion",
"environment",
"fluentbitImage",
"gatewayDomains",
"gatewayFeatures",
"gatewayLogLevel",
"gatewayMdsdConfigVersion",
"gatewayOtelCollectorImage",
"keyvaultDNSSuffix",
"keyvaultPrefix",
"mdmFrontendUrl",
Expand All @@ -223,6 +254,10 @@ func (g *generator) gatewayVMSS() *arm.Resource {
"'MDMIMAGE=''"+version.MdmImage("")+"''\n'",
)

parts = append(parts,
"'CLUSTERMDSDIMAGE=''"+version.MdsdImage("")+"''\n'",
)

parts = append(parts,
"'LOCATION=$(base64 -d <<<'''",
"base64(resourceGroup().location)",
Expand All @@ -241,6 +276,12 @@ func (g *generator) gatewayVMSS() *arm.Resource {
"''')\n'",
)

parts = append(parts,
"'GATEWAYUSERASSIGNEDIDENTITYRESOURCEID=$(base64 -d <<<'''",
"base64(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', concat('aro-gateway-', resourceGroup().location)))",
"''')\n'",
)

// VMSS extensions only support one custom script
// Because of this, the util-*.sh scripts are prefixed to the bootstrapping script
// main is called at the end of the bootstrapping script, so appending them will not work
Expand Down
81 changes: 81 additions & 0 deletions pkg/deploy/generator/scripts/gatewayVMSS.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,12 +56,18 @@ main() {
local -r rpimage="$RPIMAGE"
# shellcheck disable=SC2034
local -r fluentbit_image="$FLUENTBITIMAGE"
# shellcheck disable=SC2034
local -r otel_collector_image="$GATEWAYOTELCOLLECTORIMAGE"
# shellcheck disable=SC2034
local -r cluster_mdsd_image="${RPIMAGE%%/*}/${CLUSTERMDSDIMAGE#*/}"
# values are references to variables, they should not be dereferenced here
# shellcheck disable=SC2034
local -rA aro_images=(
["mdm"]="mdmimage"
["rp"]="rpimage"
["fluentbit"]="fluentbit_image"
["otelcollector"]="otel_collector_image"
["clustermdsd"]="cluster_mdsd_image"
Comment thread
hlipsig marked this conversation as resolved.
)

pull_container_images aro_images
Expand All @@ -74,6 +80,9 @@ main() {
"443/tcp"
# JIT ssh
"22/tcp"
# OTel collector
"4317/tcp"
"13133/tcp"
)

firewalld_configure enable_ports
Expand Down Expand Up @@ -115,19 +124,88 @@ ENVIRONMENT='$ENVIRONMENT'"
# shellcheck disable=SC2034
local -r mdsd_config_version="$GATEWAYMDSDCONFIGVERSION"

# shellcheck disable=SC2034
local -r cluster_mdsd_conf_file="MDSDIMAGE='$cluster_mdsd_image'
MONITORING_GCS_ENVIRONMENT='$MDSDENVIRONMENT'
MONITORING_GCS_ACCOUNT='$CLUSTERMDSDACCOUNT'
MONITORING_GCS_REGION='$LOCATION'
MONITORING_GCS_AUTH_ID_TYPE=AuthMSIToken
MONITORING_GCS_AUTH_ID=mi_res_id#\${GATEWAYUSERASSIGNEDIDENTITYRESOURCEID}
MONITORING_GCS_NAMESPACE='$CLUSTERMDSDNAMESPACE'
MONITORING_CONFIG_VERSION='$OTELCLUSTERMDSDCONFIGVERSION'
MONITORING_USE_GENEVA_CONFIG_SERVICE=true
MONITORING_TENANT='$LOCATION'
MONITORING_ROLE=cluster
MONITORING_ROLE_INSTANCE=\"\$(hostname)\"
MONITORING_ENVIRONMENT='$ENVIRONMENT'
ENABLE_GIG_BRIDGE_MODE=1"

# shellcheck disable=SC2034
local -r gateway_otel_collector_conf="extensions:
health_check:
endpoint: 0.0.0.0:13133
gatewayauth:
tls:
cert_file: /etc/otel-collector/tls/tls-cert.pem
key_file: /etc/otel-collector/tls/tls-key.pem

receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
middlewares:

@swiencki swiencki Jun 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The receivers.otlp.protocols.grpc block includes middlewares and auth.authenticator fields that belong on the service pipeline or extension config, not the receiver; this will likely fail collector config validation.

- id: gatewayauth
auth:
authenticator: gatewayauth

exporters:
otlp/cluster-mdsd:
endpoint: localhost:2020
tls:
insecure: true

@bennerv bennerv Jun 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

intentional insecure?

@tsatam tsatam Jun 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This communication is between the otel and mdsd containers running directly on the VMSS instance, over localhost. We can enable TLS for this communication but it will require additional configuration of all components.


processors:
attributes/cluster:
actions:
- key: resourceid
from_context: \"auth.clusterResourceID\"
action: upsert

memory_limiter:
check_interval: 1s
limit_mib: 512

batch:
timeout: 10s
send_batch_size: 1024

service:
extensions:
- health_check
- gatewayauth
pipelines:
logs:
receivers: [otlp]
processors: [memory_limiter, attributes/cluster, batch]
exporters: [otlp/cluster-mdsd]"

# values are references to variables, they should not be dereferenced here
# shellcheck disable=SC2034
local -rA aro_configs=(
["gateway_config"]="aro_gateway_conf_file"
["fluentbit"]="fluentbit_conf_file"
["mdsd"]="mdsd_config_version"
["gateway_otel_collector"]="gateway_otel_collector_conf"
["cluster_mdsd"]="cluster_mdsd_conf_file"
["static_ip_address"]="static_ip_addresses"
)

# shellcheck disable=SC2034
# use default podman network with range 10.88.0.0/16
local -rA static_ip_addresses=(
["gateway"]="10.88.0.2"
["otelcollector"]="10.88.0.9"
["mdm"]="10.88.0.8"
)

Expand All @@ -143,8 +221,11 @@ ENVIRONMENT='$ENVIRONMENT'"
"mdm"
"chronyd"
"fluentbit"
"gateway-otel-collector"
"cluster-mdsd"
"download-mdsd-credentials.timer"
"download-mdm-credentials.timer"
"download-gateway-otel-credentials.timer"
"firewalld"
)

Expand Down
Loading
Loading