Skip to content

Commit 7cda4ae

Browse files
jmsundarclaude
jmsundar
and
claude
committed
feat: [DEVOPS-2394] add Mastra receiver + Arize headers_setter pipeline
Split trace pipelines in cx-arize and cx-arize-s3 configs so Mastra AI spans take a dedicated path to Arize without polluting the Coralogix/S3 flow: - New otlp/mastra receiver on localhost:4319/4320 with include_metadata - headers_setter extension reads api_key/space_id/project_name from inbound OTLP headers (Lambda-provided) and forwards them upstream to Arize; api_key/space_id have collector-side defaults from env/secret - otlp/arize exporter swaps its static header block for auth: { authenticator: headers_setter } - traces/mastra pipeline routes otlp/mastra → otlp/arize; the standard traces pipeline stays Coralogix-only (+ S3 in the -s3 variant) Registers headerssetterextension in the collector build (v0.150.0, matching awss3exporter's pin) via default.go and a new custom-build sibling file mirroring sigv4auth.go. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 299cdf9 commit 7cda4ae

6 files changed

Lines changed: 133 additions & 16 deletions

File tree

‎collector/go.mod‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,8 @@ require (
105105
github.com/open-telemetry/opentelemetry-collector-contrib/connector/spanmetricsconnector v0.109.0 // indirect
106106
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awss3exporter v0.150.0 // indirect
107107
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter v0.109.0 // indirect
108+
github.com/open-telemetry/opentelemetry-collector-contrib/extension/headerssetterextension v0.150.0 // indirect
109+
github.com/open-telemetry/opentelemetry-collector-contrib/extension/internal/credentialsfile v0.150.0 // indirect
108110
github.com/open-telemetry/opentelemetry-collector-contrib/extension/sigv4authextension v0.109.0 // indirect
109111
github.com/open-telemetry/opentelemetry-collector-contrib/internal/coreinternal v0.109.0 // indirect
110112
github.com/open-telemetry/opentelemetry-collector-contrib/internal/filter v0.109.0 // indirect
@@ -178,6 +180,7 @@ require (
178180
go.opentelemetry.io/collector/extension v1.56.0 // indirect
179181
go.opentelemetry.io/collector/extension/auth v0.109.0 // indirect
180182
go.opentelemetry.io/collector/extension/experimental/storage v0.109.0 // indirect
183+
go.opentelemetry.io/collector/extension/extensionauth v1.56.0 // indirect
181184
go.opentelemetry.io/collector/extension/extensioncapabilities v0.150.0 // indirect
182185
go.opentelemetry.io/collector/extension/xextension v0.150.0 // indirect
183186
go.opentelemetry.io/collector/featuregate v1.56.0 // indirect

‎collector/go.sum‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -225,6 +225,10 @@ github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awss3exporter
225225
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awss3exporter v0.150.0/go.mod h1:67FJXUfds/OXpKAZ6FDUeN9JXUJqfeFBXUyc2YqlMWA=
226226
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter v0.109.0 h1:3pOBB7MIHs0CempWSjZLekRKt9Oc3OO2lr2eE7R/IeQ=
227227
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter v0.109.0/go.mod h1:keP8lTqP228+Bj2C0we9/risZU3jMxI1vS9L5fkyxoE=
228+
github.com/open-telemetry/opentelemetry-collector-contrib/extension/headerssetterextension v0.150.0 h1:KLQv93WapOZK9pDA4cL/usMQxVcTu6WlG0EPwA0zgUs=
229+
github.com/open-telemetry/opentelemetry-collector-contrib/extension/headerssetterextension v0.150.0/go.mod h1:XAQyALiM1kP9YI5MFAVLujW4zSSfZrPzojzc5lDZ2FY=
230+
github.com/open-telemetry/opentelemetry-collector-contrib/extension/internal/credentialsfile v0.150.0 h1:lw1/8T5k7JH3RRHATYqXVtPgmn483wmRp19rDgZivNU=
231+
github.com/open-telemetry/opentelemetry-collector-contrib/extension/internal/credentialsfile v0.150.0/go.mod h1:neAzGFqd93Rg4m1DDJm9rAWg0+0Jk76fHXHgzAfczgM=
228232
github.com/open-telemetry/opentelemetry-collector-contrib/extension/sigv4authextension v0.109.0 h1:wXorKr/8dt2A1x0DGW+Uh53sX1ERdJp6IUMmUaagR20=
229233
github.com/open-telemetry/opentelemetry-collector-contrib/extension/sigv4authextension v0.109.0/go.mod h1:3riREDwyXx+8g+IK27BaqVxMqRaGU6jpeBZ+RPNBB0w=
230234
github.com/open-telemetry/opentelemetry-collector-contrib/internal/common v0.109.0 h1:3a/Y3/uaLVdu/1QHjxgaCZu8dI6VLffI9N6j80oiAbw=
@@ -461,6 +465,8 @@ go.opentelemetry.io/collector/extension/auth v0.109.0 h1:yKUMCUG3IkjuOnHriNj0nqF
461465
go.opentelemetry.io/collector/extension/auth v0.109.0/go.mod h1:wOIv49JhXIfol8CRmQvLve05ft3nZQUnTfcnuZKxdbo=
462466
go.opentelemetry.io/collector/extension/experimental/storage v0.109.0 h1:kIJiOXHHBgMCvuDNA602dS39PJKB+ryiclLE3V5DIvM=
463467
go.opentelemetry.io/collector/extension/experimental/storage v0.109.0/go.mod h1:6cGr7MxnF72lAiA7nbkSC8wnfIk+L9CtMzJWaaII9vs=
468+
go.opentelemetry.io/collector/extension/extensionauth v1.56.0 h1:w+SjfUd38NGKZfL0QsrW4bke5jVkZdtMD+6scHW5K+0=
469+
go.opentelemetry.io/collector/extension/extensionauth v1.56.0/go.mod h1:iXhR9e5eC2XbdDf/Z17QJIV+wQx1E5DTth2oE3MmVMA=
464470
go.opentelemetry.io/collector/extension/extensioncapabilities v0.109.0 h1:gwLiFySAlYvaaITqmF3EkblrxwcfaDxntUmflf3YdrA=
465471
go.opentelemetry.io/collector/extension/extensioncapabilities v0.109.0/go.mod h1:4zf6xx0Qtv9OHJg599Ah48URgpNCPP6tiSakwIRcRfk=
466472
go.opentelemetry.io/collector/extension/extensioncapabilities v0.150.0 h1:8R7+oocYCmUKJ57mB49wcSBHkxv1npXFd2HYyc/a41Q=

‎collector/lambdacomponents/default.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ package lambdacomponents
1919
import (
2020
"github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awss3exporter"
2121
"github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter"
22+
"github.com/open-telemetry/opentelemetry-collector-contrib/extension/headerssetterextension"
2223
"github.com/open-telemetry/opentelemetry-collector-contrib/extension/sigv4authextension"
2324
"github.com/open-telemetry/opentelemetry-collector-contrib/processor/attributesprocessor"
2425
"github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor"
@@ -84,6 +85,7 @@ func Components(extensionID string) (otelcol.Factories, error) {
8485

8586
extensions, err := extension.MakeFactoryMap(
8687
sigv4authextension.NewFactory(),
88+
headerssetterextension.NewFactory(),
8789
)
8890
if err != nil {
8991
errs = append(errs, err)

‎collector/lambdacomponents/extension/headerssetter.go‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
//go:build lambdacomponents.custom && (lambdacomponents.all || lambdacomponents.extension.all || lambdacomponents.extension.headerssetter)
2+
3+
// Copyright The OpenTelemetry Authors
4+
//
5+
// Licensed under the Apache License, Version 2.0 (the "License");
6+
// you may not use this file except in compliance with the License.
7+
// You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing, software
12+
// distributed under the License is distributed on an "AS IS" BASIS,
13+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
// See the License for the specific language governing permissions and
15+
// limitations under the License.
16+
17+
package extension
18+
19+
import (
20+
"github.com/open-telemetry/opentelemetry-collector-contrib/extension/headerssetterextension"
21+
"go.opentelemetry.io/collector/extension"
22+
)
23+
24+
func init() {
25+
Factories = append(Factories, func(extensionId string) extension.Factory {
26+
return headerssetterextension.NewFactory()
27+
})
28+
}

‎extend/collector-config-cx-arize-s3.yaml‎

Lines changed: 47 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,24 @@
88
# at collector startup by the secretsmanagerprovider confmap provider
99
# (registered in collector/internal/collector/collector.go:77).
1010
#
11+
# Trace pipelines are split:
12+
# traces — standard OTLP spans on 4317/4318 → Coralogix + S3
13+
# traces/mastra — Mastra AI OTLP spans on 4319/4320 → Arize
14+
#
15+
# Arize credentials ride inbound on the 4319/4320 receivers as OTLP headers
16+
# (api_key, space_id, project_name) and are forwarded upstream via the
17+
# headers_setter extension. api_key/space_id fall back to collector env/secret;
18+
# project_name must come from the Lambda (consumers register N projects per
19+
# service, one env var can't serve N).
20+
#
1121
# Required env vars (set by extend-cdk-lib):
1222
# CX_SECRET — Coralogix API key secret name/ARN
1323
# CX_APPLICATION — Coralogix application tag (AWS account env name)
1424
# CX_SUBSYSTEM — Coralogix subsystem tag (service name)
15-
# ARIZE_API_KEY_SECRET — Arize OTel API key secret name/ARN
16-
# ARIZE_SPACE_ID — Arize space ID (Relay global ID)
17-
# ARIZE_PROJECT_NAME — Arize project name
25+
# ARIZE_API_KEY_SECRET — Arize OTel API key secret name/ARN (also used as
26+
# collector-side fallback for inbound api_key header)
27+
# ARIZE_SPACE_ID — Arize space ID (also used as collector-side
28+
# fallback for inbound space_id header)
1829
# ARIZE_S3_BUCKET_NAME — S3 bucket for archival
1930
# AWS_REGION — provided by Lambda runtime
2031
#
@@ -23,13 +34,37 @@
2334
# ARIZE_COLLECTOR_ENDPOINT — default otlp.arize.com:443 (gRPC)
2435
# ARIZE_S3_PREFIX — default traces/${CX_APPLICATION}/${CX_SUBSYSTEM}
2536

37+
extensions:
38+
headers_setter:
39+
headers:
40+
- action: upsert
41+
key: api_key
42+
from_context: api_key
43+
default_value: ${secretsmanager:${env:ARIZE_API_KEY_SECRET}}
44+
- action: upsert
45+
key: space_id
46+
from_context: space_id
47+
default_value: ${env:ARIZE_SPACE_ID}
48+
- action: upsert
49+
key: project_name
50+
from_context: project_name
51+
# no default — must come from the Lambda
52+
2653
receivers:
2754
otlp:
2855
protocols:
2956
grpc:
3057
endpoint: localhost:4317
3158
http:
3259
endpoint: localhost:4318
60+
otlp/mastra:
61+
protocols:
62+
grpc:
63+
endpoint: localhost:4319
64+
include_metadata: true
65+
http:
66+
endpoint: localhost:4320
67+
include_metadata: true
3368
telemetryapi:
3469

3570
processors:
@@ -64,12 +99,11 @@ exporters:
6499

65100
# Arize — OTLP gRPC (default transport per Arize docs)
66101
# https://arize.com/docs/ax/integrations/opentelemetry/opentelemetry-arize-otel
102+
# headers_setter supplies api_key / space_id / project_name per request.
67103
otlp/arize:
68104
endpoint: ${env:ARIZE_COLLECTOR_ENDPOINT:-otlp.arize.com:443}
69-
headers:
70-
api_key: ${secretsmanager:${env:ARIZE_API_KEY_SECRET}}
71-
space_id: ${env:ARIZE_SPACE_ID}
72-
project_name: ${env:ARIZE_PROJECT_NAME}
105+
auth:
106+
authenticator: headers_setter
73107

74108
# S3 archival — partitioned OTLP JSON
75109
awss3:
@@ -82,11 +116,16 @@ exporters:
82116
marshaler: otlp_json
83117

84118
service:
119+
extensions: [headers_setter]
85120
pipelines:
86121
traces:
87122
receivers: [otlp]
88123
processors: [batch, decouple]
89-
exporters: [otlp/coralogix, otlp/arize, awss3]
124+
exporters: [otlp/coralogix, awss3]
125+
traces/mastra:
126+
receivers: [otlp/mastra]
127+
processors: [batch, decouple]
128+
exporters: [otlp/arize]
90129
metrics:
91130
receivers: [otlp]
92131
processors: [batch, decouple]

‎extend/collector-config-cx-arize.yaml‎

Lines changed: 47 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -3,25 +3,60 @@
33
# Selected via OPENTELEMETRY_COLLECTOR_CONFIG_URI when arizeTracingProps is
44
# provided WITHOUT s3Archival. For CX+Arize+S3 see collector-config-cx-arize-s3.yaml.
55
#
6+
# Trace pipelines are split:
7+
# traces — standard OTLP spans on 4317/4318 → Coralogix
8+
# traces/mastra — Mastra AI OTLP spans on 4319/4320 → Arize
9+
#
10+
# Arize credentials ride inbound on the 4319/4320 receivers as OTLP headers
11+
# (api_key, space_id, project_name) and are forwarded upstream via the
12+
# headers_setter extension. api_key/space_id fall back to collector env/secret;
13+
# project_name must come from the Lambda (consumers register N projects per
14+
# service, one env var can't serve N).
15+
#
616
# Required env vars (set by extend-cdk-lib):
717
# CX_SECRET — Coralogix API key secret name/ARN
818
# CX_APPLICATION — Coralogix application tag (AWS account env name)
919
# CX_SUBSYSTEM — Coralogix subsystem tag (service name)
10-
# ARIZE_API_KEY_SECRET — Arize OTel API key secret name/ARN
11-
# ARIZE_SPACE_ID — Arize space ID (Relay global ID)
12-
# ARIZE_PROJECT_NAME — Arize project name
20+
# ARIZE_API_KEY_SECRET — Arize OTel API key secret name/ARN (also used as
21+
# collector-side fallback for inbound api_key header)
22+
# ARIZE_SPACE_ID — Arize space ID (also used as collector-side
23+
# fallback for inbound space_id header)
1324
#
1425
# Optional:
1526
# CX_ENDPOINT — default otel-traces.coralogix.com:443
1627
# ARIZE_COLLECTOR_ENDPOINT — default otlp.arize.com:443 (gRPC)
1728

29+
extensions:
30+
headers_setter:
31+
headers:
32+
- action: upsert
33+
key: api_key
34+
from_context: api_key
35+
default_value: ${secretsmanager:${env:ARIZE_API_KEY_SECRET}}
36+
- action: upsert
37+
key: space_id
38+
from_context: space_id
39+
default_value: ${env:ARIZE_SPACE_ID}
40+
- action: upsert
41+
key: project_name
42+
from_context: project_name
43+
# no default — must come from the Lambda
44+
1845
receivers:
1946
otlp:
2047
protocols:
2148
grpc:
2249
endpoint: localhost:4317
2350
http:
2451
endpoint: localhost:4318
52+
otlp/mastra:
53+
protocols:
54+
grpc:
55+
endpoint: localhost:4319
56+
include_metadata: true
57+
http:
58+
endpoint: localhost:4320
59+
include_metadata: true
2560
telemetryapi:
2661

2762
processors:
@@ -56,19 +91,23 @@ exporters:
5691

5792
# Arize — OTLP gRPC (default transport per Arize docs)
5893
# https://arize.com/docs/ax/integrations/opentelemetry/opentelemetry-arize-otel
94+
# headers_setter supplies api_key / space_id / project_name per request.
5995
otlp/arize:
6096
endpoint: ${env:ARIZE_COLLECTOR_ENDPOINT:-otlp.arize.com:443}
61-
headers:
62-
api_key: ${secretsmanager:${env:ARIZE_API_KEY_SECRET}}
63-
space_id: ${env:ARIZE_SPACE_ID}
64-
project_name: ${env:ARIZE_PROJECT_NAME}
97+
auth:
98+
authenticator: headers_setter
6599

66100
service:
101+
extensions: [headers_setter]
67102
pipelines:
68103
traces:
69104
receivers: [otlp]
70105
processors: [batch, decouple]
71-
exporters: [otlp/coralogix, otlp/arize]
106+
exporters: [otlp/coralogix]
107+
traces/mastra:
108+
receivers: [otlp/mastra]
109+
processors: [batch, decouple]
110+
exporters: [otlp/arize]
72111
metrics:
73112
receivers: [otlp]
74113
processors: [batch, decouple]

0 commit comments

Comments
 (0)