Skip to content

Commit 2ae8674

Browse files
jmsundarclaude
jmsundar
and
claude
committed
feat: [DEVOPS-2394] add Extend collector configs + awss3 exporter
Scaffolds Extend's in-house OTel Lambda layer diverging from coralogix-nodejs-autoinstrumentation: - extend/collector-config-cx-only.yaml — default CX-only config (parity) - extend/collector-config-cx-arize-s3.yaml — CX + Arize OTLP/gRPC + S3 archival - collector/lambdacomponents/default.go — register awss3exporter - extend/README.md — fork rationale + consumer contract Secret resolution is native via collector's secretsmanagerprovider (${secretsmanager:<name-or-arn>} syntax resolved at startup). No bash wrapper. Follow-ups: go.mod upstream breakage (pre-existing), GH Actions publish flow, SSM layer-version registration. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 8838714 commit 2ae8674

6 files changed

Lines changed: 577 additions & 103 deletions

File tree

‎collector/go.mod‎

Lines changed: 120 additions & 103 deletions
Large diffs are not rendered by default.

‎collector/go.sum‎

Lines changed: 244 additions & 0 deletions
Large diffs are not rendered by default.

‎collector/lambdacomponents/default.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
package lambdacomponents
1818

1919
import (
20+
"github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awss3exporter"
2021
"github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter"
2122
"github.com/open-telemetry/opentelemetry-collector-contrib/extension/sigv4authextension"
2223
"github.com/open-telemetry/opentelemetry-collector-contrib/processor/attributesprocessor"
@@ -55,6 +56,7 @@ func Components(extensionID string) (otelcol.Factories, error) {
5556
}
5657

5758
exporters, err := exporter.MakeFactoryMap(
59+
awss3exporter.NewFactory(),
5860
debugexporter.NewFactory(),
5961
loggingexporter.NewFactory(),
6062
otlpexporter.NewFactory(),

‎extend/README.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# Extend OTel Lambda Layer
2+
3+
Extend's fork of [coralogix/opentelemetry-lambda](https://github.com/coralogix/opentelemetry-lambda) (branch `coralogix-nodejs-autoinstrumentation`) — published as our own Lambda layer to replace `coralogix-nodejs-wrapper-and-exporter-{arch}` and support dual export to Coralogix + Arize + S3 archival.
4+
5+
Ticket: [DEVOPS-2394](https://helloextend.atlassian.net/browse/DEVOPS-2394)
6+
7+
## What differs from upstream
8+
9+
1. **Dual-backend collector configs** (`extend/collector-config-*.yaml`):
10+
- `collector-config-cx-only.yaml` — default, CX-only export (parity with existing CX layer)
11+
- `collector-config-cx-arize-s3.yaml` — CX + Arize OTLP/gRPC + S3 archival
12+
13+
2. **awss3exporter registration** in `collector/lambdacomponents/default.go` for trace archival to S3.
14+
15+
3. **Native secret resolution** — configs use `${secretsmanager:<name-or-arn>}` syntax, resolved by the collector's `secretsmanagerprovider` at startup (registered in `collector/internal/collector/collector.go:77`). No bash wrapper.
16+
17+
## Consumer contract (via `extend-cdk-lib` NodeLambdaBuilder)
18+
19+
`OPENTELEMETRY_COLLECTOR_CONFIG_URI` selects which config to load (set by CDK). Required env vars:
20+
21+
| Var | Source | Purpose |
22+
|-----|--------|---------|
23+
| `CX_SECRET` | existing | CX API key — Secrets Manager name or ARN |
24+
| `CX_APPLICATION` | existing | CX application tag |
25+
| `CX_SUBSYSTEM` | existing | CX subsystem tag |
26+
| `ARIZE_API_KEY_SECRET` | new | Arize OTel API key — Secrets Manager name or ARN |
27+
| `ARIZE_SPACE_ID` | new | Arize space ID (Relay global ID) |
28+
| `ARIZE_PROJECT_NAME` | new | Arize project name |
29+
| `ARIZE_S3_BUCKET_NAME` | new | S3 bucket for archival |
30+
| `ARIZE_COLLECTOR_ENDPOINT` | optional | default `otlp.arize.com:443` (gRPC) |
31+
32+
## Build (pending workflow setup)
33+
34+
Follows upstream: `./ci-scripts/build_nodejs_layer.sh` — requires sibling checkouts of `coralogix/opentelemetry-js-contrib` and `coralogix/opentelemetry-js`. See `.github/workflows/publish-nodejs.yml` for the published flow.
35+
36+
**Extend-specific follow-ups** (tracked in DEVOPS-2394):
37+
- [ ] Add `awss3exporter` to `collector/go.mod` via `go mod tidy`
38+
- [ ] Add GitHub Actions workflow to publish to Extend AWS accounts
39+
- [ ] Publish layer version to SSM `/extend/otel-lambda/layer-version/{arch}` for CDK lookup
40+
41+
## Upstream sync
42+
43+
```
44+
git fetch upstream coralogix-nodejs-autoinstrumentation
45+
git merge upstream/coralogix-nodejs-autoinstrumentation
46+
```

‎extend/collector-config-cx-arize-s3.yaml‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
# Extend OTel Lambda collector config — dual export to Coralogix + Arize + S3 archival.
2+
#
3+
# Consumed by services that integrate Arize tracing. Selected via the
4+
# OPENTELEMETRY_COLLECTOR_CONFIG_URI env var set by extend-cdk-lib
5+
# NodeLambdaBuilder when `arizeTracingProps` is provided.
6+
#
7+
# Secret resolution is native: `${secretsmanager:<name-or-arn>}` is expanded
8+
# at collector startup by the secretsmanagerprovider confmap provider
9+
# (registered in collector/internal/collector/collector.go:77).
10+
#
11+
# Required env vars (set by extend-cdk-lib):
12+
# CX_SECRET — Coralogix API key secret name/ARN
13+
# CX_APPLICATION — Coralogix application tag (AWS account env name)
14+
# CX_SUBSYSTEM — Coralogix subsystem tag (service name)
15+
# ARIZE_API_KEY_SECRET — Arize OTel API key secret name/ARN
16+
# ARIZE_SPACE_ID — Arize space ID (Relay global ID)
17+
# ARIZE_PROJECT_NAME — Arize project name
18+
# ARIZE_S3_BUCKET_NAME — S3 bucket for archival
19+
# AWS_REGION — provided by Lambda runtime
20+
#
21+
# Optional:
22+
# CX_ENDPOINT — default otel-traces.coralogix.com:443
23+
# ARIZE_COLLECTOR_ENDPOINT — default otlp.arize.com:443 (gRPC)
24+
25+
receivers:
26+
otlp:
27+
protocols:
28+
grpc:
29+
endpoint: localhost:4317
30+
http:
31+
endpoint: localhost:4318
32+
telemetryapi:
33+
34+
processors:
35+
batch:
36+
timeout: 1s
37+
decouple:
38+
39+
exporters:
40+
# Coralogix — OTLP gRPC to CX traces endpoint
41+
otlp/coralogix:
42+
endpoint: ${env:CX_ENDPOINT:-otel-traces.coralogix.com:443}
43+
headers:
44+
Authorization: Bearer ${secretsmanager:${env:CX_SECRET}}
45+
CX-Application-Name: ${env:CX_APPLICATION}
46+
CX-Subsystem-Name: ${env:CX_SUBSYSTEM}
47+
48+
# Coralogix — metrics endpoint
49+
otlp/coralogix_metrics:
50+
endpoint: otel-metrics.coralogix.com:443
51+
headers:
52+
Authorization: Bearer ${secretsmanager:${env:CX_SECRET}}
53+
CX-Application-Name: ${env:CX_APPLICATION}
54+
CX-Subsystem-Name: ${env:CX_SUBSYSTEM}
55+
56+
# Coralogix — logs endpoint
57+
otlp/coralogix_logs:
58+
endpoint: otel-logs.coralogix.com:443
59+
headers:
60+
Authorization: Bearer ${secretsmanager:${env:CX_SECRET}}
61+
CX-Application-Name: ${env:CX_APPLICATION}
62+
CX-Subsystem-Name: ${env:CX_SUBSYSTEM}
63+
64+
# Arize — OTLP gRPC (default transport per Arize docs)
65+
# https://arize.com/docs/ax/integrations/opentelemetry/opentelemetry-arize-otel
66+
otlp/arize:
67+
endpoint: ${env:ARIZE_COLLECTOR_ENDPOINT:-otlp.arize.com:443}
68+
headers:
69+
api_key: ${secretsmanager:${env:ARIZE_API_KEY_SECRET}}
70+
space_id: ${env:ARIZE_SPACE_ID}
71+
project_name: ${env:ARIZE_PROJECT_NAME}
72+
73+
# S3 archival — partitioned OTLP JSON
74+
awss3:
75+
s3uploader:
76+
region: ${env:AWS_REGION}
77+
s3_bucket: ${env:ARIZE_S3_BUCKET_NAME}
78+
s3_prefix: traces/${env:CX_APPLICATION}/${env:CX_SUBSYSTEM}
79+
s3_partition: minute
80+
compression: gzip
81+
marshaler: otlp_json
82+
83+
service:
84+
pipelines:
85+
traces:
86+
receivers: [otlp]
87+
processors: [batch, decouple]
88+
exporters: [otlp/coralogix, otlp/arize, awss3]
89+
metrics:
90+
receivers: [otlp]
91+
processors: [batch, decouple]
92+
exporters: [otlp/coralogix_metrics]
93+
logs:
94+
receivers: [otlp, telemetryapi]
95+
processors: [batch, decouple]
96+
exporters: [otlp/coralogix_logs]
97+
telemetry:
98+
metrics:
99+
address: localhost:8888

‎extend/collector-config-cx-only.yaml‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# Extend OTel Lambda collector config — Coralogix-only export.
2+
#
3+
# Default config used when a service has NOT opted in to Arize tracing.
4+
# Maintains parity with the existing CX-managed layer behavior.
5+
#
6+
# Required env vars (set by extend-cdk-lib NodeLambdaBuilder):
7+
# CX_SECRET — Coralogix API key secret name/ARN
8+
# CX_APPLICATION — Coralogix application tag
9+
# CX_SUBSYSTEM — Coralogix subsystem tag
10+
#
11+
# Optional:
12+
# CX_ENDPOINT — default otel-traces.coralogix.com:443
13+
14+
receivers:
15+
otlp:
16+
protocols:
17+
grpc:
18+
endpoint: localhost:4317
19+
http:
20+
endpoint: localhost:4318
21+
telemetryapi:
22+
23+
processors:
24+
batch:
25+
timeout: 1s
26+
decouple:
27+
28+
exporters:
29+
otlp/coralogix:
30+
endpoint: ${env:CX_ENDPOINT:-otel-traces.coralogix.com:443}
31+
headers:
32+
Authorization: Bearer ${secretsmanager:${env:CX_SECRET}}
33+
CX-Application-Name: ${env:CX_APPLICATION}
34+
CX-Subsystem-Name: ${env:CX_SUBSYSTEM}
35+
36+
otlp/coralogix_metrics:
37+
endpoint: otel-metrics.coralogix.com:443
38+
headers:
39+
Authorization: Bearer ${secretsmanager:${env:CX_SECRET}}
40+
CX-Application-Name: ${env:CX_APPLICATION}
41+
CX-Subsystem-Name: ${env:CX_SUBSYSTEM}
42+
43+
otlp/coralogix_logs:
44+
endpoint: otel-logs.coralogix.com:443
45+
headers:
46+
Authorization: Bearer ${secretsmanager:${env:CX_SECRET}}
47+
CX-Application-Name: ${env:CX_APPLICATION}
48+
CX-Subsystem-Name: ${env:CX_SUBSYSTEM}
49+
50+
service:
51+
pipelines:
52+
traces:
53+
receivers: [otlp]
54+
processors: [batch, decouple]
55+
exporters: [otlp/coralogix]
56+
metrics:
57+
receivers: [otlp]
58+
processors: [batch, decouple]
59+
exporters: [otlp/coralogix_metrics]
60+
logs:
61+
receivers: [otlp, telemetryapi]
62+
processors: [batch, decouple]
63+
exporters: [otlp/coralogix_logs]
64+
telemetry:
65+
metrics:
66+
address: localhost:8888

0 commit comments

Comments
 (0)