Skip to content

Commit dd172ba

Browse files
kylehounslowkylehounslow
committed
feat(compat): add vendor compatibility translation edge
Adds compat/ package and docker-compose.compat.yml overlay that accepts telemetry from Datadog, Jaeger (legacy wire protocol), and Splunk HEC agents and forwards OTLP to the base observability-stack collector. Includes customer-facing documentation under docs/starlight-docs. Architecture: dedicated otel-collector-compat edge collector runs the upstream collector-contrib receivers for each vendor, batches, and forwards OTLP to the base collector — which handles all existing processor/exporter logic unchanged. No drift risk with the base config. Modern OTel-SDK apps (including Jaeger v1.42+) bypass the compat hop and send OTLP directly to the base collector. Activation uses the repo's existing INCLUDE_COMPOSE_* convention: echo 'INCLUDE_COMPOSE_COMPAT=docker-compose.compat.yml' >> .env docker compose up -d Pipelines wired into compat collector: - traces: [datadog, jaeger] - metrics: [datadog, statsd, splunk_hec] - logs: [datadog, splunk_hec] Public documentation added under /docs/send-data/from-vendor/: - index.md (overview + architecture + quickstart) - datadog.md, jaeger.md, splunk.md (per-vendor migration guides) - 'From Vendor Agents' sidebar entry added via astro.config.mjs - Link added from /docs/send-data/ overview Scope and framing: - This is protocol-compatible ingest, not a 1:1 vendor platform replacement. - We defer to upstream collector-contrib READMEs and vendor documentation for attribute schemas and translation behavior. The vendor pages here document only 3-5 canonical attributes per vendor and link out to the authoritative sources. - Component receivers have varying stability tiers upstream (alpha/beta/ development). Vendor pages document the specifics. SignalFx deliberately omitted — upstream signalfxreceiver was deprecated 2026-02-13 with explicit guidance to migrate to OTLP. Bundles Jaeger's hotrod demo on port 8080 as a built-in OTLP trace generator (Jaeger's canonical demo, pointed at the base collector). Validated: starlight-docs astro build passes with all internal links valid (115 pages built successfully). Signed-off-by: Kyle Hounslow <kylhouns@amazon.com>
1 parent aa0fb14 commit dd172ba

14 files changed

Lines changed: 843 additions & 0 deletions

File tree

‎compat/README.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# Vendor Compatibility Overlay
2+
3+
Adds a dedicated OpenTelemetry Collector that accepts Datadog, Jaeger, and Splunk HEC wire protocols. Vendor agents can be pointed at observability-stack by changing an endpoint URL — no application code changes required.
4+
5+
## Architecture
6+
7+
```
8+
vendor agents ──▶ otel-collector-compat ──OTLP──▶ otel-collector (base, unchanged) ──▶ Data Prepper / Prometheus ──▶ OpenSearch
9+
(this overlay)
10+
11+
OTLP apps ─────────────────────────────────▶ (direct to base, no compat hop)
12+
(incl. modern Jaeger)
13+
```
14+
15+
The compat collector uses upstream [`opentelemetry-collector-contrib`](https://github.com/open-telemetry/opentelemetry-collector-contrib) receivers to translate vendor wire protocols to OTLP, then forwards to the base collector. All enrichment and downstream routing happens in the base pipeline.
16+
17+
## Activation
18+
19+
```bash
20+
echo "INCLUDE_COMPOSE_COMPAT=docker-compose.compat.yml" >> .env
21+
docker compose up -d
22+
```
23+
24+
This adds `otel-collector-compat` and the bundled Jaeger `hotrod` demo to the stack.
25+
26+
## Supported vendors
27+
28+
| Vendor | Receiver(s) | Default ports | Details |
29+
|--------|-------------|---------------|---------|
30+
| Datadog | `datadogreceiver`, `statsdreceiver` | 8126/tcp, 8125/udp | [vendors/datadog/](vendors/datadog/) |
31+
| Jaeger (legacy wire protocol) | `jaegerreceiver` | 14250/tcp, 14268/tcp | [vendors/jaeger/](vendors/jaeger/) |
32+
| Splunk HEC | `splunkhecreceiver` | 8088/tcp | [vendors/splunk/](vendors/splunk/) |
33+
34+
Modern Jaeger apps emit OTLP natively and bypass this overlay. See [vendors/jaeger/](vendors/jaeger/).
35+
36+
SignalFx is not supported. The upstream `signalfxreceiver` is deprecated with explicit guidance to migrate to OTLP.
37+
38+
## Deployment modes
39+
40+
Each vendor integration supports:
41+
42+
1. **Greenfield** — only observability-stack runs on the target port.
43+
2. **Side-by-side** — vendor agent and observability-stack run simultaneously via port remapping (useful for A/B validation).
44+
3. **Full replacement** — vendor agent decommissioned, observability-stack on the default vendor port.
45+
46+
## Port customization
47+
48+
Vendor ports are remappable via environment variables. Useful when a real vendor agent already occupies the default port on the host.
49+
50+
| Variable | Default | Receiver |
51+
|----------|---------|----------|
52+
| `COMPAT_DATADOG_APM_PORT` | 8126 | Datadog trace-agent |
53+
| `COMPAT_DATADOG_STATSD_PORT` | 8125 | DogStatsD |
54+
| `COMPAT_JAEGER_GRPC_PORT` | 14250 | Jaeger gRPC |
55+
| `COMPAT_JAEGER_THRIFT_HTTP_PORT` | 14268 | Jaeger Thrift HTTP |
56+
| `COMPAT_SPLUNK_HEC_PORT` | 8088 | Splunk HEC |
57+
| `COMPAT_COLLECTOR_MEMORY_LIMIT` | 256M | Compat collector memory limit |
58+
59+
## Attribute translation
60+
61+
Each receiver translates vendor-specific data to the OpenTelemetry data model. Translation behavior is defined by the upstream OpenTelemetry Collector receivers and varies per vendor. For schema details, consult:
62+
63+
- The upstream receiver READMEs under [`opentelemetry-collector-contrib/receiver/`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver)
64+
- The vendor's own instrumentation and tagging documentation
65+
66+
Per-vendor pages in this directory list the canonical attributes users typically encounter and link to upstream sources.
67+
68+
## Directory layout
69+
70+
```
71+
compat/
72+
├── README.md ← this file
73+
├── collector/
74+
│ ├── config.compat.yaml ← compat collector config (receivers + forward)
75+
│ └── README.md ← compat collector design notes
76+
└── vendors/
77+
├── datadog/README.md
78+
├── jaeger/README.md
79+
└── splunk/README.md
80+
81+
docker-compose.compat.yml ← overlay service definitions
82+
```
83+
84+
## Adding a vendor
85+
86+
1. Create `vendors/<name>/README.md`.
87+
2. Add the receiver stanza to `collector/config.compat.yaml` and wire it into the appropriate pipeline.
88+
3. Add port mappings to `docker-compose.compat.yml`.

‎compat/collector/README.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Compat Collector Config
2+
3+
`config.compat.yaml` configures `otel-collector-compat`, an edge collector that runs when the compat overlay is active.
4+
5+
## Role
6+
7+
Accepts vendor wire protocols (Datadog, Jaeger, Splunk HEC) on their native ports, translates to the OTLP data model via upstream `opentelemetry-collector-contrib` receivers, and forwards OTLP to the base collector. All enrichment, filtering, and downstream routing (Data Prepper, Prometheus, etc.) happens in the base collector config.
8+
9+
```
10+
vendor apps ──▶ otel-collector-compat ──OTLP──▶ otel-collector (base)
11+
[this config] [unchanged]
12+
```
13+
14+
The compat collector config contains only receivers, the `batch` processor, and an OTLP exporter pointed at the base collector. No transforms.
15+
16+
## Receivers
17+
18+
| Receiver | Config key | Pipelines | Default port | Upstream stability |
19+
|----------|-----------|-----------|--------------|--------------------|
20+
| Datadog APM | `datadog` | traces, metrics, logs | 8126/tcp | alpha |
21+
| DogStatsD | `statsd` | metrics | 8125/udp | beta |
22+
| Jaeger | `jaeger` | traces | 14250/grpc, 14268/http | beta |
23+
| Splunk HEC | `splunk_hec` | logs, metrics | 8088/tcp | beta |
24+
25+
`splunk_hec` is wired into both logs and metrics pipelines — the upstream receiver routes events to the correct signal type based on payload contents.
26+
27+
SignalFx is not supported. The upstream `signalfxreceiver` is deprecated; SignalFx customers should migrate to OTLP.
28+
29+
## Pipelines
30+
31+
```yaml
32+
service:
33+
pipelines:
34+
traces:
35+
receivers: [datadog, jaeger]
36+
processors: [batch]
37+
exporters: [otlp_grpc, debug]
38+
39+
metrics:
40+
receivers: [datadog, statsd, splunk_hec]
41+
processors: [batch]
42+
exporters: [otlp_grpc, debug]
43+
44+
logs:
45+
receivers: [datadog, splunk_hec]
46+
processors: [batch]
47+
exporters: [otlp_grpc, debug]
48+
```
49+
50+
The `batch` processor reduces RPC volume against the base collector. No other processing.
51+
52+
## Activation
53+
54+
```bash
55+
echo "INCLUDE_COMPOSE_COMPAT=docker-compose.compat.yml" >> .env
56+
docker compose up -d
57+
```
58+
59+
Docker Compose's `include:` directive reads the env var and pulls in the overlay, which adds `otel-collector-compat` as a new service.
60+
61+
## Resource usage
62+
63+
Default memory limit: 256MB, configurable via `COMPAT_COLLECTOR_MEMORY_LIMIT`. Idle usage is typically under 100MB.

‎compat/collector/config.compat.yaml‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# config.compat.yaml
2+
#
3+
# Edge collector config. Translates vendor wire protocols to OTLP and forwards
4+
# to the base collector, which handles enrichment and downstream export.
5+
#
6+
# vendor apps ──▶ otel-collector-compat ──OTLP──▶ otel-collector (base)
7+
# (this config) (unchanged)
8+
9+
receivers:
10+
# Datadog trace-agent protocol. Also accepts metrics and logs endpoints;
11+
# all three are wired into the service pipelines below.
12+
# See: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/datadogreceiver
13+
datadog:
14+
endpoint: 0.0.0.0:8126
15+
read_timeout: 60s
16+
trace_id_cache_size: 100
17+
18+
# StatsD / DogStatsD over UDP.
19+
# See: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/statsdreceiver
20+
statsd:
21+
endpoint: 0.0.0.0:8125
22+
aggregation_interval: 60s
23+
enable_metric_type: true
24+
is_monotonic_counter: false
25+
26+
# Jaeger native wire protocol (Thrift HTTP + gRPC). For legacy
27+
# jaeger-client-* applications. Modern Jaeger apps emit OTLP and send
28+
# directly to the base collector on 4317/4318 without this hop.
29+
# See: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/jaegerreceiver
30+
jaeger:
31+
protocols:
32+
grpc:
33+
endpoint: 0.0.0.0:14250
34+
thrift_http:
35+
endpoint: 0.0.0.0:14268
36+
37+
# Splunk HTTP Event Collector (logs and metrics).
38+
# See: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/splunkhecreceiver
39+
splunk_hec:
40+
endpoint: 0.0.0.0:8088
41+
42+
processors:
43+
# Batch spans/events to reduce RPC volume against the base collector.
44+
batch:
45+
timeout: 10s
46+
send_batch_size: 1024
47+
48+
exporters:
49+
# Forward to the base collector. Downstream routing (Data Prepper,
50+
# Prometheus, etc.) is configured there.
51+
otlp_grpc:
52+
endpoint: otel-collector:4317
53+
tls:
54+
insecure: true
55+
56+
debug:
57+
verbosity: basic
58+
59+
service:
60+
pipelines:
61+
traces:
62+
receivers: [datadog, jaeger]
63+
processors: [batch]
64+
exporters: [otlp_grpc, debug]
65+
66+
metrics:
67+
receivers: [datadog, statsd, splunk_hec]
68+
processors: [batch]
69+
exporters: [otlp_grpc, debug]
70+
71+
logs:
72+
receivers: [datadog, splunk_hec]
73+
processors: [batch]
74+
exporters: [otlp_grpc, debug]
75+
76+
telemetry:
77+
logs:
78+
level: info

‎compat/vendors/datadog/README.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# Datadog → observability-stack
2+
3+
When the compat overlay is enabled, observability-stack accepts telemetry from Datadog-instrumented applications via the upstream [`datadogreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/datadogreceiver) and [`statsdreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/statsdreceiver).
4+
5+
## Protocol support
6+
7+
| Signal | Default port | Upstream stability |
8+
|--------|--------------|--------------------|
9+
| Traces | 8126/tcp | Alpha |
10+
| Metrics | 8126/tcp | Development |
11+
| Logs | 8126/tcp | Development |
12+
| DogStatsD metrics | 8125/udp | Beta |
13+
14+
For the full supported-endpoint list and current behavior, consult the upstream [`datadogreceiver` README](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/datadogreceiver).
15+
16+
## Configuration
17+
18+
Datadog tracer libraries (`dd-trace-*`) support endpoint override via environment variables. The common ones:
19+
20+
| Variable | Default |
21+
|----------|---------|
22+
| `DD_AGENT_HOST` | `localhost` |
23+
| `DD_TRACE_AGENT_PORT` | `8126` |
24+
| `DD_TRACE_AGENT_URL` | — (full URL override) |
25+
26+
```bash
27+
export DD_AGENT_HOST=<observability-stack-host>
28+
export DD_TRACE_AGENT_PORT=8126
29+
```
30+
31+
DogStatsD clients point at port 8125/udp on the same host. Consult each tracer library's documentation for configuration specifics.
32+
33+
## Canonical attribute mapping
34+
35+
| Datadog | OTel |
36+
|---------|------|
37+
| `service` | `service.name` |
38+
| `env` | `deployment.environment` |
39+
| `version` | `service.version` |
40+
41+
For all other attributes, consult the upstream receiver README and Datadog's [tagging documentation](https://docs.datadoghq.com/getting_started/tagging/).
42+
43+
## Deployment modes
44+
45+
1. **Greenfield** — observability-stack runs on port 8126.
46+
2. **Side-by-side** — real Datadog Agent on 8126, compat collector on a remapped port (`COMPAT_DATADOG_APM_PORT=8127`). Useful for A/B validation during migration.
47+
3. **Full replacement** — Datadog Agent removed, compat collector on 8126.
48+
49+
## Not covered
50+
51+
- Live processes, profiling, network monitoring — no open-source OTel receivers exist.
52+
- Synthetic monitoring — not a telemetry-ingest concern.
53+
- Datadog UI features (notebooks, SLOs, monitors) — use OpenSearch Dashboards equivalents.
54+
55+
## Caveats
56+
57+
- 128-bit trace IDs from Datadog-instrumented services are feature-flag-gated upstream (`receiver.datadogreceiver.Enable128BitTraceID`, off by default). Traces that originate in an OTel-instrumented service and pass through a Datadog-instrumented service may not correlate without this flag.
58+
- Metric temporality may require a [`deltatocumulativeprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/deltatocumulativeprocessor) for backends expecting cumulative temporality. Validate before production use.
59+
60+
## References
61+
62+
- [`datadogreceiver` README](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/datadogreceiver)
63+
- [`statsdreceiver` README](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/statsdreceiver)
64+
- [Datadog unified service tagging](https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging/)

‎compat/vendors/jaeger/README.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# Jaeger → observability-stack
2+
3+
## Modern Jaeger apps (OTLP)
4+
5+
Modern Jaeger versions support OpenTelemetry SDK instrumentation and OTLP export. Applications already configured for OTLP do not need the compat overlay — they send directly to the base collector on port 4317 (gRPC) or 4318 (HTTP).
6+
7+
```bash
8+
# Point existing OTLP exporters at observability-stack
9+
OTEL_EXPORTER_OTLP_ENDPOINT=http://<observability-stack-host>:4318
10+
```
11+
12+
The bundled Jaeger `hotrod` demo validates this path. With the compat overlay enabled:
13+
14+
```bash
15+
curl http://localhost:8080/dispatch?customer=123
16+
```
17+
18+
Produces a multi-service trace (frontend → customer → driver → route → redis → mysql) that lands in OpenSearch Trace Analytics.
19+
20+
## Legacy Jaeger clients
21+
22+
Applications using archived `jaeger-client-*` libraries send the native Jaeger wire protocol. When the compat overlay is enabled, observability-stack accepts these via the upstream [`jaegerreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/jaegerreceiver).
23+
24+
| Protocol | Default port | Upstream stability |
25+
|----------|--------------|--------------------|
26+
| Thrift HTTP | 14268/tcp | Beta |
27+
| gRPC (Jaeger protobuf) | 14250/tcp | Beta |
28+
29+
Point each legacy client at the compat collector via the client library's endpoint configuration.
30+
31+
The `jaeger-client-*` libraries are archived upstream and no longer receive patches. Long-term, migrate instrumentation to OpenTelemetry SDKs.
32+
33+
## Canonical attribute mapping
34+
35+
| Jaeger | OTel |
36+
|--------|------|
37+
| `service` (from `Process` tag) | `service.name` resource attribute |
38+
| Span kind tag | `span.kind` |
39+
| 64-bit trace/span IDs | 128-bit trace ID / 64-bit span ID |
40+
41+
For all other attributes, consult the upstream [`pkg/translator/jaeger`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/translator/jaeger).
42+
43+
## Deployment modes
44+
45+
1. **OTLP path** — no overlay required; OTLP apps send directly to the base collector.
46+
2. **Greenfield legacy** — compat collector on Jaeger ports 14250/14268.
47+
3. **Side-by-side legacy** — Jaeger Collector AND compat collector run simultaneously via port remapping (`COMPAT_JAEGER_THRIFT_HTTP_PORT`, `COMPAT_JAEGER_GRPC_PORT`).
48+
4. **Full replacement legacy** — Jaeger Collector decommissioned, compat collector on 14250/14268.
49+
50+
## Not covered
51+
52+
- Jaeger query UI — use OpenSearch Dashboards' Trace Analytics.
53+
- Jaeger-specific storage backends (Cassandra, Elasticsearch, badger) — data writes to OpenSearch via Data Prepper.
54+
- Jaeger's dependency graph — OpenSearch Dashboards has a service map.
55+
56+
## References
57+
58+
- [`jaegerreceiver` README](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/jaegerreceiver)
59+
- [`pkg/translator/jaeger`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/translator/jaeger)
60+
- [Jaeger hotrod demo](https://github.com/jaegertracing/jaeger/tree/main/examples/hotrod)

0 commit comments

Comments
 (0)