Skip to content

Commit 97aedf8

Browse files
kylehounslowkylehounslow
committed
feat(compat): add vendor compatibility translation edge
Adds compat/ package and docker-compose.compat.yml overlay that accepts telemetry from Datadog, Jaeger (legacy wire protocol), and Splunk HEC agents and forwards OTLP to the base observability-stack collector. Includes customer-facing documentation under docs/starlight-docs. Architecture: dedicated otel-collector-compat edge collector runs the upstream collector-contrib receivers for each vendor, batches, and forwards OTLP to the base collector — which handles all existing processor/exporter logic unchanged. No drift risk with the base config. Modern OTel-SDK apps (including Jaeger v1.42+) bypass the compat hop and send OTLP directly to the base collector. Activation uses the repo's existing INCLUDE_COMPOSE_* convention: echo 'INCLUDE_COMPOSE_COMPAT=docker-compose.compat.yml' >> .env docker compose up -d Pipelines wired into compat collector: - traces: [datadog, jaeger] - metrics: [datadog, statsd, splunk_hec] - logs: [datadog, splunk_hec] Public documentation added under /docs/send-data/from-vendor/: - index.md (overview + architecture + quickstart) - datadog.md, jaeger.md, splunk.md (per-vendor migration guides) - 'From Vendor Agents' sidebar entry added via astro.config.mjs - Link added from /docs/send-data/ overview Scope and framing: - This is protocol-compatible ingest, not a 1:1 vendor platform replacement. - We defer to upstream collector-contrib READMEs and vendor documentation for attribute schemas and translation behavior. The vendor pages here document only 3-5 canonical attributes per vendor and link out to the authoritative sources. - Component receivers have varying stability tiers upstream (alpha/beta/ development). Vendor pages document the specifics. SignalFx deliberately omitted — upstream signalfxreceiver was deprecated 2026-02-13 with explicit guidance to migrate to OTLP. Bundles Jaeger's hotrod demo on port 8080 as a built-in OTLP trace generator (Jaeger's canonical demo, pointed at the base collector). Validated: starlight-docs astro build passes with all internal links valid (115 pages built successfully). Signed-off-by: Kyle Hounslow <kylhouns@amazon.com>
1 parent aa0fb14 commit 97aedf8

14 files changed

Lines changed: 733 additions & 0 deletions

File tree

‎compat/README.md‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
# Vendor Compatibility Overlay
2+
3+
User-facing migration guides: https://observability.opensearch.org/docs/send-data/from-vendor/
4+
5+
This overlay adds a dedicated OpenTelemetry Collector that accepts Datadog, Jaeger, and Splunk HEC wire protocols, translates them to OTLP, and forwards to the base collector. No application code changes are required — vendor agents are pointed at observability-stack by changing an endpoint URL.
6+
7+
## Do I need this overlay?
8+
9+
| If your apps emit... | Need this overlay? |
10+
|----------------------|--------------------|
11+
| OpenTelemetry OTLP (gRPC or HTTP) | No. Send directly to the base collector on 4317 or 4318. |
12+
| Datadog (dd-trace-*, DogStatsD) | Yes. |
13+
| Jaeger native wire protocol (`jaeger-client-*`) | Yes. |
14+
| Splunk HEC | Yes. |
15+
| Jaeger via modern OpenTelemetry SDK + OTLP | No. |
16+
17+
## Architecture
18+
19+
```
20+
vendor agents ──▶ otel-collector-compat ──OTLP──▶ otel-collector (base, unchanged) ──▶ Data Prepper / Prometheus ──▶ OpenSearch
21+
(this overlay)
22+
23+
OTLP apps ─────────────────────────────────▶ (direct to base, no compat hop)
24+
```
25+
26+
The compat collector uses upstream [`opentelemetry-collector-contrib`](https://github.com/open-telemetry/opentelemetry-collector-contrib) receivers. All enrichment and downstream routing happens in the base pipeline — the compat config is purely ingest + forward.
27+
28+
## Activation
29+
30+
```bash
31+
echo "INCLUDE_COMPOSE_COMPAT=docker-compose.compat.yml" >> .env
32+
docker compose up -d
33+
```
34+
35+
Adds `otel-collector-compat` and the bundled Jaeger `hotrod` demo to the stack.
36+
37+
### Verify it's running
38+
39+
```bash
40+
docker compose ps otel-collector-compat
41+
curl -sI http://localhost:8126/info # HTTP 200 = Datadog receiver is live
42+
```
43+
44+
## Supported vendors
45+
46+
| Vendor | Receiver(s) | Default ports | Repo notes |
47+
|--------|-------------|---------------|------------|
48+
| Datadog | `datadogreceiver`, `statsdreceiver` | 8126/tcp, 8125/udp | [vendors/datadog/](vendors/datadog/) |
49+
| Jaeger (legacy wire protocol) | `jaegerreceiver` | 14250/tcp, 14268/tcp | [vendors/jaeger/](vendors/jaeger/) |
50+
| Splunk HEC | `splunkhecreceiver` | 8088/tcp | [vendors/splunk/](vendors/splunk/) |
51+
52+
User-facing migration guides live at https://observability.opensearch.org/docs/send-data/from-vendor/.
53+
54+
SignalFx is not supported. The upstream `signalfxreceiver` is deprecated with explicit guidance to migrate to OTLP.
55+
56+
## Deployment modes
57+
58+
Each vendor supports greenfield, side-by-side, and full-replacement modes. See the public migration guide for each vendor for specifics.
59+
60+
## Port customization
61+
62+
Ports are remappable via environment variables. Useful when a real vendor agent already occupies the default port on the host.
63+
64+
| Variable | Default | Receiver |
65+
|----------|---------|----------|
66+
| `COMPAT_DATADOG_APM_PORT` | 8126 | Datadog trace-agent |
67+
| `COMPAT_DATADOG_STATSD_PORT` | 8125 | DogStatsD |
68+
| `COMPAT_JAEGER_GRPC_PORT` | 14250 | Jaeger gRPC |
69+
| `COMPAT_JAEGER_THRIFT_HTTP_PORT` | 14268 | Jaeger Thrift HTTP |
70+
| `COMPAT_SPLUNK_HEC_PORT` | 8088 | Splunk HEC |
71+
| `COMPAT_COLLECTOR_MEMORY_LIMIT` | 256M | Compat collector memory limit |
72+
73+
## Attribute translation
74+
75+
Each receiver translates vendor-specific data to the OpenTelemetry data model. Translation behavior is defined by the upstream receivers. For schema details, consult:
76+
77+
- The upstream receiver READMEs under [`opentelemetry-collector-contrib/receiver/`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver)
78+
- The vendor's own instrumentation and tagging documentation
79+
80+
## Directory layout
81+
82+
```
83+
compat/
84+
├── README.md ← this file
85+
├── collector/
86+
│ ├── config.compat.yaml ← compat collector config
87+
│ └── README.md ← compat collector design notes
88+
└── vendors/
89+
├── datadog/README.md ← developer notes + link to migration guide
90+
├── jaeger/README.md
91+
└── splunk/README.md
92+
93+
docker-compose.compat.yml ← overlay service definitions
94+
```
95+
96+
## Adding a vendor
97+
98+
1. Create `vendors/<name>/README.md` with a link to the (forthcoming) public migration guide and developer notes (receiver used, config location, quick local test).
99+
2. Add the receiver stanza to `collector/config.compat.yaml` and wire it into the appropriate pipeline(s).
100+
3. Add port mappings to `docker-compose.compat.yml`.
101+
4. Add a page at `docs/starlight-docs/src/content/docs/send-data/from-vendor/<name>.md` with the user migration guide.
102+
5. Verify end-to-end: send vendor-format data → confirm it lands in OpenSearch.

‎compat/collector/README.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Compat Collector Config
2+
3+
`config.compat.yaml` configures `otel-collector-compat`, an edge collector that runs when the compat overlay is active.
4+
5+
## Role
6+
7+
Accepts vendor wire protocols (Datadog, Jaeger, Splunk HEC) on their native ports, translates to the OTLP data model via upstream `opentelemetry-collector-contrib` receivers, and forwards OTLP to the base collector. All enrichment, filtering, and downstream routing (Data Prepper, Prometheus, etc.) happens in the base collector config.
8+
9+
```
10+
vendor apps ──▶ otel-collector-compat ──OTLP──▶ otel-collector (base)
11+
[this config] [unchanged]
12+
```
13+
14+
The compat collector config contains only receivers, the `batch` processor, and an OTLP exporter pointed at the base collector. No transforms.
15+
16+
## Receivers
17+
18+
| Receiver | Config key | Pipelines | Default port | Upstream stability |
19+
|----------|-----------|-----------|--------------|--------------------|
20+
| Datadog APM | `datadog` | traces, metrics, logs | 8126/tcp | alpha |
21+
| DogStatsD | `statsd` | metrics | 8125/udp | beta |
22+
| Jaeger | `jaeger` | traces | 14250/grpc, 14268/http | beta |
23+
| Splunk HEC | `splunk_hec` | logs, metrics | 8088/tcp | beta |
24+
25+
`splunk_hec` is wired into both logs and metrics pipelines — the upstream receiver routes events to the correct signal type based on payload contents.
26+
27+
SignalFx is not supported. The upstream `signalfxreceiver` is deprecated; SignalFx customers should migrate to OTLP.
28+
29+
## Pipelines
30+
31+
```yaml
32+
service:
33+
pipelines:
34+
traces:
35+
receivers: [datadog, jaeger]
36+
processors: [batch]
37+
exporters: [otlp_grpc, debug]
38+
39+
metrics:
40+
receivers: [datadog, statsd, splunk_hec]
41+
processors: [batch]
42+
exporters: [otlp_grpc, debug]
43+
44+
logs:
45+
receivers: [datadog, splunk_hec]
46+
processors: [batch]
47+
exporters: [otlp_grpc, debug]
48+
```
49+
50+
The `batch` processor reduces RPC volume against the base collector. No other processing.
51+
52+
## Activation
53+
54+
```bash
55+
echo "INCLUDE_COMPOSE_COMPAT=docker-compose.compat.yml" >> .env
56+
docker compose up -d
57+
```
58+
59+
Docker Compose's `include:` directive reads the env var and pulls in the overlay, which adds `otel-collector-compat` as a new service.
60+
61+
## Resource usage
62+
63+
Default memory limit: 256MB, configurable via `COMPAT_COLLECTOR_MEMORY_LIMIT`. Idle usage is typically under 100MB.

‎compat/collector/config.compat.yaml‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# config.compat.yaml
2+
#
3+
# Edge collector config. Translates vendor wire protocols to OTLP and forwards
4+
# to the base collector, which handles enrichment and downstream export.
5+
#
6+
# vendor apps ──▶ otel-collector-compat ──OTLP──▶ otel-collector (base)
7+
# (this config) (unchanged)
8+
9+
receivers:
10+
# Datadog trace-agent protocol. Also accepts metrics and logs endpoints;
11+
# all three are wired into the service pipelines below.
12+
# See: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/datadogreceiver
13+
datadog:
14+
endpoint: 0.0.0.0:8126
15+
read_timeout: 60s
16+
trace_id_cache_size: 100
17+
18+
# StatsD / DogStatsD over UDP.
19+
# See: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/statsdreceiver
20+
statsd:
21+
endpoint: 0.0.0.0:8125
22+
aggregation_interval: 60s
23+
enable_metric_type: true
24+
is_monotonic_counter: false
25+
26+
# Jaeger native wire protocol (Thrift HTTP + gRPC). For legacy
27+
# jaeger-client-* applications. Modern Jaeger apps emit OTLP and send
28+
# directly to the base collector on 4317/4318 without this hop.
29+
# See: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/jaegerreceiver
30+
jaeger:
31+
protocols:
32+
grpc:
33+
endpoint: 0.0.0.0:14250
34+
thrift_http:
35+
endpoint: 0.0.0.0:14268
36+
37+
# Splunk HTTP Event Collector (logs and metrics).
38+
# See: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/splunkhecreceiver
39+
splunk_hec:
40+
endpoint: 0.0.0.0:8088
41+
42+
processors:
43+
# Batch spans/events to reduce RPC volume against the base collector.
44+
batch:
45+
timeout: 10s
46+
send_batch_size: 1024
47+
48+
exporters:
49+
# Forward to the base collector. Downstream routing (Data Prepper,
50+
# Prometheus, etc.) is configured there.
51+
otlp_grpc:
52+
endpoint: otel-collector:4317
53+
tls:
54+
insecure: true
55+
56+
debug:
57+
verbosity: basic
58+
59+
service:
60+
pipelines:
61+
traces:
62+
receivers: [datadog, jaeger]
63+
processors: [batch]
64+
exporters: [otlp_grpc, debug]
65+
66+
metrics:
67+
receivers: [datadog, statsd, splunk_hec]
68+
processors: [batch]
69+
exporters: [otlp_grpc, debug]
70+
71+
logs:
72+
receivers: [datadog, splunk_hec]
73+
processors: [batch]
74+
exporters: [otlp_grpc, debug]
75+
76+
telemetry:
77+
logs:
78+
level: info

‎compat/vendors/datadog/README.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Datadog
2+
3+
Migration guide and user-facing documentation: https://observability.opensearch.org/docs/send-data/from-vendor/datadog/
4+
5+
## Receivers used
6+
7+
- [`datadogreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/datadogreceiver) — traces, metrics, logs (8126/tcp)
8+
- [`statsdreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/statsdreceiver) — DogStatsD (8125/udp)
9+
10+
Config: [`../../collector/config.compat.yaml`](../../collector/config.compat.yaml) — `datadog:` and `statsd:` receiver blocks.
11+
12+
## Quick local test
13+
14+
```bash
15+
# DogStatsD metric
16+
echo "test.metric:1|c|#env:dev" | nc -u -w1 localhost 8125
17+
18+
# HEC-style trace payload (msgpack) — see upstream receiver README for format details
19+
```
20+
21+
Traces are best exercised by pointing a `dd-trace-*` SDK application at `localhost:8126`.

‎compat/vendors/jaeger/README.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Jaeger
2+
3+
Migration guide and user-facing documentation: https://observability.opensearch.org/docs/send-data/from-vendor/jaeger/
4+
5+
Jaeger is an open-source CNCF project rather than a proprietary vendor. This directory exists because users migrating from Jaeger deployments typically look for it alongside other vendor integrations.
6+
7+
## Receiver used
8+
9+
- [`jaegerreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/jaegerreceiver) — Jaeger native wire protocol (Thrift HTTP on 14268/tcp, gRPC on 14250/tcp)
10+
11+
Config: [`../../collector/config.compat.yaml`](../../collector/config.compat.yaml) — `jaeger:` receiver block.
12+
13+
Modern Jaeger apps emit OTLP natively and bypass this overlay. The bundled `hotrod` demo (port 8080) is configured this way and exercises the base collector directly.
14+
15+
## Quick local test
16+
17+
```bash
18+
# Trigger the bundled hotrod demo (OTLP path)
19+
curl http://localhost:8080/dispatch?customer=123
20+
21+
# View traces at http://localhost:5601 → Trace Analytics
22+
```
23+
24+
The legacy Thrift HTTP path is harder to exercise without a `jaeger-client-*` app. See the upstream receiver README for wire format details.

‎compat/vendors/splunk/README.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Splunk HEC
2+
3+
Migration guide and user-facing documentation: https://observability.opensearch.org/docs/send-data/from-vendor/splunk/
4+
5+
## Receiver used
6+
7+
- [`splunkhecreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/splunkhecreceiver) — Splunk HTTP Event Collector (8088/tcp)
8+
9+
Config: [`../../collector/config.compat.yaml`](../../collector/config.compat.yaml) — `splunk_hec:` receiver block.
10+
11+
Wired into the logs and metrics pipelines. Traces are not wired in the default compat config.
12+
13+
## Quick local test
14+
15+
```bash
16+
curl -X POST http://localhost:8088/services/collector \
17+
-H "Authorization: Splunk any-token" \
18+
-H "Content-Type: application/json" \
19+
-d '{"event":"hello","sourcetype":"manual","source":"curl"}'
20+
# {"text": "Success", "code": 0}
21+
```
22+
23+
Events land in OpenSearch under `logs-otel-v1-*`. View in Dashboards → Discover.

‎docker-compose.compat.yml‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# docker-compose.compat.yml
2+
#
3+
# Vendor compatibility overlay. Adds an edge OTel Collector that accepts
4+
# Datadog, Jaeger, and Splunk HEC wire protocols and forwards OTLP to the
5+
# base collector.
6+
#
7+
# Activation:
8+
# echo "INCLUDE_COMPOSE_COMPAT=docker-compose.compat.yml" >> .env
9+
# docker compose up -d
10+
#
11+
# vendor apps ──▶ otel-collector-compat ──OTLP──▶ otel-collector (base)
12+
# (this overlay) (unchanged)
13+
14+
services:
15+
otel-collector-compat:
16+
image: otel/opentelemetry-collector-contrib:${OTEL_COLLECTOR_VERSION}
17+
container_name: otel-collector-compat
18+
pull_policy: always
19+
command: ["--config=/etc/otelcol-config.yml"]
20+
volumes:
21+
- ./compat/collector/config.compat.yaml:/etc/otelcol-config.yml
22+
ports:
23+
# Datadog trace-agent (datadogreceiver)
24+
- "${COMPAT_DATADOG_APM_PORT:-8126}:8126"
25+
# DogStatsD (statsdreceiver, UDP)
26+
- "${COMPAT_DATADOG_STATSD_PORT:-8125}:8125/udp"
27+
# Jaeger native wire protocol. Modern Jaeger apps emit OTLP and should
28+
# send directly to the base collector on 4317/4318 instead of this hop.
29+
- "${COMPAT_JAEGER_GRPC_PORT:-14250}:14250"
30+
- "${COMPAT_JAEGER_THRIFT_HTTP_PORT:-14268}:14268"
31+
# Splunk HEC
32+
- "${COMPAT_SPLUNK_HEC_PORT:-8088}:8088"
33+
networks:
34+
- observability-stack-network
35+
restart: unless-stopped
36+
depends_on:
37+
- otel-collector
38+
deploy:
39+
resources:
40+
limits:
41+
memory: ${COMPAT_COLLECTOR_MEMORY_LIMIT:-256M}
42+
43+
# Jaeger hotrod demo app. Emits OTLP directly to the base collector, so it
44+
# does not exercise the compat collector. Included as a built-in trace
45+
# generator for verifying the base pipeline end-to-end.
46+
#
47+
# http://localhost:8080 hotrod UI
48+
# http://localhost:5601 OpenSearch Dashboards → Trace Analytics
49+
hotrod:
50+
image: jaegertracing/example-hotrod:1.60
51+
container_name: hotrod
52+
environment:
53+
- OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
54+
command: ["all"]
55+
networks:
56+
- observability-stack-network
57+
ports:
58+
- "8080:8080"
59+
depends_on:
60+
- otel-collector

‎docker-compose.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ include:
99
- path: ${INCLUDE_COMPOSE_AGENT_EVAL_LLM:-docker-compose/util/docker-compose.empty.yml}
1010
- path: ${INCLUDE_COMPOSE_LOCAL_OPENSEARCH:-docker-compose/util/docker-compose.empty.yml}
1111
- path: ${INCLUDE_COMPOSE_LOCAL_OPENSEARCH_DASHBOARDS:-docker-compose/util/docker-compose.empty.yml}
12+
- path: ${INCLUDE_COMPOSE_COMPAT:-docker-compose/util/docker-compose.empty.yml}
1213

1314
x-default-logging: &logging
1415
driver: "json-file"

‎docs/starlight-docs/astro.config.mjs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,10 @@ export default defineConfig({
107107
label: 'Infrastructure',
108108
autogenerate: { directory: 'send-data/infrastructure' },
109109
},
110+
{
111+
label: 'From Vendor Agents',
112+
autogenerate: { directory: 'send-data/from-vendor' },
113+
},
110114
{
111115
label: 'Data Pipeline',
112116
autogenerate: { directory: 'send-data/data-pipeline' },

0 commit comments

Comments
 (0)